The Wayback Machine - https://web.archive.org/web/20220322203405/https://github.com/codertimo/BERT-pytorch/issues/56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chooses 15% of token #56

Open

makcedward opened this issue Feb 22, 2019 · 1 comment

Labels

good first issue

makcedward commented Feb 22, 2019

From paper, it mentioned

Instead, the training data generator chooses 15% of tokens at random, e.g., in the sentence my
dog is hairy it chooses hairy.

It means that 15% of token will be choose for sure.

From https://github.com/codertimo/BERT-pytorch/blob/master/bert_pytorch/dataset/dataset.py#L68,
for every single token, it has 15% of chance that go though the followup procedure. Does it aligned with 15% of token will be chosen?

Owner

codertimo commented Apr 8, 2019

Sorry for the late response, I think you are right.
I'll fix it ASAP

codertimo added the good first issue label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment