The Wayback Machine - https://web.archive.org/web/20220322203352/https://github.com/codertimo/BERT-pytorch/issues/53
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PositionalEmbedding #53

Open
12yang12 opened this issue Jan 15, 2019 · 3 comments
Open

PositionalEmbedding #53

12yang12 opened this issue Jan 15, 2019 · 3 comments
Labels
good first issue

Comments

@12yang12
Copy link

@12yang12 12yang12 commented Jan 15, 2019

The position embedding in the BERT is not the same as in the transformer. Why not use the form in bert?
@codertimo
Copy link
Owner

@codertimo codertimo commented Apr 8, 2019 •
edited

@Yang92to Great Point, I'll check out the BERT positional embedding method, and update ASAP

@codertimo codertimo added the good first issue label Apr 8, 2019
@jacklanchantin
Copy link

@jacklanchantin jacklanchantin commented Nov 26, 2019

@codertimo the BERT positional embedding method is to just learn an embedding for each position. So you can use nn.Embedding with a constant input sequence [0,1,2,...,L-1] where L is the maximum sequence length.

@yonghee12
Copy link

@yonghee12 yonghee12 commented Sep 21, 2020

@codertimo
Since BERT uses learned positional embeddings and it is one of the biggest difference between original transformers and BERT, I think it is quite urgent to modify the positional embedding part.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
@jacklanchantin @codertimo @yonghee12 @12yang12