Skip to content

Conversation

graykode
Copy link
Owner

@graykode graykode commented Oct 3, 2019

In ALBERT(Lan at el), There is not detail about 80% mask
image

But, from n-gram masking (Joshi et al., 2019), they said about 80/10/10

As in BERT, we also mask 15% of the tokens in total: replacing 80% of the masked tokens with [MASK], 10% with random tokens and 10% with the original tokens. However, we perform this replacement at the span level and not for each token individually; i.e. all the tokens in a span are replaced with [MASK]or sampled tokens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant