/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

0 vote185 views1 answer
Détruire une variable python
Python
4 votes190 views1 answer
Render audio content without page view in flask
Python
0 vote181 views1 answer
Python pandas how to select range of data
Python
0 vote232 views1 answer
405 status code django
Python Django
1 vote175 views1 answer
_set in django
Python
0 vote201 views1 answer
Print format round python
Python
1 vote179 views1 answer
Torch tensor equal to
Python
10 votes267 views6 answers
Flask – Environment
Python Flask
3 votes240 views2 answers
Reciprocal python
Python
0 vote234 views1 answer
Get category discord.py
Python