/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

18 votes139 views1 answer
List to dic
Python
4 votes165 views2 answers
Python soup get href
Python
1 vote149 views1 answer
How to create routes in flask
Python
3 votes147 views2 answers
Python json check if key exist
Python
3 votes147 views1 answer
Open cv read webcam
Python
24 votes150 views3 answers
Flask vs django
Python
0 vote186 views1 answer
Python - How To Check Operating System
Python
6 votes141 views2 answers
Python datetime get all days between two dates
Python
-2 votes137 views1 answer
How to pass function parameter in decorator
Python
0 vote149 views2 answers
What does manage.py do
Python