/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

1 vote178 views1 answer
Cv2 videocapture nth frame
Python
0 vote218 views1 answer
How to add multiple items in a list in python
Python
1 vote192 views3 answers
List of prime numbers in python
Python
0 vote200 views2 answers
Python sorting array without inbuilt sort
Python
0 vote208 views1 answer
Python svg viewing
Python
0 vote202 views1 answer
``` discord newline doesn't work python
Python
0 vote199 views1 answer
Python select columns with no na
Python
0 vote207 views2 answers
Numpy random in python
Python
3 votes274 views6 answers
Get multiple inputs in python
Python
0 vote225 views1 answer
How to see the functions of a library in python
Python