Removing tokens from the tokenizer

[snoop2head]

Any methods that I can remove unwanted tokens from the tokenizer?

Referring to #4827 , I tried to remove tokens from the tokenizer with the following code.

First, I fetch the tokenizer from huggingface hub.

from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("klue/roberta-base")
print(len(tokenizer.vocab))

32000

From the fetched tokenizer, I tried to remove tokens such as [unused363].
So I first extracted tokens with 'unused' and deleted afterwards.

# get all tokens with "unused" in target_tokenizer
unwanted_words = []
for word in tokenizer.vocab:
    if "unused" in word:
        unwanted_words.append(word)

# remove all unwanted tokens from target_tokenizer
for word in unwanted_words:
    del tokenizer.vocab[word]

print(len(tokenizer.vocab))

32000

Apparently, del didn't do its job.
The list unwanted_words has 500 elements but none of which are removed from the tokenizer.

Any other methods that I can refer to?

[SaulLu]

@snoop2head , I will answer your question purely "technically" because in general it can be quite complicated to remove tokens from a tokenizer that has been trained with a certain algorithm.

Indeed, it can seems a bit complicated to modify a tokenizer fast because they wrap the tokenizers library which is not executed in python but in rust. In particular the attribute vocab cannot be modified because it is in fact just the reading of an attribute of an object which runs in rust. So when we want to modify a part of the tokenizer that runs in rust we have to recreate it.

Here are the steps to remove tokens from your vocabulary:

1. Get your tokenizer and the list of tokens you want to remove

import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("klue/roberta-base")

# get all tokens with "unused" in target_tokenizer
unwanted_words = []
for word in tokenizer.vocab:
    if "unused" in word:
        unwanted_words.append(word)

2. Get the arguments that allowed to initialize the "model" component of the backend_tokenizer.

model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))
# 32000

3. Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want

# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))
# 31500

4. Intitialize again the "model" component of the backend_tokenizer with the new vocabulary

from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))
# 31500

I take this opportunity, could you tell us more about why you want to be able to remove tokens from your vocabulary?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[SaulLu]

I will close this issue as there is not much activity. But don't hesitate to answer later if you want to bring us more information. 🙂

[kaushalshetty]

@SaulLu How can we remove stale terms and add newer terms? I am interested to train a model on evolving text where words go through a vocabulary and semantic shift. In such a case, I would train a new tokenizer on the latest data and create a new vocabulary of common terms between old and new tokenizer along with removal of stale tokens and adding newer ones from the new tokenizer.

[ajesujoba]

@SaulLu, thanks for the example you gave above. I tried using the same approach with GPT but it fails at the point of initializing the "model" component of the backend_tokenizer with the new vocabulary

#1. Get your tokenizer and the list of tokens you want to remove

import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

# get all tokens with "unused" in target_tokenizer
unwanted_words = [ 'ply', 'Ġmor','Ġprovide','IC','ung','Ġparty', 'Ġexist', 'Ġmag',]


#2. Get the arguments that allowed to initialize the "model" component of the backend_tokenizer.
model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))


#3. Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want

# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))


#4. Intitialize again the "model" component of the backend_tokenizer with the new vocabulary

from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))

The error I have is below:

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

[<ipython-input-21-fa908d23c419>](https://localhost:8080/#) in <module>
     30 model_class = getattr(models, model_state.pop("type"))
     31 
---> 32 tokenizer.backend_tokenizer.model = model_class(**model_state)
     33 
     34 print(len(tokenizer.vocab))

TypeError: argument 'merges': failed to extract enum PyMerges ('Merges | Filename')
- variant Merges (Merges): TypeError: failed to extract field PyMerges::Merges.0, caused by TypeError: 'str' object cannot be converted to 'PyTuple'
- variant Filename (Filename): TypeError: failed to extract field PyMerges::Filename.0, caused by TypeError: 'list' object cannot be converted to 'PyString'

Is there any thing I can do to get it to work?

[AmeniHmd]

@SaulLu, thanks for the example you gave above. I tried using the same approach with GPT but it fails at the point of initializing the "model" component of the backend_tokenizer with the new vocabulary

#1. Get your tokenizer and the list of tokens you want to remove

import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

# get all tokens with "unused" in target_tokenizer
unwanted_words = [ 'ply', 'Ġmor','Ġprovide','IC','ung','Ġparty', 'Ġexist', 'Ġmag',]


#2. Get the arguments that allowed to initialize the "model" component of the backend_tokenizer.
model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))


#3. Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want

# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))


#4. Intitialize again the "model" component of the backend_tokenizer with the new vocabulary

from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))

The error I have is below:

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

[<ipython-input-21-fa908d23c419>](https://localhost:8080/#) in <module>
     30 model_class = getattr(models, model_state.pop("type"))
     31 
---> 32 tokenizer.backend_tokenizer.model = model_class(**model_state)
     33 
     34 print(len(tokenizer.vocab))

TypeError: argument 'merges': failed to extract enum PyMerges ('Merges | Filename')
- variant Merges (Merges): TypeError: failed to extract field PyMerges::Merges.0, caused by TypeError: 'str' object cannot be converted to 'PyTuple'
- variant Filename (Filename): TypeError: failed to extract field PyMerges::Filename.0, caused by TypeError: 'list' object cannot be converted to 'PyString'

Is there any thing I can do to get it to work?

@ajesujoba did you find a solution

[denadai2]

@SaulLu, thanks for the example you gave above. I tried using the same approach with GPT but it fails at the point of initializing the "model" component of the backend_tokenizer with the new vocabulary

#1. Get your tokenizer and the list of tokens you want to remove

import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

# get all tokens with "unused" in target_tokenizer
unwanted_words = [ 'ply', 'Ġmor','Ġprovide','IC','ung','Ġparty', 'Ġexist', 'Ġmag',]


#2. Get the arguments that allowed to initialize the "model" component of the backend_tokenizer.
model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))


#3. Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want

# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))


#4. Intitialize again the "model" component of the backend_tokenizer with the new vocabulary

from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))

The error I have is below:

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

[<ipython-input-21-fa908d23c419>](https://localhost:8080/#) in <module>
     30 model_class = getattr(models, model_state.pop("type"))
     31 
---> 32 tokenizer.backend_tokenizer.model = model_class(**model_state)
     33 
     34 print(len(tokenizer.vocab))

TypeError: argument 'merges': failed to extract enum PyMerges ('Merges | Filename')
- variant Merges (Merges): TypeError: failed to extract field PyMerges::Merges.0, caused by TypeError: 'str' object cannot be converted to 'PyTuple'
- variant Filename (Filename): TypeError: failed to extract field PyMerges::Filename.0, caused by TypeError: 'list' object cannot be converted to 'PyString'

Is there any thing I can do to get it to work?

@ajesujoba did you find a solution

same error
:(

[AlfredTino]

@ajesujoba The successful example model is roberta, whose tokenizer is Wordpiece, meaning that it needs not merging. However, the failed example uses gpt2, whose tokenizer is BPE -- merging is necessary for byte-pair encoding.

[1kkiRen]

@SaulLu, thanks for the example you gave above. I tried using the same approach with GPT but it fails at the point of initializing the "model" component of the backend_tokenizer with the new vocabulary

#1. Get your tokenizer and the list of tokens you want to remove

import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

# get all tokens with "unused" in target_tokenizer
unwanted_words = [ 'ply', 'Ġmor','Ġprovide','IC','ung','Ġparty', 'Ġexist', 'Ġmag',]


#2. Get the arguments that allowed to initialize the "model" component of the backend_tokenizer.
model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))


#3. Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want

# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))


#4. Intitialize again the "model" component of the backend_tokenizer with the new vocabulary

from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))

The error I have is below:

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

[<ipython-input-21-fa908d23c419>](https://localhost:8080/#) in <module>
     30 model_class = getattr(models, model_state.pop("type"))
     31 
---> 32 tokenizer.backend_tokenizer.model = model_class(**model_state)
     33 
     34 print(len(tokenizer.vocab))

TypeError: argument 'merges': failed to extract enum PyMerges ('Merges | Filename')
- variant Merges (Merges): TypeError: failed to extract field PyMerges::Merges.0, caused by TypeError: 'str' object cannot be converted to 'PyTuple'
- variant Filename (Filename): TypeError: failed to extract field PyMerges::Filename.0, caused by TypeError: 'list' object cannot be converted to 'PyString'

Is there any thing I can do to get it to work?

@ajesujoba did you find a solution

same error :(

Ig I solved this issue. Check this repo

[huyiwen]

import json
from copy import deepcopy

with open("minicpm-2.4b/tokenizer.json") as f:
    tokenizer = json.load(f)

vocab = deepcopy(tokenizer["model"]["vocab"])
merges = deepcopy(tokenizer["model"]["merges"])

vocab_list = list(vocab.items())
vocab_list = sorted(vocab_list, key=lambda x: x[1])
vocab_list = [x[0] for x in vocab_list]

total = 122753
target = 99000

# prepare del_tokens
with open("del_tokens.txt") as f:
    del_tokens = set(f.read().splitlines())

# add least frequent tokens
for rule in reversed(merges):
    token = rule.replace(" ", "")
    if token in vocab and len(del_tokens) < total - target:
        # remove
        del_tokens.add(token)

# remove all merge rules
new_merges = []
for merge in merges:
    token_merge = merge.replace(" ", "")
    token_left, token_right = merge.split(" ")
    if token_left in del_tokens or token_right in del_tokens or token_merge in del_tokens:
        continue
    new_merges.append(merge)

# build new vocab
new_vocab = [v for v in vocab_list if v not in del_tokens]

# save del_tokens for preview
with open("new-removed.txt", "w") as f:
    f.write("\n".join(del_tokens))
    f.write("\n")

# write back
tokenizer["model"]["vocab"] = {k: i for i, k in enumerate(new_vocab)}
tokenizer["model"]["merges"] = new_merges
with open("new-tokenizer/tokenizer.json", "w") as f:
    json.dump(tokenizer, f, ensure_ascii=False, indent=2)

I use this snippet to remove tokens. I am not sure if this action will have any side effects.