Troubleshoot

Sometimes errors occur, but we are here to help! This guide covers some of the most common issues we’ve seen and how you can resolve them. However, this guide isn’t meant to be a comprehensive collection of every 🤗 Transformers issue. For more help with troubleshooting your issue, try:

1. Asking for help on the forums. There are specific categories you can post your question to, like Beginners or 🤗 Transformers. Make sure you write a good descriptive forum post with some reproducible code to maximize the likelihood that your problem is solved!

2. Create an Issue on the 🤗 Transformers repository if it is a bug related to the library. Try to include as much information describing the bug as possible to help us better figure out what’s wrong and how we can fix it.

3. Check the Migration guide if you use an older version of 🤗 Transformers since some important changes have been introduced between versions.

For more details about troubleshooting and getting help, take a look at Chapter 8 of the Model Database course.

Firewalled environments

Some GPU instances on cloud and intranet setups are firewalled to external connections, resulting in a connection error. When your script attempts to download model weights or datasets, the download will hang and then timeout with the following message:

ValueError: Connection error, and we cannot find the requested files in the cached path.
Please try again or make sure your Internet connection is on.

In this case, you should try to run 🤗 Transformers on offline mode to avoid the connection error.

CUDA out of memory

Training large models with millions of parameters can be challenging without the appropriate hardware. A common error you may encounter when the GPU runs out of memory is:

CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.17 GiB total capacity; 9.70 GiB already allocated; 179.81 MiB free; 9.85 GiB reserved in total by PyTorch)

Here are some potential solutions you can try to lessen memory use:

• Reduce the per_device_train_batch_size value in TrainingArguments.
• Try using gradient_accumulation_steps in TrainingArguments to effectively increase overall batch size.

Unable to load a saved TensorFlow model

TensorFlow’s model.save method will save the entire model - architecture, weights, training configuration - in a single file. However, when you load the model file again, you may run into an error because 🤗 Transformers may not load all the TensorFlow-related objects in the model file. To avoid issues with saving and loading TensorFlow models, we recommend you:

• Save the model weights as a h5 file extension with model.save_weights and then reload the model with from_pretrained():

>>> from transformers import TFPreTrainedModel
>>> from tensorflow import keras

>>> model.save_weights("some_folder/tf_model.h5")
>>> model = TFPreTrainedModel.from_pretrained("some_folder")

• Save the model with ~TFPretrainedModel.save_pretrained and load it again with from_pretrained():

>>> from transformers import TFPreTrainedModel

>>> model.save_pretrained("path_to/model")
>>> model = TFPreTrainedModel.from_pretrained("path_to/model")

ImportError

Another common error you may encounter, especially if it is a newly released model, is ImportError:

ImportError: cannot import name 'ImageGPTImageProcessor' from 'transformers' (unknown location)

For these error types, check to make sure you have the latest version of 🤗 Transformers installed to access the most recent models:

pip install transformers --upgrade

CUDA error: device-side assert triggered

Sometimes you may run into a generic CUDA error about an error in the device code.

RuntimeError: CUDA error: device-side assert triggered

You should try to run the code on a CPU first to get a more descriptive error message. Add the following environment variable to the beginning of your code to switch to a CPU:

>>> import os

>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""

Another option is to get a better traceback from the GPU. Add the following environment variable to the beginning of your code to get the traceback to point to the source of the error:

>>> import os

>>> os.environ["CUDA_LAUNCH_BLOCKING"] = "1"

Incorrect output when padding tokens aren't masked

In some cases, the output hidden_state may be incorrect if the input_ids include padding tokens. To demonstrate, load a model and tokenizer. You can access a model’s pad_token_id to see its value. The pad_token_id may be None for some models, but you can always manually set it.

>>> from transformers import AutoModelForSequenceClassification
>>> import torch

>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
>>> model.config.pad_token_id
0

The following example shows the output without masking the padding tokens:

>>> input_ids = torch.tensor([[7592, 2057, 2097, 2393, 9611, 2115], [7592, 0, 0, 0, 0, 0]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [ 0.1317, -0.1683]], grad_fn=<AddmmBackward0>)

Here is the actual output of the second sequence:

>>> input_ids = torch.tensor([[7592]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)

Most of the time, you should provide an attention_mask to your model to ignore the padding tokens to avoid this silent error. Now the output of the second sequence matches its actual output:

>>> attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 0, 0, 0, 0, 0]])
>>> output = model(input_ids, attention_mask=attention_mask)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)

🤗 Transformers doesn’t automatically create an attention_mask to mask a padding token if it is provided because:

• Some models don’t have a padding token.
• For some use-cases, users want a model to attend to a padding token.

ValueError: Unrecognized configuration class XYZ for this kind of AutoModel

Generally, we recommend using the AutoModel class to load pretrained instances of models. This class can automatically infer and load the correct architecture from a given checkpoint based on the configuration. If you see this ValueError when loading a model from a checkpoint, this means the Auto class couldn’t find a mapping from the configuration in the given checkpoint to the kind of model you are trying to load. Most commonly, this happens when a checkpoint doesn’t support a given task. For instance, you’ll see this error in the following example because there is no GPT2 for question answering:

>>> from transformers import AutoProcessor, AutoModelForQuestionAnswering

>>> processor = AutoProcessor.from_pretrained("gpt2-medium")
>>> model = AutoModelForQuestionAnswering.from_pretrained("gpt2-medium")
ValueError: Unrecognized configuration class <class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'> for this kind of AutoModel: AutoModelForQuestionAnswering.
Model type should be one of AlbertConfig, BartConfig, BertConfig, BigBirdConfig, BigBirdPegasusConfig, BloomConfig, ...