# Pretrained models that work with **finetune_transformers_pytorch.ipynb**

|Status|Architecture|Shortcut name|
 |:-|:-|:-|
 |Worked|BERT|bert-base-uncased|
 |Worked|BERT|bert-large-uncased|
 |Worked|BERT|bert-base-cased|
 |Worked|BERT|bert-large-cased|
 |Worked|BERT|bert-base-multilingual-uncased|
 |Worked|BERT|bert-base-multilingual-cased|
 |Worked|BERT|bert-base-chinese|
 |Worked|BERT|bert-base-german-cased|
 |Worked|BERT|bert-large-uncased-whole-word-masking|
 |Worked|BERT|bert-large-cased-whole-word-masking|
 |Worked|BERT|bert-large-uncased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-large-cased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-base-cased-finetuned-mrpc|
 |Worked|BERT|bert-base-german-dbmdz-cased|
 |Worked|BERT|bert-base-german-dbmdz-uncased|
 |Failed|BERT|cl-tohoku/bert-base-japanese|
 |Failed|BERT|cl-tohoku/bert-base-japanese-whole-word-masking|
 |Failed|BERT|cl-tohoku/bert-base-japanese-char|
 |Failed|BERT|cl-tohoku/bert-base-japanese-char-whole-word-masking|
 |Worked|BERT|TurkuNLP/bert-base-finnish-cased-v1|
 |Worked|BERT|TurkuNLP/bert-base-finnish-uncased-v1|
 |Worked|BERT|wietsedv/bert-base-dutch-cased|
 |Failed|GPT|openai-gpt|
 |Failed|GPT-2|gpt2|
 |Failed|GPT-2|gpt2-medium|
 |Failed|GPT-2|gpt2-large|
 |Failed|GPT-2|gpt2-xl|
 |Failed|Transformer-XL|transfo-xl-wt103|
 |Worked|XLNet|xlnet-base-cased|
 |Worked|XLNet|xlnet-large-cased|
 |Worked|XLM|xlm-mlm-en-2048|
 |Worked|XLM|xlm-mlm-ende-1024|
 |Worked|XLM|xlm-mlm-enfr-1024|
 |Worked|XLM|xlm-mlm-enro-1024|
 |Worked|XLM|xlm-mlm-xnli15-1024|
 |Worked|XLM|xlm-mlm-tlm-xnli15-1024|
 |Worked|XLM|xlm-clm-enfr-1024|
 |Worked|XLM|xlm-clm-ende-1024|
 |Worked|XLM|xlm-mlm-17-1280|
 |Failed|XLM|xlm-mlm-100-1280|
 |Worked|RoBERTa|roberta-base|
 |Worked|RoBERTa|roberta-large|
 |Failed|RoBERTa|roberta-large-mnli|
 |Worked|RoBERTa|distilroberta-base|
 |Worked|RoBERTa|roberta-base-openai-detector|
 |Worked|RoBERTa|roberta-large-openai-detector|
 |Worked|DistilBERT|distilbert-base-uncased|
 |Worked|DistilBERT|distilbert-base-uncased-distilled-squad|
 |Worked|DistilBERT|distilbert-base-cased|
 |Worked|DistilBERT|distilbert-base-cased-distilled-squad|
 |Failed|DistilBERT|distilgpt2|
 |Worked|DistilBERT|distilbert-base-german-cased|
 |Worked|DistilBERT|distilbert-base-multilingual-cased|
 |Failed|CTRL|ctrl|
 |Worked|CamemBERT|camembert-base|
 |Worked|ALBERT|albert-base-v1|
 |Worked|ALBERT|albert-large-v1|
 |Worked|ALBERT|albert-xlarge-v1|
 |Worked|ALBERT|albert-xxlarge-v1|
 |Worked|ALBERT|albert-base-v2|
 |Worked|ALBERT|albert-large-v2|
 |Worked|ALBERT|albert-xlarge-v2|
 |Worked|ALBERT|albert-xxlarge-v2|
 |Failed|T5|t5-small|
 |Failed|T5|t5-base|
 |Failed|T5|t5-large|
 |Failed|T5|t5-3B|
 |Failed|T5|t5-11B|
 |Worked|XLM-RoBERTa|xlm-roberta-base|
 |Worked|XLM-RoBERTa|xlm-roberta-large|
 |Worked|FlauBERT|flaubert/flaubert_small_cased|
 |Worked|FlauBERT|flaubert/flaubert_base_uncased|
 |Worked|FlauBERT|flaubert/flaubert_base_cased|
 |Worked|FlauBERT|flaubert/flaubert_large_cased|
 |Worked|Bart|facebook/bart-large|
 |Worked|Bart|facebook/bart-base|
 |Failed|Bart|facebook/bart-large-mnli|
 |Worked|Bart|facebook/bart-large-cnn|
 |Failed|DialoGPT|DialoGPT-small|
 |Failed|DialoGPT|DialoGPT-medium|
 |Failed|DialoGPT|DialoGPT-large|
 |Failed|Reformer|reformer-enwik8|
 |Failed|Reformer|reformer-crime-and-punishment|
 |Failed|MarianMT|Helsinki-NLP/opus-mt-{src}-{tgt}|
 |Failed|Pegasus|google/pegasus-{dataset}|
 |Worked|Longformer|allenai/longformer-base-4096|
 |Worked|Longformer|allenai/longformer-large-4096|
 |Failed|MBart|facebook/mbart-large-cc25|
 |Failed|MBart|facebook/mbart-large-en-ro|
 |Failed|Lxmert|lxmert-base-uncased|
 |Worked|Funnel Transformer|funnel-transformer/small|
 |Worked|Funnel Transformer|funnel-transformer/small-base|
 |Worked|Funnel Transformer|funnel-transformer/medium|
 |Worked|Funnel Transformer|funnel-transformer/medium-base|
 |Worked|Funnel Transformer|funnel-transformer/intermediate|
 |Worked|Funnel Transformer|funnel-transformer/intermediate-base|
 |Worked|Funnel Transformer|funnel-transformer/large|
 |Worked|Funnel Transformer|funnel-transformer/large-base|
 |Worked|Funnel Transformer|funnel-transformer/xlarge|
 |Worked|Funnel Transformer|funnel-transformer/xlarge-base|
 |Failed|LayoutLM|microsoft/layoutlm-base-uncased|
 |Failed|LayoutLM|microsoft/layoutlm-large-uncased|
 |Worked|DeBERTa|microsoft/deberta-base|
 |Worked|DeBERTa|microsoft/deberta-large|
 |Worked|SqueezeBERT|squeezebert/squeezebert-uncased|
 |Failed|SqueezeBERT|squeezebert/squeezebert-mnli|
 |Worked|BERT|bert-base-uncased|
 |Worked|BERT|bert-large-uncased|
 |Worked|BERT|bert-base-cased|
 |Worked|BERT|bert-large-cased|
 |Worked|BERT|bert-base-multilingual-uncased|
 |Worked|BERT|bert-base-multilingual-cased|
 |Worked|BERT|bert-base-chinese|
 |Worked|BERT|bert-base-german-cased|
 |Worked|BERT|bert-large-uncased-whole-word-masking|
 |Worked|BERT|bert-large-cased-whole-word-masking|
 |Worked|BERT|bert-large-uncased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-large-cased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-base-cased-finetuned-mrpc|
 |Worked|BERT|bert-base-german-dbmdz-cased|
 |Worked|BERT|bert-base-german-dbmdz-uncased|
 |Failed|BERT|cl-tohoku/bert-base-japanese|
 |Failed|BERT|cl-tohoku/bert-base-japanese-whole-word-masking|
 |Failed|BERT|cl-tohoku/bert-base-japanese-char|
 |Failed|BERT|cl-tohoku/bert-base-japanese-char-whole-word-masking|
 |Worked|BERT|TurkuNLP/bert-base-finnish-cased-v1|
 |Worked|BERT|TurkuNLP/bert-base-finnish-uncased-v1|
 |Worked|BERT|wietsedv/bert-base-dutch-cased|
 |Failed|GPT|openai-gpt|
 |Failed|GPT-2|gpt2|
 |Failed|GPT-2|gpt2-medium|
 |Failed|GPT-2|gpt2-large|
 |Failed|GPT-2|gpt2-xl|
 |Failed|Transformer-XL|transfo-xl-wt103|
 |Worked|XLNet|xlnet-base-cased|
 |Worked|XLNet|xlnet-large-cased|
 |Worked|XLM|xlm-mlm-en-2048|
 |Worked|XLM|xlm-mlm-ende-1024|
 |Worked|XLM|xlm-mlm-enfr-1024|
 |Worked|XLM|xlm-mlm-enro-1024|
 |Worked|XLM|xlm-mlm-xnli15-1024|
 |Worked|XLM|xlm-mlm-tlm-xnli15-1024|
 |Worked|XLM|xlm-clm-enfr-1024|
 |Worked|XLM|xlm-clm-ende-1024|
 |Worked|XLM|xlm-mlm-17-1280|
 |Failed|XLM|xlm-mlm-100-1280|
 |Worked|RoBERTa|roberta-base|
 |Worked|RoBERTa|roberta-large|
 |Failed|RoBERTa|roberta-large-mnli|
 |Worked|RoBERTa|distilroberta-base|
 |Worked|RoBERTa|roberta-base-openai-detector|
 |Worked|RoBERTa|roberta-large-openai-detector|
 |Worked|DistilBERT|distilbert-base-uncased|
 |Worked|DistilBERT|distilbert-base-uncased-distilled-squad|
 |Worked|DistilBERT|distilbert-base-cased|
 |Worked|DistilBERT|distilbert-base-cased-distilled-squad|
 |Failed|DistilBERT|distilgpt2|
 |Worked|DistilBERT|distilbert-base-german-cased|
 |Worked|DistilBERT|distilbert-base-multilingual-cased|
 |Failed|CTRL|ctrl|
 |Worked|CamemBERT|camembert-base|
 |Worked|ALBERT|albert-base-v1|
 |Worked|ALBERT|albert-large-v1|
 |Worked|ALBERT|albert-xlarge-v1|
 |Worked|ALBERT|albert-xxlarge-v1|
 |Worked|ALBERT|albert-base-v2|
 |Worked|ALBERT|albert-large-v2|
 |Worked|ALBERT|albert-xlarge-v2|
 |Worked|ALBERT|albert-xxlarge-v2|
 |Failed|T5|t5-small|
 |Failed|T5|t5-base|
 |Failed|T5|t5-large|
 |Failed|T5|t5-3B|
 |Failed|T5|t5-11B|
 |Worked|XLM-RoBERTa|xlm-roberta-base|
 |Worked|XLM-RoBERTa|xlm-roberta-large|
 |Worked|FlauBERT|flaubert/flaubert_small_cased|
 |Worked|FlauBERT|flaubert/flaubert_base_uncased|
 |Worked|FlauBERT|flaubert/flaubert_base_cased|
 |Worked|FlauBERT|flaubert/flaubert_large_cased|
 |Worked|Bart|facebook/bart-large|
 |Worked|Bart|facebook/bart-base|
 |Failed|Bart|facebook/bart-large-mnli|
 |Worked|Bart|facebook/bart-large-cnn|
 |Failed|DialoGPT|DialoGPT-small|
 |Failed|DialoGPT|DialoGPT-medium|
 |Failed|DialoGPT|DialoGPT-large|
 |Failed|Reformer|reformer-enwik8|
 |Failed|Reformer|reformer-crime-and-punishment|
 |Failed|MarianMT|Helsinki-NLP/opus-mt-{src}-{tgt}|
 |Failed|Pegasus|google/pegasus-{dataset}|
 |Worked|Longformer|allenai/longformer-base-4096|
 |Worked|Longformer|allenai/longformer-large-4096|
 |Failed|MBart|facebook/mbart-large-cc25|
 |Failed|MBart|facebook/mbart-large-en-ro|
 |Failed|Lxmert|lxmert-base-uncased|
 |Worked|Funnel Transformer|funnel-transformer/small|
 |Worked|Funnel Transformer|funnel-transformer/small-base|
 |Worked|Funnel Transformer|funnel-transformer/medium|
 |Worked|Funnel Transformer|funnel-transformer/medium-base|
 |Worked|Funnel Transformer|funnel-transformer/intermediate|
 |Worked|Funnel Transformer|funnel-transformer/intermediate-base|
 |Worked|Funnel Transformer|funnel-transformer/large|
 |Worked|Funnel Transformer|funnel-transformer/large-base|
 |Worked|Funnel Transformer|funnel-transformer/xlarge|
 |Worked|Funnel Transformer|funnel-transformer/xlarge-base|
 |Failed|LayoutLM|microsoft/layoutlm-base-uncased|
 |Failed|LayoutLM|microsoft/layoutlm-large-uncased|
 |Worked|DeBERTa|microsoft/deberta-base|
 |Worked|DeBERTa|microsoft/deberta-large|
 |Worked|SqueezeBERT|squeezebert/squeezebert-uncased|
 |Failed|SqueezeBERT|squeezebert/squeezebert-mnli|
 |Worked|BERT|bert-base-uncased|
 |Worked|BERT|bert-large-uncased|
 |Worked|BERT|bert-base-cased|
 |Worked|BERT|bert-large-cased|
 |Worked|BERT|bert-base-multilingual-uncased|
 |Worked|BERT|bert-base-multilingual-cased|
 |Worked|BERT|bert-base-chinese|
 |Worked|BERT|bert-base-german-cased|
 |Worked|BERT|bert-large-uncased-whole-word-masking|
 |Worked|BERT|bert-large-cased-whole-word-masking|
 |Worked|BERT|bert-large-uncased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-large-cased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-base-cased-finetuned-mrpc|
 |Worked|BERT|bert-base-german-dbmdz-cased|
 |Worked|BERT|bert-base-german-dbmdz-uncased|
 |Failed|BERT|cl-tohoku/bert-base-japanese|
 |Failed|BERT|cl-tohoku/bert-base-japanese-whole-word-masking|
 |Failed|BERT|cl-tohoku/bert-base-japanese-char|
 |Failed|BERT|cl-tohoku/bert-base-japanese-char-whole-word-masking|
 |Worked|BERT|TurkuNLP/bert-base-finnish-cased-v1|
 |Worked|BERT|TurkuNLP/bert-base-finnish-uncased-v1|
 |Worked|BERT|wietsedv/bert-base-dutch-cased|
 |Failed|GPT|openai-gpt|
 |Failed|GPT-2|gpt2|
 |Failed|GPT-2|gpt2-medium|
 |Failed|GPT-2|gpt2-large|
 |Failed|GPT-2|gpt2-xl|
 |Failed|Transformer-XL|transfo-xl-wt103|
 |Worked|XLNet|xlnet-base-cased|
 |Worked|XLNet|xlnet-large-cased|
 |Worked|XLM|xlm-mlm-en-2048|
 |Worked|XLM|xlm-mlm-ende-1024|
 |Worked|XLM|xlm-mlm-enfr-1024|
 |Worked|XLM|xlm-mlm-enro-1024|
 |Worked|XLM|xlm-mlm-xnli15-1024|
 |Worked|XLM|xlm-mlm-tlm-xnli15-1024|
 |Worked|XLM|xlm-clm-enfr-1024|
 |Worked|XLM|xlm-clm-ende-1024|
 |Worked|XLM|xlm-mlm-17-1280|
 |Failed|XLM|xlm-mlm-100-1280|
 |Worked|RoBERTa|roberta-base|
 |Worked|RoBERTa|roberta-large|
 |Failed|RoBERTa|roberta-large-mnli|
 |Worked|RoBERTa|distilroberta-base|
 |Worked|RoBERTa|roberta-base-openai-detector|
 |Worked|RoBERTa|roberta-large-openai-detector|
 |Worked|DistilBERT|distilbert-base-uncased|
 |Worked|DistilBERT|distilbert-base-uncased-distilled-squad|
 |Worked|DistilBERT|distilbert-base-cased|
 |Worked|DistilBERT|distilbert-base-cased-distilled-squad|
 |Failed|DistilBERT|distilgpt2|
 |Worked|DistilBERT|distilbert-base-german-cased|
 |Worked|DistilBERT|distilbert-base-multilingual-cased|
 |Failed|CTRL|ctrl|
 |Worked|CamemBERT|camembert-base|
 |Worked|ALBERT|albert-base-v1|
 |Worked|ALBERT|albert-large-v1|
 |Worked|ALBERT|albert-xlarge-v1|
 |Worked|ALBERT|albert-xxlarge-v1|
 |Worked|ALBERT|albert-base-v2|
 |Worked|ALBERT|albert-large-v2|
 |Worked|ALBERT|albert-xlarge-v2|
 |Worked|ALBERT|albert-xxlarge-v2|
 |Failed|T5|t5-small|
 |Failed|T5|t5-base|
 |Failed|T5|t5-large|
 |Failed|T5|t5-3B|
 |Failed|T5|t5-11B|
 |Worked|XLM-RoBERTa|xlm-roberta-base|
 |Worked|XLM-RoBERTa|xlm-roberta-large|
 |Worked|FlauBERT|flaubert/flaubert_small_cased|
 |Worked|FlauBERT|flaubert/flaubert_base_uncased|
 |Worked|FlauBERT|flaubert/flaubert_base_cased|
 |Worked|FlauBERT|flaubert/flaubert_large_cased|
 |Worked|Bart|facebook/bart-large|
 |Worked|Bart|facebook/bart-base|
 |Failed|Bart|facebook/bart-large-mnli|
 |Worked|Bart|facebook/bart-large-cnn|
 |Failed|DialoGPT|DialoGPT-small|
 |Failed|DialoGPT|DialoGPT-medium|
 |Failed|DialoGPT|DialoGPT-large|
 |Failed|Reformer|reformer-enwik8|
 |Failed|Reformer|reformer-crime-and-punishment|
 |Failed|MarianMT|Helsinki-NLP/opus-mt-{src}-{tgt}|
 |Failed|Pegasus|google/pegasus-{dataset}|
 |Worked|Longformer|allenai/longformer-base-4096|
 |Worked|Longformer|allenai/longformer-large-4096|
 |Failed|MBart|facebook/mbart-large-cc25|
 |Failed|MBart|facebook/mbart-large-en-ro|
 |Failed|Lxmert|lxmert-base-uncased|
 |Worked|Funnel Transformer|funnel-transformer/small|
 |Worked|Funnel Transformer|funnel-transformer/small-base|
 |Worked|Funnel Transformer|funnel-transformer/medium|
 |Worked|Funnel Transformer|funnel-transformer/medium-base|
 |Worked|Funnel Transformer|funnel-transformer/intermediate|
 |Worked|Funnel Transformer|funnel-transformer/intermediate-base|
 |Worked|Funnel Transformer|funnel-transformer/large|
 |Worked|Funnel Transformer|funnel-transformer/large-base|
 |Worked|Funnel Transformer|funnel-transformer/xlarge|
 |Worked|Funnel Transformer|funnel-transformer/xlarge-base|
 |Failed|LayoutLM|microsoft/layoutlm-base-uncased|
 |Failed|LayoutLM|microsoft/layoutlm-large-uncased|
 |Worked|DeBERTa|microsoft/deberta-base|
 |Worked|DeBERTa|microsoft/deberta-large|
 |Worked|SqueezeBERT|squeezebert/squeezebert-uncased|
 |Failed|SqueezeBERT|squeezebert/squeezebert-mnli|
 |Worked|BERT|bert-base-uncased|
 |Worked|BERT|bert-large-uncased|
 |Worked|BERT|bert-base-cased|
 |Worked|BERT|bert-large-cased|
 |Worked|BERT|bert-base-multilingual-uncased|
 |Worked|BERT|bert-base-multilingual-cased|
 |Worked|BERT|bert-base-chinese|
 |Worked|BERT|bert-base-german-cased|
 |Worked|BERT|bert-large-uncased-whole-word-masking|
 |Worked|BERT|bert-large-cased-whole-word-masking|
 |Worked|BERT|bert-large-uncased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-large-cased-whole-word-masking-finetuned-squad|
 |Worked|BERT|bert-base-cased-finetuned-mrpc|
 |Worked|BERT|bert-base-german-dbmdz-cased|
 |Worked|BERT|bert-base-german-dbmdz-uncased|
 |Worked|BERT|TurkuNLP/bert-base-finnish-cased-v1|
 |Worked|BERT|TurkuNLP/bert-base-finnish-uncased-v1|
 |Worked|BERT|wietsedv/bert-base-dutch-cased|
 |Worked|XLNet|xlnet-base-cased|
 |Worked|XLNet|xlnet-large-cased|
 |Worked|XLM|xlm-mlm-en-2048|
 |Worked|XLM|xlm-mlm-ende-1024|
 |Worked|XLM|xlm-mlm-enfr-1024|
 |Worked|XLM|xlm-mlm-enro-1024|
 |Worked|XLM|xlm-mlm-xnli15-1024|
 |Worked|XLM|xlm-mlm-tlm-xnli15-1024|
 |Worked|XLM|xlm-clm-enfr-1024|
 |Worked|XLM|xlm-clm-ende-1024|
 |Worked|XLM|xlm-mlm-17-1280|
 |Worked|RoBERTa|roberta-base|
 |Worked|RoBERTa|roberta-large|
 |Worked|RoBERTa|distilroberta-base|
 |Worked|RoBERTa|roberta-base-openai-detector|
 |Worked|RoBERTa|roberta-large-openai-detector|
 |Worked|DistilBERT|distilbert-base-uncased|
 |Worked|DistilBERT|distilbert-base-uncased-distilled-squad|
 |Worked|DistilBERT|distilbert-base-cased|
 |Worked|DistilBERT|distilbert-base-cased-distilled-squad|
 |Worked|DistilBERT|distilbert-base-german-cased|
 |Worked|DistilBERT|distilbert-base-multilingual-cased|
 |Worked|CamemBERT|camembert-base|
 |Worked|ALBERT|albert-base-v1|
 |Worked|ALBERT|albert-large-v1|
 |Worked|ALBERT|albert-xlarge-v1|
 |Worked|ALBERT|albert-xxlarge-v1|
 |Worked|ALBERT|albert-base-v2|
 |Worked|ALBERT|albert-large-v2|
 |Worked|ALBERT|albert-xlarge-v2|
 |Worked|ALBERT|albert-xxlarge-v2|
 |Worked|XLM-RoBERTa|xlm-roberta-base|
 |Worked|XLM-RoBERTa|xlm-roberta-large|
 |Worked|FlauBERT|flaubert/flaubert_small_cased|
 |Worked|FlauBERT|flaubert/flaubert_base_uncased|
 |Worked|FlauBERT|flaubert/flaubert_base_cased|
 |Worked|FlauBERT|flaubert/flaubert_large_cased|
 |Worked|Bart|facebook/bart-large|
 |Worked|Bart|facebook/bart-base|
 |Worked|Bart|facebook/bart-large-cnn|
 |Worked|Longformer|allenai/longformer-base-4096|
 |Worked|Longformer|allenai/longformer-large-4096|
 |Worked|Funnel Transformer|funnel-transformer/small|
 |Worked|Funnel Transformer|funnel-transformer/small-base|
 |Worked|Funnel Transformer|funnel-transformer/medium|
 |Worked|Funnel Transformer|funnel-transformer/medium-base|
 |Worked|Funnel Transformer|funnel-transformer/intermediate|
 |Worked|Funnel Transformer|funnel-transformer/intermediate-base|
 |Worked|Funnel Transformer|funnel-transformer/large|
 |Worked|Funnel Transformer|funnel-transformer/large-base|
 |Worked|Funnel Transformer|funnel-transformer/xlarge|
 |Worked|Funnel Transformer|funnel-transformer/xlarge-base|
 |Worked|DeBERTa|microsoft/deberta-base|
 |Worked|DeBERTa|microsoft/deberta-large|
 |Worked|SqueezeBERT|squeezebert/squeezebert-uncased|
