r/LanguageTechnology • • Mar 31 '21

A Python library to boost T5 models speed up to 5x & reduce the model size by 3x.

I wanted to share this new library I've been working on and that I open-sourced!.

here are some links to the library:

💻 GitHub Repository

🐍 PyPi project

as the title suggests, you can increase the inference speed of any pretrained T5 model and also decrease the models' size, in a single line of code.

The library can be installed with pip install fastt5. This code snippet from the repository's README gives a concise overview:

from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer

model_name = 't5-small'
model = export_and_get_onnx_model(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)
t_input = "translate English to French: The universe is a dark forest."
token = tokenizer(t_input, return_tensors='pt')

tokens = model.generate(input_ids=token['input_ids'],
               attention_mask=token['attention_mask'],
               num_beams=2)

output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)

The fastT5 library exports the T5 model to onnx with past_key_values, then quantizes it and runs it on onnxruntime.

The exported onnx models support the generate() method of huggingface transformers for inferencing.

for more information on the project refer to the repository here.

69 Upvotes

6 comments sorted by

1

u/jtoma5 Mar 31 '21

Is the a knowledge distillation technique like tinybert?

1

u/strngelet Apr 01 '21

no, it exports PyTorch T5 models to onnx and runs on onnxruntime.

1

u/jtoma5 Apr 01 '21

Thanks. Took a look at this while I was in line today, so good to know about this. Your application looks focused on the fine-tuning stage, but apart from quantizing, onnx looks to bring improvements to pretraining to?

1

u/strngelet Apr 01 '21

the library is only for inference, not for fine-tuning or pretraining, It exports an already pretrained torch model to onnx. (The onnx models can be converted to any other frameworks like TensorFlow, caffe2.. etc.) Then we can quantize the onnx for decreasing the model size (it also increases the inference speed). finally, the model is run on onnxruntime to speed up the inference.

you can also train models on onnxruntime.

3

u/ajan1019 Mar 31 '21

This looks great. Thanks for sharing.

1

u/MultiheadAttention Jan 22 '23

Does this library supports batched inference on GPU?