r/LanguageTechnology • u/strngelet • Mar 31 '21
A Python library to boost T5 models speed up to 5x & reduce the model size by 3x.
I wanted to share this new library I've been working on and that I open-sourced!.
here are some links to the library:
as the title suggests, you can increase the inference speed of any pretrained T5 model and also decrease the models' size, in a single line of code.
The library can be installed with pip install fastt5. This code snippet from the repository's README gives a concise overview:
from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer
model_name = 't5-small'
model = export_and_get_onnx_model(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
t_input = "translate English to French: The universe is a dark forest."
token = tokenizer(t_input, return_tensors='pt')
tokens = model.generate(input_ids=token['input_ids'],
attention_mask=token['attention_mask'],
num_beams=2)
output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)The fastT5 library exports the T5 model to onnx with past_key_values, then quantizes it and runs it on onnxruntime.
The exported onnx models support the generate() method of huggingface transformers for inferencing.
for more information on the project refer to the repository here.
3
1
1
u/jtoma5 Mar 31 '21
Is the a knowledge distillation technique like tinybert?