t5-sl-large

t5-sl-large model is a Slovene T5 model. It has 24 encoder and 24 decoder layers, in total about 750 million parameters. It was trained for 3 epochs on the following corpora:

Corpora

The following corpora were used for training the model:

• Gigafida 2.0
• Kas 1.0
• Janes 1.0 (only Janes-news, Janes-forum, Janes-blog, Janes-wiki subcorpora)
• Slovenian parliamentary corpus siParl 2.0
• slWaC

Evaluation

The model is described in detail and evaluated in our paper "Sequence to sequence pretraining for a less-resourced Slovenian language"