MaksudSharipov/Uzbek-MWE-Tokenizer-uzBERT

Uzbek MWE Tokenizer (UzBERT)

The First AI-based Uzbek MWE Tokenizer

This model identifies Multi-Word Expressions (MWEs) in Uzbek language texts using a transformer-based intelligent architecture based on UzBERT. It handles token classification for idiomatic expressions, compound nouns, and auxiliary verb constructions.

This model is fine-tuned on a Cyrillic-script dataset, enabling stable and consistent identification due to monographemic structures.

Citation

If you use this model, please cite the following papers:

Sharipov, M. S. (2026). Transformer-based intelligent model for identifying multi-word lexical units and reducing syntactic ambiguities in Uzbek language texts. Bulletin of TUIT: Management and Communication Technologies, 2(14), 103-107. DOI: 10.61663/262tuitmct14
Link: https://uzjurnal.uz/2/2026/2/index?issue=14

Plain Text:

Saidov, A., & Sharipov, M. (2025). Development of a Lemmatization Algorithm: Interpreting Open Compound Words. In Proceedings of the 17th International Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE). IEEE. DOI: 10.1109/APEIE66761.2025.11289280

BibTeX:

@inproceedings{saidov2025development,
  title={Development of a Lemmatization Algorithm: Interpreting Open Compound Words},
  author={Saidov, Abdusobir and Sharipov, Maksud},
  booktitle={Proceedings of the 17th International Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE 2025)},
  year={2025},
  organization={IEEE},
  doi={10.1109/APEIE66761.2025.11289280},
  url={[https://www.scopus.com/pages/publications/105031773662](https://www.scopus.com/pages/publications/105031773662)}
}

## How to use

We recommend using the official PyPI package, which handles both Latin and Cyrillic script conversion seamlessly:

```bash
pip install uzbek-mwe-tokenizer

Usage with the Package

from uzbek_mwe_tokenizer import UzbekMWETokenizer

# Latin mode example
tokenizer = UzbekMWETokenizer(mode="lot")
mwes = tokenizer.extract_mwe("Natijani ko'rib, hamma o'quvchilarning birdaniga tarvuzi qo'ltig'idan tushdi.")
print(mwes)

# Cyrillic mode example
tokenizer_cyr = UzbekMWETokenizer(mode="cyr")
mwes_cyr = tokenizer_cyr.extract_mwe("Рақибларимизни кўриб бизнинг асло тепа сочимиз тик бўлмади.")
print(mwes_cyr)

Expected Output:

[{'mwe': "tarvuzi qo'ltig'idan tushdi", 'confidence': 100.0}]

Direct Usage with HuggingFace Transformers

from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = "MaksudSharipov/Uzbek-MWE-Tokenizer-uzBERT"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
model = AutoModelForTokenClassification.from_pretrained(model_name)

(Note: Using the PyPI package is highly recommended as it contains built-in logic to group split subwords back into MWE phrases and supports automatic transliteration for Latin scripts).