kaan84/sinhala-print-trocr-pilot

TrOCR Sinhala Printed Lines — Pilot Weights

Exploratory pilot run. Not the pre-registered confirmatory experiment. Evaluated only on a held-out split of the same synthetic dataset used for training. No real-document evaluation has been performed. Do not cite these numbers as Sinhala OCR performance.

These are the weights from a single exploratory run completed before the project protocol was pre-registered. They are published for transparency and reproducibility of the pilot, and to make the tokenizer findings below independently verifiable.

The pre-registered protocol, scope, risk register, and release gates live in the main project repository: kaan84/sinhala-print-trocr.

Status

Item	Status
Run type	Exploratory pilot (1 seed)
Confirmatory runs (3 seeds, pre-registered config)	Seeds 42 and 123 complete; seed 2026 pending
Real-document evaluation	Not started
Dataset license (sinhala_synthetic_ocr-large)	MIT, confirmed
Checkpoint license (Ransaka/TrOCR-Sinhala)	Confirmed MIT (author, 2026-07-30)
Public redistribution of weights	Permitted under MIT

Relationship to the pre-registered protocol

The pilot used a configuration that differs from the pre-registered confirmatory experiment. Both are reported to keep the record honest:

Parameter	Pilot (this model)	Pre-registered confirmatory
Learning rate	5e-5	4e-5
Epochs	8	10
Split	80/10/10 synthetic	90/10 synthetic train/val; real test
Runs	1 (seed 42)	3 seeds (42, 123, 2026), mean ± SD
Test set	Synthetic held-out	Real line-level benchmark

The pilot's purpose was to establish that the pipeline runs end to end and to surface data and tokenizer issues. It succeeded at both: the tokenizer audit below directly shaped the pre-registered protocol. Pilot and confirmatory results are not comparable and will never be pooled.

Pilot results (synthetic only)

Test set: 697 held-out samples from Ransaka/sinhala_synthetic_ocr-large (80/10/10 split, seed 42) — the same generator, fonts, and text distribution as the training data. Metrics use NFC normalization and whitespace collapsing, with predictions and references decoded through the same tokenizer.

Metric	Zero-shot Ransaka/TrOCR-Sinhala	Pilot fine-tune
CER ↓	36.18%	16.98%
WER ↓	55.93%	32.11%
Exact match ↑	0.00%	3.87%

Relative CER reduction: 53.1% under identical evaluation. Validation CER was still falling at the final epoch (17.72%), indicating the run was data/training-limited rather than overfitted.

Confirmatory runs since completed (seed 42: 14.39% CER; seed 123: 14.75% CER on a 90/10 validation split) are reported in the project repository, not here. They are not comparable to the pilot figures above.

model-index metadata is deliberately omitted: per the project's release gates, structured metrics are added only once real-document results exist, so synthetic figures are not mistaken for benchmark results.

Context, not comparison: published page-level results on real Sinhala documents (Tesseract v5 10.69% CER; Surya 8.84% CER / 26.64% WER; Google Document AI 2.06% CER; fine-tuned LightOnOCR-2-1B 1.05% CER / 5.63% WER, arXiv:2606.29378) use a different evaluation unit and real pages. They are not directly comparable to the synthetic line-level figures above.

Tokenizer audit (key pilot finding)

Run on 5,574 training texts before training:

Diagnostic	Result	Verdict
Unknown-token rate	0.0000%	Pass
Tokens per grapheme cluster (mean)	0.301	Pass
Encode/decode round-trip failure	30.14% (1,680 texts)	Fail
— spacing-only differences	1,313 (23.6%)	Metric-neutral here
— residual character loss	367 (6.6%)	Hard ceiling
— residual involving ZWJ/ZWNJ	357 (6.4%)	Hard ceiling

Most-lost characters: U+200D ZERO WIDTH JOINER (×309), U+0020 SPACE (×293), U+200C ZERO WIDTH NON-JOINER (×180).

Interpretation. Vocabulary coverage is fine (no unknown tokens), but the inherited tokenizer cannot represent ZWJ/ZWNJ, so Sinhala conjunct forms that require them (e.g. ශ්‍රී, මාධ්‍ය) can never be produced correctly regardless of training — a hard ceiling on roughly 6.6% of texts. Separately, original punctuation spacing is destroyed and re-inserted on decode; this cancels out in the metrics above because predictions and references pass through the same tokenizer, but it will not cancel when comparing against external systems on raw ground truth. Anyone building on these weights inherits both issues. A scoped fix (vocabulary addition plus decoder embedding resize) is planned.

Training configuration

Parameter	Value
Base checkpoint	Ransaka/TrOCR-Sinhala (315.22M params)
Epochs / LR	8 / 5e-5, linear, 10% warmup
Effective batch	8 (2 × grad-accum 4)
Weight decay / precision	0.01 / FP16
Max target length	128 tokens (99th-percentile rule, clamped to [128, 256])
Decoding	beam 4, no-repeat trigram
Checkpoint selection	lowest validation CER (epoch 8)
Seed / hardware	42 / 1× NVIDIA T4 (Colab), ~1h55m

Preprocessing: NFC normalization and whitespace collapsing of labels; padding masked to −100. One record with null text was removed before splitting (6,969 → 6,968). TensorBoard logs are in runs/.

Usage

from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

repo_id = "kaan84/sinhala-print-trocr-pilot"
processor = TrOCRProcessor.from_pretrained(repo_id)
model = VisionEncoderDecoderModel.from_pretrained(repo_id)

image = Image.open("sinhala-line.png").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values, num_beams=4, no_repeat_ngram_size=3, max_length=128)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

Input must be a cropped single line of printed Sinhala text. Note that the AutoModel / pipeline("image-to-text") snippets auto-generated by the Hub do not load this architecture correctly — use VisionEncoderDecoderModel as shown.

Intended use and out of scope

Intended: reproducing the pilot; research baseline; a transfer-learning starting point.

Out of scope: real scanned or photographed documents (untested); handwriting; full pages, multi-line layouts, tables (no detection or layout stage); fonts outside the five training fonts; degraded, skewed, or low-contrast images; any production, commercial, or high-stakes use. Do not treat unreviewed OCR output as authoritative in legal, medical, financial, identity, or public-administration contexts.

Limitations and bias

1. Synthetic-only evaluation. Train, validation, and test come from one generator with five fonts and clean renders. Published Sinhala evidence shows roughly an order-of-magnitude degradation from clean synthetic to real degraded pages, so these figures overstate real-world capability.
2. Tokenizer fidelity (inherited). See the audit above: ZWJ/ZWNJ loss and spacing destruction; metrics are computed in tokenizer-normalized space.
3. Font and domain bias. Training text is web/news-style Sinhala in five modern fonts; expect degradation on legacy fonts, historical print, and formal or legal registers.
4. Single run. One seed, no confidence intervals. Confirmatory runs will report mean ± SD across three seeds.
5. Strict exact match. One spacing or joiner difference fails an entire sample.

Licensing and attribution

The full chain is MIT: microsoft/trocr-base-printed → Ransaka/TrOCR-Sinhala → Ransaka/sinhala_synthetic_ocr-large → these weights. The training dataset carries an MIT LICENSE, and the intermediate checkpoint's author confirmed MIT on 2026-07-30 and added the declaration to that repository's metadata. The correspondence record is archived in ATTRIBUTION_AND_LICENSING.md.

Attribution: Ransaka Ravihara (base checkpoint; dataset DOI 10.57967/hf/1809) and Microsoft TrOCR (Li et al., 2021, arXiv:2109.10282).

Citation

@misc{sinhala_print_trocr_pilot,
  title        = {TrOCR Sinhala Printed Lines --- Pilot Weights},
  author       = {kaan84},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/kaan84/sinhala-print-trocr-pilot}},
  note         = {Exploratory pilot; synthetic-only evaluation}
}

@misc{li2021trocr,
  title         = {TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models},
  author        = {Li, Minghao and Lv, Tengchao and Chen, Jingye and Cui, Lei and Lu, Yijuan and Florencio, Dinei and Zhang, Cha and Li, Zhoujun and Wei, Furu},
  year          = {2021},
  eprint        = {2109.10282},
  archivePrefix = {arXiv}
}

@misc{ransaka_sinhala_synthetic_ocr,
  title     = {sinhala\_synthetic\_ocr-large},
  author    = {Ravihara, Ransaka},
  year      = {2024},
  doi       = {10.57967/hf/1809},
  publisher = {Hugging Face}
}

Project roadmap

• ✅ Phase 0 — scope, pre-registered protocol, licensing review
• ✅ Pilot — synthetic baseline and tokenizer audit (this release)
• 🔄 Phase 1 confirmatory — seeds 42 and 123 done, 2026 pending (tracked in the project repo)
• ⬜ Phase 2 — deterministic line-level benchmark from avishadilhara/sinhala-ocr-lk-acts-1010 (evaluation only, never training), plus Tesseract 5.5 and optional Surya baselines
• ⬜ Phase 3 — degradation augmentation and real-data fine-tuning
• ⬜ Phase 4 — tokenizer fix (ZWJ/ZWNJ support)