finetune_cls_vs_content_12class_v5

TL;DR

A multi-task Vietnamese web content classifier built on PhoBERT. Given a text segment extracted from a Vietnamese website, the model simultaneously predicts:

Task 1 (Binary): Whether the content constitutes a copyright violation (0 = safe, 1 = violation) — F1 macro 0.983
Task 2 (12-class): The content category of the website — F1 macro 0.790

This is the production-grade v5 checkpoint, the best-performing model in a 5-version iterative fine-tuning project.

Model Details

Model Description

finetune_cls_vs_content_12class_v5 is a dual-task text classifier fine-tuned on Vietnamese web content segments. It extends the 10-class backbone ttqdunggg/finetune_cls_vs_content_ronbackbone_100k (PhoBERT + multi-task head) to cover 12 content categories including two newly added classes: Phim ảnh (Film/Entertainment) and Thể thao (Sports).

The model is the culmination of 5 progressive fine-tuning iterations (v1–v5), each addressing specific data quality issues identified from misclassification analysis. v5 achieved the best overall Macro F1 (0.790) across all versions, with particular improvements in MXH (+0.157 vs v4) driven by strict data cleaning strategies.

Developed by: phucleDio
Model type: Multi-task Sequence Classification (PhoBERT / RoBERTa-based)
Languages: Vietnamese (vi)
License: MIT
Fine-tuned from: ttqdunggg/finetune_cls_vs_content_ronbackbone_100k
Base backbone: vinai/phobert-base-v2 (RoBERTa architecture, ~135M parameters)

Model Sources

Repository: phucleDio/finetune_cls_vs_content_12class_v5
Predecessor (10-class backbone): ttqdunggg/finetune_cls_vs_content_ronbackbone_100k

Uses

Direct Use

This model is designed for automated classification of Vietnamese website content segments. Each input is a text chunk (up to 256 Vietnamese word-piece tokens after ViTokenizer segmentation) extracted from a webpage.

Primary use cases:

Copyright violation detection in Vietnamese web crawling pipelines (Task 1)
Content category labeling for downstream filtering, moderation, or analytics (Task 2)
Identifying high-risk content domains (gambling, adult content, lending, etc.)

Downstream Use

The model can be plugged into:

Web crawling and content indexing pipelines as a pre-filter
Copyright enforcement systems that need to classify the nature of flagged content
Content moderation dashboards
Recommender system pre-processing to filter out certain content categories

Out-of-Scope Use

Non-Vietnamese content: The model is trained exclusively on Vietnamese web text and will produce unreliable results on other languages.
Very short inputs (< 20 tokens): The model may not have sufficient context to classify reliably.
Individual social media posts or comments: The training data consists of full webpage segments, not micro-text.
Image, audio, or video content: Text-only model; does not process multimedia.
Fine-grained sub-category classification: E.g., distinguishing specific sports types or film genres.

The 12 Content Categories (Task 2)

Index	Label	CSV `type`	Description
0	Báo chí	1	News media, journalism, online newspapers
1	18+	2	Adult/explicit content
2	Cờ bạc	3	Gambling, betting, lottery
3	Vay	4	Lending, loans, financial credit services
4	Tiền ảo	5	Cryptocurrency, NFT, blockchain
5	Tổ chức	6	Organizations, government agencies, educational institutions
6	E-commerce	7	Online shopping, product listings, retail
7	MXH	8	Social media platforms and community pages
8	Game	9	Online gaming, game portals
9	Chưa xác định	10	Unclassified / ambiguous content
10	Phim ảnh	11	Film, TV, video entertainment (new in v1)
11	Thể thao	12	Sports news, results, commentary (new in v1)

Bias, Risks, and Limitations

Known Limitations

E-commerce ↔ Tổ chức confusion (structural): The most persistent misclassification (≈27–32% of E-commerce predictions) is E-commerce → Tổ chức. Many Vietnamese e-commerce websites include company registration info, tax IDs, and organizational-looking footer boilerplate that overlaps with organizational content. This is partly an annotation ambiguity issue baked into the original dataset.
MXH underrepresentation: With only 551 training samples, MXH (social media) is the smallest class. While v5 recovers to F1=0.600, it remains the lowest-performing class and is sensitive to data distribution shifts.
Báo chí ↔ Phim ảnh / Thể thao overlap: Vietnamese entertainment news sites (e.g., kenh14.vn, 2sao.vn) produce content that straddles Báo chí and Phim ảnh. Sports news on general newspapers overlaps with Thể thao.
18+ recall sensitivity: With only 32 validation samples, the 18+ class F1 (0.889) has high variance and may not generalize reliably to unseen adult content domains.
Vietnamese-only: The model uses ViTokenizer word segmentation and PhoBERT's Vietnamese vocabulary. Non-Vietnamese text will be poorly tokenized.

Recommendations

For production use, consider post-processing rules based on URL/domain patterns (e.g., .gov.vn → Tổ chức) to resolve E-commerce ↔ Tổ chức ambiguity.
Apply a confidence threshold on Task 2 predictions; for low-confidence outputs (softmax max < 0.5), treat as Chưa xác định.
Monitor MXH performance carefully in deployment; add more training data if precision drops below 0.50.

How to Get Started with the Model

import torch
from transformers import AutoTokenizer
from pyvi import ViTokenizer

# Load tokenizer from PhoBERT base
tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base-v2")

# Load model
from transformers import AutoModel
import torch.nn as nn

class PhoBERTMultiTask12(nn.Module):
    def __init__(self, base_model_name="vinai/phobert-base-v2",
                 num_labels_task1=2, num_labels_task2=12):
        super().__init__()
        self.roberta = AutoModel.from_pretrained(base_model_name)
        hidden = self.roberta.config.hidden_size
        self.classifier_task1 = nn.Sequential(
            nn.Linear(hidden, hidden), nn.Tanh(), nn.Dropout(0.1),
            nn.Linear(hidden, num_labels_task1)
        )
        self.classifier_task2 = nn.Sequential(
            nn.Linear(hidden, hidden), nn.Tanh(), nn.Dropout(0.1),
            nn.Linear(hidden, num_labels_task2)
        )

    def forward(self, input_ids, attention_mask, **kwargs):
        outputs = self.roberta(input_ids=input_ids, attention_mask=attention_mask)
        cls = outputs.last_hidden_state[:, 0, :]
        return self.classifier_task1(cls), self.classifier_task2(cls)

# Inference
LABEL_NAMES = {
    0: "Bao chi", 1: "18+", 2: "Co bac", 3: "Vay", 4: "Tien ao",
    5: "To chuc", 6: "E-commerce", 7: "MXH", 8: "Game",
    9: "Chua xac dinh", 10: "Phim anh", 11: "The thao"
}

def predict(text: str, model, tokenizer, device="cpu"):
    # Vietnamese word segmentation (required for PhoBERT)
    segmented = ViTokenizer.tokenize(text)
    
    inputs = tokenizer(
        segmented,
        return_tensors="pt",
        max_length=256,
        padding="max_length",
        truncation=True
    ).to(device)
    
    model.eval()
    with torch.no_grad():
        logits1, logits2 = model(**inputs)
    
    binary_pred = torch.argmax(logits1, dim=-1).item()
    category_pred = torch.argmax(logits2, dim=-1).item()
    
    return {
        "copyright_violation": bool(binary_pred),      # Task 1
        "content_category": LABEL_NAMES[category_pred], # Task 2
        "confidence_task1": torch.softmax(logits1, dim=-1).max().item(),
        "confidence_task2": torch.softmax(logits2, dim=-1).max().item(),
    }

# Example
model = PhoBERTMultiTask12()
# Load weights from HuggingFace Hub
# model.load_state_dict(torch.load(...))

text = "Mua ngay! Thêm vào giỏ hàng. Giá ưu đãi 199.000đ. Miễn phí giao hàng toàn quốc."
result = predict(text, model, tokenizer)
# {'copyright_violation': False, 'content_category': 'E-commerce', ...}

Training Details

Training Data

Dataset: CLS_segment_25K_12class_v5.csv — 24,186 segments from Vietnamese websites.

Class	Count	%	Weight (v5)
Báo chí	1,411	5.8%	5.11
18+	214	0.9%	20.00 (capped)
Cờ bạc	836	3.5%	8.62
Vay	1,356	5.6%	5.31
Tiền ảo	584	2.4%	10.00 (capped)
Tổ chức	4,787	19.8%	1.51
E-commerce	7,206	29.8%	1.00 (ref)
MXH	551	2.3%	10.00 (capped)
Game	676	2.8%	10.00 (capped)
Chưa xác định	4,288	17.7%	1.68
Phim ảnh	850	3.5%	8.48
Thể thao	1,427	5.9%	5.05

Train / Val split: 85% / 15%, stratified by class (type).

Training set: 20,558 segments
Validation set: 3,628 segments

Data provenance and key cleaning decisions across versions:

Version	Key Data Change
v1	23,916 rows — 10-class base (21,983) + keyword-matched Phim ảnh (433) + Thể thao (1,500) from 90k pool
v2	24,351 rows — Phim ảnh expanded to 850 via 3-source combine (pool + Báo chí_22k + Chua_xd_22k)
v3	24,006 rows — Removed 345 contradictory-label rows (same domain labeled as both Báo chí and Phim ảnh). Pool-only strategy adopted
v4	24,895 rows — +500 E-commerce hard samples + 500 MXH enrichment (later found noisy)
v5	24,186 rows — MXH reverted to clean v3 (551); Footer boilerplate stripped from 290 E-commerce segments; 229 low-quality E-commerce rows removed via Strict Shopping-Feature Rule

v5 Strict Shopping-Feature Rule (E-commerce filter): A segment is kept in E-commerce training only if it contains at least one of: giỏ hàng, thêm vào giỏ, xem giỏ hàng, bảng giá, mua ngay, đơn hàng. 229 segments without these signals were discarded.

Footer Boilerplate Stripping: 290 E-commerce segments had boilerplate text (Giấy chứng nhận ĐKKD, MST, Địa chỉ trụ sở, Powered by) stripped before training to prevent false Tổ chức associations.

Training Procedure

Architecture

PhoBERT Backbone (vinai/phobert-base-v2, ~135M params)
         │  [CLS] embedding
    ┌────┴────┐
    ▼         ▼
Task 1      Task 2
Linear→Tanh→Dropout→Linear(2)   Linear→Tanh→Dropout→Linear(12)
Binary (0=safe, 1=violation)     12-class content category

Weight transfer from the 10-class backbone:

roberta.* and classifier_task1.*: copied 1:1
classifier_task2.out_proj.weight: first 10 rows copied, 2 new rows randomly initialized
classifier_task2.out_proj.bias: first 10 copied, 2 new = 0

Training Hyperparameters (v5 Final)

The training followed a 2-phase schedule:

Phase 1 — Warmup (3 epochs, backbone frozen):

Parameter	Value
Learning rate	`1e-4`
LR scheduler	`linear`
Label smoothing	`0.05`
Frozen	`roberta.` + `classifier_task1.`
Task1 / Task2 loss weight	0.1 / 0.9
Batch size (effective)	128 (64 × 2 grad accum)
Sequence length	256
Precision	fp16

Phase 2 — End-to-End Fine-tune (7 epochs, all parameters trainable):

Parameter	Value
Learning rate	`1.5e-5`
LR scheduler	`linear`
Warmup steps	3 × steps/epoch
Label smoothing	`0.08`
Max grad norm	`1.0`
Task1 / Task2 loss weight	0.35 / 0.65
Early stopping patience	4 epochs
Batch size (effective)	128 (64 × 2 grad accum)
Weight decay	0.01
Precision	fp16

Loss function:

Total Loss = 0.35 × CrossEntropy(logits1, labels_task1)
           + 0.65 × WeightedCrossEntropy(logits2, labels_task2, class_weights)
Label smoothing applied to both tasks.

Combined metric for early stopping:

f1_combined = harmonic_mean(F1_task1_macro, F1_task2_macro)

Speeds, Sizes, Times

Property	Value
Model size	~542 MB (safetensors)
Training compute	Google Colab A100 GPU
Approximate training time	~2–3 hours per version
Inference (CPU)	~30–80ms per segment
Inference (GPU)	~5–15ms per segment

Evaluation

Testing Data

Validation set: 3,628 segments (15% stratified split from CLS_segment_25K_12class_v5.csv), held out during training.

Metrics

F1 macro: Unweighted mean of per-class F1. Primary metric for Task 2 due to class imbalance.
Accuracy: Overall prediction accuracy.
Per-class precision / recall / F1: Reported for each of the 12 content categories.

Results

Task 1 — Binary Classification (Copyright Violation Detection)

              precision    recall  f1-score   support

           0     0.9931    0.9964    0.9948      3053
           1     0.9805    0.9635    0.9719       575

    accuracy                         0.9912      3628
   macro avg     0.9868    0.9799    0.9833      3628

Task 2 — 12-class Content Classification

                precision    recall  f1-score   support

      Bao chi     0.8000    0.7736    0.7866       212
          18+     0.9032    0.8750    0.8889        32
       Co bac     0.8815    0.9520    0.9154       125
          Vay     0.9256    0.9803    0.9522       203
      Tien ao     0.7812    0.8523    0.8152        88
      To chuc     0.6127    0.8593    0.7154       718
   E-commerce     0.9463    0.6031    0.7367      1081
          MXH     0.5327    0.6867    0.6000        83
         Game     0.8056    0.8614    0.8325       101
Chua xac dinh     0.7857    0.7869    0.7863       643
     Phim anh     0.6118    0.7266    0.6643       128
     The thao     0.7490    0.8364    0.7903       214

     accuracy                         0.7652      3628
    macro avg     0.7779    0.8161    0.7903      3628

Cross-task Overlap Matrix (Báo chí / Chưa xác định / Phim ảnh / Thể thao)

                 Bao chi  Chua xd  Phim anh  The thao
Bao chi            0.90     0.04      0.03      0.03
Chua xac dinh      0.02     0.94      0.02      0.01
Phim anh           0.02     0.05      0.89      0.05
The thao           0.02     0.01      0.02      0.96

Version History (Task 2 Macro F1)

Version	Macro F1	Key Change
v1	0.797	First 12-class extension; Phim ảnh (433) + Thể thao (1,500) from keyword search
v2	0.777	+417 Phim ảnh from 3-source combine; weight capping at 10.0; cosine LR
v3	0.795	Contradictory label removal (345 rows); Pool-only strategy — Phim ảnh F1 ↑ to 0.615
v4	0.770	+500 MXH (noisy) + 500 E-com hard samples; MXH F1 collapsed to 0.443
v5	0.790	MXH reverted to clean v3; footer stripping; strict E-com filter; MXH ↑ +0.157

Summary

v5 achieves the best overall Macro F1 (0.790) across all 5 versions. Key highlights:

Task 1 (copyright binary): F1 0.983, Accuracy 99.12% — consistently excellent
Top-performing classes: Vay (0.952), Cờ bạc (0.915), 18+ (0.889), Game (0.833)
Weakest class: MXH (0.600) due to limited training data (551 samples)
Persistent challenge: E-commerce ↔ Tổ chức confusion (~28% of E-com predicted as Tổ chức)

Environmental Impact

Carbon emissions estimated using the Machine Learning Impact calculator.

Hardware Type: NVIDIA A100 (Google Colab Pro+)
Hours used: ~3 hours per training run × 5 versions ≈ 15 GPU-hours total
Cloud Provider: Google Colab
Compute Region: Unknown (Colab managed infrastructure)
Carbon Emitted: ~0.5–1.5 kg CO₂eq estimated (15 A100-hours)

Technical Specifications

Model Architecture

Backbone: PhoBERT (RoBERTa-based), 12 transformer layers, 768 hidden size, ~135M parameters
Task 1 Head: Linear(768 → 768) → Tanh → Dropout(0.1) → Linear(768 → 2)
Task 2 Head: Linear(768 → 768) → Tanh → Dropout(0.1) → Linear(768 → 12)
Classifier input: [CLS] token representation
Tokenization: ViTokenizer (Vietnamese word segmentation) → PhoBERT BPE tokenizer, max_length=256

Software Stack

Component	Version
Python	3.10
PyTorch	2.x
Transformers (HuggingFace)	4.40+
pyvi (ViTokenizer)	0.1.x
scikit-learn	1.x
Accelerate	latest

Glossary

Term	Definition
Segment	A text chunk extracted from a Vietnamese webpage, typically 50–300 words
Task 1	Binary classification: does this content violate copyright? (0=no, 1=yes)
Task 2	12-class classification: what content category is this website?
Macro F1	Unweighted average of per-class F1 — treats all classes equally regardless of support
Class weight	Inverse-frequency weight applied to loss function to compensate for class imbalance
Footer stripping	Removal of boilerplate legal/company registration text from webpage segments
ViTokenizer	Vietnamese word segmenter from the `pyvi` library; required for PhoBERT input

Citation

If you use this model in your work, please cite:

BibTeX:

@misc{phucleDio2026contentclassifier,
  author       = {phucleDio},
  title        = {Vietnamese Web Content Classifier (12-class, v5)},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/phucleDio/finetune_cls_vs_content_12class_v5}},
  note         = {Fine-tuned from ttqdunggg/finetune_cls_vs_content_ronbackbone_100k}
}