itcen-entec/gemma-3-4b-tax-law-lora-gguf

Gemma 3 4B Tax Law LoRA (Korean) - GGUF

🇰🇷 Korean tax-law Q/A specialized LoRA fine-tuned model based on Gemma 3 4B, converted to GGUF format for efficient inference.

Model Details

• Developed by: hongjoong-entec
• Model type: Causal LM with LoRA fine-tuning (GGUF format)
• Language(s): ['ko']
• License: apache-2.0
• Finetuned from: google/gemma-3-4b-it
• Original model: hongjoong-entec/gemma-3-4b-tax-law-lora
• Format: GGUF (for llama.cpp, Ollama, etc.)

Uses

Direct Use

• Korean tax-law Q/A and related explanations in Korean
• Tax consulting assistance (with professional review)
• Efficient inference with llama.cpp, Ollama, or other GGUF-compatible tools

Out-of-Scope Use

• Legal advice without professional review
• Sensitive decisions without human validation

How to Get Started with the Model

Using with Ollama

# Download the model
ollama pull hongjoong-entec/gemma-3-4b-tax-law-lora-gguf

# Or use locally if you have the GGUF file
ollama create gemma3-tax-law -f Modelfile

# Run the model
ollama run gemma3-tax-law "접대비 한도는 어떻게 계산하나요?"

Using with llama.cpp

# Download the GGUF file
wget https://huggingface.co/hongjoong-entec/gemma-3-4b-tax-law-lora-gguf/resolve/main/gemma-3-4b-tax-law-lora.gguf

# Run with llama.cpp
./main -m gemma-3-4b-tax-law-lora.gguf -p "접대비 한도는 어떻게 계산하나요?" -n 256

Using with Python (llama-cpp-python)

from llama_cpp import Llama

# Load the GGUF model
llm = Llama(
    model_path="./gemma-3-4b-tax-law-lora.gguf",
    n_ctx=2048,
    n_threads=8,
)

# Prompt template for best results
prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Respond in Korean and provide only the final answer without chain-of-thought, analysis, section headers, or disclaimers.
If needed, include a brief basis in one concise sentence.

### Question:
{}

### Response:
{}"""

# Example usage
question = "접대비 한도는 어떻게 계산하나요?"
prompt = prompt_style.format(question, "")

# Generate response
output = llm(
    prompt,
    max_tokens=1024,
    temperature=0.01,
    top_p=0.95,
    stop=["###", "Question:", "Response:"],
    echo=False
)

print(output['choices'][0]['text'].strip())

Model Files

This repository contains the following files:

• gemma-3-4b-tax-law-lora.gguf: The quantized GGUF model file
• Modelfile: Ollama configuration file

Training Details

Original Training Data

• Internal processed Korean tax-law QA subset
• Dataset format: legalqa_fino1_format.jsonl
• Focus: Korean tax regulations and Q/A pairs

Original Training Procedure (LoRA)

• Base model: google/gemma-3-4b-it
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Hyperparameters:
  • r=64
  • alpha=16
  • dropout=0.05
  • Learning rate: 2e-4
  • Batch size: 1 (with gradient accumulation)

GGUF Conversion

• Converted from the original LoRA model: hongjoong-entec/gemma-3-4b-tax-law-lora
• Format: GGUF for efficient CPU/GPU inference
• Compatible with llama.cpp, Ollama, and other GGUF-based tools

Performance

Inference Performance

• CPU inference: Optimized for consumer hardware
• Memory usage: Reduced compared to original PyTorch model
• Speed: Faster inference on CPU with quantization benefits

Model Quality

• Maintains the quality of the original LoRA fine-tuned model
• Specialized for Korean tax-law queries
• Manual evaluation shows good accuracy on tax-related questions

Evaluation

• Qualitative evaluation on Korean tax-law prompts
• ROUGE scoring against reference answers (from original model)
• Manual review for accuracy and relevance

Limitations

⚠️ Important Disclaimers:

• This model is for informational purposes only and does not constitute legal advice
• Always consult with qualified tax professionals for official guidance
• Outputs may be outdated or incomplete and require human validation
• Tax laws change frequently - verify current regulations
• GGUF quantization may introduce minor quality differences compared to the original model

Technical Specifications

Model Architecture

• Based on Google Gemma 3 4B architecture
• LoRA fine-tuning applied to attention and MLP layers
• Converted to GGUF format for efficient inference

Hardware Requirements

• Minimum RAM: 8GB (for CPU inference)
• Recommended RAM: 16GB or more
• GPU: Optional, but recommended for faster inference

Usage Tips

1. Prompt Template: Use the provided prompt template for best results
2. Temperature: Low temperature (0.01-0.1) recommended for factual responses
3. Context Length: Model supports up to 8192 tokens context
4. Stop Tokens: Use appropriate stop tokens to prevent over-generation

Model Performance

The GGUF model demonstrates good performance on Korean tax-related queries while offering:

• Faster inference compared to the original PyTorch model
• Lower memory requirements
• Better compatibility with edge deployment scenarios

Use this model as a starting point for research rather than definitive answers.