Morteza89/llama-3.3-70b-instruct-int4-ov

Llama 3.3 70B Instruct OpenVINO INT4

Built with Llama.

This repository contains an unofficial OpenVINO™ IR conversion of meta-llama/Llama-3.3-70B-Instruct with INT4 weight compression.

The model is intended for local text generation using OpenVINO GenAI. OpenVINO GenAI provides an optimized generative inference API for OpenVINO models and is the recommended runtime path here for getting strong performance on Intel hardware, including Intel CPUs and supported Intel GPUs.

Original model

• Original model: meta-llama/Llama-3.3-70B-Instruct
• Original creator: Meta
• License: Llama 3.3 Community License
• Converted format: OpenVINO IR
• Weight format: INT4
• Task: text generation

This is an unofficial converted model repository. It is not an official Meta or OpenVINO release.

Please refer to the original model card and the included LICENSE file for full license terms, usage restrictions, acceptable use policy, attribution requirements, and limitations.

License and redistribution

This model is distributed under the Llama 3.3 Community License.

The original license permits use, reproduction, distribution, copying, creation of derivative works, and modification of the Llama materials, subject to the license terms. Redistribution requires providing the license agreement, prominently displaying “Built with Llama,” and retaining the required attribution notice. This repository includes a NOTICE file for that attribution.

Model summary

Llama 3.3 70B Instruct is a multilingual instruction-tuned text-only large language model optimized for dialogue and general assistant-style use cases. The upstream model is distributed by Meta through Hugging Face.

This OpenVINO version is designed for efficient local inference with OpenVINO GenAI.

Why OpenVINO GenAI?

This repository is designed to run with openvino-genai, specifically openvino_genai.LLMPipeline.

OpenVINO GenAI is optimized for generative AI inference with OpenVINO. Its LLM pipeline can load an OpenVINO-converted model folder directly, including the model, tokenizer, detokenizer, chat template, and generation configuration. This makes it a clean and efficient runtime path for local LLM inference on Intel hardware.

Conversion

This model was converted with Optimum Intel using the OpenVINO export path.

optimum-cli export openvino \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --weight-format int4 \
  --trust-remote-code \
  ov_Llama-3.3-70B-Instruct_INT4
Quantization
Weight format: INT4
Export tool: Optimum Intel
Runtime: OpenVINO GenAI
Model format: OpenVINO IR

INT4 compression is intended to reduce model size and memory usage compared with higher precision weights. As with any converted and quantized model, quality and numerical behavior may differ from the original model and should be validated for your use case.
Usage
Run from inside the model directory
python examples/run_llama33_openvino_genai.py \
  --model-dir . \
  --device CPU \
  --max-new-tokens 128 \
  --prompt "Explain what OpenVINO is in one short paragraph."

Tested environment

Example tested environment:

Environment: ov_hf_env
OpenVINO: 2026.1.0-21367-63e31528c62-releases/2026/1
OpenVINO GenAI: 2026.1.0.0-2957-1dabb8c2255
Notes
This model is text-only.
This repository uses OpenVINO GenAI for inference instead of a Transformers runtime.
The example script supports deterministic decoding by default and sampling through --do-sample.
OpenVINO Model Server compatibility is not claimed unless separately validated.
Limitations

This repository inherits the limitations and restrictions of the original meta-llama/Llama-3.3-70B-Instruct model and the Llama 3.3 Community License. Additional differences may arise from OpenVINO conversion, INT4 compression, and runtime configuration.

Attribution

Built with Llama.

This is an unofficial OpenVINO conversion of the original Meta Llama 3.3 model. All rights to the original model, training, and licensing remain with the original authors.