Instructions to use lightblue/suzume-llama-3-8B-multilingual with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lightblue/suzume-llama-3-8B-multilingual with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="lightblue/suzume-llama-3-8B-multilingual")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("lightblue/suzume-llama-3-8B-multilingual")
model = AutoModelForCausalLM.from_pretrained("lightblue/suzume-llama-3-8B-multilingual", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lightblue/suzume-llama-3-8B-multilingual with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lightblue/suzume-llama-3-8B-multilingual"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightblue/suzume-llama-3-8B-multilingual",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/lightblue/suzume-llama-3-8B-multilingual

SGLang

How to use lightblue/suzume-llama-3-8B-multilingual with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lightblue/suzume-llama-3-8B-multilingual" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightblue/suzume-llama-3-8B-multilingual",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lightblue/suzume-llama-3-8B-multilingual" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightblue/suzume-llama-3-8B-multilingual",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use lightblue/suzume-llama-3-8B-multilingual with Docker Model Runner:
```
docker model run hf.co/lightblue/suzume-llama-3-8B-multilingual
```

How to add multilinguality to llama 3 tokenizer?

by richard-park - opened May 13, 2024

Discussion

richard-park

May 13, 2024

I am a newbie in the LLM field.
I'm trying to add Korean language to llama 3 tokenizer,
I have no idea where the problem is in the code.
I would really appreciate it if you could tell me what's wrong with the code.

I added the token as below, but tokenize gives me the same result.
I'm currently using a dataset of about 10G.

I used this article by huggingface as a reference.
https://huggingface.co/learn/nlp-course/chapter6/2

from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset
base_model = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model)

ds = load_dataset("richard-park/sapie-dataset-for-tokenizer", cache_dir="./cache")

def get_training_corpus():
return (
ds["train"][i : i + 1000]["text"]
for i in range(0, len(ds["train"]), 1000)
)

training_corpus = get_training_corpus()
new_tokenizer = tokenizer.train_new_from_iterator(training_corpus, 12_000)
new_tokens = set(new_tokenizer.vocab) - set(tokenizer.vocab)
print("new vocab size: ", len(new_tokens))

sample_text = ["요즘 날씨가 너무 오락가락해서 아직도 겨울옷을 못치웠어요.." ,
"맛있는 밥을 드셨습니까? 맛이 궁금하네요",
"대법원부터 하급심 판례까지 원하는 판례를 찾는 가장 빠른 방법 - 서면 검색, 요청 판례, 유사 판례, AI 추천, 판례 및 법령 검색." ,
"본 발명은 금속판의 다수 부분을 에칭시켜 특정 무늬모양을 형성하는 건축용 금속재 장식판으로 이루어진 것에 특징이 있다.",
"골다공증은 왜 생기는거에요? 그리고 치료하려면 어떻게해야하죠?",
]

tokenizer.add_tokens(list(new_tokens))
base_tokenizer = AutoTokenizer.from_pretrained(base_model)
for text in sample_text:
print("old: ", [base_tokenizer.decode([id]) for id in base_tokenizer.encode(text)])
print("new: ", [tokenizer.decode([id]) for id in tokenizer.encode(text)])
print("-"*100)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment