Instructions to use bigscience/bloom-560m-intermediate with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bigscience/bloom-560m-intermediate with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bigscience/bloom-560m-intermediate")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-560m-intermediate")
model = AutoModelForCausalLM.from_pretrained("bigscience/bloom-560m-intermediate")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use bigscience/bloom-560m-intermediate with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bigscience/bloom-560m-intermediate"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/bloom-560m-intermediate",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/bigscience/bloom-560m-intermediate

SGLang

How to use bigscience/bloom-560m-intermediate with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bigscience/bloom-560m-intermediate" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/bloom-560m-intermediate",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bigscience/bloom-560m-intermediate" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bigscience/bloom-560m-intermediate",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use bigscience/bloom-560m-intermediate with Docker Model Runner:
```
docker model run hf.co/bigscience/bloom-560m-intermediate
```

There might be some thing wrong with the 500,000 and 600,000 step checkpoint

by TingchenFu - opened May 21, 2023

Discussion

TingchenFu

May 21, 2023

I tried to evaluate the intermediate checkpoint by perplexity on WMT21 zh-en translation test set. Specifically, I ran all checkpoints from 1000step to 600,000 step and the ppl on the English translation (only the English reference is fed into the model to calculate ppl) are 392, 71, 53, 51, 50, 50, 71, 320322 respectively. It seems that the 500,000step checkpoint is the same as the 10,000 step checkpoint and there are some errors with the 600,000 step checkpoint.

ErikaaWang

Oct 7, 2023

Yes agree! I also found 10k and 500k the same. Btw, why do you think there is an error on 600k?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment