Instructions to use h2oai/h2ogpt-gm-oasst1-en-1024-12b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2ogpt-gm-oasst1-en-1024-12b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="h2oai/h2ogpt-gm-oasst1-en-1024-12b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-1024-12b")
model = AutoModelForCausalLM.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-1024-12b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2ogpt-gm-oasst1-en-1024-12b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2ogpt-gm-oasst1-en-1024-12b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2ogpt-gm-oasst1-en-1024-12b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/h2oai/h2ogpt-gm-oasst1-en-1024-12b

SGLang

How to use h2oai/h2ogpt-gm-oasst1-en-1024-12b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2ogpt-gm-oasst1-en-1024-12b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2ogpt-gm-oasst1-en-1024-12b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2ogpt-gm-oasst1-en-1024-12b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2ogpt-gm-oasst1-en-1024-12b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use h2oai/h2ogpt-gm-oasst1-en-1024-12b with Docker Model Runner:
```
docker model run hf.co/h2oai/h2ogpt-gm-oasst1-en-1024-12b
```

Adding Evaluation Results

by leaderboard-pr-bot - opened Nov 17, 2023

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+14

-1

Files changed (1) hide show

README.md +14 -1

README.md CHANGED Viewed

@@ -196,4 +196,17 @@ Please read this disclaimer carefully before using the large language model prov
 - Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
 - Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
-By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.

 - Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
 - Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
+By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_h2oai__h2ogpt-gm-oasst1-en-1024-12b)
+| Metric                | Value                     |
+|-----------------------|---------------------------|
+| Avg.                  | 35.6   |
+| ARC (25-shot)         | 43.09          |
+| HellaSwag (10-shot)   | 69.75    |
+| MMLU (5-shot)         | 25.87         |
+| TruthfulQA (0-shot)   | 38.0   |
+| Winogrande (5-shot)   | 66.14   |
+| GSM8K (5-shot)        | 1.06        |
+| DROP (3-shot)         | 5.27         |