Instructions to use TheBloke/CodeLlama-34B-Python-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/CodeLlama-34B-Python-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/CodeLlama-34B-Python-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("TheBloke/CodeLlama-34B-Python-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheBloke/CodeLlama-34B-Python-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use TheBloke/CodeLlama-34B-Python-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/CodeLlama-34B-Python-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-34B-Python-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

SGLang

How to use TheBloke/CodeLlama-34B-Python-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/CodeLlama-34B-Python-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-34B-Python-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/CodeLlama-34B-Python-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-34B-Python-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use TheBloke/CodeLlama-34B-Python-GGUF with Ollama:
```
ollama run hf.co/TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use TheBloke/CodeLlama-34B-Python-GGUF with Docker Model Runner:
```
docker model run hf.co/TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M
```

Lemonade

How to use TheBloke/CodeLlama-34B-Python-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheBloke/CodeLlama-34B-Python-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.CodeLlama-34B-Python-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

CodeLlama-34B-Python-GGUF

Commit History

Sep 27, 2023

12:46
Update base_model formatting

TheBloke

3f2c2ff

Sep 19, 2023

01:11
Upload README.md

TheBloke

a4103e4

Sep 7, 2023

11:13
Upload README.md

TheBloke

1c2b591

Sep 5, 2023

14:12
Upload README.md

TheBloke

1dafb0a
13:11
Upload README.md

TheBloke

eccfa9d
13:11
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

dbdeaaf
13:11
Add Llama 2 license files

TheBloke

a7d5b81
13:10
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

065bd00
13:09
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

35077b7
13:06
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

3491683
13:03
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

3c2462e
13:01
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

8c9e563
12:59
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

17834bd
12:56
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

9ffd8b4
12:53
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

34ffc25
12:51
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

a35d5af
12:49
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

896cfff
12:47
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

4debd48
12:45
Initial GGUF model commit (model made with llama.cpp commit d59bd97)

TheBloke

f2616a0
02:12
Upload README.md

TheBloke

d79f382
02:12
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

0f600bd
02:12
Add Llama 2 license files

TheBloke

ce9f56d
02:12
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

877190e
02:09
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

b1e53f8
02:07
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

7fdfa47
02:04
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

a59d733
02:01
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

e86f95c
01:59
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

d956cd4
01:56
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

25b3b55
01:53
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

7e6e3f4
01:51
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

531133a
01:49
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

cca1526
01:46
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

c4a2f60
01:45
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

842e41f

Aug 26, 2023

15:46
Initial GGUF model commit

TheBloke

1bd20e4
15:46
Initial GGUF model commit

TheBloke

0d3378e
15:46
Add Llama 2 license files

TheBloke

1ccb186
15:46
Add Llama 2 license files

TheBloke

09b763d
15:46
Add Llama 2 license files

TheBloke

fbf52d6
15:43
Initial GGUF model commit

TheBloke

25973b8
15:43
Initial GGUF model commit

TheBloke

d9f6e8f
15:43
Add Llama 2 license files

TheBloke

4c0c8b2
15:43
Add Llama 2 license files

TheBloke

cdaffb2
15:43
Add Llama 2 license files

TheBloke

dca511f
11:14
Initial GGUF model commit

TheBloke

ba908f8
11:14
Initial GGUF model commit

TheBloke

0cde384
11:14
Add Llama 2 license files

TheBloke

0234b54
11:14
Add Llama 2 license files

TheBloke

21cb3d3
11:14
Add Llama 2 license files

TheBloke

5507b15

Aug 25, 2023

15:33
Initial GGUF model commit

TheBloke

4c8b8c2

Commit History

Sep 27, 2023

Update base_model formatting

Sep 19, 2023

Upload README.md

Sep 7, 2023

Upload README.md

Sep 5, 2023

Upload README.md

Upload README.md

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Add Llama 2 license files

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Initial GGUF model commit (model made with llama.cpp commit d59bd97)

Upload README.md

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Add Llama 2 license files

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Aug 26, 2023

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Aug 25, 2023

Initial GGUF model commit