Instructions to use facebook/KernelLLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use facebook/KernelLLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="facebook/KernelLLM") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("facebook/KernelLLM") model = AutoModelForCausalLM.from_pretrained("facebook/KernelLLM", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use facebook/KernelLLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "facebook/KernelLLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "facebook/KernelLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/facebook/KernelLLM
- SGLang
How to use facebook/KernelLLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "facebook/KernelLLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "facebook/KernelLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "facebook/KernelLLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "facebook/KernelLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use facebook/KernelLLM with Docker Model Runner:
docker model run hf.co/facebook/KernelLLM
Commit History
-
Update README.md
Verified signature26d1191 -
Update README.md
Verified signature2260245
-
Update README.md
Verified signature41e3dbf -
Update README.md
Verified signaturec9e070b
-
Update README.md
Verified signatureee6d3c4 -
Added Updates section
Verified signature52bf86f
-
Update README.md
Verified signature13e2283
-
Update README.md
Verified signature8bbf4cb
-
Adding library tag for better download tracking as well as task tags (#4)
Verified signature- Adding library tag for better download tracking as well as task tags (dbe1d8f8cc88e21a9244e497b18904a88fdd3657) Co-authored-by: Vaibhav Srivastav <[email protected]>347beb3 -
Update README.md
Verified signature1a5b094 -
Update README.md
Verified signatureb6d6b91
-
Upload folder using huggingface_hub
Verified signaturea74f195 -
Update README.md
Verified signatureff49b9c -
Update README.md
Verified signature428cf76 -
Update README.md
Verified signature19545c6 -
Upload llm_performance_comparison.png
Verified signaturef5663dc -
Upload kernelllm_pass_at_k_scaling.png
Verified signature924a822 -
Upload blog_post_model_performance.png
Verified signature86ad378
-
Update README.md
Verified signature86d9bc4 -
Update README.md
Verified signature1019e2b -
Rename triton-kernel-workflow.png to media/triton-kernel-workflow.png
Verified signature61aae47 -
Rename llm_performance_comparison.png to media/llm_performance_comparison.png
Verified signaturec3a6615 -
Rename blog_post_model_performance.png to media/blog_post_model_performance.png
Verified signatureb81de95 -
Rename kernelllm_pass_at_k_scaling_PASS_AT_200.png to media/kernelllm_pass_at_k_scaling.png
Verified signature2c6b244 -
Rename blog_post_model_performance_rev_4_RC.png to blog_post_model_performance.png
Verified signaturedb2abc2 -
Rename llm_performance_comparison_rev_5_RC.png to llm_performance_comparison.png
Verified signature73ce538 -
Upload triton-kernel-workflow.png
Verified signature1086969 -
Upload 3 files
Verified signaturec56bd9c -
Update README.md
Verified signature30bdfe4 -
Create kernelllm.py
Verified signature36a8d36
-
Update base_model to `Llama-3.1-8B-Instruct` (#1)
Verified signature- Update base_model to `Llama-3.1-8B-Instruct` (9e29264936095f19bb6437c655adf23c76579942) Co-authored-by: Zacharias Fisches <[email protected]>da6c75f -
Fix generated response text
Verified signaturejoin="" does not exist for python 3.10 And a regular print was printing text with jumbled \n This PR instead just make the code look nice ```python import torch import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() def forward(self, a, b): return a + b def get_inputs(): # randomly generate input tensors based on the model architecture a = torch.randn(1, 128).cuda() b = torch.randn(1, 128).cuda() return [a, b] def get_init_inputs(): # randomly generate tensors required for initialization based on the model architecture return [] ``` The example new arch with custom Triton kernels looks like this: ``` import torch import torch.nn as nn import torch.nn.functional as F import triton import triton.language as tl @triton.jit def add_kernel( x_ptr, # Pointer to first input y_ptr, # Pointer to second input out_ptr, ```3c9bd97
-
Update README.md
Verified signature8064ae2 -
Update README.md
Verified signatured20aedd -
Upload LICENSE.pdf with huggingface_hub
Verified signature78816ff -
Update README.md
Verified signature3544c26
-
Upload folder using huggingface_hub
Verified signaturef43ffe9 -
initial commit
Verified signaturee156f25