Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,149 @@
+---
+language:
+  - en
+  - zh
+library_name: transformers
+license: mit
+pipeline_tag: text-generation
+---
+
+# GLM-5-FP8
+
+<div align="center">
+<img src=https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/logo.svg width="15%"/>
+</div>
+<p align="center">
+    👋 Join our <a href="https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/wechat.png" target="_blank">WeChat</a> or <a href="https://discord.gg/QR7SARHRxK" target="_blank">Discord</a> community.
+    <br>
+    📖 Check out the GLM-5 <a href="https://z.ai/blog/glm-5" target="_blank">technical blog</a>.
+    <br>
+    📍 Use GLM-5 API services on <a href="https://docs.z.ai/guides/llm/glm-5">Z.ai API Platform. </a>
+    <br>
+    👉 One click to <a href="https://chat.z.ai">GLM-5</a>.
+</p>
+
+## Introduction
+
+We are launching GLM-5, targeting complex systems engineering and long-horizon agentic tasks. Scaling is still one of the most important ways to improve the intelligence efficiency of Artificial General Intelligence (AGI). Compared to GLM-4.5, GLM-5 scales from 355B parameters (32B active) to 744B parameters (40B active), and increases pre-training data from 23T to 28.5T tokens. GLM-5 also integrates DeepSeek Sparse Attention (DSA), largely reducing deployment cost while preserving long-context capacity.
+
+Reinforcement learning aims to bridge the gap between competence and excellence in pre-trained models. However, deploying it at scale for LLMs is a challenge due to the RL training inefficiency. To this end, we developed [slime](https://github.com/THUDM/slime), a novel **asynchronous RL infrastructure** that substantially improves training throughput and efficiency, enabling more fine-grained post-training iterations. With advances in both pre-training and post-training, GLM-5 delivers significant improvement compared to GLM-4.7 across a wide range of academic benchmarks and achieves best-in-class performance among all open-source models in the world on reasoning, coding, and agentic tasks,  closing the gap with frontier models.
+
+## Benchmark
+
+|                                  | GLM-5                  | GLM-4.7   | DeepSeek-V3.2 | Kimi K2.5 | Claude Opus 4.5 | Gemini 3 Pro | GPT-5.2 (xhigh) |
+| -------------------------------- | ---------------------- | --------- | ------------- |-----------| --------------- | ------------ | --------------- |
+| HLE                              | 30.5                   | 24.8      | 25.1          | 31.5      | 28.4            | 37.2         | 35.4            |
+| HLE (w/ Tools)                   | 50.4                   | 42.8      | 40.8          | 51.8      | 43.4*           | 45.8*        | 45.5*           |
+| AIME 2026 I                      | 92.7                   | 92.9      | 92.7          | 92.5      | 93.3            | 90.6         | -               |
+| HMMT Nov. 2025                   | 96.9                   | 93.5      | 90.2          | 91.1      | 91.7            | 93.0         | 97.1            |
+| IMOAnswerBench                   | 82.5                   | 82.0      | 78.3          | 81.8      | 78.5            | 83.3         | 86.3            |
+| GPQA-Diamond                     | 86.0                   | 85.7      | 82.4          | 87.6      | 87.0            | 91.9         | 92.4            |
+| SWE-bench Verified               | 77.8                   | 73.8      | 73.1          | 76.8      | 80.9            | 76.2         | 80.0            |
+| SWE-bench Multilingual           | 73.3                   | 66.7      | 70.2          | 73.0      | 77.5            | 65.0         | 72.0            |
+| Terminal-Bench 2.0 (Terminus 2)  | 56.2 / 60.7 † | 41.0      | 39.3          | 50.8      | 59.3            | 54.2         | 54.0            |
+| Terminal-Bench 2.0 (Claude Code) | 56.2 / 61.1 †  | 32.8      | 46.4          | -         | 57.9            | -            | -               |
+| CyberGym                         | 43.2                   | 23.5      | 17.3          | 41.3      | 50.6            | 39.9         | -               |
+| BrowseComp                       | 62.0                   | 52.0      | 51.4          | 60.6      | 37.0            | 37.8         | -               |
+| BrowseComp (w/ Context Manage)   | 75.9                   | 67.5      | 67.6          | 74.9      | 67.8            | 59.2         | 65.8            |
+| BrowseComp-Zh                    | 72.7                   | 66.6      | 65.0          | 62.3      | 62.4            | 66.8         | 76.1            |
+| τ²-Bench                         | 89.7                   | 87.4      | 85.3          | 80.2      | 91.6            | 90.7         | 85.5            |
+| MCP-Atlas (Public Set)           | 67.8                   | 52.0      | 62.2          | 63.8      | 65.2            | 66.6         | 68.0            |
+| Tool-Decathlon                   | 38.0                   | 23.8      | 35.2          | 27.8      | 43.5            | 36.4         | 46.3            |
+| Vending Bench 2                  | $4,432.12              | $2,376.82 | $1,034.00     | $1,198.46 | $4,967.06       | $5,478.16    | $3,591.33       |
+
+> *: refers to their scores of full set.
+> 
+> †: A verified version of Terminal-Bench 2.0 that fixes some ambiguous instructions.
+See footnote for more evaluation details.
+
+### Footnote
+
+* **Humanity’s Last Exam (HLE) & other reasoning tasks**: We evaluate with a maximum generation length of 131,072 tokens (`temperature=1.0, top_p=0.95, max_new_tokens=131072`). By default, we report the text-only subset; results marked with * are from the full set. We use GPT-5.2 (medium) as the judge model. For HLE-with-tools, we use a maximum context length of 202,752 tokens.
+* **SWE-bench & SWE-bench Multilingual**: We run the SWE-bench suite with OpenHands using a tailored instruction prompt. Settings: `temperature=0.7, top_p=0.95, max_new_tokens=16384`, with a 200K context window.
+* **BrowserComp**: Without context management, we retain details from the most recent 5 turns. With context management, we use the same discard-all strategy as DeepSeek-v3.2 and Kimi K2.5.
+* **Terminal-Bench 2.0 (Terminus 2)**: We evaluate with the Terminus framework using `timeout=2h, temperature=0.7, top_p=1.0, max_new_tokens=8192`, with a 128K context window. Resource limits are capped at 16 CPUs and 32 GB RAM.
+* **Terminal-Bench 2.0 (Claude Code)**: We evaluate in Claude Code 2.1.14 (think mode, default effort) with `temperature=1.0, top_p=0.95, max_new_tokens=65536`. We remove wall-clock time limits due to generation speed, while preserving per-task CPU and memory constraints. Scores are averaged over 5 runs. We fix environment issues introduced by Claude Code and also report results on a verified Terminal-Bench 2.0 dataset that resolves ambiguous instructions (see: [https://huggingface.co/datasets/zai-org/terminal-bench-2-verified](https://huggingface.co/datasets/zai-org/terminal-bench-2-verified)).
+* **CyberGym**: We evaluate in Claude Code 2.1.18 (think mode, no web tools) with (`temperature=1.0, top_p=1.0, max_new_tokens=32000`) and a 250-minute timeout per task. Results are single-run Pass@1 over 1,507 tasks.
+* **MCP-Atlas**: All models are evaluated in think mode on the 500-task public subset with a 10-minute timeout per task. We use Gemini 3 Pro as the judge model.
+* **τ²-bench**: We add a small prompt adjustment in Retail and Telecom to avoid failures caused by premature user termination. For Airline, we apply the domain fixes proposed in the Claude Opus 4.5 system card.
+* **Vending Bench 2**: Runs are conducted independently by [Andon Labs](https://andonlabs.com/evals/vending-bench-2).
+
+
+## Serve GLM-5 Locally
+
+### Prepare environment
+
+vLLM, SGLang, and xLLM all support local deployment of GLM-5. A simple deployment guide is provided here.
+
++ vLLM
+
+    Using Docker as:
+
+    ```shell
+    docker pull vllm/vllm-openai:nightly 
+    ```
+
+    or using pip:
+
+    ```shell
+    pip install -U vllm --pre --index-url https://pypi.org/simple --extra-index-url https://wheels.vllm.ai/nightly
+    ```
+
+    then upgrade transformers:
+
+    ```
+    pip install git+https://github.com/huggingface/transformers.git
+    ```
+
++ SGLang
+
+    Using Docker as:
+    ```bash
+    docker pull lmsysorg/sglang:glm5-hopper # For Hopper GPU
+    docker pull lmsysorg/sglang:glm5-blackwell # For Blackwell GPU
+    ```
+
+### Deploy
+
++ vLLM
+
+    ```shell
+    vllm serve zai-org/GLM-5-FP8 \
+         --tensor-parallel-size 8 \
+         --gpu-memory-utilization 0.85 \
+         --speculative-config.method mtp \
+         --speculative-config.num_speculative_tokens 1 \
+         --tool-call-parser glm47 \
+         --reasoning-parser glm45 \
+         --enable-auto-tool-choice \
+         --served-model-name glm-5-fp8
+    ```
+
+    Check the [recipes](https://github.com/vllm-project/recipes/blob/main/GLM/GLM5.md) for more details.
+
++ SGLang
+
+    ```shell
+    python3 -m sglang.launch_server \
+      --model-path zai-org/GLM-5-FP8 \
+      --tp-size 8 \
+      --tool-call-parser glm47  \
+      --reasoning-parser glm45 \
+      --speculative-algorithm EAGLE \
+      --speculative-num-steps 3 \
+      --speculative-eagle-topk 1 \
+      --speculative-num-draft-tokens 4 \
+      --mem-fraction-static 0.85 \
+      --served-model-name glm-5-fp8
+    ```
+    
+    Check the [sglang cookbook](https://cookbook.sglang.io/autoregressive/GLM/GLM-5) for more details.
+
++ xLLM and other Ascend NPU
+
+    Please check the deployment guide [here](https://github.com/zai-org/GLM-5/blob/main/example/ascend.md).
+
+
+## Citation
+
+Our technical report is coming soon.

chat_template.jinja

@@ -0,0 +1,86 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content -%}
+{{ '<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '</think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '<tool_response>' }}
+{{- m.content }}
+{{- '</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+<tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '</think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,782 @@
+{
+  "architectures": [
+    "GlmMoeDsaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "ep_size": 1,
+  "first_k_dense_replace": 3,
+  "hidden_act": "silu",
+  "head_dim": 64,
+  "hidden_size": 6144,
+  "index_head_dim": 128,
+  "index_n_heads": 32,
+  "index_topk": 2048,
+  "indexer_rope_interleave": true,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 202752,
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "model_type": "glm_moe_dsa",
+  "n_group": 1,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 78,
+  "num_key_value_heads": 64,
+  "num_nextn_predict_layers": 1,
+  "pad_token_id": 154820,
+  "pretraining_tp": 1,
+  "q_lora_rank": 2048,
+  "qk_head_dim": 256,
+  "qk_nope_head_dim": 192,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-05,
+  "rope_interleave": true,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "transformers_version": "5.0.2.dev0",
+  "use_cache": true,
+  "v_head_dim": 256,
+  "vocab_size": 154880,
+  "quantization_config": {
+    "activation_scheme": "dynamic",
+    "fmt": "e4m3",
+    "quant_method": "fp8",
+    "weight_block_size": [
+      128,
+      128
+    ],
+    "modules_to_not_convert": [
+      "lm_head",
+      "model.embed_tokens",
+      "model.layers.0.input_layernorm",
+      "model.layers.0.post_attention_layernorm",
+      "model.layers.0.self_attn.indexer.k_norm",
+      "model.layers.0.self_attn.indexer.k_norm.bias",
+      "model.layers.0.self_attn.indexers_proj",
+      "model.layers.0.self_attn.kv_a_layernorm",
+      "model.layers.0.self_attn.q_a_layernorm",
+      "model.layers.1.input_layernorm",
+      "model.layers.1.post_attention_layernorm",
+      "model.layers.1.self_attn.indexer.k_norm",
+      "model.layers.1.self_attn.indexer.k_norm.bias",
+      "model.layers.1.self_attn.indexers_proj",
+      "model.layers.1.self_attn.kv_a_layernorm",
+      "model.layers.1.self_attn.q_a_layernorm",
+      "model.layers.2.input_layernorm",
+      "model.layers.2.post_attention_layernorm",
+      "model.layers.2.self_attn.indexer.k_norm",
+      "model.layers.2.self_attn.indexer.k_norm.bias",
+      "model.layers.2.self_attn.indexers_proj",
+      "model.layers.2.self_attn.kv_a_layernorm",
+      "model.layers.2.self_attn.q_a_layernorm",
+      "model.layers.3.input_layernorm",
+      "model.layers.3.mlp.gate",
+      "model.layers.3.mlp.gate.e_score_correction_bias",
+      "model.layers.3.post_attention_layernorm",
+      "model.layers.3.self_attn.indexer.k_norm",
+      "model.layers.3.self_attn.indexer.k_norm.bias",
+      "model.layers.3.self_attn.indexers_proj",
+      "model.layers.3.self_attn.kv_a_layernorm",
+      "model.layers.3.self_attn.q_a_layernorm",
+      "model.layers.4.input_layernorm",
+      "model.layers.4.mlp.gate",
+      "model.layers.4.mlp.gate.e_score_correction_bias",
+      "model.layers.4.post_attention_layernorm",
+      "model.layers.4.self_attn.indexer.k_norm",
+      "model.layers.4.self_attn.indexer.k_norm.bias",
+      "model.layers.4.self_attn.indexers_proj",
+      "model.layers.4.self_attn.kv_a_layernorm",
+      "model.layers.4.self_attn.q_a_layernorm",
+      "model.layers.5.input_layernorm",
+      "model.layers.5.mlp.gate",
+      "model.layers.5.mlp.gate.e_score_correction_bias",
+      "model.layers.5.post_attention_layernorm",
+      "model.layers.5.self_attn.indexer.k_norm",
+      "model.layers.5.self_attn.indexer.k_norm.bias",
+      "model.layers.5.self_attn.indexers_proj",
+      "model.layers.5.self_attn.kv_a_layernorm",
+      "model.layers.5.self_attn.q_a_layernorm",
+      "model.layers.6.input_layernorm",
+      "model.layers.6.mlp.gate",
+      "model.layers.6.mlp.gate.e_score_correction_bias",
+      "model.layers.6.post_attention_layernorm",
+      "model.layers.6.self_attn.indexer.k_norm",
+      "model.layers.6.self_attn.indexer.k_norm.bias",
+      "model.layers.6.self_attn.indexers_proj",
+      "model.layers.6.self_attn.kv_a_layernorm",
+      "model.layers.6.self_attn.q_a_layernorm",
+      "model.layers.7.input_layernorm",
+      "model.layers.7.mlp.gate",
+      "model.layers.7.mlp.gate.e_score_correction_bias",
+      "model.layers.7.post_attention_layernorm",
+      "model.layers.7.self_attn.indexer.k_norm",
+      "model.layers.7.self_attn.indexer.k_norm.bias",
+      "model.layers.7.self_attn.indexers_proj",
+      "model.layers.7.self_attn.kv_a_layernorm",
+      "model.layers.7.self_attn.q_a_layernorm",
+      "model.layers.8.input_layernorm",
+      "model.layers.8.mlp.gate",
+      "model.layers.8.mlp.gate.e_score_correction_bias",
+      "model.layers.8.post_attention_layernorm",
+      "model.layers.8.self_attn.indexer.k_norm",
+      "model.layers.8.self_attn.indexer.k_norm.bias",
+      "model.layers.8.self_attn.indexers_proj",
+      "model.layers.8.self_attn.kv_a_layernorm",
+      "model.layers.8.self_attn.q_a_layernorm",
+      "model.layers.9.input_layernorm",
+      "model.layers.9.mlp.gate",
+      "model.layers.9.mlp.gate.e_score_correction_bias",
+      "model.layers.9.post_attention_layernorm",
+      "model.layers.9.self_attn.indexer.k_norm",
+      "model.layers.9.self_attn.indexer.k_norm.bias",
+      "model.layers.9.self_attn.indexers_proj",
+      "model.layers.9.self_attn.kv_a_layernorm",
+      "model.layers.9.self_attn.q_a_layernorm",
+      "model.layers.10.input_layernorm",
+      "model.layers.10.mlp.gate",
+      "model.layers.10.mlp.gate.e_score_correction_bias",
+      "model.layers.10.post_attention_layernorm",
+      "model.layers.10.self_attn.indexer.k_norm",
+      "model.layers.10.self_attn.indexer.k_norm.bias",
+      "model.layers.10.self_attn.indexers_proj",
+      "model.layers.10.self_attn.kv_a_layernorm",
+      "model.layers.10.self_attn.q_a_layernorm",
+      "model.layers.11.input_layernorm",
+      "model.layers.11.mlp.gate",
+      "model.layers.11.mlp.gate.e_score_correction_bias",
+      "model.layers.11.post_attention_layernorm",
+      "model.layers.11.self_attn.indexer.k_norm",
+      "model.layers.11.self_attn.indexer.k_norm.bias",
+      "model.layers.11.self_attn.indexers_proj",
+      "model.layers.11.self_attn.kv_a_layernorm",
+      "model.layers.11.self_attn.q_a_layernorm",
+      "model.layers.12.input_layernorm",
+      "model.layers.12.mlp.gate",
+      "model.layers.12.mlp.gate.e_score_correction_bias",
+      "model.layers.12.post_attention_layernorm",
+      "model.layers.12.self_attn.indexer.k_norm",
+      "model.layers.12.self_attn.indexer.k_norm.bias",
+      "model.layers.12.self_attn.indexers_proj",
+      "model.layers.12.self_attn.kv_a_layernorm",
+      "model.layers.12.self_attn.q_a_layernorm",
+      "model.layers.13.input_layernorm",
+      "model.layers.13.mlp.gate",
+      "model.layers.13.mlp.gate.e_score_correction_bias",
+      "model.layers.13.post_attention_layernorm",
+      "model.layers.13.self_attn.indexer.k_norm",
+      "model.layers.13.self_attn.indexer.k_norm.bias",
+      "model.layers.13.self_attn.indexers_proj",
+      "model.layers.13.self_attn.kv_a_layernorm",
+      "model.layers.13.self_attn.q_a_layernorm",
+      "model.layers.14.input_layernorm",
+      "model.layers.14.mlp.gate",
+      "model.layers.14.mlp.gate.e_score_correction_bias",
+      "model.layers.14.post_attention_layernorm",
+      "model.layers.14.self_attn.indexer.k_norm",
+      "model.layers.14.self_attn.indexer.k_norm.bias",
+      "model.layers.14.self_attn.indexers_proj",
+      "model.layers.14.self_attn.kv_a_layernorm",
+      "model.layers.14.self_attn.q_a_layernorm",
+      "model.layers.15.input_layernorm",
+      "model.layers.15.mlp.gate",
+      "model.layers.15.mlp.gate.e_score_correction_bias",
+      "model.layers.15.post_attention_layernorm",
+      "model.layers.15.self_attn.indexer.k_norm",
+      "model.layers.15.self_attn.indexer.k_norm.bias",
+      "model.layers.15.self_attn.indexers_proj",
+      "model.layers.15.self_attn.kv_a_layernorm",
+      "model.layers.15.self_attn.q_a_layernorm",
+      "model.layers.16.input_layernorm",
+      "model.layers.16.mlp.gate",
+      "model.layers.16.mlp.gate.e_score_correction_bias",
+      "model.layers.16.post_attention_layernorm",
+      "model.layers.16.self_attn.indexer.k_norm",
+      "model.layers.16.self_attn.indexer.k_norm.bias",
+      "model.layers.16.self_attn.indexers_proj",
+      "model.layers.16.self_attn.kv_a_layernorm",
+      "model.layers.16.self_attn.q_a_layernorm",
+      "model.layers.17.input_layernorm",
+      "model.layers.17.mlp.gate",
+      "model.layers.17.mlp.gate.e_score_correction_bias",
+      "model.layers.17.post_attention_layernorm",
+      "model.layers.17.self_attn.indexer.k_norm",
+      "model.layers.17.self_attn.indexer.k_norm.bias",
+      "model.layers.17.self_attn.indexers_proj",
+      "model.layers.17.self_attn.kv_a_layernorm",
+      "model.layers.17.self_attn.q_a_layernorm",
+      "model.layers.18.input_layernorm",
+      "model.layers.18.mlp.gate",
+      "model.layers.18.mlp.gate.e_score_correction_bias",
+      "model.layers.18.post_attention_layernorm",
+      "model.layers.18.self_attn.indexer.k_norm",
+      "model.layers.18.self_attn.indexer.k_norm.bias",
+      "model.layers.18.self_attn.indexers_proj",
+      "model.layers.18.self_attn.kv_a_layernorm",
+      "model.layers.18.self_attn.q_a_layernorm",
+      "model.layers.19.input_layernorm",
+      "model.layers.19.mlp.gate",
+      "model.layers.19.mlp.gate.e_score_correction_bias",
+      "model.layers.19.post_attention_layernorm",
+      "model.layers.19.self_attn.indexer.k_norm",
+      "model.layers.19.self_attn.indexer.k_norm.bias",
+      "model.layers.19.self_attn.indexers_proj",
+      "model.layers.19.self_attn.kv_a_layernorm",
+      "model.layers.19.self_attn.q_a_layernorm",
+      "model.layers.20.input_layernorm",
+      "model.layers.20.mlp.gate",
+      "model.layers.20.mlp.gate.e_score_correction_bias",
+      "model.layers.20.post_attention_layernorm",
+      "model.layers.20.self_attn.indexer.k_norm",
+      "model.layers.20.self_attn.indexer.k_norm.bias",
+      "model.layers.20.self_attn.indexers_proj",
+      "model.layers.20.self_attn.kv_a_layernorm",
+      "model.layers.20.self_attn.q_a_layernorm",
+      "model.layers.21.input_layernorm",
+      "model.layers.21.mlp.gate",
+      "model.layers.21.mlp.gate.e_score_correction_bias",
+      "model.layers.21.post_attention_layernorm",
+      "model.layers.21.self_attn.indexer.k_norm",
+      "model.layers.21.self_attn.indexer.k_norm.bias",
+      "model.layers.21.self_attn.indexers_proj",
+      "model.layers.21.self_attn.kv_a_layernorm",
+      "model.layers.21.self_attn.q_a_layernorm",
+      "model.layers.22.input_layernorm",
+      "model.layers.22.mlp.gate",
+      "model.layers.22.mlp.gate.e_score_correction_bias",
+      "model.layers.22.post_attention_layernorm",
+      "model.layers.22.self_attn.indexer.k_norm",
+      "model.layers.22.self_attn.indexer.k_norm.bias",
+      "model.layers.22.self_attn.indexers_proj",
+      "model.layers.22.self_attn.kv_a_layernorm",
+      "model.layers.22.self_attn.q_a_layernorm",
+      "model.layers.23.input_layernorm",
+      "model.layers.23.mlp.gate",
+      "model.layers.23.mlp.gate.e_score_correction_bias",
+      "model.layers.23.post_attention_layernorm",
+      "model.layers.23.self_attn.indexer.k_norm",
+      "model.layers.23.self_attn.indexer.k_norm.bias",
+      "model.layers.23.self_attn.indexers_proj",
+      "model.layers.23.self_attn.kv_a_layernorm",
+      "model.layers.23.self_attn.q_a_layernorm",
+      "model.layers.24.input_layernorm",
+      "model.layers.24.mlp.gate",
+      "model.layers.24.mlp.gate.e_score_correction_bias",
+      "model.layers.24.post_attention_layernorm",
+      "model.layers.24.self_attn.indexer.k_norm",
+      "model.layers.24.self_attn.indexer.k_norm.bias",
+      "model.layers.24.self_attn.indexers_proj",
+      "model.layers.24.self_attn.kv_a_layernorm",
+      "model.layers.24.self_attn.q_a_layernorm",
+      "model.layers.25.input_layernorm",
+      "model.layers.25.mlp.gate",
+      "model.layers.25.mlp.gate.e_score_correction_bias",
+      "model.layers.25.post_attention_layernorm",
+      "model.layers.25.self_attn.indexer.k_norm",
+      "model.layers.25.self_attn.indexer.k_norm.bias",
+      "model.layers.25.self_attn.indexers_proj",
+      "model.layers.25.self_attn.kv_a_layernorm",
+      "model.layers.25.self_attn.q_a_layernorm",
+      "model.layers.26.input_layernorm",
+      "model.layers.26.mlp.gate",
+      "model.layers.26.mlp.gate.e_score_correction_bias",
+      "model.layers.26.post_attention_layernorm",
+      "model.layers.26.self_attn.indexer.k_norm",
+      "model.layers.26.self_attn.indexer.k_norm.bias",
+      "model.layers.26.self_attn.indexers_proj",
+      "model.layers.26.self_attn.kv_a_layernorm",
+      "model.layers.26.self_attn.q_a_layernorm",
+      "model.layers.27.input_layernorm",
+      "model.layers.27.mlp.gate",
+      "model.layers.27.mlp.gate.e_score_correction_bias",
+      "model.layers.27.post_attention_layernorm",
+      "model.layers.27.self_attn.indexer.k_norm",
+      "model.layers.27.self_attn.indexer.k_norm.bias",
+      "model.layers.27.self_attn.indexers_proj",
+      "model.layers.27.self_attn.kv_a_layernorm",
+      "model.layers.27.self_attn.q_a_layernorm",
+      "model.layers.28.input_layernorm",
+      "model.layers.28.mlp.gate",
+      "model.layers.28.mlp.gate.e_score_correction_bias",
+      "model.layers.28.post_attention_layernorm",
+      "model.layers.28.self_attn.indexer.k_norm",
+      "model.layers.28.self_attn.indexer.k_norm.bias",
+      "model.layers.28.self_attn.indexers_proj",
+      "model.layers.28.self_attn.kv_a_layernorm",
+      "model.layers.28.self_attn.q_a_layernorm",
+      "model.layers.29.input_layernorm",
+      "model.layers.29.mlp.gate",
+      "model.layers.29.mlp.gate.e_score_correction_bias",
+      "model.layers.29.post_attention_layernorm",
+      "model.layers.29.self_attn.indexer.k_norm",
+      "model.layers.29.self_attn.indexer.k_norm.bias",
+      "model.layers.29.self_attn.indexers_proj",
+      "model.layers.29.self_attn.kv_a_layernorm",
+      "model.layers.29.self_attn.q_a_layernorm",
+      "model.layers.30.input_layernorm",
+      "model.layers.30.mlp.gate",
+      "model.layers.30.mlp.gate.e_score_correction_bias",
+      "model.layers.30.post_attention_layernorm",
+      "model.layers.30.self_attn.indexer.k_norm",
+      "model.layers.30.self_attn.indexer.k_norm.bias",
+      "model.layers.30.self_attn.indexers_proj",
+      "model.layers.30.self_attn.kv_a_layernorm",
+      "model.layers.30.self_attn.q_a_layernorm",
+      "model.layers.31.input_layernorm",
+      "model.layers.31.mlp.gate",
+      "model.layers.31.mlp.gate.e_score_correction_bias",
+      "model.layers.31.post_attention_layernorm",
+      "model.layers.31.self_attn.indexer.k_norm",
+      "model.layers.31.self_attn.indexer.k_norm.bias",
+      "model.layers.31.self_attn.indexers_proj",
+      "model.layers.31.self_attn.kv_a_layernorm",
+      "model.layers.31.self_attn.q_a_layernorm",
+      "model.layers.32.input_layernorm",
+      "model.layers.32.mlp.gate",
+      "model.layers.32.mlp.gate.e_score_correction_bias",
+      "model.layers.32.post_attention_layernorm",
+      "model.layers.32.self_attn.indexer.k_norm",
+      "model.layers.32.self_attn.indexer.k_norm.bias",
+      "model.layers.32.self_attn.indexers_proj",
+      "model.layers.32.self_attn.kv_a_layernorm",
+      "model.layers.32.self_attn.q_a_layernorm",
+      "model.layers.33.input_layernorm",
+      "model.layers.33.mlp.gate",
+      "model.layers.33.mlp.gate.e_score_correction_bias",
+      "model.layers.33.post_attention_layernorm",
+      "model.layers.33.self_attn.indexer.k_norm",
+      "model.layers.33.self_attn.indexer.k_norm.bias",
+      "model.layers.33.self_attn.indexers_proj",
+      "model.layers.33.self_attn.kv_a_layernorm",
+      "model.layers.33.self_attn.q_a_layernorm",
+      "model.layers.34.input_layernorm",
+      "model.layers.34.mlp.gate",
+      "model.layers.34.mlp.gate.e_score_correction_bias",
+      "model.layers.34.post_attention_layernorm",
+      "model.layers.34.self_attn.indexer.k_norm",
+      "model.layers.34.self_attn.indexer.k_norm.bias",
+      "model.layers.34.self_attn.indexers_proj",
+      "model.layers.34.self_attn.kv_a_layernorm",
+      "model.layers.34.self_attn.q_a_layernorm",
+      "model.layers.35.input_layernorm",
+      "model.layers.35.mlp.gate",
+      "model.layers.35.mlp.gate.e_score_correction_bias",
+      "model.layers.35.post_attention_layernorm",
+      "model.layers.35.self_attn.indexer.k_norm",
+      "model.layers.35.self_attn.indexer.k_norm.bias",
+      "model.layers.35.self_attn.indexers_proj",
+      "model.layers.35.self_attn.kv_a_layernorm",
+      "model.layers.35.self_attn.q_a_layernorm",
+      "model.layers.36.input_layernorm",
+      "model.layers.36.mlp.gate",
+      "model.layers.36.mlp.gate.e_score_correction_bias",
+      "model.layers.36.post_attention_layernorm",
+      "model.layers.36.self_attn.indexer.k_norm",
+      "model.layers.36.self_attn.indexer.k_norm.bias",
+      "model.layers.36.self_attn.indexers_proj",
+      "model.layers.36.self_attn.kv_a_layernorm",
+      "model.layers.36.self_attn.q_a_layernorm",
+      "model.layers.37.input_layernorm",
+      "model.layers.37.mlp.gate",
+      "model.layers.37.mlp.gate.e_score_correction_bias",
+      "model.layers.37.post_attention_layernorm",
+      "model.layers.37.self_attn.indexer.k_norm",
+      "model.layers.37.self_attn.indexer.k_norm.bias",
+      "model.layers.37.self_attn.indexers_proj",
+      "model.layers.37.self_attn.kv_a_layernorm",
+      "model.layers.37.self_attn.q_a_layernorm",
+      "model.layers.38.input_layernorm",
+      "model.layers.38.mlp.gate",
+      "model.layers.38.mlp.gate.e_score_correction_bias",
+      "model.layers.38.post_attention_layernorm",
+      "model.layers.38.self_attn.indexer.k_norm",
+      "model.layers.38.self_attn.indexer.k_norm.bias",
+      "model.layers.38.self_attn.indexers_proj",
+      "model.layers.38.self_attn.kv_a_layernorm",
+      "model.layers.38.self_attn.q_a_layernorm",
+      "model.layers.39.input_layernorm",
+      "model.layers.39.mlp.gate",
+      "model.layers.39.mlp.gate.e_score_correction_bias",
+      "model.layers.39.post_attention_layernorm",
+      "model.layers.39.self_attn.indexer.k_norm",
+      "model.layers.39.self_attn.indexer.k_norm.bias",
+      "model.layers.39.self_attn.indexers_proj",
+      "model.layers.39.self_attn.kv_a_layernorm",
+      "model.layers.39.self_attn.q_a_layernorm",
+      "model.layers.40.input_layernorm",
+      "model.layers.40.mlp.gate",
+      "model.layers.40.mlp.gate.e_score_correction_bias",
+      "model.layers.40.post_attention_layernorm",
+      "model.layers.40.self_attn.indexer.k_norm",
+      "model.layers.40.self_attn.indexer.k_norm.bias",
+      "model.layers.40.self_attn.indexers_proj",
+      "model.layers.40.self_attn.kv_a_layernorm",
+      "model.layers.40.self_attn.q_a_layernorm",
+      "model.layers.41.input_layernorm",
+      "model.layers.41.mlp.gate",
+      "model.layers.41.mlp.gate.e_score_correction_bias",
+      "model.layers.41.post_attention_layernorm",
+      "model.layers.41.self_attn.indexer.k_norm",
+      "model.layers.41.self_attn.indexer.k_norm.bias",
+      "model.layers.41.self_attn.indexers_proj",
+      "model.layers.41.self_attn.kv_a_layernorm",
+      "model.layers.41.self_attn.q_a_layernorm",
+      "model.layers.42.input_layernorm",
+      "model.layers.42.mlp.gate",
+      "model.layers.42.mlp.gate.e_score_correction_bias",
+      "model.layers.42.post_attention_layernorm",
+      "model.layers.42.self_attn.indexer.k_norm",
+      "model.layers.42.self_attn.indexer.k_norm.bias",
+      "model.layers.42.self_attn.indexers_proj",
+      "model.layers.42.self_attn.kv_a_layernorm",
+      "model.layers.42.self_attn.q_a_layernorm",
+      "model.layers.43.input_layernorm",
+      "model.layers.43.mlp.gate",
+      "model.layers.43.mlp.gate.e_score_correction_bias",
+      "model.layers.43.post_attention_layernorm",
+      "model.layers.43.self_attn.indexer.k_norm",
+      "model.layers.43.self_attn.indexer.k_norm.bias",
+      "model.layers.43.self_attn.indexers_proj",
+      "model.layers.43.self_attn.kv_a_layernorm",
+      "model.layers.43.self_attn.q_a_layernorm",
+      "model.layers.44.input_layernorm",
+      "model.layers.44.mlp.gate",
+      "model.layers.44.mlp.gate.e_score_correction_bias",
+      "model.layers.44.post_attention_layernorm",
+      "model.layers.44.self_attn.indexer.k_norm",
+      "model.layers.44.self_attn.indexer.k_norm.bias",
+      "model.layers.44.self_attn.indexers_proj",
+      "model.layers.44.self_attn.kv_a_layernorm",
+      "model.layers.44.self_attn.q_a_layernorm",
+      "model.layers.45.input_layernorm",
+      "model.layers.45.mlp.gate",
+      "model.layers.45.mlp.gate.e_score_correction_bias",
+      "model.layers.45.post_attention_layernorm",
+      "model.layers.45.self_attn.indexer.k_norm",
+      "model.layers.45.self_attn.indexer.k_norm.bias",
+      "model.layers.45.self_attn.indexers_proj",
+      "model.layers.45.self_attn.kv_a_layernorm",
+      "model.layers.45.self_attn.q_a_layernorm",
+      "model.layers.46.input_layernorm",
+      "model.layers.46.mlp.gate",
+      "model.layers.46.mlp.gate.e_score_correction_bias",
+      "model.layers.46.post_attention_layernorm",
+      "model.layers.46.self_attn.indexer.k_norm",
+      "model.layers.46.self_attn.indexer.k_norm.bias",
+      "model.layers.46.self_attn.indexers_proj",
+      "model.layers.46.self_attn.kv_a_layernorm",
+      "model.layers.46.self_attn.q_a_layernorm",
+      "model.layers.47.input_layernorm",
+      "model.layers.47.mlp.gate",
+      "model.layers.47.mlp.gate.e_score_correction_bias",
+      "model.layers.47.post_attention_layernorm",
+      "model.layers.47.self_attn.indexer.k_norm",
+      "model.layers.47.self_attn.indexer.k_norm.bias",
+      "model.layers.47.self_attn.indexers_proj",
+      "model.layers.47.self_attn.kv_a_layernorm",
+      "model.layers.47.self_attn.q_a_layernorm",
+      "model.layers.48.input_layernorm",
+      "model.layers.48.mlp.gate",
+      "model.layers.48.mlp.gate.e_score_correction_bias",
+      "model.layers.48.post_attention_layernorm",
+      "model.layers.48.self_attn.indexer.k_norm",
+      "model.layers.48.self_attn.indexer.k_norm.bias",
+      "model.layers.48.self_attn.indexers_proj",
+      "model.layers.48.self_attn.kv_a_layernorm",
+      "model.layers.48.self_attn.q_a_layernorm",
+      "model.layers.49.input_layernorm",
+      "model.layers.49.mlp.gate",
+      "model.layers.49.mlp.gate.e_score_correction_bias",
+      "model.layers.49.post_attention_layernorm",
+      "model.layers.49.self_attn.indexer.k_norm",
+      "model.layers.49.self_attn.indexer.k_norm.bias",
+      "model.layers.49.self_attn.indexers_proj",
+      "model.layers.49.self_attn.kv_a_layernorm",
+      "model.layers.49.self_attn.q_a_layernorm",
+      "model.layers.50.input_layernorm",
+      "model.layers.50.mlp.gate",
+      "model.layers.50.mlp.gate.e_score_correction_bias",
+      "model.layers.50.post_attention_layernorm",
+      "model.layers.50.self_attn.indexer.k_norm",
+      "model.layers.50.self_attn.indexer.k_norm.bias",
+      "model.layers.50.self_attn.indexers_proj",
+      "model.layers.50.self_attn.kv_a_layernorm",
+      "model.layers.50.self_attn.q_a_layernorm",
+      "model.layers.51.input_layernorm",
+      "model.layers.51.mlp.gate",
+      "model.layers.51.mlp.gate.e_score_correction_bias",
+      "model.layers.51.post_attention_layernorm",
+      "model.layers.51.self_attn.indexer.k_norm",
+      "model.layers.51.self_attn.indexer.k_norm.bias",
+      "model.layers.51.self_attn.indexers_proj",
+      "model.layers.51.self_attn.kv_a_layernorm",
+      "model.layers.51.self_attn.q_a_layernorm",
+      "model.layers.52.input_layernorm",
+      "model.layers.52.mlp.gate",
+      "model.layers.52.mlp.gate.e_score_correction_bias",
+      "model.layers.52.post_attention_layernorm",
+      "model.layers.52.self_attn.indexer.k_norm",
+      "model.layers.52.self_attn.indexer.k_norm.bias",
+      "model.layers.52.self_attn.indexers_proj",
+      "model.layers.52.self_attn.kv_a_layernorm",
+      "model.layers.52.self_attn.q_a_layernorm",
+      "model.layers.53.input_layernorm",
+      "model.layers.53.mlp.gate",
+      "model.layers.53.mlp.gate.e_score_correction_bias",
+      "model.layers.53.post_attention_layernorm",
+      "model.layers.53.self_attn.indexer.k_norm",
+      "model.layers.53.self_attn.indexer.k_norm.bias",
+      "model.layers.53.self_attn.indexers_proj",
+      "model.layers.53.self_attn.kv_a_layernorm",
+      "model.layers.53.self_attn.q_a_layernorm",
+      "model.layers.54.input_layernorm",
+      "model.layers.54.mlp.gate",
+      "model.layers.54.mlp.gate.e_score_correction_bias",
+      "model.layers.54.post_attention_layernorm",
+      "model.layers.54.self_attn.indexer.k_norm",
+      "model.layers.54.self_attn.indexer.k_norm.bias",
+      "model.layers.54.self_attn.indexers_proj",
+      "model.layers.54.self_attn.kv_a_layernorm",
+      "model.layers.54.self_attn.q_a_layernorm",
+      "model.layers.55.input_layernorm",
+      "model.layers.55.mlp.gate",
+      "model.layers.55.mlp.gate.e_score_correction_bias",
+      "model.layers.55.post_attention_layernorm",
+      "model.layers.55.self_attn.indexer.k_norm",
+      "model.layers.55.self_attn.indexer.k_norm.bias",
+      "model.layers.55.self_attn.indexers_proj",
+      "model.layers.55.self_attn.kv_a_layernorm",
+      "model.layers.55.self_attn.q_a_layernorm",
+      "model.layers.56.input_layernorm",
+      "model.layers.56.mlp.gate",
+      "model.layers.56.mlp.gate.e_score_correction_bias",
+      "model.layers.56.post_attention_layernorm",
+      "model.layers.56.self_attn.indexer.k_norm",
+      "model.layers.56.self_attn.indexer.k_norm.bias",
+      "model.layers.56.self_attn.indexers_proj",
+      "model.layers.56.self_attn.kv_a_layernorm",
+      "model.layers.56.self_attn.q_a_layernorm",
+      "model.layers.57.input_layernorm",
+      "model.layers.57.mlp.gate",
+      "model.layers.57.mlp.gate.e_score_correction_bias",
+      "model.layers.57.post_attention_layernorm",
+      "model.layers.57.self_attn.indexer.k_norm",
+      "model.layers.57.self_attn.indexer.k_norm.bias",
+      "model.layers.57.self_attn.indexers_proj",
+      "model.layers.57.self_attn.kv_a_layernorm",
+      "model.layers.57.self_attn.q_a_layernorm",
+      "model.layers.58.input_layernorm",
+      "model.layers.58.mlp.gate",
+      "model.layers.58.mlp.gate.e_score_correction_bias",
+      "model.layers.58.post_attention_layernorm",
+      "model.layers.58.self_attn.indexer.k_norm",
+      "model.layers.58.self_attn.indexer.k_norm.bias",
+      "model.layers.58.self_attn.indexers_proj",
+      "model.layers.58.self_attn.kv_a_layernorm",
+      "model.layers.58.self_attn.q_a_layernorm",
+      "model.layers.59.input_layernorm",
+      "model.layers.59.mlp.gate",
+      "model.layers.59.mlp.gate.e_score_correction_bias",
+      "model.layers.59.post_attention_layernorm",
+      "model.layers.59.self_attn.indexer.k_norm",
+      "model.layers.59.self_attn.indexer.k_norm.bias",
+      "model.layers.59.self_attn.indexers_proj",
+      "model.layers.59.self_attn.kv_a_layernorm",
+      "model.layers.59.self_attn.q_a_layernorm",
+      "model.layers.60.input_layernorm",
+      "model.layers.60.mlp.gate",
+      "model.layers.60.mlp.gate.e_score_correction_bias",
+      "model.layers.60.post_attention_layernorm",
+      "model.layers.60.self_attn.indexer.k_norm",
+      "model.layers.60.self_attn.indexer.k_norm.bias",
+      "model.layers.60.self_attn.indexers_proj",
+      "model.layers.60.self_attn.kv_a_layernorm",
+      "model.layers.60.self_attn.q_a_layernorm",
+      "model.layers.61.input_layernorm",
+      "model.layers.61.mlp.gate",
+      "model.layers.61.mlp.gate.e_score_correction_bias",
+      "model.layers.61.post_attention_layernorm",
+      "model.layers.61.self_attn.indexer.k_norm",
+      "model.layers.61.self_attn.indexer.k_norm.bias",
+      "model.layers.61.self_attn.indexers_proj",
+      "model.layers.61.self_attn.kv_a_layernorm",
+      "model.layers.61.self_attn.q_a_layernorm",
+      "model.layers.62.input_layernorm",
+      "model.layers.62.mlp.gate",
+      "model.layers.62.mlp.gate.e_score_correction_bias",
+      "model.layers.62.post_attention_layernorm",
+      "model.layers.62.self_attn.indexer.k_norm",
+      "model.layers.62.self_attn.indexer.k_norm.bias",
+      "model.layers.62.self_attn.indexers_proj",
+      "model.layers.62.self_attn.kv_a_layernorm",
+      "model.layers.62.self_attn.q_a_layernorm",
+      "model.layers.63.input_layernorm",
+      "model.layers.63.mlp.gate",
+      "model.layers.63.mlp.gate.e_score_correction_bias",
+      "model.layers.63.post_attention_layernorm",
+      "model.layers.63.self_attn.indexer.k_norm",
+      "model.layers.63.self_attn.indexer.k_norm.bias",
+      "model.layers.63.self_attn.indexers_proj",
+      "model.layers.63.self_attn.kv_a_layernorm",
+      "model.layers.63.self_attn.q_a_layernorm",
+      "model.layers.64.input_layernorm",
+      "model.layers.64.mlp.gate",
+      "model.layers.64.mlp.gate.e_score_correction_bias",
+      "model.layers.64.post_attention_layernorm",
+      "model.layers.64.self_attn.indexer.k_norm",
+      "model.layers.64.self_attn.indexer.k_norm.bias",
+      "model.layers.64.self_attn.indexers_proj",
+      "model.layers.64.self_attn.kv_a_layernorm",
+      "model.layers.64.self_attn.q_a_layernorm",
+      "model.layers.65.input_layernorm",
+      "model.layers.65.mlp.gate",
+      "model.layers.65.mlp.gate.e_score_correction_bias",
+      "model.layers.65.post_attention_layernorm",
+      "model.layers.65.self_attn.indexer.k_norm",
+      "model.layers.65.self_attn.indexer.k_norm.bias",
+      "model.layers.65.self_attn.indexers_proj",
+      "model.layers.65.self_attn.kv_a_layernorm",
+      "model.layers.65.self_attn.q_a_layernorm",
+      "model.layers.66.input_layernorm",
+      "model.layers.66.mlp.gate",
+      "model.layers.66.mlp.gate.e_score_correction_bias",
+      "model.layers.66.post_attention_layernorm",
+      "model.layers.66.self_attn.indexer.k_norm",
+      "model.layers.66.self_attn.indexer.k_norm.bias",
+      "model.layers.66.self_attn.indexers_proj",
+      "model.layers.66.self_attn.kv_a_layernorm",
+      "model.layers.66.self_attn.q_a_layernorm",
+      "model.layers.67.input_layernorm",
+      "model.layers.67.mlp.gate",
+      "model.layers.67.mlp.gate.e_score_correction_bias",
+      "model.layers.67.post_attention_layernorm",
+      "model.layers.67.self_attn.indexer.k_norm",
+      "model.layers.67.self_attn.indexer.k_norm.bias",
+      "model.layers.67.self_attn.indexers_proj",
+      "model.layers.67.self_attn.kv_a_layernorm",
+      "model.layers.67.self_attn.q_a_layernorm",
+      "model.layers.68.input_layernorm",
+      "model.layers.68.mlp.gate",
+      "model.layers.68.mlp.gate.e_score_correction_bias",
+      "model.layers.68.post_attention_layernorm",
+      "model.layers.68.self_attn.indexer.k_norm",
+      "model.layers.68.self_attn.indexer.k_norm.bias",
+      "model.layers.68.self_attn.indexers_proj",
+      "model.layers.68.self_attn.kv_a_layernorm",
+      "model.layers.68.self_attn.q_a_layernorm",
+      "model.layers.69.input_layernorm",
+      "model.layers.69.mlp.gate",
+      "model.layers.69.mlp.gate.e_score_correction_bias",
+      "model.layers.69.post_attention_layernorm",
+      "model.layers.69.self_attn.indexer.k_norm",
+      "model.layers.69.self_attn.indexer.k_norm.bias",
+      "model.layers.69.self_attn.indexers_proj",
+      "model.layers.69.self_attn.kv_a_layernorm",
+      "model.layers.69.self_attn.q_a_layernorm",
+      "model.layers.70.input_layernorm",
+      "model.layers.70.mlp.gate",
+      "model.layers.70.mlp.gate.e_score_correction_bias",
+      "model.layers.70.post_attention_layernorm",
+      "model.layers.70.self_attn.indexer.k_norm",
+      "model.layers.70.self_attn.indexer.k_norm.bias",
+      "model.layers.70.self_attn.indexers_proj",
+      "model.layers.70.self_attn.kv_a_layernorm",
+      "model.layers.70.self_attn.q_a_layernorm",
+      "model.layers.71.input_layernorm",
+      "model.layers.71.mlp.gate",
+      "model.layers.71.mlp.gate.e_score_correction_bias",
+      "model.layers.71.post_attention_layernorm",
+      "model.layers.71.self_attn.indexer.k_norm",
+      "model.layers.71.self_attn.indexer.k_norm.bias",
+      "model.layers.71.self_attn.indexers_proj",
+      "model.layers.71.self_attn.kv_a_layernorm",
+      "model.layers.71.self_attn.q_a_layernorm",
+      "model.layers.72.input_layernorm",
+      "model.layers.72.mlp.gate",
+      "model.layers.72.mlp.gate.e_score_correction_bias",
+      "model.layers.72.post_attention_layernorm",
+      "model.layers.72.self_attn.indexer.k_norm",
+      "model.layers.72.self_attn.indexer.k_norm.bias",
+      "model.layers.72.self_attn.indexers_proj",
+      "model.layers.72.self_attn.kv_a_layernorm",
+      "model.layers.72.self_attn.q_a_layernorm",
+      "model.layers.73.input_layernorm",
+      "model.layers.73.mlp.gate",
+      "model.layers.73.mlp.gate.e_score_correction_bias",
+      "model.layers.73.post_attention_layernorm",
+      "model.layers.73.self_attn.indexer.k_norm",
+      "model.layers.73.self_attn.indexer.k_norm.bias",
+      "model.layers.73.self_attn.indexers_proj",
+      "model.layers.73.self_attn.kv_a_layernorm",
+      "model.layers.73.self_attn.q_a_layernorm",
+      "model.layers.74.input_layernorm",
+      "model.layers.74.mlp.gate",
+      "model.layers.74.mlp.gate.e_score_correction_bias",
+      "model.layers.74.post_attention_layernorm",
+      "model.layers.74.self_attn.indexer.k_norm",
+      "model.layers.74.self_attn.indexer.k_norm.bias",
+      "model.layers.74.self_attn.indexers_proj",
+      "model.layers.74.self_attn.kv_a_layernorm",
+      "model.layers.74.self_attn.q_a_layernorm",
+      "model.layers.75.input_layernorm",
+      "model.layers.75.mlp.gate",
+      "model.layers.75.mlp.gate.e_score_correction_bias",
+      "model.layers.75.post_attention_layernorm",
+      "model.layers.75.self_attn.indexer.k_norm",
+      "model.layers.75.self_attn.indexer.k_norm.bias",
+      "model.layers.75.self_attn.indexers_proj",
+      "model.layers.75.self_attn.kv_a_layernorm",
+      "model.layers.75.self_attn.q_a_layernorm",
+      "model.layers.76.input_layernorm",
+      "model.layers.76.mlp.gate",
+      "model.layers.76.mlp.gate.e_score_correction_bias",
+      "model.layers.76.post_attention_layernorm",
+      "model.layers.76.self_attn.indexer.k_norm",
+      "model.layers.76.self_attn.indexer.k_norm.bias",
+      "model.layers.76.self_attn.indexers_proj",
+      "model.layers.76.self_attn.kv_a_layernorm",
+      "model.layers.76.self_attn.q_a_layernorm",
+      "model.layers.77.input_layernorm",
+      "model.layers.77.mlp.gate",
+      "model.layers.77.mlp.gate.e_score_correction_bias",
+      "model.layers.77.post_attention_layernorm",
+      "model.layers.77.self_attn.indexer.k_norm",
+      "model.layers.77.self_attn.indexer.k_norm.bias",
+      "model.layers.77.self_attn.indexers_proj",
+      "model.layers.77.self_attn.kv_a_layernorm",
+      "model.layers.77.self_attn.q_a_layernorm",
+      "model.layers.78.eh_proj",
+      "model.layers.78.enorm",
+      "model.layers.78.hnorm",
+      "model.layers.78.input_layernorm",
+      "model.layers.78.mlp.gate",
+      "model.layers.78.mlp.gate.e_score_correction_bias",
+      "model.layers.78.post_attention_layernorm",
+      "model.layers.78.self_attn.indexer.k_norm",
+      "model.layers.78.self_attn.indexer.k_norm.bias",
+      "model.layers.78.self_attn.indexers_proj",
+      "model.layers.78.self_attn.kv_a_layernorm",
+      "model.layers.78.self_attn.q_a_layernorm",
+      "model.layers.78.shared_head.norm",
+      "model.norm"
+    ]
+  }
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "5.0.2.dev0"
+}

model-00001-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c75e4dc9f7b1e345170f5cd3c47c764dfe631d7895008f5efdc82b2dde92de4c
+size 5363940952

model-00002-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a17170e28d81d697955ae340c0927b648e8e1f6b06a6ae8051f286c8f74f725
+size 5361736696

model-00003-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1370f0035979329b828d21b11369784f52dcdb19bf0268ad934c18f79c0f44c6
+size 5363339120

model-00004-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e72700bd7b0f1bf6fd96ee23f8b8895a8d00ac9b8b9c11446f297d82c91015a4
+size 5361736640

model-00005-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e518a59a00bc1ec97d36f19734e21152e6a9faeb193d98e1278356be40101cc9
+size 5363339176

model-00006-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d37be93a5270cf8f7e4c20e5a733a31aff2bd83bc9656e9b3d7dc1b9a2d5926
+size 5361736504

model-00007-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f096b79ea10167b5e9fc66e99f0654c684893009cef9342ed72a363631b552d
+size 5363339304

model-00008-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39cdaf4f718d51bd65c5cc812cce5fd116742ba34904925b2251fe92e380a3ab
+size 5361736368

model-00009-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a294efbbf5c7f0eca60797c7a6902045d84896371cc5815ca431ce52afcc567
+size 5363339440

model-00010-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:390eaf7f2c80d1ac017521f4a5113677abfed056753d05c451a7c2a546384998
+size 5361736232

model-00011-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f0dcfd2ac425ac53d1f8496738a1d589abf751ed417f8208bc806ec64e2ebc2
+size 5363339592

model-00012-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3162c33985af89993a70da3530ff552c4885e569f8ca06e84b3bead004488a8
+size 5363339104

model-00013-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2d1702e2f1958d30eafb81c69f0abf0989c3d8bc82b752793a246c6cc60aa1c
+size 5361736696

model-00014-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81603c43fd54122915e648c6760efb06b2074ddf979220702f0a408776a8749f
+size 5363339120

model-00015-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a9a7e63e77ebefad25c210f3448528febd9233c66764e772fa0d87673f35ba1
+size 5361736688

model-00016-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9868c6b52a816ef19ee78bdb40d3a9f588dd7d8b55e07ef669ceb7cc8e215405
+size 5363339128

model-00017-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b15acba1b715f59fd5faff35837c873a5c2ba3cbd18f761e230cd0162305a34
+size 5361736552

model-00018-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52507d77cd4eadb12479d79acb5d860fa782de2ddc244d11f9e5a80a63127750
+size 5363339256

model-00019-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57cb5e9b3ca010298d6449f011dc04db5c66c70bac463dce08d63a8d60654858
+size 5361736416

model-00020-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18b9acd03945bbf30331843a692e68e0ac009355471a5b2c3d887a51d79d5f08
+size 5361791448

model-00021-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7d4fcf67f0abda1ac39e21cdf4d004f90109a33ee456ce39e53ea744c2c2fd1
+size 5361736352

model-00022-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7329ef4f994a7ac22901e465f943665708ad88236e630cae0532cf75ee4cc474
+size 5363339456

model-00023-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d546f87e34309770629fdac04b16e16fee4abe435b9613feb1236ec95a0380fc
+size 5361736224

model-00024-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79b755e74f642cb8ec7627f50414a6efe617485200f29acb65b340c78a603a79
+size 5363339608

model-00025-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b7c7a8ac34e62424109fdc5d58bd5667adae8dcc8bf80254142ca14113f575d
+size 5363339104

model-00026-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18d098486d2b0fe98046868dfc5c052f752768df47571eb250c8629e99a90f4b
+size 5361736696

model-00027-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15dfa9511f0446684a5afc4a7afff701a2c03c72ff16750c61e6d11f19e63f22
+size 5363339112

model-00028-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d3d1c850ccb27716365a4e7a6078f7a589d2b94b780f0fd7f6ac5222d75402d
+size 5361736664

model-00029-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b061a3c1f07728dedc62e70e1382882c6f28c9d4c3657e68c7edb1f9d8b3c7a9
+size 5363339152

model-00030-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2de093fc6246082803e35e0d9f4a44f674a3182157df6afa1616aa1981ddc8d
+size 5361736528

model-00031-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58f45b0fbdcec9210e20e21d83d85c916f4c7a8f82589950960b17b0ea882eb6
+size 5363339280

model-00032-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5107083d264ed50fa120c4a527c80c61c79406918b16200a9db23bb5ff798c6d
+size 5361736392

model-00033-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cd86b5c670b8f1cf16a69e25e96ff9f09cc495eb7978ada4ac8eb14729f4586
+size 5363339416

model-00034-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6d39453d8ca90f5e38e0d51a95fd585766ad95488fb79d9f5d651bbcfe765bd
+size 5361736264

model-00035-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9d557ec5601653b8be2bb8ed3121159b368a53e6c6e27a7a901f2be859a3e32
+size 5363339544

model-00036-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e162b3ad7bb627cd8984e41a62d049f070321b4b30ee7deb335f88a9ab93de9
+size 5363339104

model-00037-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f980f59d1f5bc3dd9a717824b652045da1ea20b32aee9593811453e97d75c64
+size 5361736696

model-00038-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74963234bf95c73eb5d2fcfefb91e266b8174d3bef480f1080dd7548f3e43110
+size 5363338920

model-00039-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9e10c68987f919c926469f7adb27d7279989dac736bb7115e9d8f19f9e7d15b
+size 5361735840

model-00040-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ea19f319667a7ff62aea9182eb39e676c44a1b2792464ede0dff578ca255030
+size 5363338584

model-00041-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff662658a03396680e04029d8d2fd8963505a3731d8abbaa3112fccaa75c7af7
+size 5361736576

model-00042-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77fdeae54c5306335c557fe56a2d4065fe7f8bcbe38f89751b98584490f88900
+size 5363339232

model-00043-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53dba89973165ece2063a47af49ffddbd92393728164eec82e522d99f360dbe9
+size 5361736440

model-00044-of-00142.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d2aa1c3abc6c15bdc46c0a675729715add84dd7d9218165618a728ad6a3e5dc
+size 5363339368

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>"
+  ],
+  "is_local": true,
+  "model_max_length": 202752,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "TokenizersBackend"
+}