mistralai

/

Mistral-Medium-3.5-128B

like 418

Follow

Mistral AI_ 18.8k

I hit the same issue. It's the way the parser code is identifying the think tags. The parser is looking for a special mistral token for [THINK] and [/THINK] rather than the strings (which makes sense). The issue is in streaming the string is not consistently coming through as that specific special token. I created a fix to match on the string instead, which works great. It's here: https://github.com/vllm-project/vllm/pull/45609

Hey thanks for the feedback, and sorry for the late response.
it's quite concerning tbh because

Seems like vLLM is picking the wrong tokenizer by default for some reason.

should not happen and didn't experiment it recently but i'll investigate

after the first tool call however all subsequent reasoning blocks end up as [THINK] [/THINK] strings in the content blob.

The parser is looking for a special mistral token for [THINK] and [/THINK] rather than the strings (which makes sense).

If I understand correctly both feedbacks, are you saying that the model through opencode is outputting tool calls and then start building a new thinking trace for the same assistant message ? If it is the case it is something that I would deem unexpected because the model should call tools, receive tool messages and then think again.

Seems like vLLM is picking the wrong tokenizer by default for some reason.

should not happen and didn't experiment it recently but i'll investigate

I'm using this quant though:
https://huggingface.co/RecViking/Mistral-Medium-3.5-128B-NVFP4

I can't test the original unquantized Mistral model because of being limited to 128GB VRAM.

If I understand correctly both feedbacks, are you saying that the model through opencode is outputting tool calls and then start building a new thinking trace for the same assistant message ? If it is the case it is something that I would deem unexpected because the model should call tools, receive tool messages and then think again.

No, that's working fine as expected. It's like this:

1. [correct]: OpenCode sends initial prompt.
2. [correct]: vLLM responds with correct reasoning block and tool calls array, no stop condition.
3. [correct]: As no stop condition, OpenCode sends next request to vLLM with tool call result, etc.
4. [INCORRECT]: vLLM responds with raw [THINK] ... [/THINK] text string within the content blob, the rest is fine (tool call array, no stop condition).
5. [correct]: As no stop condition, OpenCode sends next request to vLLM with tool call result, etc.
6. [INCORRECT]: vLLM responds with raw [THINK] ... [/THINK] text string within the content blob, ...
   ... repeats until stop condition

But as all reasoning blocks end up in the context that way, I actually can't even do any kind of simple Mistral test with OpenCode, because it immediately hits the max. context. -> auto compaction -> agent starts from zero -> auto compaction, and so forth ...

Arf the issue is probably due to the fact that this model falls under the hf format that is not the one we maintain. I'm actually trying to sync with Transformers team to ease switching format and changing this.

The fix i have in mind here is to try to remap the weights names to what they'd be in consolidated format and rebuild the params.json with the same as original model but add the quantization_config of the config.json (and change layer namings for ignored ones)

Oi

I'm having the same issue, using the consolidated safetensors in vLLM v0.24.0. I'm seeing it in Opencode, and my own harness. Notably, I also see it when using the model via Cortecs.

Every assistant response, except for the first, has the reasoning coming through as content wrapped in [THINK][/THINK] tags. The first response always returns properly.

You can reproduce the issue with the following script, which is a slighly amended version of vLLM's streaming reasoning example from https://docs.vllm.ai/en/stable/examples/reasoning/openai_chat_completion_with_reasoning_streaming/to emulate the second assistant response in a conversation.

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
An example shows how to generate chat completions from reasoning models
like DeepSeekR1.

To run this example, you need to start the vLLM server with the reasoning
parser:

Unlike openai_chat_completion_with_reasoning.py, this example demonstrates the
streaming chat completions feature.

The streaming chat completions feature allows you to receive chat completions
in real-time as they are generated by the model. This is useful for scenarios
where you want to display chat completions to the user as they are generated
by the model.

Remember to check content and reasoning exist in `ChatCompletionChunk`,
content may not exist leading to errors if you try to access it.
"""

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

messages = [
        {"role": "user", "content": "9.11 and 9.8, which is greater?"},
        {"role": "assistant", "content": "9.11 is greater", "reasoning": "9.11 is greater than 9.8"},
        {"role": "user", "content": "8.11 and 8.8, which is greater?"}
    ]

def main():
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    models = client.models.list()
    model = models.data[0].id

    # ruff: noqa: E501
    # For granite: add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
     stream = client.chat.completions.create(
            model=model,
            messages=messages,
            stream=True,
            extra_body={"reasoning_effort": "high"}
        )

    print("client: Start streaming chat completions...")
    printed_reasoning = False
    printed_content = False

    for chunk in stream:
        # Safely extract reasoning and content from delta,
        # defaulting to None if attributes don't exist or are empty strings
        reasoning = getattr(chunk.choices[0].delta, "reasoning", None) or None
        content = getattr(chunk.choices[0].delta, "content", None) or None

        if reasoning is not None:
            if not printed_reasoning:
                printed_reasoning = True
                print("reasoning:", end="", flush=True)
            print(reasoning, end="", flush=True)
        elif content is not None:
            if not printed_content:
                printed_content = True
                print("\ncontent:", end="", flush=True)
            # Extract and print the content
            print(content, end="", flush=True)


if __name__ == "__main__":
    main()

I'm aware of an issue regarding reasoning and tool parsing in vLLM, i'm working on a fix and will update here once a PR is submitted and again when landed ! In the meantime you might have to sadly use an earlier vLLM version sorry for the inconvenience

Thanks Julien, much appreciated! No problem to use an earlier version in the meantime.

Hi,
first of all thanks to Mistral team for publicly releasing this model.

I have the same issue, but I can't find version of vLLM which works without this issue. @juliendenize suggested to use older version of vLLM, but which one?
I tried 0.25.1, 0.25.0, 0.24.0, 0.23.0 and 0.22.1 - all have the same issue. Can someone share which vLLM version worked for him and which command was used to run it?

Hey FYI here is the pr we're trying to land to make the fix:
https://github.com/vllm-project/vllm/pull/48947
hopefully it will be merged soon but if you'd like to give it a try please let us know if it solves your problem :)

Hi the PR has been merged lmk if it fixes your issue !
Edit: this is for Mistral format

Thanks Julien, it's working great for me now.