Bug in vLLM inference code: Answer never set due to elif check
I think the Python code example presented for the vLLM inference does not work as intended. Correct me if I’m wrong, but when serving the model with the given parameters and using the OpenAI API, the stream chunks will always have a reasoning_content field, even when the model only outputs content. If there is no reasoning for a chunk, the value of the field is just None, but the field itself is still present. So, the elif ensures that the content variable is actually never set, because the if condition, which checks if the attribute reasoning_content is present, will always be true.
if hasattr(chunk.choices[0].delta, "reasoning_content"):
reasoning_content = chunk.choices[0].delta.reasoning_content
elif hasattr(chunk.choices[0].delta, "content"):
content = chunk.choices[0].delta.content
Because content is never set, it can never be added to answer, thus answer will always stay empty, and it will always appear like the model never actuallly generated an answer. So i would recommend changing the elif to an if.
Yep makes sense thanks for reaching out, will update the readme.
Edit: I can't remember if at time the vLLM version was not 100% of the time outputting a reasoning_content so for backward compatibility i check if it's there AND if it's not None.