Unexpected outputs
Hello! I’ve been experiencing some issues while using the M-Prometheus models for Direct Assessment on French texts. The 3B version works perfectly, but I’ve encountered many instances of unexpected behaviors with the 7B and 14B models.
For example, here are some outputs I got from the 14B model:
" Feedback: <|im_end|>
<|im_start|>.AFDMETHODIMP CMyClass::MyMethod() {
// Your implementation here
}<|im_end|> "
" Feedback: <|im_end|>
<|im_start|>.Formsubmit(true,"score", 5)<|im_end|> "
" Feedback: <|im_end|>
<|im_start|>.Formsubmission
<|im_start|><|im_start|>？
<|im_start|><|im_start|>？
…
<|im_start|><|im_start|>？
<|im_start|>คณะกรรมกรรมการ
<|im_start|>คณะกรรมกรรมการ
<|im_end|> "
I prompted the models using Qwen’s chat template, with the prompt you specified. Do you have any idea why I’m getting these unexpected outputs? Thanks in advance!
Hello,
I've never encountered this sort of issue. Are you using vllm for inference?
Hello,
I used the Transformers library for inference, not vllm. I was able to find the cause of my problem: I was preparing the inputs and decoding the outputs similarly to how it is done for Prometheus-2, which is a bit different from the code currently shown on the M-Prometheus model card. Now, all three models work perfectly!
Great! Closing the issue, then.