add note on fim tokens
#42
by loubnabnl HF Staff - opened
README.md
CHANGED
|
@@ -243,6 +243,7 @@ inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
|
|
| 243 |
outputs = model.generate(inputs)
|
| 244 |
print(tokenizer.decode(outputs[0]))
|
| 245 |
```
|
|
|
|
| 246 |
|
| 247 |
### Load other checkpoints
|
| 248 |
We upload the checkpoint of each experiment to a separate branch as well as the intermediate checkpoints as commits on the branches. You can load them with the `revision` flag:
|
|
|
|
| 243 |
outputs = model.generate(inputs)
|
| 244 |
print(tokenizer.decode(outputs[0]))
|
| 245 |
```
|
| 246 |
+
Make sure to use `<fim-prefix>, <fim-suffix>, <fim-middle>` and not `<fim_prefix>, <fim_suffix>, <fim_middle>` as in StarCoder models.
|
| 247 |
|
| 248 |
### Load other checkpoints
|
| 249 |
We upload the checkpoint of each experiment to a separate branch as well as the intermediate checkpoints as commits on the branches. You can load them with the `revision` flag:
|