Problems with Ollama API

[alexkh]

Greetings!

I'm writing Inforno - a Rust GUI for chatting LLM's. Was having issues running Ornith-1.0 on Ollama in Linux, so I decided to test this Fox app in Linux, since it claims to be a drop-in replacement. The CLI interface seems to works perfectly.

Now, since my app already uses Ollama API, I decided to connect it directly by creating Fox config file and setting port to 11434.
I installed ornith-1.0-35b-Q4_K_M which should be runnable with moe_cpu = true so my whole config.toml is this:

port = 11435
moe_cpu = true
type_kv = "turbo3"

Now, the pipeline seems to work, but I'm getting partial or empty replies.

My prompt is simple: "is 189 divisible by 9?"
I'm getting this reply:

Agent1:
<|im_end｜><｜fim begin｜>189  <response_group index="0">
<|mask_start|>**Yes**, let's check using both methods:



### Method
<response_group index="0">

*   **Direct Division**: 
$$\r

or this reply:

Agent1:
To determine if 189 is divisible by 9, you can use the divisibility rule for <response>
<response_group index="0">
<|mask_start|>**

*   **Divisibility Rule for 9**: A number is divisible by 9 if the sum of its digits is divisible by 9.

**Step-by-Step Check:**

1.
 Add the digits of <response>
<|mask_end|><｜fim begin｜>**Yes**, let's check using both methods:



### Method

Two obvious problems: incomplete responses and weird html tags which are not present when using cli api.

I also tried turning off reasoning, turning off streaming - did not help much.

Best regards,

-Alexandre

[ManuelSLemos]

Those aren't stray HTML tags, they're the model's own. <|mask_start|> and <response_group> say Ornith-1.0 is a diffusion model, and those don't generate left to right � they unmask a sequence over a fixed number of steps. llama.cpp ships a separate llama-diffusion-cli for them.

fox's decode loop is autoregressive and never checked, so it loaded the model, ran it token by token anyway, and streamed the intermediate state to you as a reply. Mask tokens in the text, fragments out of order, truncation, and your second reply ending with the exact opening of your first � all the same cause. Nothing wrong with your config or your client; fox just can't serve this model. 0.21.0 refuses it at load and points you at llama-diffusion-cli instead of producing nonsense.

Two smaller things. type_kv = "turbo3" isn't a value fox has � it takes f16, q8_0 or q4_0, and everything else silently became f16, so that setting did nothing and never told you. Now it fails at startup. moe_cpu is real and was working.

Also, you said you set the port to 11434 but the config says 11435. Worth checking what your app is actually connecting to � if Ollama is still up on 11434, you may have been testing Ollama.

If Ornith-1.0 isn't a diffusion model after all, say so and I'll look again � the repeated text would then point at KV reuse between requests, and --kv-reuse false would settle it.

[alexkh]

Greetings!
Ornith is not a diffusion model, it's a LLM built for agentic coding.
I have had some success by wrapping the cli-run of Fox into a thin Ollama API server which gave some meaningful output below. It works for me, I just wish I could ask it to disable reasoning to save some context space. Unfortunately, there is no such command line parameter in Fox. Otherwise it's workable in CLI, as long as I keep wrapping the attached files so they are passed in a single line...

Ornith_chat.md