Hi — I reviewed OpenJev for a write-up on mrjev.com, at 443b679. I ran it in node:24 in Docker with OPENAI_BASE_URL pointed at a stand-in OpenAI-compatible server of mine, so I could log every request and control exactly what the "model" answered. No real provider was called.
The per-option idea is worth building: asking for one {"p": 0-1} under a strict json_schema per option genuinely removes the dropped-key and invented-label failures that a single big JSON blob has, and tsc --noEmit is clean. Four things I'd raise, in the order I'd fix them.
1. There's no licence. No LICENSE file, no license field in package.json (it's "private": true), and no mention in the README. With a GitHub description of "OpenSource Jev", that's the one that surprises people — as it stands nobody can legally reuse any of it. MIT or Apache-2.0 would take a minute.
2. A model that is certain about everything and one that knows nothing produce identical output. Two runs of the same 3-option choice, with my stand-in answering a constant:
| stand-in answers |
choice |
probabilities |
confidence |
p = 0.5 for every option |
billing |
0.333 / 0.333 / 0.333 |
0.417 |
p = 1.0 for every option |
billing |
0.333 / 0.333 / 0.333 |
0.417 |
That follows from scoresToDistribution — softmax over logits is normalised odds, so a constant cancels — but nothing downstream can tell the two apart. Worth considering: carry the raw per-option probabilities through to the response alongside the normalised ones (you already have them in results), so a caller can see "every option scored 1.0" and treat it as a non-answer.
Related: confidence is 0.5 × top + 0.5 × (gap + 0.5), which has a floor of 0.25 — a perfectly uniform three-way answer reads 0.417 and a four-way one 0.375. When the model does discriminate it behaves (0.976 vs 0.012 in my test), so it's specifically the "no information" end that's mislabelled. 1 − normalised entropy would go to 0 where this goes to 0.375, if you want a number a threshold can use.
And because the tie-break is >, the winner of a uniform distribution is whichever key comes first in criteria: reordering the three keys with nothing else changed moved the answer from billing to technical.
3. Two paths swallow a failed parse. If the reply isn't JSON, the regex fallback in scoreProposition takes over. I answered with prose — "I think billing is most likely, maybe 70 percent" — and /0?\.\d+|[01](?:\.0+)?/ matched the 0 inside 70, so that option scored 0.0. Nothing in the response says a parse failed; a warnings: [] array on the response, or simply propagating the failure, would be better than a silent 0.
oneshot has the same shape at String(data.choice ?? Object.keys(q.criteria)[0]): a model that omits the key gets the first option at confidence 0.5 with a probability map made up on the spot. I saw {"billing": 0.5, "technical": 0} come back from a reply that mentioned neither.
(Credit where it's due: when my stand-in returned HTTP 500 for one of three option calls, the OpenAI SDK's own retry covered it and the answer came back complete.)
4. app.use(cors()) plus the .env key. The server falls back to the key in your .env when the request doesn't carry one, and CORS is fully open with no auth:
$ curl -D- -X POST localhost:3001/api/evaluate -H 'Origin: https://evil.example' \
-d '{"state":"hi","questions":{"q":{"type":"noul","instructions":"Is this a test?"}}}'
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Served, using the server's key. While npm run dev is running, any page open in the browser can spend it. Binding to 127.0.0.1 and restricting cors() to the dev origin would close it; requiring the key in the request when one isn't in .env is already the behaviour, so this is only about the fallback path.
Minor: there's an empty file named sed2Mso9e committed at the repository root, presumably the debris of a sed command.
Happy to send a PR for any of these — the licence and the CORS origin are both a couple of lines, and I'd be glad to do the entropy-based confidence as a separate one if you want it.
Hi — I reviewed OpenJev for a write-up on mrjev.com, at
443b679. I ran it innode:24in Docker withOPENAI_BASE_URLpointed at a stand-in OpenAI-compatible server of mine, so I could log every request and control exactly what the "model" answered. No real provider was called.The per-option idea is worth building: asking for one
{"p": 0-1}under a strictjson_schemaper option genuinely removes the dropped-key and invented-label failures that a single big JSON blob has, andtsc --noEmitis clean. Four things I'd raise, in the order I'd fix them.1. There's no licence. No
LICENSEfile, nolicensefield inpackage.json(it's"private": true), and no mention in the README. With a GitHub description of "OpenSource Jev", that's the one that surprises people — as it stands nobody can legally reuse any of it. MIT or Apache-2.0 would take a minute.2. A model that is certain about everything and one that knows nothing produce identical output. Two runs of the same 3-option choice, with my stand-in answering a constant:
p = 0.5for every optionbillingp = 1.0for every optionbillingThat follows from
scoresToDistribution— softmax over logits is normalised odds, so a constant cancels — but nothing downstream can tell the two apart. Worth considering: carry the raw per-option probabilities through to the response alongside the normalised ones (you already have them inresults), so a caller can see "every option scored 1.0" and treat it as a non-answer.Related:
confidenceis0.5 × top + 0.5 × (gap + 0.5), which has a floor of 0.25 — a perfectly uniform three-way answer reads 0.417 and a four-way one 0.375. When the model does discriminate it behaves (0.976 vs 0.012 in my test), so it's specifically the "no information" end that's mislabelled.1 − normalised entropywould go to 0 where this goes to 0.375, if you want a number a threshold can use.And because the tie-break is
>, the winner of a uniform distribution is whichever key comes first incriteria: reordering the three keys with nothing else changed moved the answer frombillingtotechnical.3. Two paths swallow a failed parse. If the reply isn't JSON, the regex fallback in
scorePropositiontakes over. I answered with prose — "I think billing is most likely, maybe 70 percent" — and/0?\.\d+|[01](?:\.0+)?/matched the0inside70, so that option scored 0.0. Nothing in the response says a parse failed; awarnings: []array on the response, or simply propagating the failure, would be better than a silent 0.oneshothas the same shape atString(data.choice ?? Object.keys(q.criteria)[0]): a model that omits the key gets the first option at confidence 0.5 with a probability map made up on the spot. I saw{"billing": 0.5, "technical": 0}come back from a reply that mentioned neither.(Credit where it's due: when my stand-in returned HTTP 500 for one of three option calls, the OpenAI SDK's own retry covered it and the answer came back complete.)
4.
app.use(cors())plus the.envkey. The server falls back to the key in your.envwhen the request doesn't carry one, and CORS is fully open with no auth:Served, using the server's key. While
npm run devis running, any page open in the browser can spend it. Binding to127.0.0.1and restrictingcors()to the dev origin would close it; requiring the key in the request when one isn't in.envis already the behaviour, so this is only about the fallback path.Minor: there's an empty file named
sed2Mso9ecommitted at the repository root, presumably the debris of asedcommand.Happy to send a PR for any of these — the licence and the CORS origin are both a couple of lines, and I'd be glad to do the entropy-based confidence as a separate one if you want it.