You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Agentic SQL & routing: higher tiers unsolvable, Verify solve can be passed without the tool path #31
Functional testing of /agentic-tools (done for #29) found four problems that need backend or UI changes, so they are not in that content PR.
1. F1–F4 cannot be solved through the tools
From F1 upward the fragment filter rejects %, _, anything containing bob, and fragments under 3 alphanumeric characters. Since the only username to match is bob, no allowed fragment can return Bob's row. An exhaustive search of ~1.5M fragments (all strings up to 5 characters from a set including b, o, case variants, wildcards and punctuation) found zero leaks at F1–F4; F0 leaks with b. At F3/F4 the table name must exactly match an allowlisted name, so it is not injectable either.
Question for maintainers: are F1–F4 meant to be defenses that hold, or challenges with an intended bypass? If the latter, the F1 filter is too strict as written. The test named test_catering_sql_f1_partial_fragment_still_exfil_mocked asserts solved is False, which suggests this changed at some point.
2. "Verify solve" can be passed without exploiting the tool path
The solve check can be satisfied at every tier, including F4, without any tool call. Because the lab is used for CTFs and workshops, I am not posting the reproduction here; I will share it with the maintainers directly.
Expected: a solve counts only when another user's token was returned by a tool result.
3. generated_sql, level_name and level_hint are returned but never shown
The page says each response includes generated_sql, but the chat only renders the combined reply. Seeing the executed SQL is the core learning point of the lab; currently it is only visible in DevTools.
4. A missing Ollama model shows up as a bare chat reply
With no model pulled, the assistant replies Error: Ollama status 404 with no guidance. Pointing to Lab Setup would help workshop attendees.
Activity