Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions crates/tinytools-agent/src/parse/grammar/tagged.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,13 @@
//! templates);
//! * sentinel pipes leaked into the markers, in any position:
//! `<|tool_call>…<tool_call|>`, `<|tool_call|>…<|tool_call|>`,
//! `…</tool_call|>`;
//! `…</tool_call|>`, including the fullwidth `|` those templates
//! actually emit;
//! * a `DeepSeek` DSML marker on the tag itself,
//! `<|DSML|tool_call>…</|DSML|tool_call>` — the same marker
Comment on lines +14 to +15

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the grammar table for DSML tagged calls

This adds a new TaggedJson spelling, but crates/tinytools-agent/README.md still omits DSML from the TaggedJson row and mentions it only under InvokeXml. Update the consumer-facing grammar table in this commit so it remains aligned with the parser and module documentation.

AGENTS.md reference: AGENTS.md:L204-L205

Useful? React with 👍 / 👎.

//! [`super::invoke_xml`] already accepts on `<invoke>`, which this family
//! used to miss, so a `deepseek` turn that chose the tag form over the
//! invoke form parsed as prose and the call was silently dropped;
//! * a `call:` prefix before the body;
//! * a fenced block instead of a tag, ```` ```tool_call … ``` ````, sometimes
//! closed by a stray `</tool_call>`;
Expand Down Expand Up @@ -39,11 +45,17 @@ use crate::types::{CallSource, ParseOptions, ParsedToolCall};
pub(crate) struct Tagged;

/// Any tag-family marker: `<tool_call>`, `<toolcall>`, `<tool-call>`, with
/// pipes, a slash, or whitespace leaked in, and an optional attribute list.
/// `<tool_calls>` (plural, a JSON key) and `<tool_callable>` do not match:
/// the name must end at a pipe, slash, whitespace, or `>`.
static TAG_RE: LazyLock<Option<Regex>> =
LazyLock::new(|| Regex::new(r"(?i)<[|/\s]*tool[_-]?call(?:[|/\s]*|\s+[^>]*)>").ok());
/// pipes (ASCII `|` or the fullwidth `|` chat templates emit), a slash,
/// whitespace, or a `DeepSeek` DSML marker leaked in, and an optional
/// attribute list. `<tool_calls>` (plural, a JSON key, and the DSML wrapper
/// element) and `<tool_callable>` do not match: the name must end at a pipe,
/// slash, whitespace, or `>`.
static TAG_RE: LazyLock<Option<Regex>> = LazyLock::new(|| {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

priority high tests confident

Update openers and probe for DSML and fullwidth variants

The new regex accepts <|DSML|tool_call>, <|DSML|tool_call>, and fullwidth bar variants, but openers() (used to hold partial markers during streaming) and the probe() pending_opener call only list ASCII-bar forms like <|tool_call. Without these entries, a DSML-prefixed or fullwidth-bar tag split across chunks is not held, so the incomplete tag fragment escapes as narrative and the call may be silently dropped. Add the corresponding strings (e.g., <|DSML|tool_call, <|tool_call, <|DSML|tool_call) to both lists.


Additional critique observation

priority high likely

Update opener detection for DSML tag markers

[RULE] incomplete-parser-integration

Adding DSML spellings only to TAG_RE is insufficient if probe/openers first locate candidate tags using a fixed literal list. The new inputs such as <|DSML|tool_call> and <|DSML|tool_call> can therefore remain prose and never reach this regex, contradicting the added tests and the stated behavior. Add the DSML opener variants (including doubled bars and any supported whitespace forms) to the grammar's candidate detection, or make probing use the same regex when searching for the next marker.

[RULE] incomplete-streaming-support ·

Regex::new(
r"(?i)<[|\u{ff5c}/\s]*(?:DSML[|\u{ff5c}/\s]*)?tool[_-]?call(?:[|\u{ff5c}/\s]*|\s+[^>]*)>",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Classify fullwidth-pipe closing markers

When the newly accepted fullwidth-pipe spelling puts the slash after the pipe (<|/tool_call|>), TAG_RE matches it but is_closing_marker still strips only ASCII |. In a doubled block followed by narrative, swallow_extra_closers therefore treats the second closer as a new opener and leaves <|/tool_call|> in the user-visible text; extend closer classification to handle U+FF5C as well.

Useful? React with 👍 / 👎.

)
.ok()
});

/// Openers a fenced block can carry. `` ```tool_calls `` (plural) is listed
/// separately from `` ```tool_call `` rather than relying on a prefix match:
Expand Down
60 changes: 60 additions & 0 deletions crates/tinytools-agent/src/parse/test/tagged.rs
Original file line number Diff line number Diff line change
Expand Up @@ -513,3 +513,63 @@ fn a_bare_trailing_opener_is_dropped_not_shown() {
let (text, _) = parse("before <tool-call>not-json");
assert_eq!(text, "before <tool-call>not-json");
}

// ── DeepSeek DSML on the tag family ─────────────────────────────────────────

/// `deepseek` emits its DSML marker on the `tool_call` tag, not only on
/// `<invoke>`, and with the fullwidth bar its template actually uses.
/// Observed live from `deepseek-v4-flash` driving the code dialect: the call
/// parsed as prose, so the turn ended with the model's lead-in and the tool
/// was never run.
#[test]
fn dsml_marker_on_the_tool_call_tag_parses() {
let raw = concat!(
"I'll look it up.\n\n",
"<|DSML|tool_call>\n",
"{\"name\": \"GMAIL_FETCH_EMAILS\", \"arguments\": {\"max_results\": 1}}\n",
"</|DSML|tool_call>"
);
let (text, calls) = crate::parse::parse_tool_calls(raw);
assert_eq!(calls.len(), 1, "DSML tool_call tag must parse: {calls:?}");
assert_eq!(calls[0].name, "GMAIL_FETCH_EMAILS");
assert_eq!(text.trim(), "I'll look it up.");
}

/// Doubled bars and the ASCII spelling are the same marker.
#[test]
fn dsml_marker_variants_on_the_tag_parse() {
for open_close in [
("<||DSML||tool_call>", "</||DSML||tool_call>"),
("<|DSML|tool_call>", "</|DSML|tool_call>"),
("<|DSML|tool_call>", "<|DSML|tool_call>"),
] {
let raw = format!(
"{}\n{{\"name\": \"echo\", \"arguments\": {{}}}}\n{}",
open_close.0, open_close.1
);
let (_, calls) = crate::parse::parse_tool_calls(&raw);
assert_eq!(
calls.len(),
1,
"variant {open_close:?} must parse: {calls:?}"
);
assert_eq!(calls[0].name, "echo");
}
}

/// The DSML *wrapper* element is plural and is not a call marker; treating it
/// as one would open a block on the wrapper and close it on the first inner
/// tag, losing the call inside.
#[test]
fn the_plural_dsml_wrapper_is_not_a_tag_marker() {
let raw = concat!(
"<|DSML|tool_calls>\n",
"<|DSML|tool_call>\n",
"{\"name\": \"echo\", \"arguments\": {}}\n",
"</|DSML|tool_call>\n",
"</|DSML|tool_calls>"
);
let (_, calls) = crate::parse::parse_tool_calls(raw);
assert_eq!(calls.len(), 1, "the inner call is the only call: {calls:?}");
assert_eq!(calls[0].name, "echo");
}
Loading