Skip to content

URLUtils.splitUrlArgs truncates on '#' inside a LoadOptions value, silently dropping the rest of the options #8

Description

@platonai

Summary

URLUtils.splitUrlArgs treats # anywhere in the configured-url string as the start of a fragment, so everything after it is dropped before LoadOptions is parsed. Any option value containing # therefore truncates the option string, silently disabling that option and every option that follows it.

Reproduction

Submit a URL with an id-selector gate (the documented way to require a non-blank element):

https://www.amazon.com/dp/B0XXXXXXX?th=1 -refresh -parse -requireNotBlank '#productTitle' -nMaxRetry 3

Server log (browser4 backend, PulsarOptions.parse):

WARN a.p.p.s.c.o.PulsarOptions - Failed to parse
-parse -refresh -parse -requireNotBlank
com.beust.jcommander.ParameterException: Expected a value after parameter -requireNotBlank
    at ai.platon.pulsar.skeleton.common.options.PulsarOptions.doParse(PulsarOptions.kt:95)
    ...
    at ai.platon.pulsar.skeleton.common.options.LoadOptions$Companion.normalize(LoadOptions.kt:1414)

What arrives at the parser is -parse -refresh -parse -requireNotBlank: the value '#productTitle' and the trailing -nMaxRetry 3 are gone — cut at the #.

Impact

  • Every LoadOptions value containing # is unusable: -requireNotBlank '#id', -outLink '#main a', -requireNotBlank '#productTitle, #title', etc.
  • The failure mode is a warning log only (parse() returns false and the options fall back to defaults), so callers believe their quality gate / retry policy is active when it is not. A -nMaxRetry 3 placed after such a value silently disappears too.
  • Batch/scrape submissions that embed the URL and the options in one string (Browser4's swarm load_and_select('<url> <args>', ':root') payload) are the most affected, because the URL and the args are parsed as one configured url.

Suggest fix

Split the URL from its arguments before any fragment handling, and strip a fragment only from the URL token, e.g.:

val (urlToken, args) = splitUrlArgs0(configuredUrl)
val url = urlToken.substringBefore('#')

Alternatively support quoted values when tokenizing args (so '#productTitle' survives). Either way, a value that cannot be parsed should not silently drop the remaining options — LoadOptions.parse should surface it (or the caller should fail).

Workaround used today

Browser4 side: use an attribute selector instead of an id selector — -requireNotBlank "[id=productTitle]" — and the CLI prints a warning when --load-options contains # (commit d924559079). That mitigates the footgun for CLI users but does not fix the parser.

Environment

  • Browser4 4.13.18-SNAPSHOT against browser4-base URLUtils/LoadOptions as consumed by browser4-core/browser4-skeleton.
  • Found while submitting a 100-URL Amazon batch with -requireNotBlank '#productTitle' -nMaxRetry 3.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions