Summary
URLUtils.splitUrlArgs treats # anywhere in the configured-url string as the start of a fragment, so everything after it is dropped before LoadOptions is parsed. Any option value containing # therefore truncates the option string, silently disabling that option and every option that follows it.
Reproduction
Submit a URL with an id-selector gate (the documented way to require a non-blank element):
https://www.amazon.com/dp/B0XXXXXXX?th=1 -refresh -parse -requireNotBlank '#productTitle' -nMaxRetry 3
Server log (browser4 backend, PulsarOptions.parse):
WARN a.p.p.s.c.o.PulsarOptions - Failed to parse
-parse -refresh -parse -requireNotBlank
com.beust.jcommander.ParameterException: Expected a value after parameter -requireNotBlank
at ai.platon.pulsar.skeleton.common.options.PulsarOptions.doParse(PulsarOptions.kt:95)
...
at ai.platon.pulsar.skeleton.common.options.LoadOptions$Companion.normalize(LoadOptions.kt:1414)
What arrives at the parser is -parse -refresh -parse -requireNotBlank: the value '#productTitle' and the trailing -nMaxRetry 3 are gone — cut at the #.
Impact
- Every LoadOptions value containing
# is unusable: -requireNotBlank '#id', -outLink '#main a', -requireNotBlank '#productTitle, #title', etc.
- The failure mode is a warning log only (
parse() returns false and the options fall back to defaults), so callers believe their quality gate / retry policy is active when it is not. A -nMaxRetry 3 placed after such a value silently disappears too.
- Batch/scrape submissions that embed the URL and the options in one string (Browser4's swarm
load_and_select('<url> <args>', ':root') payload) are the most affected, because the URL and the args are parsed as one configured url.
Suggest fix
Split the URL from its arguments before any fragment handling, and strip a fragment only from the URL token, e.g.:
val (urlToken, args) = splitUrlArgs0(configuredUrl)
val url = urlToken.substringBefore('#')
Alternatively support quoted values when tokenizing args (so '#productTitle' survives). Either way, a value that cannot be parsed should not silently drop the remaining options — LoadOptions.parse should surface it (or the caller should fail).
Workaround used today
Browser4 side: use an attribute selector instead of an id selector — -requireNotBlank "[id=productTitle]" — and the CLI prints a warning when --load-options contains # (commit d924559079). That mitigates the footgun for CLI users but does not fix the parser.
Environment
- Browser4 4.13.18-SNAPSHOT against
browser4-base URLUtils/LoadOptions as consumed by browser4-core/browser4-skeleton.
- Found while submitting a 100-URL Amazon batch with
-requireNotBlank '#productTitle' -nMaxRetry 3.
Summary
URLUtils.splitUrlArgstreats#anywhere in the configured-url string as the start of a fragment, so everything after it is dropped beforeLoadOptionsis parsed. Any option value containing#therefore truncates the option string, silently disabling that option and every option that follows it.Reproduction
Submit a URL with an id-selector gate (the documented way to require a non-blank element):
Server log (browser4 backend,
PulsarOptions.parse):What arrives at the parser is
-parse -refresh -parse -requireNotBlank: the value'#productTitle'and the trailing-nMaxRetry 3are gone — cut at the#.Impact
#is unusable:-requireNotBlank '#id',-outLink '#main a',-requireNotBlank '#productTitle, #title', etc.parse()returnsfalseand the options fall back to defaults), so callers believe their quality gate / retry policy is active when it is not. A-nMaxRetry 3placed after such a value silently disappears too.load_and_select('<url> <args>', ':root')payload) are the most affected, because the URL and the args are parsed as one configured url.Suggest fix
Split the URL from its arguments before any fragment handling, and strip a fragment only from the URL token, e.g.:
Alternatively support quoted values when tokenizing args (so
'#productTitle'survives). Either way, a value that cannot be parsed should not silently drop the remaining options —LoadOptions.parseshould surface it (or the caller should fail).Workaround used today
Browser4 side: use an attribute selector instead of an id selector —
-requireNotBlank "[id=productTitle]"— and the CLI prints a warning when--load-optionscontains#(commitd924559079). That mitigates the footgun for CLI users but does not fix the parser.Environment
browser4-baseURLUtils/LoadOptionsas consumed bybrowser4-core/browser4-skeleton.-requireNotBlank '#productTitle' -nMaxRetry 3.