Skip to content

fix(run): survive malformed links and explain unreachable companies - #12

Merged
anusbutt merged 1 commit into
mainfrom
fix/malformed-href-and-skip-reasons
Jul 30, 2026
Merged

fix(run): survive malformed links and explain unreachable companies#12
anusbutt merged 1 commit into
mainfrom
fix/malformed-href-and-skip-reasons

Conversation

@anusbutt

Copy link
Copy Markdown
Owner

Found while running a 54-company sweep: one company crashed and 50 were misreported.

A single malformed href took down a whole company

dryerventheroes.com ships an unfilled WordPress placeholder in its nav:

<a href="http://[BookingLink]" target="_blank">Schedule Service</a>

Python's urlsplit (3.11.4+) reads the brackets as an IPv6 literal and calls ipaddress.ip_address("BookingLink"), which raises ValueError. discover_extra_pages did not guard the parse, so the exception escaped and the company failed outright:

error: dryer-vent-superheroes-of-dallas: 'BookingLink' does not appear to be an IPv4 or IPv6 address

Per-company isolation contained it — the batch finished — but the prospect was lost over someone else's broken link. The link is now skipped.

Verified against the live site: discover_extra_pages now returns [('contact', 'https://dryerventheroes.com/contact-us/')] instead of raising.

Every unreachable company reported "blank email"

The skip reason echoed the ingest-time bucket_reason, which describes the input row rather than what research found — so all 50 unreachable companies reported blank email, reading as though address recovery never ran.

The reason is now judged after recovery, separating the cases that call for different responses:

Condition Reason
No website resolved no website could be resolved
Website resolved, nothing readable no page could be fetched from <url>
Pages read, no address published no published address on any fetched page

The first and third are what the README already documented, so this makes the code match the docs rather than the other way round. On the sweep above, those 50 rows now read no published address on any fetched page — which says the keyword is fine and the vertical prefers contact forms.

Guarantees

  • No company is silently dropped — strengthened. One company moves from failed to a named, explained skip, and the remaining skips gain a reason that distinguishes a sourcing problem from an unreachable vertical.
  • No change to network behaviour, the Facebook host guard, citation validation, the locked fallback, or approval-gated sending. Recovery still issues no new requests.

Tests

  • tests/unit/test_extract.py — a bracketed placeholder href alongside a good link; the good link still comes back
  • tests/unit/test_pipeline_reasons.py (new) — all three reason branches
  • tests/integration/test_email_recovery_batch.py — end-to-end skip reason describes the outcome, not the input row

Full suite: 612 passed.

🤖 Generated with Claude Code

Two companies were lost or misreported on a 54-row sweep.

A single malformed href took down a whole company. An unfilled WordPress
placeholder — <a href="http://[BookingLink]"> — reads as an IPv6 literal to
urlsplit (3.11.4+), which raises from ipaddress. discover_extra_pages did not
guard the parse, so someone else's broken nav link cost us the prospect.
Per-company isolation contained it, but the company was still lost for no
reason. Skip the link instead.

The skip reason echoed the ingest-time bucket_reason, so every unreachable
company reported "blank email" — a restatement of the input row that reads as
though address recovery never ran. Judge the reason after recovery instead, and
separate the cases that call for different responses: no website resolved is a
sourcing problem, while pages read with nothing published is a vertical that
prefers contact forms. This is what the README already documented.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@anusbutt
anusbutt merged commit 15b93fe into main Jul 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant