fix(run): survive malformed links and explain unreachable companies - #12
Merged
Conversation
Two companies were lost or misreported on a 54-row sweep. A single malformed href took down a whole company. An unfilled WordPress placeholder — <a href="http://[BookingLink]"> — reads as an IPv6 literal to urlsplit (3.11.4+), which raises from ipaddress. discover_extra_pages did not guard the parse, so someone else's broken nav link cost us the prospect. Per-company isolation contained it, but the company was still lost for no reason. Skip the link instead. The skip reason echoed the ingest-time bucket_reason, so every unreachable company reported "blank email" — a restatement of the input row that reads as though address recovery never ran. Judge the reason after recovery instead, and separate the cases that call for different responses: no website resolved is a sourcing problem, while pages read with nothing published is a vertical that prefers contact forms. This is what the README already documented. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while running a 54-company sweep: one company crashed and 50 were misreported.
A single malformed href took down a whole company
dryerventheroes.comships an unfilled WordPress placeholder in its nav:Python's
urlsplit(3.11.4+) reads the brackets as an IPv6 literal and callsipaddress.ip_address("BookingLink"), which raisesValueError.discover_extra_pagesdid not guard the parse, so the exception escaped and the company failed outright:Per-company isolation contained it — the batch finished — but the prospect was lost over someone else's broken link. The link is now skipped.
Verified against the live site:
discover_extra_pagesnow returns[('contact', 'https://dryerventheroes.com/contact-us/')]instead of raising.Every unreachable company reported "blank email"
The skip reason echoed the ingest-time
bucket_reason, which describes the input row rather than what research found — so all 50 unreachable companies reportedblank email, reading as though address recovery never ran.The reason is now judged after recovery, separating the cases that call for different responses:
no website could be resolvedno page could be fetched from <url>no published address on any fetched pageThe first and third are what the README already documented, so this makes the code match the docs rather than the other way round. On the sweep above, those 50 rows now read
no published address on any fetched page— which says the keyword is fine and the vertical prefers contact forms.Guarantees
failedto a named, explained skip, and the remaining skips gain a reason that distinguishes a sourcing problem from an unreachable vertical.Tests
tests/unit/test_extract.py— a bracketed placeholder href alongside a good link; the good link still comes backtests/unit/test_pipeline_reasons.py(new) — all three reason branchestests/integration/test_email_recovery_batch.py— end-to-end skip reason describes the outcome, not the input rowFull suite: 612 passed.
🤖 Generated with Claude Code