Skip to content

Say why a Yahoo download came back empty instead of blaming the symbol - #52

Merged
stefan-jansen merged 1 commit into
mainfrom
fix/yahoo-rate-limit-not-invalid-symbol
Sep 11, 2026
Merged

Say why a Yahoo download came back empty instead of blaming the symbol#52
stefan-jansen merged 1 commit into
mainfrom
fix/yahoo-rate-limit-not-invalid-symbol

Conversation

@stefan-jansen

Copy link
Copy Markdown
Contributor

An empty frame from yf.download was read as proof the symbol does not exist. It is not proof of anything.

yf.download reports every per-symbol failure identically: it calls Ticker.history without raise_errors, catches whatever comes back inside _download_one, records it on a context object local to that call, logs it, and hands back an empty frame. So any cause - a throttle, a Yahoo outage, a malformed response, a transient miss - reached the user as Symbol 'AAPL' not found or invalid, sending them to debug the one thing that was correct.

Rate limiting is the instance that reached CI. A chapter-2 notebook job failed on 2026-09-11 with that message while the log beneath it read YFRateLimitError('Too Many Requests. Rate limited. Try after a while.'). The class is wider than the instance, and this fixes the class.

What changed

On an empty download the provider asks again through the path download itself uses, with raise_errors=True, and reports what comes back:

second ask result
YFRateLimitError RateLimitError - retryable
another YFException SymbolNotFoundError carrying Yahoo's own reason in details
empty SymbolNotFoundError, exactly as before
data returned - the first result was a transient miss

It costs one request, only on a path that has already failed. A non-empty download never asks twice, and a test pins that.

Retry comes with the mapping and adds no mechanism. BaseProvider.fetch_ohlcv already wraps in tenacity with retry_if_exception(_provider_error_is_retryable) - retryable being a NetworkError whose retryable is set - and RateLimitError(NetworkError) defaults it True and honours retry_after. A SymbolNotFoundError is not a NetworkError, so the old reporting made a 429 fatal as well as misleading.

NetworkError is added to the re-raise tuple in _fetch_and_transform_data for the same reason: the catch-all below it converts anything unnamed into DataValidationError, which would have turned a survivable error into a fatal one two lines after it was raised.

Two things that look like the obvious fix and are not

Recorded in the tests so the next reader does not spend the time finding out again.

  • yf.download has no raise_errors parameter. Ticker.history does, and it is the path download calls internally.
  • yfinance.shared._ERRORS is declared in 1.5.2 and never written. It reads as a channel for per-symbol errors and carries none.

Verification

Measured against yfinance 1.5.2. The behaviour depended on is upstream's, so test_ticker_history_still_takes_raise_errors fails if that channel disappears rather than letting the probe quietly stop raising and the defect return unnoticed.

$ uv run pytest tests/test_yahoo_empty_download_reporting.py -q
9 passed

$ uv run pytest tests/test_yahoo_provider.py tests/test_yahoo_provider_unit.py \
      tests/test_yahoo_provider_edge_cases.py tests/test_yahoo_empty_download_reporting.py -q
70 passed

Four mutations, each failing the cases that own it:

mutation cases that fail
restore the original raise the five diagnosis cases
drop NetworkError from the re-raise tuple the three rate-limit cases
probe even when the download succeeded a non-empty download never asks twice
map a throttle back to the symbol error the three rate-limit cases

Not in this change

Reading the chapter-2 notebooks' universe from a test fixture instead of the network would make that CI job deterministic, but the chapter's subject is incremental downloading, so a fixture may defeat its point. That is a decision about what the chapter demonstrates, not test infrastructure.

🤖 Generated with Claude Code

`_fetch_and_transform_data` read an empty frame from `yf.download` as proof the symbol does
not exist. It is not proof of anything. `yf.download` reports every per-symbol failure
identically: it calls `Ticker.history` without `raise_errors`, catches whatever comes back
inside `_download_one`, records it on a context object local to that call, logs it, and hands
back an empty frame. So ANY cause - a throttle, a Yahoo outage, a malformed response, a
transient miss - was reported to the reader as `Symbol 'AAPL' not found or invalid`, sending
them to debug the one thing that was correct.

Rate limiting is the instance that reached CI. The public repository's `ch02-03` job failed on
2026-09-11 with that message while the log beneath it read `YFRateLimitError('Too Many
Requests. Rate limited. Try after a while.')`. The class is wider than the instance.

On an empty download the provider now asks again through the path `download` itself uses, with
`raise_errors=True`, and reports what comes back: a throttle as RateLimitError, a refusal Yahoo
named as SymbolNotFoundError carrying that reason in `details`, an empty answer as
SymbolNotFoundError exactly as before. If the second ask returns data, the first result was a
transient miss and the data is returned rather than discarded. It costs one request, only on a
path that has already failed.

Retry comes with the mapping and needs no new mechanism. `BaseProvider.fetch_ohlcv` already
wraps in tenacity with `retry_if_exception(_provider_error_is_retryable)`, retryable being a
NetworkError whose `retryable` is set, and RateLimitError(NetworkError) defaults it True and
honours `retry_after`. A SymbolNotFoundError is not a NetworkError, so the old reporting made a
429 fatal as well as misleading. NetworkError is added to the re-raise tuple in
`_fetch_and_transform_data` for the same reason: the catch-all below it converts anything
unnamed into DataValidationError, which would have turned a survivable error into a fatal one
two lines after it was raised.

Two things that look like the obvious fix and are not, recorded in the tests so the next reader
does not re-derive them. `yf.download` has no `raise_errors` parameter, though `Ticker.history`
does. And `yfinance.shared._ERRORS` is declared and never written, so it reads as a channel for
per-symbol errors and carries none.

Measured against yfinance 1.5.2. The behaviour depended on is upstream's, so
`test_ticker_history_still_takes_raise_errors` fails if that channel disappears rather than
letting the probe quietly stop raising and the defect return unnoticed.

Nine tests in tests/test_yahoo_empty_download_reporting.py, and four mutations each fail the
cases that own them: restoring the original raise, dropping NetworkError from the re-raise
tuple, probing when the download succeeded, and mapping a throttle back to the symbol error.
The existing Yahoo suites pass unchanged, 70 tests in total.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 11, 2026 16:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@stefan-jansen
stefan-jansen merged commit 05249ec into main Sep 11, 2026
22 checks passed
@stefan-jansen
stefan-jansen deleted the fix/yahoo-rate-limit-not-invalid-symbol branch September 11, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants