Say why a Yahoo download came back empty instead of blaming the symbol - #52
Merged
Merged
Conversation
`_fetch_and_transform_data` read an empty frame from `yf.download` as proof the symbol does
not exist. It is not proof of anything. `yf.download` reports every per-symbol failure
identically: it calls `Ticker.history` without `raise_errors`, catches whatever comes back
inside `_download_one`, records it on a context object local to that call, logs it, and hands
back an empty frame. So ANY cause - a throttle, a Yahoo outage, a malformed response, a
transient miss - was reported to the reader as `Symbol 'AAPL' not found or invalid`, sending
them to debug the one thing that was correct.
Rate limiting is the instance that reached CI. The public repository's `ch02-03` job failed on
2026-09-11 with that message while the log beneath it read `YFRateLimitError('Too Many
Requests. Rate limited. Try after a while.')`. The class is wider than the instance.
On an empty download the provider now asks again through the path `download` itself uses, with
`raise_errors=True`, and reports what comes back: a throttle as RateLimitError, a refusal Yahoo
named as SymbolNotFoundError carrying that reason in `details`, an empty answer as
SymbolNotFoundError exactly as before. If the second ask returns data, the first result was a
transient miss and the data is returned rather than discarded. It costs one request, only on a
path that has already failed.
Retry comes with the mapping and needs no new mechanism. `BaseProvider.fetch_ohlcv` already
wraps in tenacity with `retry_if_exception(_provider_error_is_retryable)`, retryable being a
NetworkError whose `retryable` is set, and RateLimitError(NetworkError) defaults it True and
honours `retry_after`. A SymbolNotFoundError is not a NetworkError, so the old reporting made a
429 fatal as well as misleading. NetworkError is added to the re-raise tuple in
`_fetch_and_transform_data` for the same reason: the catch-all below it converts anything
unnamed into DataValidationError, which would have turned a survivable error into a fatal one
two lines after it was raised.
Two things that look like the obvious fix and are not, recorded in the tests so the next reader
does not re-derive them. `yf.download` has no `raise_errors` parameter, though `Ticker.history`
does. And `yfinance.shared._ERRORS` is declared and never written, so it reads as a channel for
per-symbol errors and carries none.
Measured against yfinance 1.5.2. The behaviour depended on is upstream's, so
`test_ticker_history_still_takes_raise_errors` fails if that channel disappears rather than
letting the probe quietly stop raising and the defect return unnoticed.
Nine tests in tests/test_yahoo_empty_download_reporting.py, and four mutations each fail the
cases that own them: restoring the original raise, dropping NetworkError from the re-raise
tuple, probing when the download succeeded, and mapping a throttle back to the symbol error.
The existing Yahoo suites pass unchanged, 70 tests in total.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An empty frame from
yf.downloadwas read as proof the symbol does not exist. It is not proof of anything.yf.downloadreports every per-symbol failure identically: it callsTicker.historywithoutraise_errors, catches whatever comes back inside_download_one, records it on a context object local to that call, logs it, and hands back an empty frame. So any cause - a throttle, a Yahoo outage, a malformed response, a transient miss - reached the user asSymbol 'AAPL' not found or invalid, sending them to debug the one thing that was correct.Rate limiting is the instance that reached CI. A chapter-2 notebook job failed on 2026-09-11 with that message while the log beneath it read
YFRateLimitError('Too Many Requests. Rate limited. Try after a while.'). The class is wider than the instance, and this fixes the class.What changed
On an empty download the provider asks again through the path
downloaditself uses, withraise_errors=True, and reports what comes back:YFRateLimitErrorRateLimitError- retryableYFExceptionSymbolNotFoundErrorcarrying Yahoo's own reason indetailsSymbolNotFoundError, exactly as beforeIt costs one request, only on a path that has already failed. A non-empty download never asks twice, and a test pins that.
Retry comes with the mapping and adds no mechanism.
BaseProvider.fetch_ohlcvalready wraps in tenacity withretry_if_exception(_provider_error_is_retryable)- retryable being aNetworkErrorwhoseretryableis set - andRateLimitError(NetworkError)defaults itTrueand honoursretry_after. ASymbolNotFoundErroris not aNetworkError, so the old reporting made a 429 fatal as well as misleading.NetworkErroris added to the re-raise tuple in_fetch_and_transform_datafor the same reason: the catch-all below it converts anything unnamed intoDataValidationError, which would have turned a survivable error into a fatal one two lines after it was raised.Two things that look like the obvious fix and are not
Recorded in the tests so the next reader does not spend the time finding out again.
yf.downloadhas noraise_errorsparameter.Ticker.historydoes, and it is the pathdownloadcalls internally.yfinance.shared._ERRORSis declared in 1.5.2 and never written. It reads as a channel for per-symbol errors and carries none.Verification
Measured against yfinance 1.5.2. The behaviour depended on is upstream's, so
test_ticker_history_still_takes_raise_errorsfails if that channel disappears rather than letting the probe quietly stop raising and the defect return unnoticed.Four mutations, each failing the cases that own it:
NetworkErrorfrom the re-raise tupleNot in this change
Reading the chapter-2 notebooks' universe from a test fixture instead of the network would make that CI job deterministic, but the chapter's subject is incremental downloading, so a fixture may defeat its point. That is a decision about what the chapter demonstrates, not test infrastructure.
🤖 Generated with Claude Code