The Yahoo fetch and batch paths agree about the current session's placeholder bar - #47
Merged
Merged
Conversation
added 2 commits
September 8, 2026 16:04
From the US close until Yahoo consolidates the daily bar, `yf.download` returns a
final row for the current exchange date carrying volume and null OHLC.
`fetch_ohlcv` passed it to `_validate_no_nulls`, which raised on the whole fetch
("yahoo: Column 'open' contains 1 null values"), while `fetch_ohlcv_batch` had
always dropped it. The same bar therefore succeeded through `batch_load` and
failed through `fetch`, `load` and `update` in the same run, and `update()` could
not avoid it because `StorageManager.update` always asks for the current session.
Both paths now share `_drop_priceless_rows`, so they cannot drift again. A row with
prices and zero volume is a halted session and is kept; a window that holds nothing
but placeholders raises `SymbolNotFoundError` rather than reporting an empty fetch
as a successful one. `_validate_no_nulls` stays strict.
Closes ml4t/agent-workspace#1027.
Every adjustment assertion on `ContinuousContractBuilder.build` mocked the adjustment method, so nothing checked that the series a reader gets back has the roll gap removed. The paired-`RollEvent` rewrite that fixed the adjustment classes is therefore unguarded at the level the reader uses. Four tests on a two-contract panel in 10-point contango: the unadjusted series moves 11.0 across the roll (10.0 of spread, 1.0 genuine), back-adjustment moves 1.0, ratio adjustment returns 1/103 - the outgoing contract's own return - and both anchor the last bar on its traded price. Checked against a deliberately gap-leaving adjuster: it is rejected by both assertions. Refs ml4t/agent-workspace#12.
The audit on this PR failed on gitpython 3.1.58, which carries CVE-2026-78679 with eight known vulnerabilities, fixed in 3.1.59. It arrives transitively through mkdocs-git-revision-date-localized-plugin, so the lock is the only place to move it. The last main run predates the advisory, which is why main was green. `uv run pytest tests/ -q`: 3,619 passed, 280 deselected, unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects a reader of the book hits through the public API.
The Yahoo single-symbol path raised on the current session's placeholder bar
From the US close until Yahoo consolidates the daily bar,
yf.downloadreturns a final row for the current exchange date carrying volume and null OHLC.fetch_ohlcvpassed it to_validate_no_nulls, which raised on the whole fetch:fetch_ohlcv_batchhad always dropped that row. So the same bar succeeded throughbatch_loadand failed throughfetch,loadandupdatein the same run - which is what a reader of18_data_managementsaw, itsbatch_loadcell passing and itsupdatecell failing on the same three symbols.update()could not avoid it:StorageManager.updatesetsfetch_endto now and has no end-date parameter, so an incremental update always asks for the current session.Both paths now share
_drop_priceless_rows, so they cannot drift again. A row with prices and zero volume is a halted session and is kept; a window holding nothing but placeholders raisesSymbolNotFoundErrorrather than reporting an empty fetch as a successful one._validate_no_nullsstays strict - a row with volume and no prices is not a session, and accepting it would write null prices into storage and into everything computed from them.Four regression tests reproduce the reported frame exactly (MultiIndex columns, volume present, NaN OHLC, the 2026-09-02/03 AAPL rows from the report). Before the fix three of them fail with the exact error above.
Closes ml4t/agent-workspace#1027.
The continuous-contract builder had no end-to-end roll-gap assertion
Every adjustment assertion on
ContinuousContractBuilder.buildmocked the adjustment method, so nothing checked that the series a reader gets back has the roll gap removed. The paired-RollEventrewrite that fixed the adjustment classes was unguarded at the level the reader uses.Four tests on a two-contract panel in 10-point contango: the unadjusted series moves 11.0 across the roll (10.0 of spread, 1.0 genuine), back-adjustment moves 1.0, ratio adjustment returns 1/103 - the outgoing contract's own return - and both anchor the last bar on its traded price. Checked against a deliberately gap-leaving adjuster: both assertions reject it.
Refs ml4t/agent-workspace#12. The defects that issue reports are already fixed and released; verified against the published 0.1.3 wheel, which reproduces the issue's own expected output (
[600, 600, 600, 600, 600], no roll-day return) where the version it was filed against printed[5.5556, 5.5556, 66.6667, 66.6667, 600.0].Verification
uv run pytest tests/ -q: 3,619 passed, 280 deselected.ruff check,ruff format --checkandty checkclean.🤖 Generated with Claude Code
https://claude.ai/code/session_01WfX5GbAHc5EVwdUnRSoUPT