Repository navigation
Conversation
df_to_basemodel called pandas.to_datetime once per row through Series.apply, then walked the column again in Python for the all-midnight check and the YYYY-MM-DD formatting. For a 5,000-row daily frame that was about 250 ms, nearly all of the conversion. Parse the column in one call and use vectorized comparisons and strftime, the same change openbq-org#7622 made to basemodel_to_df. A column that mixes UTC offsets (a series crossing a daylight-saving boundary) still falls back to the per-element parse, so its values are unchanged.
isihussain
reviewed
Nov 16, 2026
isihussain
left a comment
Contributor
There was a problem hiding this comment.
Code Review: Multi-Axis Evaluation
Great performance optimization @b84330808! Vectorizing the date parsing and midnight formatting in df_to_basemodel replaces an
1. Performance & Vectorization Mechanics
- Vectorized Parsing: Replacing
df["date"].apply(to_datetime)withto_datetime(df["date"])avoids per-element Python interpreter overhead, yielding order-of-magnitude improvements for high-frequency or multi-year historical series. - Vectorized Midnight Normalization: The
(dates == dates.dt.normalize()).all()check and subsequentdates.dt.strftime("%Y-%m-%d")is the canonical idiomatic pattern in pandas, avoiding iterating through datetime objects in pure Python when the Series has a nativedatetime64dtype.
2. Resilience & Edge Cases
- Mixed-Offset Fallback: Falling back to
df["date"].apply(to_datetime)specifically when a column contains mixed UTC offsets (such as across DST transition boundaries where pandas cannot infer a single unified timezone tz-aware dtype) ensures zero regressions for tricky historical datasets. - Non-Date Column Safety: Retaining the
except (TypeError, ValueError)guard preserves non-date object columns without crashing.
3. Unit Tests
- The added test cases in
test_utils.pycover:- A mixed-offset Series crossing daylight saving time boundaries (
-04:00vs-05:00). - Monkeypatched verification confirming the element-wise fallback path executes correctly when vectorized parsing fails.
- A mixed-offset Series crossing daylight saving time boundaries (
Summary: Clean, high-impact performance boost with defensive fallback handling. LGTM!
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request the OpenBB Platform
Description
df_to_basemodelconverts every DataFrame passed asdata=to a data-processing command (technical,quantitative,econometrics, viafilter_inputs->convert_to_basemodel), and the frames given tocharting.to_chart(data=...). Its date handling still callspandas.to_datetimeonce per row throughSeries.apply, then walks the column again in Python for the all-midnight check and theYYYY-MM-DDformatting:For a 1,000-row daily frame,
cProfileputs 0.175 s of the call's 0.182 s in those twoapplypasses.This parses the column in one call and uses vectorized comparisons and
strftime, the same change #7622 made tobasemodel_to_df. A column that mixes UTC offsets (a series crossing a daylight-saving boundary) has no single datetime64 dtype, so it still falls back to the per-element parse and the per-value check, and its output is unchanged.How has this been tested?
Input: the frame
obb.equity.price.historical(...).to_df()returns for daily bars (adateindex ofdatetime.date). Base and patched runs alternated, 5 rounds each, Python 3.11, pandas 3.0.6 (the version incore/uv.lock), Windows 11, i7-9700. Each figure is the median of the 5 per-round medians, with the range of those 5 in parentheses.df_to_basemodel(df)alone (median of 9 calls per round):End to end,
obb.technical.rsi(data=df, target="close")(median of 7 calls per round):DatetimeIndex,datetime64columns, month and US-style strings, mixed date/datetime strings, non-date strings, numeric dates, MultiIndex, Series) under pandas 3.0.6 and 2.2.3. The output was identical for all of them except a column with a missing date: the old code raisedValueError: NaTType does not support time, and the new one returns those rows withdate: None.YYYY-MM-DD, and the per-element fallback is used when the vectorized parse raises.openbb_platform/core/tests/app: 1276 passed, 54 skipped (two files that needhttpx2forstarlette.testclientwere not collected locally).extensions/technical/testspass.ruff format --check,ruff checkandty checkpass for the changed files.Checklist
feature/feature-nameorhotfix/hotfix-name.