Skip to content

perf(core): parse the date column once in df_to_basemodel - #7840

Open
b84330808 wants to merge 2 commits into
openbq-org:developfrom
b84330808:feature/df-to-basemodel-vectorize-dates
Open

b84330808 wants to merge 2 commits into
openbq-org:developfrom
b84330808:feature/df-to-basemodel-vectorize-dates

Conversation

@b84330808

Copy link
Copy Markdown
Contributor

Pull Request the OpenBB Platform

Description

  • Summary of the change/ bug fix.

df_to_basemodel converts every DataFrame passed as data= to a data-processing command (technical, quantitative, econometrics, via filter_inputs -> convert_to_basemodel), and the frames given to charting.to_chart(data=...). Its date handling still calls pandas.to_datetime once per row through Series.apply, then walks the column again in Python for the all-midnight check and the YYYY-MM-DD formatting:

dates = df["date"].apply(to_datetime)
...
if all(t.time() == time(0, 0) for t in df["date"]):
    df["date"] = df["date"].apply(lambda x: x.date().strftime("%Y-%m-%d"))

For a 1,000-row daily frame, cProfile puts 0.175 s of the call's 0.182 s in those two apply passes.

This parses the column in one call and uses vectorized comparisons and strftime, the same change #7622 made to basemodel_to_df. A column that mixes UTC offsets (a series crossing a daylight-saving boundary) has no single datetime64 dtype, so it still falls back to the per-element parse and the per-value check, and its output is unchanged.

How has this been tested?

Input: the frame obb.equity.price.historical(...).to_df() returns for daily bars (a date index of datetime.date). Base and patched runs alternated, 5 rounds each, Python 3.11, pandas 3.0.6 (the version in core/uv.lock), Windows 11, i7-9700. Each figure is the median of the 5 per-round medians, with the range of those 5 in parentheses.

df_to_basemodel(df) alone (median of 9 calls per round):

rows develop this PR
250 13.6 ms (13.3-13.7) 2.0 ms (2.0-3.4)
1,000 50.6 ms (49.3-51.0) 5.0 ms (4.8-6.4)
5,000 254.2 ms (249.2-258.9) 22.7 ms (22.3-23.4)

End to end, obb.technical.rsi(data=df, target="close") (median of 7 calls per round):

rows develop this PR
1,000 105.1 ms (99.6-111.8) 53.9 ms (53.5-60.1)
5,000 528.3 ms (510.9-541.0) 274.2 ms (267.2-288.5)
  • I compared the old and new function on 22 inputs (date objects, ISO strings, intraday naive and tz-aware strings, mixed offsets across DST at midnight and intraday, tz-aware and naive DatetimeIndex, datetime64 columns, month and US-style strings, mixed date/datetime strings, non-date strings, numeric dates, MultiIndex, Series) under pandas 3.0.6 and 2.2.3. The output was identical for all of them except a column with a missing date: the old code raised ValueError: NaTType does not support time, and the new one returns those rows with date: None.
  • Added two tests: a mixed-offset all-midnight column still becomes YYYY-MM-DD, and the per-element fallback is used when the vectorized parse raises.
  • openbb_platform/core/tests/app: 1276 passed, 54 skipped (two files that need httpx2 for starlette.testclient were not collected locally). extensions/technical/tests pass.
  • ruff format --check, ruff check and ty check pass for the changed files.

Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have adhered to the GitFlow naming convention and my branch name is in the format of feature/feature-name or hotfix/hotfix-name.
  • I ensure that I am following the CONTRIBUTING guidelines.

df_to_basemodel called pandas.to_datetime once per row through
Series.apply, then walked the column again in Python for the
all-midnight check and the YYYY-MM-DD formatting. For a 5,000-row daily
frame that was about 250 ms, nearly all of the conversion.

Parse the column in one call and use vectorized comparisons and
strftime, the same change openbq-org#7622 made to basemodel_to_df. A column that
mixes UTC offsets (a series crossing a daylight-saving boundary) still
falls back to the per-element parse, so its values are unchanged.

@isihussain isihussain left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review: Multi-Axis Evaluation

Great performance optimization @b84330808! Vectorizing the date parsing and midnight formatting in df_to_basemodel replaces an $O(N)$ row-wise Python loop with native pandas C/Cython vectorization, which provides substantial speedups on large financial datasets.


1. Performance & Vectorization Mechanics

  • Vectorized Parsing: Replacing df["date"].apply(to_datetime) with to_datetime(df["date"]) avoids per-element Python interpreter overhead, yielding order-of-magnitude improvements for high-frequency or multi-year historical series.
  • Vectorized Midnight Normalization: The (dates == dates.dt.normalize()).all() check and subsequent dates.dt.strftime("%Y-%m-%d") is the canonical idiomatic pattern in pandas, avoiding iterating through datetime objects in pure Python when the Series has a native datetime64 dtype.

2. Resilience & Edge Cases

  • Mixed-Offset Fallback: Falling back to df["date"].apply(to_datetime) specifically when a column contains mixed UTC offsets (such as across DST transition boundaries where pandas cannot infer a single unified timezone tz-aware dtype) ensures zero regressions for tricky historical datasets.
  • Non-Date Column Safety: Retaining the except (TypeError, ValueError) guard preserves non-date object columns without crashing.

3. Unit Tests

  • The added test cases in test_utils.py cover:
    1. A mixed-offset Series crossing daylight saving time boundaries (-04:00 vs -05:00).
    2. Monkeypatched verification confirming the element-wise fallback path executes correctly when vectorized parsing fails.

Summary: Clean, high-impact performance boost with defensive fallback handling. LGTM!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants