Skip to content

fix: Export silently yields zero GDR/DTR rows when result fits in initial QSqlQueryModel batch (closes #177)#179

Merged
noonchen merged 2 commits into
noonchen:mainfrom
seam0814:fix-empty-gdr-dtr-export-177
Jun 25, 2026
Merged

fix: Export silently yields zero GDR/DTR rows when result fits in initial QSqlQueryModel batch (closes #177)#179
noonchen merged 2 commits into
noonchen:mainfrom
seam0814:fix-empty-gdr-dtr-export-177

Conversation

@seam0814

@seam0814 seam0814 commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Summary

getDatalogForReport() is a generator the Excel exporter uses to
stream GDR/DTR rows out of the tmodel_datalog Qt model. The
previous shape silently yielded zero rows whenever the entire
query result fit in QSqlQueryModel's initial prefetch batch
(default 256 rows). DTR/GDR records in a typical STDF file are well
under 256, so the GDR & DTR Summary sheet always came out blank even
though the data was in the DB — exactly the symptom in #177. The
"Convert STDF to XLSX" path goes through DatabaseFetcher.getDTR_GDRs
which executes the SQL directly, and so was unaffected.

Closes #177.

Root cause

# previous
row = 0
while model.canFetchMore():        # <-- already False if result ≤ 256 rows
    model.fetchMore()
    while row < model.rowCount():
        ...
        yield datalogRow

QSqlQueryModel pre-populates up to 256 rows on setQuery().
canFetchMore() returns True only if there are more rows beyond
that initial batch. So for small result sets the outer loop never
runs and the generator yields nothing.

Change

# new
row = 0
while True:
    while row < model.rowCount():
        ...
        yield datalogRow
    if not model.canFetchMore():
        break
    model.fetchMore()

Yield current rows first, then fetch more, repeat until exhausted.
Streaming behaviour is preserved for large result sets; the small-
result-set case now drains the rows that are already in the model.

Verification

Reproduced and verified deterministically using a FakeModel that
mimics QSqlQueryModel's 256-row prefetch:

total=   0 | old yielded=   0 [ok ] | new yielded=   0 [ok]
total=   1 | old yielded=   0 [BUG] | new yielded=   1 [ok]
total=   5 | old yielded=   0 [BUG] | new yielded=   5 [ok]
total= 100 | old yielded=   0 [BUG] | new yielded= 100 [ok]
total= 256 | old yielded=   0 [BUG] | new yielded= 256 [ok]
total= 257 | old yielded= 257 [ok ] | new yielded= 257 [ok]
total= 500 | old yielded= 500 [ok ] | new yielded= 500 [ok]
total=1000 | old yielded=1000 [ok ] | new yielded=1000 [ok]

(The repo has no automated test harness, so this was static
reproduction; if you have a sample STDF with DTR records that
previously reproduced #177, the same file should now produce a
populated GDR & DTR Summary sheet on Export.)

Scope

getDatalogForReport is only called from
stdfExporter.getTableDataFromParent in
deps/uic_stdExporter.py, so the change is scoped to the Export
path. The in-GUI GDR & DTR table iterates the model directly via
tmodel_datalog and was not affected by the bug.

…tial batch (closes noonchen#177)

`getDatalogForReport()` is a generator that the Excel exporter uses to
stream GDR/DTR rows out of the `tmodel_datalog` Qt model. The previous
shape was

    while model.canFetchMore():
        model.fetchMore()
        while row < model.rowCount():
            ...
            yield datalogRow

`QSqlQueryModel` lazily fetches rows in batches (default 256). After
`setQuery()` returns, the model is already populated with up to 256
rows, and `canFetchMore()` only returns `True` if there are *more*
rows beyond that batch. So whenever the entire query result fits in
the initial batch — which is the common case for GDR/DTR records,
typically a few per device times a few devices — the outer loop
never enters and the generator yields zero rows. The exporter then
writes only the three column headers and the GDR & DTR Summary sheet
looks blank, even though the option is checked and the data is in
the database.

The converter path (`DatabaseFetcher.getDTR_GDRs`) executes
`DATALOG_QUERY` directly against the SQLite cursor and is unaffected,
which is why "Convert STDF to XLSX" from the Utility menu produces
the expected output for the same file.

The fix reorders the loop to "yield current rows first, then fetch
more, repeat until exhausted":

    while True:
        while row < model.rowCount():
            ...
            yield datalogRow
        if not model.canFetchMore():
            break
        model.fetchMore()

Streaming behaviour is preserved for large result sets, and the small-
result-set case now yields the rows that are already in the model
before checking for more.

Reproduction & verification (with a `FakeModel` mimicking
`QSqlQueryModel`'s 256-row prefetch):

    total=   0 | old yielded=   0 [ok ] | new yielded=   0 [ok]
    total=   1 | old yielded=   0 [BUG] | new yielded=   1 [ok]
    total=   5 | old yielded=   0 [BUG] | new yielded=   5 [ok]
    total= 100 | old yielded=   0 [BUG] | new yielded= 100 [ok]
    total= 256 | old yielded=   0 [BUG] | new yielded= 256 [ok]
    total= 257 | old yielded= 257 [ok ] | new yielded= 257 [ok]
    total= 500 | old yielded= 500 [ok ] | new yielded= 500 [ok]
    total=1000 | old yielded=1000 [ok ] | new yielded=1000 [ok]

`getDatalogForReport` is only called from
`stdfExporter.getTableDataFromParent` in `deps/uic_stdExporter.py`, so
the scope is limited to the Export path; the in-GUI GDR & DTR table
already iterates the model directly and is unaffected.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@noonchen noonchen left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes seem good, only need to update the comment :)

Comment thread STDF-Viewer.py Outdated
Comment thread STDF-Viewer.py
@noonchen-ni

Copy link
Copy Markdown
Collaborator

@seam0814 could you please update the comments as suggested? thanks!

Per PR noonchen#179 feedback: keep only the first sentence inline since the
256-row prefetch root cause is already described in the PR body, and
restore the `# method 1` commented-out block to minimize diff.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@seam0814

Copy link
Copy Markdown
Contributor Author

Updated and pushed (db25269). Inline comment trimmed and the # method 1 block restored as requested — thanks for the review!

@noonchen

Copy link
Copy Markdown
Owner

@seam0814 thanks for your contribution! merged.

The windows build error might be related to msvc version or other reasons, ignored for now.

@noonchen
noonchen merged commit 5345b2c into noonchen:main Jun 25, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GDR & DTR Summary is empty when using the Export function

3 participants