-
Notifications
You must be signed in to change notification settings - Fork 778
fix(dataset): keep CSV columns aligned for items with different keys #2079
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -203,16 +203,12 @@ async def export_csv_to_stream( | |
| if 'lineterminator' not in kwargs: | ||
| kwargs['lineterminator'] = '\n' | ||
|
|
||
| writer = csv.writer(dst, **kwargs) | ||
| write_header = True | ||
|
|
||
| # Iterate over the dataset and write to CSV. | ||
| async for item in iterator: | ||
| if not item: | ||
| continue | ||
|
|
||
| if write_header: | ||
| writer.writerow(item.keys()) | ||
| write_header = False | ||
|
|
||
| writer.writerow(item.values()) | ||
| items = [item async for item in iterator if item] | ||
| if not items: | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The |
||
| return | ||
|
Comment on lines
+206
to
+208
|
||
|
|
||
| fieldnames = list(dict.fromkeys(key for item in items for key in item)) | ||
| writer = csv.DictWriter(dst, fieldnames=fieldnames, **kwargs) | ||
| writer.writeheader() | ||
|
Comment on lines
+210
to
+212
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This conflicts with my review. I believe we shouldn't just skip the additional fields that aren't present in the first element. @vdusek, @Pijukatel. What do you think?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, probably not. This should also be checked against the JS implementation. |
||
| for item in items: | ||
| writer.writerow(item) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This buffers the whole dataset in memory instead of streaming (!). I don't think we want to do that.