Repository navigation
Commit 3f32c12
committed
perf(pyarrow): group delete positions by file_path in a single pass
_read_deletes built its result with a dict comprehension that filtered
the whole delete table once per distinct file_path, making the cost
O(rows x distinct paths). A delete file referencing many data files
therefore degraded quadratically. The ORC branch had the same shape
via unique().
Group by file_path instead, which is a single pass and lets both the
Parquet and ORC branches share one code path. Dictionary encoding of
file_path is retained, since delete files commonly repeat a small
number of long paths across many rows; dictionaries are unified first
because grouping cannot combine differing ones.
Closes #39831 parent 1312c55 commit 3f32c12
2 files changed
Lines changed: 30 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1134 | 1134 | | |
1135 | 1135 | | |
1136 | 1136 | | |
1137 | | - | |
1138 | | - | |
1139 | | - | |
1140 | | - | |
1141 | | - | |
1142 | 1137 | | |
1143 | 1138 | | |
1144 | 1139 | | |
1145 | 1140 | | |
1146 | | - | |
1147 | | - | |
1148 | | - | |
1149 | | - | |
1150 | | - | |
1151 | 1141 | | |
1152 | 1142 | | |
1153 | 1143 | | |
| |||
1156 | 1146 | | |
1157 | 1147 | | |
1158 | 1148 | | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
1159 | 1157 | | |
1160 | 1158 | | |
1161 | 1159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1822 | 1822 | | |
1823 | 1823 | | |
1824 | 1824 | | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
1825 | 1847 | | |
1826 | 1848 | | |
1827 | 1849 | | |
| |||
0 commit comments