Repository navigation
Commit 0a2c3d2
perf(deletion-vector): avoid payload copies and bitmap reserialization (#4181)
* perf(deletion-vector): use zero-copy memoryview for deserialization
Replace repeated byte string slicing with pointer offset tracking
via Python's built-in memoryview. This avoids O(N^2) memory buffer
allocations during parsing, improving throughput by up to 39x on
large payload sizes.
* Avoid reserializing deletion vectors while parsing
Use PyRoaring's portable serialized size and require memoryview-compatible versions. Cover mixed bitmap encodings and key gaps without reserialization.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify PyRoaring serialized-size comment
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>1 parent 581e716 commit 0a2c3d2
4 files changed
Lines changed: 55 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| |||
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
54 | | - | |
55 | | - | |
56 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
57 | 59 | | |
58 | 60 | | |
59 | 61 | | |
| 62 | + | |
60 | 63 | | |
61 | | - | |
| 64 | + | |
62 | 65 | | |
63 | 66 | | |
64 | 67 | | |
65 | 68 | | |
66 | 69 | | |
67 | 70 | | |
68 | | - | |
| 71 | + | |
69 | 72 | | |
70 | 73 | | |
71 | 74 | | |
72 | 75 | | |
73 | 76 | | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
78 | 81 | | |
79 | 82 | | |
80 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
167 | 204 | | |
168 | 205 | | |
169 | 206 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments