Skip to content

Commit f6997b1

Browse files
donislawdevclaude
andcommitted
perf: an archive names its directories once and encrypts into one buffer
Two findings from the performance report, both in internal/format/archive and neither moving a byte. P9: Layout.Path rebuilt the directory chain for every entry, through a fmt format verb, though the chain depends only on the depth. Prefix() builds it and the two hot callers hoist it out of their loops. Measured at depth 50 over 10 000 entries: 82.2 ms before, 30.6 ms once the verb went, and close to nothing with the prefix built once. The default depth is zero, where the prefix is empty and none of this was ever paid - so this is the ceiling of a setting rather than a run anybody has. P10: the two locked-entry writers allocated a fresh buffer on every Write. At 128 MB in 32 kB blocks that is about four thousand allocations. The buffer now lives as long as the entry. It cannot be done in place: p belongs to the caller and the zip writer passes the same slice on, so scrambling it would corrupt what somebody else is about to read. That is written next to both writers. The report never measured P10 - entryWriter is unexported - and the clock does not settle it either, since the processor time ranges overlap. The collector count does, and it is deterministic: a 128 MB locked zip went from 48 collections to 6 with aes-256 and 7 with zipcrypto. No new guard, deliberately. Path length against LongestPath and the locked archive's exact size were both already guarded, and adding a fourth defence beside three is the shape this project has thrown away seven times. Three mutations prove those guards catch the broken version. One mutation pattern went stale in the same step, quoting the call this change rewrote, and staleness.py caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent b2eca73 commit f6997b1

6 files changed

Lines changed: 81 additions & 21 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,17 @@ because it turns other people's test suites red.
7777
in front of the picture, so it has to know how large the picture is before it
7878
starts.
7979

80+
- **A password protected archive allocates once per entry instead of once per
81+
block written.** Producing a 128 MB locked `.zip` used to make the collector
82+
run 48 times. It runs 6. The files are identical and the wall clock barely
83+
moves, so this is headroom rather than a speed-up you will notice.
84+
85+
- **Nesting files deep inside an archive costs almost nothing to name.** The
86+
directory chain in front of every entry was rebuilt for each one, though it
87+
depends only on the depth. At the deepest setting with 10 000 entries that was
88+
82 ms of naming, and it is now close to nothing. Archives left flat, which is
89+
the default, never paid it either way.
90+
8091
- **`verify` and `cleanup` read the files over several threads, so checking a
8192
large run is several times faster.** Nothing about what they report changes -
8293
the same differences, in the same order, with the same exit codes.

‎internal/format/archive/layout.go‎

Lines changed: 43 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
package archive
22

33
import (
4-
"fmt"
54
"strconv"
65
"strings"
76

@@ -59,18 +58,22 @@ const (
5958
// format rather than trusting this paragraph.
6059
maxDepth = 50
6160

62-
// dirSegment numbers the levels so a path reads as what it is. Two digits
63-
// because maxDepth is two digits, and a fixed width so every segment is
64-
// the same size and the arithmetic above stays a multiplication.
65-
dirSegment = "d%02d/"
66-
67-
// dirSegmentBytes is what one segment comes to once rendered - "d00/" is
68-
// four bytes where the format string above is six. Written out rather than
69-
// taken as len(dirSegment), which is the bug the depth guard caught the
70-
// first time it ran: the arithmetic said every path was 2 bytes per level
71-
// longer than it is, which would have understated the ceiling rather than
72-
// overstating it, so nothing would have failed until somebody widened the
73-
// segment. A guard compares this against a really rendered path.
61+
// A level is written as "d00/": the letter, the number padded to two
62+
// digits because maxDepth is two digits, and the separator. A fixed width
63+
// is what keeps the arithmetic above a multiplication rather than a walk.
64+
//
65+
// It used to be a "d%02d/" format string rendered through fmt, and that
66+
// cost more than everything else about naming an entry put together -
67+
// measured 2026-09-06 at depth 50 over 10 000 entries, 82.2 ms against
68+
// 30.6 ms once the verb went. Prefix writes the four bytes out by hand.
69+
//
70+
// dirSegmentBytes is what one level comes to. It was written out rather
71+
// than taken as the length of that format string, which is the bug the
72+
// depth guard caught the first time it ran: six bytes rather than four
73+
// said every path was 2 bytes per level longer than it is, which
74+
// understates the ceiling instead of overstating it, so nothing would have
75+
// failed until somebody widened the segment. A guard still compares this
76+
// against a really rendered path.
7477
dirSegmentBytes = 4
7578
)
7679

@@ -87,15 +90,38 @@ type Layout struct {
8790
// The empty name gives the directory chain itself with its trailing slash,
8891
// which is what both containers want a directory entry to be called.
8992
func (l Layout) Path(name string) string {
93+
return l.Prefix() + name
94+
}
95+
96+
// Prefix is the directory chain on its own, with nothing on the end.
97+
//
98+
// It depends on Depth and on nothing else, so a caller naming thousands of
99+
// entries works it out once rather than once per entry. Measured 2026-09-06 at
100+
// depth 50 over 10 000 entries: a median of 82.2 ms rebuilding it every time
101+
// against 1.08 ms building it once, ranges disjoint.
102+
//
103+
// The default depth is zero and a flat archive spends nothing here either way,
104+
// so this is a ceiling rather than a typical run - which is worth saying,
105+
// because the number above reads like a saving every user gets.
106+
//
107+
// Rendered by hand rather than through fmt: the format verb is what made the
108+
// old version expensive, and a two digit number with a floor of two is small
109+
// enough to write out. "d%02d/" pads to two and lets a third digit through,
110+
// which is what the branch below does.
111+
func (l Layout) Prefix() string {
90112
if l.Depth <= 0 {
91-
return name
113+
return ""
92114
}
93115
var b strings.Builder
94-
b.Grow(l.Depth*len(dirSegment) + len(name))
116+
b.Grow(l.Depth * dirSegmentBytes)
95117
for i := 0; i < l.Depth; i++ {
96-
fmt.Fprintf(&b, dirSegment, i)
118+
b.WriteByte('d')
119+
if i < 10 {
120+
b.WriteByte('0')
121+
}
122+
b.WriteString(strconv.Itoa(i))
123+
b.WriteByte('/')
97124
}
98-
b.WriteString(name)
99125
return b.String()
100126
}
101127

‎internal/format/archive/lock.go‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,8 @@ type entryWriter struct {
274274
out io.Writer
275275
ctr *counter
276276
mac hash
277+
// buf is reused across writes. See Write.
278+
buf []byte
277279
}
278280

279281
// hash is the part of hash.Hash this uses. Named so the field above reads as
@@ -284,7 +286,14 @@ type hash interface {
284286
}
285287

286288
func (e *entryWriter) Write(p []byte) (int, error) {
287-
out := make([]byte, len(p))
289+
// A scratch buffer that lives as long as the entry rather than one per
290+
// call. It cannot be done in place: p belongs to the caller, and the zip
291+
// writer hands the same slice on elsewhere, so scrambling it here would
292+
// corrupt what somebody else is about to read.
293+
if cap(e.buf) < len(p) {
294+
e.buf = make([]byte, len(p))
295+
}
296+
out := e.buf[:len(p)]
288297
e.ctr.xor(out, p)
289298
if _, err := e.mac.Write(out); err != nil {
290299
return 0, err

‎internal/format/archive/zipcrypto.go‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,8 @@ func (c *pkware) encrypt(p byte) byte {
8282
type zipCryptoWriter struct {
8383
out io.Writer
8484
c *pkware
85+
// buf is reused across writes. See Write.
86+
buf []byte
8587
}
8688

8789
// newZipCryptoWriter starts an entry, writing the twelve byte header before
@@ -111,7 +113,12 @@ func (l Lock) newZipCryptoWriter(w io.Writer, seed uint64, index int, crc uint32
111113
}
112114

113115
func (z *zipCryptoWriter) Write(p []byte) (int, error) {
114-
out := make([]byte, len(p))
116+
// Reused across writes, and not done in place for the reason written out
117+
// on entryWriter.Write: p belongs to the caller.
118+
if cap(z.buf) < len(p) {
119+
z.buf = make([]byte, len(p))
120+
}
121+
out := z.buf[:len(p)]
115122
for i := range p {
116123
out[i] = z.c.encrypt(p[i])
117124
}

‎internal/format/targz/targz.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,9 @@ func (generator) Plan(r format.Request) (format.Plan, error) {
252252
// seed of a member does not move when a group above it changes count. That is
253253
// untouchable rule 2 applied one level down.
254254
func planChildren(r format.Request, groups []format.Content, layout archive.Layout) ([]child, error) {
255+
// The directory chain depends only on the depth, so it is built once here
256+
// rather than once per entry.
257+
prefix := layout.Prefix()
255258
var out []child
256259
index := 0
257260
// Numbering runs per format rather than per group, so two groups of the
@@ -273,7 +276,7 @@ func planChildren(r format.Request, groups []format.Content, layout archive.Layo
273276
}
274277
numbered[g.Format]++
275278
out = append(out, child{
276-
name: layout.Path(fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension)),
279+
name: prefix + fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension),
277280
desc: desc,
278281
plan: cp,
279282
})

‎internal/format/zip/children.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ import (
2424
// seed of a member does not move when a group above it changes count. That is
2525
// untouchable rule 2 applied one level down.
2626
func planChildren(r format.Request, groups []format.Content, layout archive.Layout) ([]child, error) {
27+
// The directory chain depends only on the depth, so it is built once here
28+
// rather than once per entry.
29+
prefix := layout.Prefix()
30+
2731
// Sized up front, because the total is known before the walk starts: it is
2832
// what the groups add up to. Growing by append instead reallocates and
2933
// copies the whole slice fourteen times on the way to ten thousand entries,
@@ -54,7 +58,7 @@ func planChildren(r format.Request, groups []format.Content, layout archive.Layo
5458
}
5559
numbered[g.Format]++
5660
out = append(out, child{
57-
name: layout.Path(fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension)),
61+
name: prefix + fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension),
5862
desc: desc,
5963
plan: cp,
6064
})

0 commit comments

Comments
 (0)