Skip to content

perf(ifs): read the IFS in 64 KB blocks instead of 10 KB - #109

Open
LittleGnome wants to merge 1 commit into
tryggingamidstodin:masterfrom
LittleGnome:perf/ifs-read-buffer
Open

perf(ifs): read the IFS in 64 KB blocks instead of 10 KB#109
LittleGnome wants to merge 1 commit into
tryggingamidstodin:masterfrom
LittleGnome:perf/ifs-read-buffer

Conversation

@LittleGnome

Copy link
Copy Markdown

Read the IFS in 64 KB blocks instead of 10 KB

Branch: perf/ifs-read-buffer

Rationale

Each IfsReadStream.read() call is one round-trip to the IFS host server, so
the buffer size decides how many round-trips reading a file costs:

Buffer Round-trips per MB
10000 bytes (current) ~105
65536 bytes ~16

On a link with any latency that difference dominates the transfer time.

The buffer is a single allocation per stream and does not grow with the file,
so the extra memory is bounded by the number of concurrently open IFS read
streams, not by how much data flows through them.

Configurability

Overridable with the JT400_IFS_READ_BUFFER environment variable or the
jt400.ifs.readBuffer system property, so a deployment that is memory
constrained or runs many concurrent streams can lower it. An unparsable or
non-positive value falls back to the default rather than throwing during class
initialisation.

Verified directly against the built jar:

default                             -> 65536
-Djt400.ifs.readBuffer=8192         -> 8192
JT400_IFS_READ_BUFFER=4096          -> 4096
JT400_IFS_READ_BUFFER=xyz           -> 65536

As with the fetch-size PR: an environment variable is the least invasive
option, but a config key threaded through the connection would fit this
library's style better, and I am happy to switch.

Scope

No API change. createIfsReadStream keeps its signature and the node-side
stream is unaffected beyond receiving larger chunks. One file, one constant
plus its override plumbing.

Verification

npm run build && npm test       -> 32 passing
npm run test-cjs                -> 32 passing

The unit suite passes against the rebuilt jar, but it runs on HSQLDB and never
touches the IFS. This needs ifs-spec against a real IBM i, which I cannot
run.

Note on the jar

ant was not available in my environment, so java/lib/jt400wrap.jar was
rebuilt with javac and jar directly, matching what build.xml does
(-source 8 -target 8, same classpath, same class set and bytecode version).

Re-running npm run build-java regenerates it with your own toolchain and is
the preferred way to take this change.

Each IfsReadStream.read call is one round-trip to the IFS host server, so
the buffer size decides how many round-trips reading a file costs. At the
current 10000 bytes a one-megabyte file takes roughly a hundred of them;
at 64 KB it takes about sixteen. On a link with any latency that
difference dominates the transfer.

The buffer is a single allocation per stream and does not grow with the
file, so the extra memory is bounded by the number of concurrently open
IFS read streams rather than by how much data passes through them.

The value is overridable with the JT400_IFS_READ_BUFFER environment
variable or the jt400.ifs.readBuffer system property, so a deployment
that is memory-constrained or has many concurrent streams can lower it.
An unparsable or non-positive value falls back to the default rather than
throwing during class initialisation. Verified the default, both override
mechanisms and the fallback.

No API change: createIfsReadStream keeps its signature, and the node-side
stream is unaffected beyond receiving larger chunks.

As with the fetch size, an environment variable is the least invasive way
to make this adjustable; a config key threaded through the connection
would fit the library's style better and is easy to switch to.

The unit suite passes against the rebuilt jar, but it runs on HSQLDB and
never touches the IFS, so this needs the integration suite (ifs-spec)
against a real IBM i.

Note on the jar: ant was not available here, so java/lib/jt400wrap.jar
was rebuilt with javac and jar directly, matching what build.xml does
(-source 8 -target 8, same classpath). Re-running npm run build-java
regenerates it with your own toolchain and is the preferred way to take
this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant