perf(ifs): read the IFS in 64 KB blocks instead of 10 KB - #109
Open
LittleGnome wants to merge 1 commit into
Open
perf(ifs): read the IFS in 64 KB blocks instead of 10 KB#109LittleGnome wants to merge 1 commit into
LittleGnome wants to merge 1 commit into
Conversation
Each IfsReadStream.read call is one round-trip to the IFS host server, so the buffer size decides how many round-trips reading a file costs. At the current 10000 bytes a one-megabyte file takes roughly a hundred of them; at 64 KB it takes about sixteen. On a link with any latency that difference dominates the transfer. The buffer is a single allocation per stream and does not grow with the file, so the extra memory is bounded by the number of concurrently open IFS read streams rather than by how much data passes through them. The value is overridable with the JT400_IFS_READ_BUFFER environment variable or the jt400.ifs.readBuffer system property, so a deployment that is memory-constrained or has many concurrent streams can lower it. An unparsable or non-positive value falls back to the default rather than throwing during class initialisation. Verified the default, both override mechanisms and the fallback. No API change: createIfsReadStream keeps its signature, and the node-side stream is unaffected beyond receiving larger chunks. As with the fetch size, an environment variable is the least invasive way to make this adjustable; a config key threaded through the connection would fit the library's style better and is easy to switch to. The unit suite passes against the rebuilt jar, but it runs on HSQLDB and never touches the IFS, so this needs the integration suite (ifs-spec) against a real IBM i. Note on the jar: ant was not available here, so java/lib/jt400wrap.jar was rebuilt with javac and jar directly, matching what build.xml does (-source 8 -target 8, same classpath). Re-running npm run build-java regenerates it with your own toolchain and is the preferred way to take this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read the IFS in 64 KB blocks instead of 10 KB
Branch:
perf/ifs-read-bufferRationale
Each
IfsReadStream.read()call is one round-trip to the IFS host server, sothe buffer size decides how many round-trips reading a file costs:
On a link with any latency that difference dominates the transfer time.
The buffer is a single allocation per stream and does not grow with the file,
so the extra memory is bounded by the number of concurrently open IFS read
streams, not by how much data flows through them.
Configurability
Overridable with the
JT400_IFS_READ_BUFFERenvironment variable or thejt400.ifs.readBuffersystem property, so a deployment that is memoryconstrained or runs many concurrent streams can lower it. An unparsable or
non-positive value falls back to the default rather than throwing during class
initialisation.
Verified directly against the built jar:
As with the fetch-size PR: an environment variable is the least invasive
option, but a config key threaded through the connection would fit this
library's style better, and I am happy to switch.
Scope
No API change.
createIfsReadStreamkeeps its signature and the node-sidestream is unaffected beyond receiving larger chunks. One file, one constant
plus its override plumbing.
Verification
The unit suite passes against the rebuilt jar, but it runs on HSQLDB and never
touches the IFS. This needs
ifs-specagainst a real IBM i, which I cannotrun.
Note on the jar
antwas not available in my environment, sojava/lib/jt400wrap.jarwasrebuilt with
javacandjardirectly, matching whatbuild.xmldoes(
-source 8 -target 8, same classpath, same class set and bytecode version).Re-running
npm run build-javaregenerates it with your own toolchain and isthe preferred way to take this change.