Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
128 lines (115 loc) · 6.91 KB
/
Copy pathindex.html
File metadata and controls
128 lines (115 loc) · 6.91 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
<!DOCTYPE html>
<html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Varka Notes</title>
<meta name="description" content="Engineering notes from Varka, a research fork of Apache Spark that compiles a projection into one vector loop inside the JVM.">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Literata:ital,opsz,wght@0,7..72,400;0,7..72,600;1,7..72,400&family=Source+Sans+3:wght@600;700&display=swap">
<style>
:root{--bg:#fbfaf7;--fg:#1f2328;--muted:#5d6672;--accent:#6741d9;--rule:#e3ddd2}
@media (prefers-color-scheme:dark){:root:not([data-theme="light"]){--bg:#15171b;--fg:#e6e3dc;--muted:#a1a7b0;--accent:#b197fc;--rule:#2b2f36}}
:root[data-theme="dark"]{--bg:#15171b;--fg:#e6e3dc;--muted:#a1a7b0;--accent:#b197fc;--rule:#2b2f36}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--fg);font-family:Literata,Georgia,serif;font-size:18px;line-height:1.6}
main{max-width:42rem;margin:0 auto;padding:4rem 16px 6rem}
h1{font-family:"Source Sans 3",system-ui,sans-serif;font-size:2rem;margin:0 0 .4rem;letter-spacing:-.01em}
.sub{color:var(--muted);margin:0 0 3rem}
a{color:var(--accent);text-decoration:none}
a:hover{text-decoration:underline}
.post{display:block;border-top:1px solid var(--rule);padding:1.4rem 0}
.post h2{font-family:"Source Sans 3",system-ui,sans-serif;font-size:1.25rem;margin:0 0 .3rem;color:var(--fg)}
.post p{margin:0;color:var(--muted);font-size:.95rem}
.date{font-size:.8rem;color:var(--muted);letter-spacing:.04em;text-transform:uppercase}
footer{border-top:1px solid var(--rule);margin-top:3rem;padding-top:1.2rem;color:var(--muted);font-size:.9rem}
</style></head>
<body><main>
<h1>Varka Notes</h1>
<p class="sub">Vectorising Apache Spark's expressions inside the JVM.</p>
<a class="post" href="your-sin-depends-on-your-jdk/">
<span class="date">9 November 2026</span>
<h2>Your <code>sin()</code> depends on your JDK and your instruction set</h2>
<p>Eight of Spark's math functions can return a different last bit on a different machine:
why <code>Math</code> and <code>StrictMath</code> part ways, what changes the answer (the
instruction set and the JDK, not the chip, the JIT or codegen), where it bites, and what to do.</p>
</a>
<a class="post" href="two-compilers-two-queues/">
<span class="date">18 October 2026</span>
<h2>Two compilers, two queues</h2>
<p>How HotSpot's five tiers, its two compilers and their two queues fit together, and why
the length of C2's queue at one instant decides whether a method is ever profiled. With the
policy's own trace of one method in a passing and a failing run, and what a warm-up has to
do about it.</p>
</a>
<a class="post" href="where-the-checking-stops/">
<span class="date">17 October 2026</span>
<h2>How a SQL engine is tested: ten methods, what each found, and what each missed</h2>
<p>For testers: the oracle is vanilla Spark, errors included. Ten methods from solver proofs
to mutation testing, each with an example, its cost and its yield, and the defects laid out by
the method that found them. One was found by none of the ten.</p>
</a>
<a class="post" href="how-spark-compiles-a-query/">
<span class="date">17 October 2026</span>
<h2>How Spark compiles a query, and why it compiles it again</h2>
<p>Why a dashboard query with a new date runs about twice as slow as a rerun with the same
one: Spark writes the value into the code it generates, compiles the stage again, and keeps
only about a hundred classes in its cache.</p>
</a>
<a class="post" href="nine-numbers/">
<span class="date">16 October 2026</span>
<h2>Nine numbers between your query and machine code</h2>
<p>The limits a Spark SQL stage passes on its way to machine code: where each one acts, its
default in Spark 4.2.0, and what happens past it, which is mostly a slower path with no error
and no log line.</p>
</a>
<a class="post" href="why-a-warmed-kernel-can-still-allocate/">
<span class="date">16 October 2026</span>
<h2>Why a warmed Vector API kernel can still allocate</h2>
<p>A kernel C2 had compiled at the top tier allocated an array on every loop iteration,
in a few warm-ups in every hundred: how C2 sums class-check traps over a whole compile,
who leaves the traps, the JVM options that remove it, and the warm-up that all but does.</p>
</a>
<a class="post" href="the-class-got-cheaper/">
<span class="date">16 October 2026</span>
<h2>The class got cheaper. The first query got slower.</h2>
<p>For people who write code generators: what changes when the generator writes bytecode
instead of Java source, why a class three to six times cheaper to make did not speed up
the first query, and where the saving that matters comes from: how classes are cached.</p>
</a>
<a class="post" href="split-only-where-it-has-to-be/">
<span class="date">12 October 2026</span>
<h2>A large CASE WHEN, split only where it has to be</h2>
<p>The sequel to the cliff: Spark 4.4.0 splits a large <code>CASE WHEN</code> inside a
stage only where its method would pass 8000 bytes; why splitting everywhere costs, how to
see the fix, the settings it adds, and what to do before you can upgrade.</p>
</a>
<a class="post" href="when-spark-stops-compiling-your-query/">
<span class="date">2 October 2026</span>
<h2>When Spark stops compiling your query</h2>
<p>The other places Spark's code generation gives up, and rarely says so: an operator that
leaves its stage, a wide projection slower inside one, a limit logged where the shell hides
it, a method interpreted instead of compiled; what each costs, and how to see it.</p>
</a>
<a class="post" href="under-8000-bytes-by-construction/">
<span class="date">1 October 2026</span>
<h2>Under 8000 bytes by construction</h2>
<p>Why a generator of Java source cannot keep a method under the JIT's limit, and how
an engine that emits bytecode measures every method of the class it built and splits
or declines before anything runs; with what that buys and costs, measured.</p>
</a>
<a class="post" href="the-8000-byte-cliff/">
<span class="date">27 September 2026</span>
<h2>The 8000-byte cliff in Spark SQL</h2>
<p>Why a query can get six times slower when a projection, a <code>CASE WHEN</code> or a
filter grows by a few terms, how to see it on your own cluster, what to set, and which
Spark release fixes which part.</p>
</a>
<a class="post" href="eight-rows-per-instruction/">
<span class="date">23 September 2026</span>
<h2>Eight rows per instruction: Spark's TIME type on Varka's 64-bit lane</h2>
<p>What a row-at-a-time loop leaves on the table, what one emitted vector loop
does instead, and what the two cost per row on a machine whose datapath was
measured rather than assumed.</p>
</a>
<footer>The engine, the benchmarks and the sources of these pages are at
<a href="https://github.com/vecbricks/varka">github.com/vecbricks/varka</a>.</footer>
</main></body></html>