Skip to content
View waytofarout's full-sized avatar
🏠
Working from home
🏠
Working from home

Block or report waytofarout

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
waytofarout/README.md

Igor

Software engineer. I build systems that have to keep running after the people who wrote them have moved on, and I take apart systems nobody has the source for anymore.

Most of my working life has been in C# and Java, with enough C and assembly to be useful in a debugger. I've been writing C# since the 1.x days and Java since before generics, which mostly means I remember why some of the odd corners of both languages exist.

Husband and father. I skate. In that order.

Lately the work is an L1 chain and the tooling around it, so the day-to-day is Go, TypeScript and Solidity. I didn't expect that, but the problems are the same problems: state, time, and other people's assumptions.

What I do

Three things, in roughly this proportion.

Architecture. Deciding where the boundaries go, what crosses them, and what is allowed to change on each side without telling the other. Everything inside a boundary can be rewritten. The boundary itself is what you're stuck with, so that's where the care goes.

Backend work. Services, data, queues, the unglamorous middle of the system. Long-lived processes under real load, and the diagnostics to understand them when they misbehave at 3 a.m.

Reverse engineering. Binaries with no source, protocols with no spec, file formats from vendors that no longer exist. Firmware, drivers, old line-of-business code that still runs the business. I work on things we own, are licensed to maintain, or need to interoperate with. Not other people's licensing.

How I think about systems

  • Boundaries are the architecture. The rest is implementation.
  • The database outlives the application. I've run three generations of app code against one schema that was designed carefully the first time. Schema first.
  • A queue is a promise that you'll handle the message later. Most outages I've worked were a broken version of that promise: no idempotency, no dead letter, no backpressure.
  • Idempotency keys cost a column. Duplicate side effects cost a weekend.
  • Timeouts, then retries with jitter, then circuit breakers. In that order. A retry without a timeout is a load multiplier.
  • Most distributed bugs that aren't network bugs are clock bugs wearing a costume. Monotonic clock for durations, wall clock for timestamps, never mix.
  • Boring technology. Postgres, a queue, a cache, something to run the containers. Every exotic piece has to earn its spot with a specific failure it prevents.
  • Monoliths are fine. A modular monolith beats most of the microservice setups I've inherited. Split on a team boundary, not on a whim.
  • The on-call engineer is the real reader of your logs. Structured, one line per event, correlation id on every line.
  • Backward compatibility is a feature. Breaking it is a release of its own with its own plan.
  • Uncontended is faster than lock-free.

Working notes

Things I've had to learn more than once.

.NET

  • Allocation rate, not GC pauses. Gen0 is cheap. What hurts is promotion. Look at gen2 count and the LOH before touching any GC setting.
  • Server GC is not a free win on a box that runs six services.
  • async void is for event handlers. Anywhere else, that's your bug.
  • ConfigureAwait(false) in libraries. In application code only if you can say why.
  • Interop struct layout: LayoutKind.Sequential, explicit Pack, and compare Marshal.SizeOf with sizeof on the C side. Every time. The one time you skip it is the time the C header has a bitfield.
  • If you think you need a finalizer, you probably need a SafeHandle.
  • Span<T> and ArrayPool<T> retired most of the unsafe parsing code I used to carry around.
  • Remoting, then WCF, then Web API, then gRPC. The wire has changed four times. The service boundaries I drew in 2005 mostly haven't.
  • dotnet-counters first, dotnet-trace second, dotnet-dump with SOS when the first two run out. PerfView when it's a Windows box and I want the truth.

JVM

  • Read the GC log before you read the code. -Xlog:gc* on 11 and up, and leave it on in production. It costs nothing.
  • Most "memory leak" tickets are a static Map somewhere, or a ThreadLocal on a pooled thread. Heap dump, dominator tree, twenty minutes.
  • String.intern() is not an optimization. It's a bet.
  • A capturing lambda allocates on every call. Check captures in hot loops.
  • Deoptimization storms: a call site goes megamorphic after warmup and latency turns spiky after a deploy. -XX:+PrintCompilation shows it.
  • Class loading in an app server is a graph, not a tree. A ClassCastException between two classes with the same name is two loaders.
  • allocateDirect is off-heap and off-limits to your heap sizing math.
  • I miss checked exceptions when I'm in C#, and curse them when I'm in Java. Both reactions are correct.
  • async-profiler and JFR. Everything else is a nicer UI over the same data.

Reverse engineering

  • Strings, then imports, then cross references. That's most of the story before the first breakpoint.
  • Rename as you go. A function still called sub_401000 after two hours means you're wandering, not reading.
  • Managed code: ILSpy or dnSpy first. If it's obfuscated, de4dot. If it's really obfuscated, let the runtime JIT it and dump the method bodies after. The runtime has to see real IL eventually.
  • JVM: javap -c -p, then CFR or Procyon. Bytecode doesn't lie. Decompilers sometimes do.
  • Native: Ghidra or IDA for static, x64dbg or WinDbg for dynamic, gdb on the other side. Ghidra's decompiler is good enough now that I rarely miss the paid one.
  • Hooks: Detours or MinHook on Windows, LD_PRELOAD on Linux, Frida when it needs to be scriptable.
  • Protocols: capture first, hypothesize second. A Lua dissector for Wireshark pays for itself on the first day.
  • Firmware: binwalk, find the loader, find the string table, find the message loop. Everything else hangs off the message loop.
  • Keep a notes file open from minute one. Addresses, guesses, what turned out wrong. Future you needs the wrong guesses too.

Go, lately

  • Context cancellation is a contract. Every blocking call takes a ctx or it owes an explanation.
  • Channels are for ownership transfer. For shared state a mutex is clearer and usually faster.
  • errors.Is and errors.As at the boundary, wrapped errors everywhere else, and the message says what was being attempted, not what failed.
  • pprof is always on. There's no reason to turn it off.

Tools

Rider and Visual Studio for .NET. IntelliJ for Java. Vim for everything else, including this file.

WinDbg with SOS, x64dbg, gdb, Ghidra, Wireshark, Procmon, strace, perf. Make, MSBuild and Gradle, and firm opinions about Maven that I keep to myself.

As few dependencies as the problem allows. Each one is a maintenance contract with a stranger.

Reading

Books I reread or hand to people.

  • Hennessy and Patterson, Computer Architecture: A Quantitative Approach
  • Richter, CLR via C#
  • Goetz, Java Concurrency in Practice
  • Russinovich and Solomon, Windows Internals
  • Eilam, Reversing
  • Dang, Gazet and Bachaalany, Practical Reverse Engineering
  • Nygard, Release It!
  • Kleppmann, Designing Data-Intensive Applications
  • Ousterhout, A Philosophy of Software Design
  • Kernighan and Pike, The Practice of Programming

Papers: Lamport on time and clocks, the Dynamo paper, Raft. I read them again every few years. They change as you do.

Off the keyboard

Rollerblading. Mostly distance on road and river paths, some park when the knees agree to it. 100 mm wheels for distance, a rockered setup for the city. Been at that about as long as the code, with the scars to prove the early years.

Family. The compiler can wait.

Here

Most of what I've written sits behind someone else's NDA. What shows up on this account is side work, tooling, and notes.

Email: lightchainai@gmail.com

Popular repositories Loading

  1. waytofarout waytofarout Public