Skip to content

Inject worker metadata to pilot + testable job args#996

Merged
brandur merged 1 commit into
masterfrom
brandur-pilot-init
Aug 3, 2025
Merged

Inject worker metadata to pilot + testable job args#996
brandur merged 1 commit into
masterfrom
brandur-pilot-init

Conversation

@brandur

@brandur brandur commented Aug 2, 2025

Copy link
Copy Markdown
Contributor

Here, add a mechanism that has the client inject worker metadata into
pilots. This gives pilots information about what jobs/workers are
registered, and helps in cases where say we want pilots to be able to
examine things like Hooks() implementations on each worker.

I also ended up adding a function for river.AddWorkerArgs that lets a
concrete args object be injected into the registration process rather
than one that's implicitly allocated. I'm a little partial on this one
because it'd be better if it wasn't necessary, but it cleans up tests
somewhat that are trying to test specific things on args like Hooks(),
and make tests on features like that possible when there's no
workaround available like storing information that'll be persisted to a
job's metadata (there's otherwise no good way to carry them out, with
the only way I can think of a global variable)

Comment thread client_test.go
}
})

AddWorkerArgs(bundle.config.Workers, jobArgs, WorkFunc(func(ctx context.Context, job *Job[JobArgsWithHooksFunc]) error {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See here and the red below for how AddWorkerArgs helps to remove boilerplate and bring implementation closer to the specific test case, but the more critical piece is that unless a hook has some "hack" workaround available like being able to add information to metadata (like this test case was doing), I couldn't find any other way to test hooks that are specific to only some job args.

The trouble is that AddWorker always implicitly constructs a JobArgs like:

func AddWorkerSafely[T JobArgs](workers *Workers, worker Worker[T]) error {
	var jobArgs T
	return workers.add(jobArgs, &workUnitFactoryWrapper[T]{worker: worker})
}

Which makes attaching a specific Hooks() implementation to it for a single test case impossible.

By bringing in something like AddWorkerArgs which is implemented like:

func AddWorkerArgs[T JobArgs](workers *Workers, jobArgs T, worker Worker[T]) {
	if err := workers.add(jobArgs, &workUnitFactoryWrapper[T]{worker: worker}); err != nil {
		panic(err)
	}
}

It lets us do what you see above, with a closure hook implementation that can set a local variable and be tested against.

The only other way I could think to do it is to have a job args specific to every test case and which sets a global variable, which sucks. So the only options I'm able to come up with are:

  1. AddWorkerArgs as you see here.
  2. Job args for each test case that set a global (or use a workaround like middleware if that's available).
  3. Don't test the hooks on specific job args case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should note also that I'm not married to this at all, but for the life of me could not find a better way.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a better idea unfortunately, annoying problem to have to hack around 😕

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K thx. And yeah :( We'll have a bit of time ahead of release probably. I'll keep mulling it over.

It'd be nice if Go allowed you to mark some API is "pretty internal, you probably don't want to use this man" or something of that nature even and at least hide it from Godoc or something. Couldn't think of any tricks here though unfortunately.

Comment thread worker.go Outdated
// AddWorkerArgs is the same as AddWorker except that it lets args be passed
// explicitly rather than being instantiated implicitly. This function is
// largely for internal use only and shoudl be used only in tests.
func AddWorkerArgs[T JobArgs](workers *Workers, jobArgs T, worker Worker[T]) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't put in a *Safely variant — I figure if we're going to add this variant that's not really suitable for normal use anyway, there's no point in multiplicatively expanding the API, especially when *Safely is very likely rarely ever used anyway.

@brandur brandur force-pushed the brandur-pilot-init branch from 84dfa87 to f39e9e7 Compare August 2, 2025 21:36
@brandur brandur requested a review from bgentry August 2, 2025 22:19
Comment thread worker.go Outdated
Comment thread client_test.go
}
})

AddWorkerArgs(bundle.config.Workers, jobArgs, WorkFunc(func(ctx context.Context, job *Job[JobArgsWithHooksFunc]) error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a better idea unfortunately, annoying problem to have to hack around 😕

Here, add a mechanism that has the client inject worker metadata into
pilots. This gives pilots information about what jobs/workers are
registered, and helps in cases where say we want pilots to be able to
examine things like `Hooks()` implementations on each worker.

I also ended up adding a function for `river.AddWorkerArgs` that lets a
concrete args object be injected into the registration process rather
than one that's implicitly allocated. I'm a little partial on this one
because it'd be better if it wasn't necessary, but it cleans up tests
somewhat that are trying to test specific things on args like `Hooks()`,
and make tests on features like that _possible_ when there's no
workaround available like storing information that'll be persisted to a
job's metadata (there's otherwise no good way to carry them out, with
the only way I can think of a global variable)
@brandur brandur force-pushed the brandur-pilot-init branch from 2bbc58b to 1dd8817 Compare August 3, 2025 00:55
@brandur

brandur commented Aug 3, 2025

Copy link
Copy Markdown
Contributor Author

Pushed a small change to add a cursory test for AddWorkerArgs for completeness.

@brandur

brandur commented Aug 3, 2025

Copy link
Copy Markdown
Contributor Author

Thanks!

@brandur brandur merged commit 0c2b5e7 into master Aug 3, 2025
10 checks passed
@brandur brandur deleted the brandur-pilot-init branch August 3, 2025 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants