Skip to content

Add after-each test hook for hawk #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
iethree wants to merge 1 commit into
base: main
Choose a base branch
from
+422 −3
Draft

Add after-each test hook for hawk #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,42 @@ they live in is loaded; this may be affected by `:only` options passed to the te

Return values of methods are ignored; they are done purely for side effects.

## Per-Test Hooks

You can run arbitrary code after each individual test finishes with an `after-each` hook:

```clj
(methodical/defmethod mb.hawk.hooks/after-each ::my-hook
[options context]
(record-test-info! context))
```

`options` are the same options passed to the test runner as a whole, just like for `before-run`/`after-run`.
`context` is a map describing the test that just ran:

| Key | Description |
| --- | --- |
| `:var` | the test var that just ran |
| `:ns` | the namespace of the test var |
| `:report-events` | all `clojure.test` report event maps emitted during the test (`:pass`, `:fail`, `:error`, `:begin-test-var`, `:end-test-var`, ...), in order. Each event additionally has `:testing-contexts` assoc'ed onto it: the value of `clojure.test/*testing-contexts*` (innermost first) at the moment the event was emitted |
| `:output` | everything the test wrote to `*out*` or `*err*`, as a string. Output is captured with a tee, so it still shows up in the normal test output too |
| `:summary` | map of `:pass`/`:fail`/`:error` counts for this test var |
| `:duration-ms` | wall-clock time the test var took, in milliseconds |
| `:parallel?` | whether the test is a `^:parallel` test (and so may run concurrently with other tests) |

Hooks run after the test var itself completes but before its `:each` fixtures finish, and run on the same thread as
the test, so for `^:parallel` tests hooks may run concurrently. If a hook throws (or a `clojure.test` assertion inside
it fails), it is reported as a test error/failure attributed to that test var -- it will fail the test suite and show
up in the JUnit output. Hooks only run for test vars that actually run: a var skipped because its `:each` fixture threw,
or because an earlier failure tripped `:fail-fast?`, does not fire after-each hooks.

Capturing test output and report events is skipped entirely when no `after-each` hooks are registered -- whether any
hooks are registered is checked once at the start of each test run -- so test runs without hooks pay no overhead.

The same caveats as for whole-suite hooks apply: dispatch values just need to be unique, hook order is indeterminate,
hooks only run if the namespace they live in gets loaded, and return values are ignored. There is no default
`after-each` method (it would run once per test); register at least one hook to enable the feature.

## Partitioning tests

You can divide a test suite into multiple partitions using the `:partition/total` and `:partition/index` keys. This is
Expand Down
94 changes: 91 additions & 3 deletions src/mb/hawk/core.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@
[mb.hawk.parallel :as hawk.parallel]
[mb.hawk.partition :as hawk.partition]
[mb.hawk.speak :as hawk.speak]
[mb.hawk.util :as u]))
[mb.hawk.util :as u])
(:import
(java.io StringWriter Writer)))

(set! *warn-on-reflection* true)

Expand Down Expand Up @@ -152,6 +154,86 @@
(def ^:private ^:dynamic *parallel-test-counter*
nil)

(def ^:private ^:dynamic *after-each-options*
"Bound to the test run options when at least one [[mb.hawk.hooks/after-each]] hook is registered, and `nil`
otherwise. Capturing per-test output and report events only happens when this is non-nil, so runs without after-each
hooks pay no overhead."
nil)

(defn- tee-writer
"Returns a Writer that forwards everything written to it to both `primary` and `copy`. Closing it flushes both but
closes neither."
^Writer [^Writer primary ^Writer copy]
(proxy [Writer] []
(write
([x]
(cond
(integer? x) (do (.write primary (int x))
(.write copy (int x)))
(string? x) (do (.write primary ^String x)
(.write copy ^String x))
:else (do (.write primary ^chars x)
(.write copy ^chars x))))
([x off len]
(if (string? x)
(do (.write primary ^String x (int off) (int len))
(.write copy ^String x (int off) (int len)))
(do (.write primary ^chars x (int off) (int len))
(.write copy ^chars x (int off) (int len))))))
(flush []
(.flush primary)
(.flush copy))
(close []
(.flush primary)
(.flush copy))))

(defn- run-test-with-after-each-hooks
"Run `test-var`, capturing its [[clojure.test]] report events and `*out*`/`*err*` output, then invoke
any [[hawk.hooks/after-each]] hooks with the run `options` and a context map describing the test that just ran.

The hooks are run as the test var's reporting window closes -- right before its `:end-test-var` event reaches the
real reporter -- rather than after [[orig-test-var]] returns. This matters: a hook exception (or a `clojure.test`
assertion inside a hook) is reported as a `clojure.test` error while `test-var` is still the var being reported on,
so it is counted and attributed to `test-var` everywhere, including in the JUnit output (which finalizes a test
var's results when it sees `:end-test-var`)."
[options test-var]
(let [events (atom [])
buf (StringWriter.)
orig-report t/report
start-ns (System/nanoTime)
run-hooks! (fn []
(let [duration-ms (/ (- (System/nanoTime) start-ns) 1e6)
summary (merge {:pass 0, :fail 0, :error 0}
(select-keys (frequencies (map :type @events))
[:pass :fail :error]))]
;; bind t/report back to the real reporter so anything the hook reports (including the
;; hook-error event below) goes straight through instead of being recaptured into `events`
(binding [t/report orig-report]
(try
(hawk.hooks/after-each options {:var test-var
:ns (:ns (meta test-var))
:report-events @events
:output (str buf)
:summary summary
:duration-ms duration-ms
:parallel? hawk.parallel/*parallel?*})
(catch Throwable e
(orig-report {:type :error
:var test-var
:message (format "Error in after-each hook for %s" test-var)
:expected nil
:actual e}))))))]
(binding [t/report (fn [event]
(swap! events conj (assoc event :testing-contexts t/*testing-contexts*))
;; run hooks while the var's reporting window is still open (before :end-test-var is
;; forwarded), so hook errors are attributed to this var
(when (= (:type event) :end-test-var)
(run-hooks!))
(orig-report event))
*out* (tee-writer *out* buf)
*err* (tee-writer *err* buf)]
(orig-test-var test-var))))

(defn run-test
"Run a single test `test-var`. Wraps/replaces [[clojure.test/test-var]]."
[test-var]
Expand All @@ -161,7 +243,9 @@
:parallel
:single-threaded)
(fnil inc 0)))
(orig-test-var test-var)))
(if-let [options *after-each-options*]
(run-test-with-after-each-hooks options test-var)
(orig-test-var test-var))))

(alter-var-root #'t/test-var (constantly run-test))

Expand Down Expand Up @@ -203,7 +287,11 @@
options)]
(when-not (every? var? test-vars)
(throw (ex-info "Invalid test vars" {:test-vars test-vars, :options options})))
(binding [*parallel-test-counter* (atom {})]
(binding [*parallel-test-counter* (atom {})
;; check for after-each hooks once per run rather than once per test; eftest conveys these bindings
;; to its worker threads
*after-each-options* (when (hawk.hooks/after-each-hooks-registered?)
options)]
(merge
(mb.eftest.runner/run-tests
test-vars
Expand Down
56 changes: 56 additions & 0 deletions src/mb/hawk/hooks.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,59 @@
"Default hook for [[after-run]]; log a message about running after-run hooks."
[_options]
(println "Running after-run hooks..."))

(methodical/defmulti after-each
"Hooks to run after each individual test var finishes (after the test itself, but before `:each` fixture teardown).
Add a new hook like this:

(methodical/defmethod mb.hawk.hooks/after-each ::my-hook
[_options context]
...)

`options` are the same options passed to the test runner as a whole, i.e. a combination of those specified in your
`deps.edn` aliases as well as additional command-line options.

`context` is a map describing the test that just ran:

| Key | Description |
|------------------|--------------------------------------------------------------------------------------------|
| `:var` | the test var that just ran |
| `:ns` | the namespace of the test var |
| `:report-events` | all `clojure.test` report event maps emitted during the test (`:pass`, `:fail`, `:error`, `:begin-test-var`, `:end-test-var`, ...), in order. Each event has `:testing-contexts` assoc'ed onto it: the value of `clojure.test/*testing-contexts*` (innermost first) when the event was emitted |
| `:output` | everything the test wrote to `*out*` or `*err*`, as a string |
| `:summary` | map of `:pass`/`:fail`/`:error` counts for this test var |
| `:duration-ms` | wall-clock time the test var took, in milliseconds |
| `:parallel?` | whether the test is a `^:parallel` test (and so may run concurrently with other tests) |

If a hook throws (or a `clojure.test` assertion inside it fails), it is reported as a test error/failure attributed
to the test var -- it will fail the test suite and show up in the JUnit output -- and other after-each hooks for that
test may not run. Hooks run on the same thread as the test, so for `^:parallel` tests they may run concurrently.
Comment on lines +83 to +85

Hooks run only for test vars that actually run: a var skipped because its `:each` fixture threw, or because an
earlier failure tripped `:fail-fast?`, does not fire after-each hooks.

Capturing test output and report events is skipped entirely when no after-each hooks are registered, so test runs
without any hooks pay no overhead. Whether any hooks are registered is checked once at the start of each test run.

Unlike [[before-run]] and [[after-run]] there is no default method (one would run once per test); register at least
one hook to enable the machinery. The dispatch value is not particularly important -- one hook will run for each
dispatch value -- but you should probably make it a namespaced keyword to avoid conflicts, and give it a docstring so
people know why it's there. The orders the hooks are run in is indeterminate. The docstring for [[after-each]] is
updated automatically as new hooks are added; you can check it to see which hooks are in use. Note that hooks will
not be ran unless the namespace they live in is loaded; this may be affected by `:only` options passed to the test
runner.
Comment on lines +95 to +99

Return values of methods are ignored; they are done purely for side effects."
{:arglists '([options context]), :defmethod-arities #{2}}
:none
:combo (methodical/do-method-combination)
:dispatcher (methodical/everything-dispatcher))

(defn after-each-hooks-registered?
"Whether any [[after-each]] hooks are currently registered. When this is false the test runner skips per-test
output/event capture entirely. There is intentionally no default [[after-each]] method, so any registered method
(including one on the `:default` dispatch value) counts as a hook."
[]
(boolean
(or (seq (methodical/primary-methods after-each))
(seq (methodical/aux-methods after-each)))))
Loading
Loading