Difficult to annotate test failures

[ChickenProp]

It would be nice to be able to add a hint when a golden test fails. Sometimes just "run hgold -u path/to/golden-files", since not everyone is familiar with the framework. Sometimes something more detailed like "you can update the golden output if ..., but otherwise the code needs to be fixed".

That seems hard right now, partly because the only way to run a golden test is with the Example instance. Here's what I've currently got:

let hint = "Run `hgold -u path/to/golden-files`"
result <- Hspec.Core.evaluateExample golden undefined ($ ()) undefined
case result.resultStatus of               
  Hspec.Core.Failure
    loc
    (Hspec.Core.ExpectedButGot Nothing expected actual) ->
      throwM $
        Hspec.Core.Failure
          loc
          ( Hspec.Core.ExpectedButGot
              (Just $ result.resultInfo ++ "\n\n" ++ hint)
              expected
              actual
          )
  Hspec.Core.Success -> pure ()                                                                                                               
  _ -> throwM result.resultStatus

with golden :: Golden Value and import Test.Hspec.Core.Spec qualified as Hspec.Core.

That is, I have to run evaluateExample manually to get an Result in IO. Then I can't do much with that directly, so I unpack it to get a ResultStatus, which has an Exception instance. So I put the failure message from the Result in an unused field in the ResultStatus, along with the hint, and throw the new status.

(This would also be less bad if hspec let me do more with a Result. There's instance Example Result, and instance Example (IO ()). But not instance Example (IO Result) or instance Exception Result.)

A simple improvement for me would be if runGolden and GoldenResult were exported. Then I could match on MismatchOutput and throw ExpectedButGot myself. I wouldn't get access to the "Files golden and actual not match" message that way but that's no big deal.

There might be some other way to improve it too.

[ChickenProp]

I guess another option I have instead of that current hack would be a wrapper like

data HintedGolden a = HintedGolden String (Golden a)
instance Example (HintedGolden a) where
  -- `evaluateExample` wraps the one from `Golden` and adds the hint to the returned `Result`

which would let me work with Result. But it's still a hack.

[ChickenProp]

(Actually, I think I slightly prefer having the info in the ResultStatus instead of directly in the Result. If it's in the Result it shows up in the list of specs that ran:

Folder.Suite
  Some test name [✘]
    Files golden and actual not match

Failures:

  src/Folder/SuiteSpec.hs:96:7: 
  1) Folder.Suite Some test name
       expected: ...
        but got: ...

but if it's in the ResultStatus it shows up in the failure message, where it's more visible. With my change:

Folder.Suite
  Some test name [✘]

Failures:

  src/Folder/SuiteSpec.hs:96:7: 
  1) Folder.Suite Some test name
       Files golden and actual not match

       Run `hgold -u path/to/golden-files`
       expected: ...
        but got: ...

[chiroptical]

I was just about to submit a similar-ish issue. When you have a golden file that changes, you aren't told what to do or where to look if you want to accept those changes. I don't have a specific suggestion for the output format, but it would be awesome to know where the golden files are being written on a failure.

For perspective, at my job we have >750 golden tests with >10 possible directories and it can be annoying to go figure out where they are written.