Could we make more local code coverage options?

[hadley]

e.g. https://github.com/NIEHS/amadeus/blob/main/.github/workflows/test-coverage-local.yaml

Makes it easier for federal government to use and avoids problems like #834

[gaborcsardi]

I am not really sure what you mean by this. That workflow has a lot of stuff. We don't necessarily need to provide ready to use workflows for everybody, but rather the actions that people can use to build their own workflows.

Re #834, is codecov upload still a problem?

[hadley]

Right, I was thinking that we don't actually use the actual service that much, and if it wasn't too hard to generate and save the data locally, maybe it's worth doing?

[gaborcsardi]

We can save the data as an artifact and/or print it, but it is a bit cumbersome to look at it.

• We could print a summary to the terminal:

  
  But the missing lines column cannot have links, and that column must be probably truncated. Of course we also cannot browse the source code with coverage information.

• We could try to create a Markdown summary for it, maybe, like this:
  https://github.com/marketplace/actions/code-coverage-summaryy#markdown-example
  We can link the missing lines, but still need to truncate them, and cannot browse code with coverage info.

• Third possibility is to create an artifact with the coverage report, but that's also a but cumbersome to download, uncompress and then open locally.

Really, the ideal situation would be if codecov.io worked a bit better, and now that we are updating it properly, hopefully it will. It gives us coverage over time and badges, and you can browse the code with coverage info.

Btw. you need to update to the latest test-coverage workflow to update.

Btw.2. there is a codecov extension for Chrome and Firefox, which is OK, then you can browse code on GH, with coverage info: https://github.com/codecov/codecov-browser-extension Unless you use another browser of course.

[gaborcsardi]

I looked at https://github.com/NIEHS/amadeus/blob/main/.github/workflows/test-coverage-local.yaml but I am not sure if it is something that we can roll out.

It has issues with concurrency as well, as it is pushing commits to the repo. Plus storing the coverage in the git tree is also not so great, e.g. you cannot run multiple coverage runs for the same commit and then take the union of the results.

IMO the current best solution is still codecov.

This said, if you (or anyone) feels like experimenting with the cited workflow, go ahead. It would be useful to see how it works in practice. I'll also start using it for a couple of repos.

[gaborcsardi]

We could also store the coverage data in a designated branch on GH, called coverage-data. Then we would do a brute-force pull-commit-push-revert loop to avoid the concurrency issues.

But storing the data there is one thing, and visualizing it is another. We could add some coverage report to the pkgdown site, but at this point it feels like a homemade reimplementation of codecov....

[hadley]

Let's keep this open, but put it on the back burner for now.

We could also consider using a hosted database (e.g. motherduck) to store the coverage coverage results. That definitely feels even more like a homegrown codecov implementation, but OTOH it's not like our needs are that complicated.

[gaborcsardi]

As long as the codecov upload is working, and it should be working right now, we can use codecov as a DB, through its api, if we don't want to interact with the web ui.

In some sense a hosted DB is even more work than storing data in a GH branch, because you need to implement auth.