Add and extend tests

[jenstroeger]

Currently only API level and worker backend unit tests exist. No integration tests or performance tests, so consider those as well.

Related discussions and changes:

• feat(test): add support for TAP (Test Anything Protocol) python-package-template#834
• feat: using pytest as a test runner, diversify the tests into unit, integration, and performance tests python-package-template#1028

Useful frameworks, for example:

• Bombardier, Chaos Monkey and ChaosToolkit, ToxiProxy

[jenstroeger]

We should also consider a more thorough approach to pen testing the deployed infrastructure, e.g. using tools like sqlmap and hping3.

[jenstroeger]

Considering the backend via PostgREST provides a Swagger spec for the REST API (docs), it makes sense to take a more structured approach to testing based on Swagger. Related projects are

• Swagger’s functional testing and contract testing seem to be paid services;
• ryancrandall801/openapi-test-generator and jhaage/swagger-test-generator and meqaio/swagger_meqa are interesting (and unmaintained?) projects;
• apiaryio/dredd is archived but seems popular;
• Discussions:
  • api integration test code generation from swagger spec
  • How To Automatically Generate API Test Scripts from Swagger Documentation

[behnazh-w]

ZAP could also be considered. While this is a pentesting framework, it is possible to only run the API fuzzer.

[behnazh-w]

RESTler can also be considered for stateful REST API fuzzing.

[jenstroeger]

It would probably make sense to combine coverage guided fuzzing (see also ClusterFuzz) with API level testing to improve automation.

[jenstroeger]

Perhaps too complex but Deterministic Simulation Testing looks interesting.

Collection of references here, and for Python’s async there is simloom. In this context see also Antithesis, which “has created the holy grail for testing distributed systems: a bespoke hypervisor that deterministically simulates an entire set of Docker containers and injects faults, created by the same people who made FoundationDB.”¹

Footnotes

1. From Deterministic Simulation Testing for Our Entire SaaS. ↩