The problem
See PR #16329, where I systematically provoked failures in automated tests I had recently added.
None of those were catched by the CI
- Screenshot tests failed, but they are broken on master anyways due to some permission issues. And these were no ScreenshotTests, so they were rightfully not included here.
- The Drone job also did not execute them.
This is very dangerous, as it conveys a false sense of security and confidence in the code quality.
It is also quite frustrating, having put effort into stuff developers usually don't like (automating tests), to prevent regressions, and none of it would actually matter.
The cause
Unclear right now - something seems to be wrong in the test selection setup, where we differentiate between
- Unit tests
- Screenshot tests
- Drone tests
Many tests seem to be picked up in neither category
The problem
See PR #16329, where I systematically provoked failures in automated tests I had recently added.
None of those were catched by the CI
This is very dangerous, as it conveys a false sense of security and confidence in the code quality.
It is also quite frustrating, having put effort into stuff developers usually don't like (automating tests), to prevent regressions, and none of it would actually matter.
The cause
Unclear right now - something seems to be wrong in the test selection setup, where we differentiate between
Many tests seem to be picked up in neither category