My test suite takes quite a bit of time, not because the code base is huge, but because it consists of a variety of mathematical models that should work under a range of conditions.
This makes it very quick to write a test that’s basically “check that every pair of models gives the same output for the same conditions” or “check that re-ordering the inputs in a certain way does not change the output”.
If you have 10 models, with three inputs that can be ordered 6 ways, you now suddenly have 60 tests that take maybe 2-3 sec each.
Scaling up: It becomes very easy to write automated testing for a lot of stuff, so even if each individual test is relatively quick, they suddenly take 10-15 min to run total.
The test suite now is ≈2000 unit/integration tests, and I have experienced uncovering an obscure bug because a single one of them failed.
I used to have to use a CI pipeline at work that had over 40 jobs and 8 stages for checking some sql syntax and formatting and also a custom python ETL library that utilized pandas and constantly got OOM errors.
They didn’t write any unit tests because “we can just do that in the CI pipeline” and if you didn’t constantly pull breaking changes into your branch you would guarantee the pipeline would fail, but if you were lucky you only had to restart 30% of your jobs.
It was the most awful thing and killed developer productivity to the point people were leaving the team because it sucks to spend 40% of your time waiting for CI scripts to fail while you are being yelled at to deliver faster.
When the CI takes longer than 10 minutes
Envious. It averages 15-30 mins
Are you like building a mobile app or have 100k tests or is it just super slow?
Still waiting on approval for more resources. It’s not a priority in the company.
I swear we have like 4 runners on a raspberry pi.
My test suite takes quite a bit of time, not because the code base is huge, but because it consists of a variety of mathematical models that should work under a range of conditions.
This makes it very quick to write a test that’s basically “check that every pair of models gives the same output for the same conditions” or “check that re-ordering the inputs in a certain way does not change the output”.
If you have 10 models, with three inputs that can be ordered 6 ways, you now suddenly have 60 tests that take maybe 2-3 sec each.
Scaling up: It becomes very easy to write automated testing for a lot of stuff, so even if each individual test is relatively quick, they suddenly take 10-15 min to run total.
The test suite now is ≈2000 unit/integration tests, and I have experienced uncovering an obscure bug because a single one of them failed.
I used to have to use a CI pipeline at work that had over 40 jobs and 8 stages for checking some sql syntax and formatting and also a custom python ETL library that utilized pandas and constantly got OOM errors.
They didn’t write any unit tests because “we can just do that in the CI pipeline” and if you didn’t constantly pull breaking changes into your branch you would guarantee the pipeline would fail, but if you were lucky you only had to restart 30% of your jobs.
It was the most awful thing and killed developer productivity to the point people were leaving the team because it sucks to spend 40% of your time waiting for CI scripts to fail while you are being yelled at to deliver faster.