The status quo
Silent But Deadly
Flaky tests start from humble beginnings and live under the radar - until they can’t be ignored. Don’t wait until they truly impact productivity and stability
Unknown Impact
What happens when you see flaky tests? You re-run them. And again. And again. Until they’re simply turned off in continuous integration or the development process.
Thousand Cuts
Flaky tests erode test confidence, and by extension confidence in tests an product stability. Don’t let this spread to customers.
Recoup lost developer time and unnecessary CI costs
BuildPulse enables you to detect, assess impact,
quarantine, and ultimately fix flaky tests.
Detect Flaky Tests
BuildPulse automatically detects flaky tests, and reports on flakiness, lost time, and test health.
Mitigate Impact
You have other burning priorities.
Once detected, BuildPulse can quarantine your tests until they’re fixed, so that they don’t affect the whole organization.
Pinpoint Root Cause
BuildPulse identifies contextual information on flaky tests: how they failed, how often, and traceability.
Reporting
Track stability over time, at any granularity. Slice & dice by repository, team, service, and more.
Monorepo Support
Segment your tests by team, environment, service, api, or any other grouping.
Notifications
Stay on top of flaky tests with Slack and email digests - at the test case level all the way to repository level.
FAQ
What is the difference between a flaky test and a false positive?
A false positive is a test failure in your test suite due to an actual error in the code being executed, or a mismatch in what the test expects from the code. A flaky test is when you have conflicting test results for the same code. For example, while running tests if you see that a test fails and passes, but the code hasn’t changed, then it’s a flaky test. There’s many causes of flakiness.
What is an example of a flaky test?
An example can be seen in growing CI CD pipelines - when pull request builds fail for changes you haven’t made. These failed tests are flaky tests.
What causes tests to be flaky?
Broken assumptions in test automation can introduce flaky tests - for example, if test data is shared between different tests whether asynchronous, high concurrency, or sequential, the results of one test can affect another. Poorly written test code can also be a factor - such as improper polling, race conditions, improper event dependency, or timeout handling for network requests or page loads. Any of these can lead to flaky test failures and test flakiness.
What is the best way to resolve or fix flaky tests?
Devops, software engineering, and software development teams will often need to compare code changes, logs, and other context from before the test instability started, and after - adding retries or reruns can also help with debugging. Test detection and test execution tooling can help automate this process as well. BuildPulse enables you to find, assess impact metrics, quarantine, and fix flaky tests.
What are some strategies for preventing flaky tests?
Paying attention and prioritizing flaky tests as they come up can be a good way to prevent them from becoming an issue. This is where a testing culture is important - if a flaky test is spotted by an engineer, it should be logged right away. This, however, takes a certain level of hygiene - BuildPulse can provide monitoring so flaky tests are caught right away.
What kind of testing strategies have flaky tests?
Flaky tests can be seen across the testing process - unit tests, integration tests, end-to-end tests, UI tests, acceptance tests.
What if my flaky tests aren’t that bad?
Flaky tests can be stealthy - often ignored by engineers and test runs are retried, they build up until they can’t be ignored anymore. These automated tests are automated disruption that slow down developer productivity, impact functionality, and reduce confidence in test results and test suites. Better to get ahead while it’s easy.
What languages does BuildPulse work with?
BuildPulse integrates with all CI providers, test frameworks, implementations to combat non-determinism, drive reliable tests, and provide the best developer experience.
