How to detect flaky tests automatically with CI pattern analysis

A flaky test is one that sometimes passes and sometimes fails for the exact same code. It's one of the most insidious forms of CI friction because it gradually erodes trust in your test suite — until engineers start ignoring red CI and merging anyway.

The signal: alternating pass/fail

The clearest indicator of a flaky test is an alternating pattern in CI run outcomes for the same repository and workflow. Pass → fail → pass → fail on the same branch, with no code changes between runs, almost always means a timing dependency, a race condition, or an external service dependency in the test.

Deviera detects this by maintaining a window of recent workflow run outcomes per repository. When it sees two or more alternations in that window — pass, fail, pass — it fires the ci_flaky_detected trigger and creates a high-severity Linear issue with:

• The repository and workflow name
• The run history showing the alternating pattern
• Links to the specific failing runs
• A suggested investigation checklist

Why not just use GitHub's built-in retry?

GitHub Actions supports retry-on-error, and some teams use it as a band-aid for flaky tests. The problem is that retrying doesn't surface the pattern — it hides it. Your CI goes green, nobody investigates, and the flakiness continues to accumulate.

Detection forces the issue. When a Linear issue is created for a flaky test pattern, it gets assigned, triaged, and fixed — rather than quietly re-run into oblivion.

Setting up flaky detection in Deviera

1. Connect your GitHub repositories via the GitHub App.
2. Go to Automations → Templates and enable the Flaky Test Alert template.
3. The template fires ci_flaky_detected → creates a Linear issue with the pattern summary.
4. Optionally, add a Slack notification action to the same trigger for immediate team visibility.

Beyond detection: fixing flaky tests

Detection gets you to triage. Fixing is a separate discipline. Common causes of flaky tests include:

• Timing assumptions — hardcoded delays or sleep() calls that assume a specific execution speed.
• Shared state — tests that modify global state without proper teardown.
• External dependencies — tests that hit real APIs or databases that have their own availability patterns.
• Order-dependent tests — tests that only pass when run in a specific sequence.

Once the Linear issue is created, your team has a structured place to investigate, assign, and close the flaky test — with auto-resolution when the alternating pattern stops.