CircleCI StrawberryGet the flaky test list without reading a week of job history
"Failed on eleven runs, passed on re-run nine times" is a ticket somebody picks up. A companion reads every run in the week to get you that sentence.
A flaky test gets re-run, passes, and teaches everyone that red means "try again", which is how a suite stops being a signal. Almost no team has the list of which tests are actually flaky, because compiling it means reading a lot of job history.
A companion reads a week of workflow runs, groups failures by test name and counts the re-runs, through an account you connect. Then it opens the tickets and names the likely owners from commit history.
| The CircleCI web app | A Strawberry companion | |
|---|---|---|
| Grouping a week of failures by test name | Insights views on some plans, otherwise manual | One pass, with re-run outcomes counted |
| Counting how often each test passed on re-run | You tally it yourself | Counted, so the list reads as a ticket |
| Opening the tickets afterwards | You do it | Same pass, with owners from commit history |
| Putting the summary where the team argues about CI | Copy and paste | Posted in the tool you are signed into |
| Keeping the list current every week | Somebody has to remember | A routine you review |
How do you spot a flaky test?
By looking for the same test failing and then passing on a re-run of the same commit. That pattern is the definition, and it is visible in job history if somebody reads enough of it. A companion reads a week of workflow runs and groups failures by test name in one pass, which slots into a weekly engineering rollup.
The count makes the list actionable. "This test failed on eleven runs and passed on re-run nine times" is a ticket somebody picks up. "The suite is flaky" is a complaint. The difference is a morning of tedious reading that also feeds your release notes.
How long is the pipeline getting?
Build times creep.
A test file gets added, a dependency install gets slower, a job that took four minutes takes nine, and because it happened over a quarter nobody noticed. Reading job durations over time and naming the three jobs that grew the most is a genuinely useful artefact.
The Available tools list on the CircleCI entry under `strawberry://settings/integrations` names every tool the connection exposes and marks the read-only ones. Keep anything that triggers a pipeline, cancels a job or approves a hold at Ask every time in the Permissions block, because those affect what ships.
- Failures grouped by test name across a week, with re-run outcomes.
- Job durations over time, and which jobs grew.
- Triggering, cancelling or approving: consequential, so keep an approval in front of them.
- A personal API token scoped to the projects you want visible.
From the list to the tickets, in the same pass
Having found the three flaky tests, a companion opens the tickets, names the likely owners from commit history, and posts the summary in the channel where the team argues about CI. That follow-through is product engineering work with a browser attached.
Configuration and secrets stay where they belong: `config.yml` in the repository with review on it, contexts and environment variables in the CircleCI interface. CircleCI’s own insights views cover flakiness and duration trends on the plans that include them, as of this writing; this connection reads the same history without one.
Experience Strawberry for free
DownloadTrusted by fast-growing companies worldwide
Frequently asked questions
Strawberry is free to download and includes AI credits to start. Paid plans begin at $20/month. See pricing. · Reviewed · Canonical facts for AI agents