Hold the environment constant
Workflow, job, matrix labels, OS, and commit form an execution fingerprint. Unknown environments are excluded rather than pooled.
acme / payments-api
Demo mode · synthetic data
Illustrative pipeline snapshot
Flaky-test intelligence
Know which tests waste your CI time.
Quarantine holds the commit and execution environment constant, finds pass/nonpass reversals, and ranks the evidence by estimated rerun cost.
Decision-ready evidence
Ranked flaky tests
| Rank | Canonical test | Nonpass evidence · reversal cohorts | CI impact | 14-day trend | Failure intensity |
|---|---|---|---|---|---|
| 01 | payment_retry_settles | 16.7% 31 / 186 nonpass within reversals · 95% CI 12.0–22.7% | 184 min estimated / 30d | ||
| 02 | refund_webhook_idempotency | 12.7% 18 / 142 nonpass within reversals · 95% CI 8.2–19.1% | 126 min estimated / 30d | ||
| 03 | ledger_reconciliation_timeout | 11.5% 11 / 96 nonpass within reversals · 95% CI 6.5–19.4% | 72 min estimated / 30d |
Demo mode. This repository and CI history are synthetic. Results illustrate the analysis workflow and do not represent production usage, measured savings, or customer data.
The correctness argument
A failure is not automatically a flake.
A test that failed Monday and passed after a code change Tuesday may simply have been fixed. Quarantine only compares outcomes when the input is held constant.
Comparable key
(test, commit SHA, workflow, job + matrix, OS)
Conflict
at least one pass ∧ at least one nonpass
(test, commit SHA, workflow, job + matrix, OS)
Conflict
at least one pass ∧ at least one nonpass
Abstain when evidence is thin
No comparable reruns means insufficient data, not a confident zero. Fail and error are normalized to one nonpass class.
Turn evidence into priority
Tests are eligible only after a same-input reversal. Estimated rerun job duration is divided among implicated tests to avoid double-counting.
Asynchronous by design
Requests stay fast. Ingestion happens elsewhere.
GitHub OAuth
PKCE, signed state, public repository metadata, and JUnit artifacts.
Hono Worker
Server-rendered routes, encrypted tokens, strict ownership checks, and CSP.
Cron + Queue
Overlapping polls, per-repo leases, at-least-once delivery, and rate-limit backoff.
D1 / SQLite
Idempotent runs, jobs, artifacts, test outcomes, and recomputed scores.
- Demo state
- Synthetic snapshot
- Consumer sample
- 2m ago
- Configured cadence
- 15 min
- Scoring window
- 30 days