Guardrails, planned tests, and auto-pause

Simple terms

Guardrails are the rules that keep tests from launching or being interpreted too casually.

Defines the statistical and operational rules that determine when tests can launch, pause, and be interpreted.

Routes

• /settings/testing
• /tests/new
• /tests/[id]

Implementation behavior

• Shop guardrail settings hold default minimum runtime, sequential significance preferences, auto-pause behavior, and planned-test defaults.
• Experiment setup can opt into planned fixed-horizon or sequential readback depending on the selected settings.
• Runtime and analytics code use these fields to label status, warnings, and decision readiness.
• SRM warnings should be reviewed as measurement-quality context before using a test result for a decision.
• Auto-pause defaults protect against obvious risk but do not replace human review of experiment quality.

How to use it

• Configure guardrails before creating a large batch of experiments.
• Choose planned-test settings during experiment creation when sample size and duration matter.
• Use sequential mode when ongoing monitoring is expected.
• Review warnings before starting, pausing, or completing a test.

When not to use it

• Do not use guardrails as a replacement for a clear hypothesis and primary metric.
• Do not disable runtime/sample warnings just to ship more tests.
• Do not switch statistical modes mid-test without documenting why.

Implementation source

• Settings UI lives under src/app/(dashboard)/settings/testing and experiment creation/detail routes.
• Settings are read and written through testing-defaults APIs and experiment update flows.
• Experiment stats and result components read planned-test and sequential fields to label decision readiness.

Data and API

• Guardrail settings on shop and experiment records.
• APIs under /api/shops/testing-defaults, experiment creation/update endpoints, and analytics readback endpoints.
• Results views read planned-test and sequential fields for interpretation.

Failure modes

• Premature winner call: test does not meet runtime, sample, or statistical assumptions.
• Unexpected pause: auto-pause threshold or guardrail was triggered.
• Inconsistent readback: experiment settings changed after launch; inspect audit history.