Before I score an idea, I check whether it can be answered.
Prioritisation frameworks like ICE and PIE are useful starting points. They help teams compare ideas on expected impact, confidence, importance, potential and effort.
But I wouldn’t treat a high score as sufficient reason to run a test.
3%of experimentation programmes run a power analysis before prioritising. The rest score ideas without knowing whether they are statistically testable.
ANZ Experimentation Maturity Report, 2026
Before prioritising, I’d check feasibility: do we have enough eligible traffic, a realistic test duration, and a measurable outcome to reach statistical significance? An idea can look excellent on paper yet be impossible to validate in a reasonable timeframe.
So my approach would be two-stage: first, filter ideas for testability using traffic and power analysis; then use a framework such as ICE or PIE to rank the viable experiments. That keeps the roadmap focused on experiments that are both valuable and capable of producing a reliable decision.
—weeks to a decision
— per variant
It fails the check. I don’t abandon the idea or force a bad test — I change the way it gets validated.
-
Ship and monitor
If it’s a low-risk, high-conviction feature, the opportunity cost of waiting two years outweighs the risk of being wrong. Ship it and watch pre- versus post- baselines — return rates, support tickets — over 30 to 60 days.
-
Pivot to micro-conversions
Move the primary metric up the funnel: add-to-cart rate, or clicks on the widget itself. The higher baseline cuts the required sample from months to weeks.
-
Validate qualitatively
Usability testing, session recordings, or a lightweight onsite micro-survey — “Was this size info helpful?”
Set the metric and the traffic you actually have.
Illustrative baselines — put your own numbers inA/B testing is a tool to manage business risk, not a dogmatic rulebook.