Planning an A/B test is less about choosing an arbitrary number of days and more about estimating how much evidence the experiment needs. This practical guide shows how to turn baseline conversion data, traffic, detectable lift, power, allocation, and business timing into a defensible test duration—then gives you a reusable worksheet for checking whether the plan is still valid.
Overview
The question “How long should we run an A/B test?” has two parts:
- How many observations are required? This is the sample-size question.
- How quickly will those observations arrive? This is the duration question.
A test that runs for a fixed period without enough conversions may produce an unstable result. Conversely, a test that reaches a nominal sample size too quickly may still need to cover normal weekday and weekend behavior, recurring campaign patterns, or a complete purchase cycle.
Sample size is influenced by the baseline conversion rate, the smallest improvement worth detecting, the acceptable false-positive risk, the desired statistical power, and the allocation between variants. Duration then depends on eligible traffic and the proportion of visitors who reach the measured conversion event.
Use the estimate as a planning tool, not as permission to stop whenever the result looks favorable. Repeatedly checking results and stopping at the first attractive outcome can increase the chance of treating random variation as a real improvement. Define the stopping rule before launch and follow it unless there is a documented data-quality or safety issue.
How to estimate A/B test duration
Work through the following sequence before creating the experiment in your testing platform.
1. Define the primary conversion event
Choose one primary outcome, such as completed checkout, qualified lead submission, account activation, or trial-to-paid conversion. Secondary metrics can explain the result, but they should not silently replace the primary metric after the test starts.
Make sure the event is measured consistently for both variants. If the experiment relies on GA4 or a tag-management setup, validate the event, parameters, deduplication, and attribution behavior before using historical rates. A useful starting point is the GA4 measurement plan and QA checklist. Also review the GA4 troubleshooting checklist if counts differ between analytics and the testing platform.
2. Establish the baseline conversion rate
The baseline is the conversion rate you expect without the change. Calculate it from a recent, representative period using the same denominator and eligibility rules planned for the experiment:
Baseline conversion rate = conversions ÷ eligible users
For example, 240 conversions from 12,000 eligible users gives a 2% baseline rate. Avoid combining unlike audiences or relying on a short period affected by a one-off promotion. If the test concerns ecommerce checkout, confirm that purchase events and revenue are not duplicated; the GA4 ecommerce tracking guide can help with that review.
3. Choose the minimum detectable effect
The minimum detectable effect (MDE) is the smallest change worth acting on. It can be expressed as a relative lift or an absolute percentage-point change. If the baseline is 2% and the target is a 10% relative improvement, the target rate is 2.2%, an increase of 0.2 percentage points.
A smaller MDE requires more observations. Set it using implementation cost, expected business value, and the decision you will make—not because a large lift produces a shorter test. If a 1% relative improvement would not change the roadmap, there is little value in designing the experiment to detect it.
4. Set power, significance, and allocation
Statistical power represents the chance of detecting the chosen effect if it is genuinely present. The significance threshold sets how much false-positive risk the analysis is designed to tolerate. Use the defaults required by your testing method or analytics team, and record them before launch rather than changing them after seeing the result.
With an even 50/50 split, observations accumulate at the same rate in each variant. Unequal allocation can be appropriate for risk management, but it increases the time required for the lower-traffic variant to reach its target sample. Do not estimate duration from total site traffic if only a subset of users is eligible for the experiment.
5. Convert sample size into calendar time
Once the required number of users or conversions is estimated, divide it by expected eligible traffic:
Estimated days = required eligible users ÷ average eligible users per day
Use a conservative traffic estimate rather than a peak-day average. If the result is 18,000 eligible users and the site typically supplies 1,500 eligible users per day, the traffic-based estimate is 12 days. Add enough calendar coverage to observe normal weekly behavior and the relevant conversion lag. The final planned duration is the longer of the statistical estimate and the operational minimum.
Inputs and assumptions worksheet
Copy this checklist into your experiment brief or spreadsheet:
- Primary event: What exactly counts as a conversion?
- Eligible population: Which users, sessions, devices, or accounts can enter?
- Baseline rate: What is the recent conversion rate, and what period produced it?
- MDE: What absolute or relative improvement would change the decision?
- Significance threshold: Which analysis convention will be used?
- Statistical power: What detection probability is required?
- Allocation: What percentage enters control and each treatment?
- Traffic rate: How many eligible users arrive on a typical day?
- Conversion lag: How long can a user take to complete the event?
- Minimum calendar coverage: How many complete weekly cycles or business periods are needed?
- Stopping rule: What sample, date, or decision condition ends the test?
Keep the denominator stable. A test measured by unique eligible users should not be compared with a forecast based on sessions or page views. Likewise, an experiment that excludes consent-restricted users may have fewer measurable observations than a general traffic report suggests. Check consent and tagging behavior with the consent mode implementation and validation guide before finalizing traffic assumptions.
Worked examples
Example 1: Traffic is the main constraint
Suppose a landing page has a 4% baseline conversion rate. The team considers a 15% relative lift meaningful, making the target rate 4.6%. The experiment is split evenly, and the planning method estimates that 10,000 eligible users are required in total.
At 800 eligible users per day, the traffic estimate is:
10,000 ÷ 800 = 12.5 days
The team might plan for at least two complete weeks, provided conversion lag and quality checks are covered. If traffic falls to 500 eligible users per day, the same sample requires about 20 days. The test duration changes even though the statistical inputs do not.
Example 2: A low conversion rate needs more patience
A signup-to-qualified-lead experiment has a 0.8% baseline rate. The team wants to detect a 0.2 percentage-point increase, from 0.8% to 1.0%. Because the event is relatively infrequent and the absolute change is small, the required sample may be much larger than for a high-volume button-click test.
If the business cannot supply enough qualified leads in a reasonable period, consider whether the primary event is too far down the funnel for this experiment. You might retain qualified lead as the decision metric but use an earlier, validated diagnostic metric to understand behavior. Do not replace the primary metric after launch merely because it reaches significance sooner.
Example 3: Revenue requires a different review
For an ecommerce test, conversion rate may be stable while average order value changes. Plan the test around the primary business question and predefine guardrails such as refunds, cancellations, or checkout errors. Revenue distributions can be more variable than binary conversion outcomes, so use the testing platform's appropriate method and avoid applying a simple conversion-rate sample estimate to every metric.
When to recalculate and what to check
Recalculate the duration estimate before launch whenever traffic, eligibility, baseline performance, allocation, or the minimum worthwhile effect changes. Revisit it during the test if a tracking defect, deployment, major campaign, pricing change, or audience rule changes the data-generating process. Document the change instead of quietly extending the test until the result is favorable.
Before reading the final result, check that:
- Both variants received the intended allocation.
- Exposure and conversion events are recorded once and under the same rules.
- No variant experienced a broken page, unusual outage, or material delivery difference.
- The planned sample and conversion lag have been covered.
- Results are reviewed by the predefined primary metric and guardrails.
Use a campaign and measurement plan consistently if traffic is segmented by channel; the UTM naming and campaign tracking template can reduce avoidable reporting ambiguity. For ongoing reporting, a Looker Studio marketing dashboard can show experiment traffic, exposure, conversions, and guardrails without replacing the formal test analysis.
Finally, save the completed worksheet with the experiment result. The inputs will change for the next test, but the structure remains useful: define the event, estimate the baseline, choose a meaningful MDE, calculate the required sample, convert it to calendar time, and stop according to the rule agreed before launch.
