A/B testing product cards

A/B testing of product cards is a method for comparing two versions of a product card by measuring buyer behavior and key metrics to pick the more effective version.

How A/B testing of product cards works

An A/B test splits the card's traffic into two (or more) groups: version A — the control, version B — the variant. Each group sees its version of the card, and you then compare preselected metrics over the experiment period.

• Random assignment. Visitors must be assigned to versions randomly to avoid systematic behavioral differences.
• Metric fixation. Commonly measured metrics are card CTR, add‑to‑cart events, conversion to order, average order value (AOV) and revenue per visitor (RPV).
• Statistical testing. Evaluate results with confidence intervals and p‑values at a chosen significance level, typically 95%.
• Control side factors. Changes in promos, seasonality, price, stock and logistics should be accounted for or neutralized during the test.

Why a Kaspi.kz seller needs A/B testing of product cards

Direct effects for a Kaspi.kz seller include higher conversion, more efficient ad spend and fewer returns due to clearer presentation. Specifically:

• Conversion increase. Even a 0.5–1% uplift in conversion can multiply profit with high traffic: at 10,000 views/day and an average check of 10 000 тг, increasing CR from 2% to 2.5% yields 250 extra orders × 10 000 тг = 2.5 mln тг extra revenue per day.
• Ad savings. Better CTR and conversion reduce cost per goal in Kaspi internal campaigns and external channels.
• Customer insights. Tests reveal which elements build trust: photos, video presence, reviews, tags like “Fast delivery” or “Discount”, etc.
• Risk reduction. Instead of mass edits to a card, testing lets you validate a hypothesis on a sample and avoid sales drops from a poor change.

Which hypotheses and metrics to test on Kaspi.kz

Prioritize hypotheses by expected impact and ease of implementation. Examples of hypotheses with typical metrics:

• Main photo. Metric: CTR, CR. Example: white background vs. lifestyle image. Often produces a visible effect on clicks and conversions.
• Additional media. Metric: CTR, time on card, CR. Adding a product video or 360° view can increase trust and conversion.
• Title and key features. Metric: CTR, CR. Test keyword order, inclusion of benefits (e.g., “2‑year warranty”), or shortening the title.
• Price presentation. Metric: CR, RPV. Displaying the discount percent, crossed‑out old price or monthly installment options may affect both CTR and average order size.
• Badges and trust signals. Metric: CR, returns rate. “Fast delivery”, “Kaspi guarantee”, seller rating or verified reviews can increase conversion and reduce returns.

Sample size calculation and test duration

Calculate sample size from baseline conversion and the minimum detectable effect (MDE) you care about. Use a power calculator with chosen significance (typically 0.05) and power (commonly 80% or 90%). Key points:

• High traffic cards need fewer days; low‑traffic cards can require weeks or months to collect enough events.
• Account for seasonality and external noise (promotions, weekends). Aim for at least 2–4 weeks to cover weekly cycles; longer if traffic is low.
• When order events are rare, consider increasing the data volume by testing several similar cards together, running cumulative tests, or raising the MDE you’re willing to detect.
• Report sample size requirements in terms of visitors and conversions (events). If expected conversions are insufficient, the test is unlikely to produce actionable results.

Real experiment examples for Kaspi.kz sellers

Concrete experiments that sellers commonly run:

• Swap main photo. Test a plain white background vs. an in‑use lifestyle photo; metric: CTR and CR.
• Add video. Short demo video vs. no video; metric: time on card, CR, returns.
• Highlight delivery or promo. Add a “Fast delivery” or “Discount” badge in the card header; metric: CTR, CR.
• Change title structure. Move the brand or key spec to the front of the title; metric: CTR from search and CR.
• Buy Box messaging. Test different seller descriptions in Buy Box (if applicable) to see impact on purchase share.

Practical tips for running A/B tests on Kaspi.kz

• One main change at a time. If possible, isolate a single hypothesis per experiment to keep results interpretable.
• Pre‑test balance checks. Verify that groups are similar by traffic source, device, and historical conversion before starting.
• Freeze price and stock when feasible. Price or availability changes during the test introduce bias; if you cannot freeze them, track them as covariates.
• Avoid running tests during known promotional events. Marketplace promos and category sales add noise; exclude those days or stratify analysis.
• Set stopping rules in advance. Decide on minimum sample, test duration, and significance thresholds before looking at results to avoid false positives.
• Use diagnostic metrics. Track CTR, add‑to‑cart, sessions per user and returns to understand the mechanism behind changes in the main metric.
• Document everything. Keep a changelog of what was tested, implementation details and rollout plan for winners and rollbacks.

Analyzing results and making a decision

After the test ends, follow a structured analysis:

• Check statistical significance and confidence intervals. A p‑value below your threshold suggests a non‑random effect; the confidence interval shows the plausible range of uplift.
• Look at the effect size. Small but statistically significant lifts may still be economically irrelevant—compare uplift to implementation cost and risks.
• Verify secondary metrics. Ensure that diagnostics like returns, average check and session length are not negatively affected.
• Consider a phased rollout. For positive results, roll out gradually and monitor KPIs in production to catch any edge cases.
• If there’s no clear winner. Either increase sample size, raise the MDE, or run a different hypothesis. Avoid making wide changes based on inconclusive data.

Examples of control metrics and pre‑launch checklist

Control metrics to monitor during the test:

• Visitors to the card (sessions or unique users)
• Click‑through rate on the card
• Adds to cart
• Conversion to order and revenue per visitor (RPV)
• Average order value (AOV)
• Return rate and complaints

Pre‑launch checklist:

• Define the primary metric and acceptable MDE.
• Calculate required sample size and expected duration.
• Ensure randomization and group assignment are implemented correctly.
• Freeze or log price, stock and active promotions.
• Prepare analytics tracking and dashboards for live monitoring.
• Agree on stopping rules and rollout plan for the winner.

Conclusion

A/B testing product cards on Kaspi.kz lets sellers make data‑driven decisions that increase conversion and reduce risk. Prioritize hypotheses by expected impact, ensure adequate sample size, control external factors and rely on both statistical and practical significance when deciding to implement changes. Document experiments and monitor rollouts to capture long‑term effects.