BlogAnalytics

How to A/B test deep link destinations without fooling yourself

· 9 min read · By the Strait team

How to A/B test deep link destinations in a mobile app: what to split, how taps are assigned, which numbers to compare, and how long to run a test.

A sale link can open the product or the collection it belongs to. An install campaign can send people without the app to your website or to the Play Store. A win-back message can open the home screen or the cart. When the link carries real traffic, you can stop guessing: A/B test the deep link destination, split taps between two screens and compare what each brought in.

This post covers how that works in Strait, what's worth testing, and the mistakes that make a destination test give you the wrong answer.

What a destination test is (and isn't)

A destination test keeps everything before the tap the same: the message, the audience, the channel, the link itself. Only the place the tap lands changes. That's what makes it clean. Change the creative and the destination together, and you can't tell which one moved the number.

It isn't a test of your message ("₹200 off" against "free delivery" is a test for your messaging tool, with two links). It answers a narrower question: given that someone tapped, where should they land?

How Strait splits taps

In the dashboard, the create-link form has an optional field, A/B variant B URL. Fill it in and the link becomes an A/B link: taps are split 50/50 between the link's normal destination (variant A) and the URL you entered (variant B). The link list marks it with an "A/B test" tag.

A few details decide how you read the results:

  • The split happens per tap, not per person. Each time a tap reaches our server, it picks a variant at random by weight. Someone who taps the same link twice can land on A the first time and B the second. Judge the test on taps, opens, installs and purchases, not on "users who saw A".
  • Variants are fixed when the link is created. The link's detail panel says so. To change what you're testing, make a new link instead of editing the old one, which also keeps the old results clean.
  • A deferred install keeps its variant. When someone without the app taps, installs, and opens it, the app opens on the destination of the variant their tap got, not a fresh pick. The tap id travels through the install (how deferred deep linking works), and the open is resolved against that tap.
  • A direct open picks at resolve time. A verified App Link can open your app without touching our server. Then the SDK resolves the link, and the variant is picked at that moment and reported with the open.
  • A/B links are always answered by our server. Plain links can be decided at the edge, but links with A/B destinations go to the origin so the pick and the record happen together (how short links work).

Variants are set in the dashboard. The REST API's create-link endpoint doesn't accept them today, so plan a destination test as a dashboard task, not a script.

Things worth testing

Test where you honestly don't know the answer, and where the answer changes what you do next.

Product screen vs collection. A link from a WhatsApp broadcast about one kurta can open that kurta or the festive collection it's part of. The product screen is one tap from the cart; the collection gives people choice. Which sells more depends on your catalogue.

Web page vs store for people without the app. For a shopping link, sending people without the app to your product page keeps the sale today. Sending them to the store may win a long-term customer, or lose them at the install screen. This is a fallback question, so read our post on what a deep link should do when the app isn't installed before you set it up.

Onboarding vs content. A content link opened by a new install can show the article first, or the sign-up step first. It's tempting to assume the sign-up step must come first; a test settles it.

What not to test: a variant that only works for some people. Both destinations must make sense for someone with the app and someone without it, on Android and on iPhone. If variant B is an in-app screen with no web page behind it, people without the app who land on B get a worse experience for reasons that have nothing to do with your question. On iPhone, the match after an App Store install is opt-in and not yet proven on a real iPhone, so check both variants on Android first.

Decide the number before you start

Pick one metric that answers the question, and write it down before the link goes out. For a product-vs-collection test, that's purchases per tap, or revenue per tap. For an onboarding test, it may be sign-ups per install. If you look at six numbers afterwards and pick the one where B won, you've found noise, not a result.

Here is what Strait gives you per variant. Analytics has an A/B variants table for each A/B link with these columns:

Column What it means
Variant A or B
Taps Taps that got this variant
Opened app Taps where the app then opened, as a count and a share of taps
Installs First launches of a new install tied to a tap on this variant
Purchases Purchase events attributed to a tap on this variant
Revenue The value of those purchases

Purchases land on a variant because each event carries the tap it came from. The app SDKs fill that in for you from the most recent link open in the last 7 days (events and revenue). Purchases that only name the link, not a tap, can't be split and appear on their own row, "Purchases without a tap". Apps on older SDKs send events without the tap, so update the SDK before you run a revenue test.

If you'd rather run the numbers yourself, the events export has a variant column for every purchase that carries its tap, and the link.clicked webhook includes the variant of each tap (webhooks).

How many taps you need

Destination tests usually compare small rates, and small rates need a lot of traffic to separate. If 3% of taps on A buy and 3.5% on B, the usual rule of thumb (5% significance, 80% power) asks for roughly 20,000 taps per variant before that half-point gap is more than chance.

Two rules keep you honest:

  1. Fix the sample size before you start. Use any A/B sample size calculator with your current rate and the smallest lift you'd act on. That gives taps per variant. Don't stop before you reach it.
  2. Don't stop the moment it looks significant. Evan Miller's well-known essay on this shows how checking results again and again and stopping at the first "win" inflates false positives. His advice: "Decide on a sample size in advance and wait until the experiment is over before you start believing the 'chance of beating original' figures."

If your link won't get that many taps in a few weeks, skip the test and ship the variant your team believes in.

Run for whole weeks

Who taps, and in what mood, can change through the week: weekend browsing isn't weekday browsing, and the days around payday can differ again. A test that runs Friday to Monday measures a weekend. Run in whole weeks, at least one and preferably two, so both variants see the same mix of days.

Watch for novelty too: a fresh screen can do well for a few days just because it's new.

Keep the test clean

  • Use one link per test, and don't reuse it. Since variants are fixed at creation, a new question means a new link, which is the cleaner habit anyway.
  • Keep campaign tags the same on both variants. If your destination URLs carry UTM tags, give both the same utm_source and utm_campaign, so channel and campaign numbers don't split along variant lines. The UTM builder for app links makes matching tags easy.
  • Don't promise a screen in the message. Since the split is per tap, a message that says "tap to see the full collection" is wrong for half the taps. Word the message so either destination fits.
  • Expect repeat taps. Someone who taps again within a short window isn't billed twice, but each tap is still recorded and gets its own variant pick. That's one more reason to compare variants on outcomes per tap rather than to count people.
  • Test both variants on a real phone first. Tap the link a few times until you've landed on each variant, with the app installed and without it. Test my link helps you check a link on your own phone.

A worked example (hypothetical numbers)

The numbers below are made up to show the method. They aren't from a real campaign.

A D2C snack brand sends a Diwali WhatsApp broadcast about a gift box. Variant A opens the gift box product screen. Variant B opens the Diwali collection, with the gift box at the top. The team picks revenue per tap as the metric and agrees to run two full weeks.

After two weeks, the hypothetical A/B variants table reads:

Variant Taps Opened app Installs Purchases Revenue
A 4,120 2,310 140 182 ₹1,27,400
B 4,080 2,295 138 158 ₹1,34,300

A sold more boxes, 182 purchases against 158. B earned more: about ₹32.9 per tap against ₹30.9 for A, because people who landed on the collection added other items. Taps, opens and installs are close, which is what you'd expect from a random split; a big gap there would point to a broken variant, not a winner.

Is ₹2 a tap real? The team checks it against the sample size they fixed before the test. If it holds, the next broadcast opens the collection. If the gap is inside the noise, either choice is fine.

When not to bother

Transactional links, like an order status or a payment reminder, have one right destination, and referral links must land on the invite every time. Save destination tests for campaign links with real volume and a real open question.

For the wider picture of what a tap is worth after it lands, read deep link analytics: taps, opens and installs.

Sources and how we checked

  • How taps are split (a weighted random pick per tap), the 50/50 split from the dashboard's variant B field, variants being fixed at creation, deferred installs keeping their tap's variant, direct opens picking at resolve time, and the variant in the link.clicked webhook are from Strait's link engine and dashboard code, read on 5 October 2026.
  • That A/B links are answered by the origin server is from the short links docs. The 7-day rule for tying events to taps is from the analytics docs. The variant column in the events export is in the REST API docs.
  • The A/B variants table and its columns are from the dashboard's Analytics page code.
  • Evan Miller, How Not To Run an A/B Test, quoted above. Read on 5 October 2026.
  • The worked example is hypothetical. We haven't run a destination test on a real campaign yet. On iPhone, Universal Links were tested on a real iPhone (iOS 26) in October 2026; install matching is opt-in (off until you turn it on in Settings → iPhone installs) and proven on iPhone simulators; the real-iPhone run waits on our App Store listing.