Should I A/B Test My Landing Page If It Only Gets a Few Visitors a Day?

For small changes, probably not yet. We ran a two-week test on a client's landing page, got one sign-up from about 500 visitors, and the test could not name a winner. Before you trust any test, check that the numbers add up, and make sure the page gets enough visitors for the result to mean something.

A client's landing page had a problem. Hundreds of people were clicking through from the emails, and almost nobody was filling in the form. So on the morning of September 25, I set up a test. There were two versions of the page, and the change was in the form. Most visitors, 75%, went to the new version, and the other 25% saw the old one.

Before the test started, we also wrote down what winning would look like. We would keep the new version only if it got at least five sign-ups and at least three times the old page's rate. We would check at 15 days and again at 30.

On October 6, I pulled the numbers. Both versions together had 533 views from 522 people, and exactly one person signed up. That is a rate of 0.19%.

The old version had 343 views and no sign-up showing. The new one had 190 views and one sign-up, which works out to 0.53%. On paper the new page was ahead, until you remember the whole lead rests on one person.

The test was far too small to tell the two pages apart. When we ran the math on one sign-up against none, it said that even if both pages were exactly the same, you would still see a gap that big about one time in three. Those are odds you would never bet your business on.

Then we asked a bigger question. How many visitors would it take to tell a page that turns 0.2% of visitors into sign-ups from one that turns 0.5% into sign-ups? The answer in our notes was about 6,000 visitors for each version. This page was getting about 38 visitors a day. At that pace, reaching 12,000 visitors would take close to a year.

That is the part most owners miss. Telling two close numbers apart takes a lot of people. With a few dozen visitors a day, a test on one small detail can run for months and still tell you nothing, while you sit and wait for an answer that is not coming.

Before I looked at who was winning, I looked at whether the scoreboard was counting right. A few things were off, and any one of them could fool you.

This is the lesson I keep coming back to. A test is only as good as the counting behind it. Check the dates, the split and the counts first, then look at the result.

From what I saw on this page, a test earns its keep when two things are true. The page gets enough visitors to reach a real answer in a time you can live with, and you know ahead of time what result would make you switch.

We got the second part right. Writing the rule down before the test kept us honest. When the two weeks were up, nobody could talk themselves into saying the new page had won, because one sign-up was nowhere near the five we had asked for.

The first part is where a lot of small businesses get stuck. If your page gets a few hundred visitors a month, it helps to know up front that a test on a small detail may sit at "no winner" for a very long time. That is worth knowing before you spend weeks waiting on it.

On October 7, the client and I talked through the page. The test with the shorter form was not bringing in sign-ups yet, so we planned bigger changes for the next round. We would swap the bullet points for a video once it came in, move the testimonials up right under the top of the page, take out the solution section, and test a black background against a white one.

By mid-October I decided I did not want to keep working on the form. A later look at the test showed one sign-up on each version, which is still a tie.

Here is what I took from it. Take what fits your business:

If you have a problem in your business that does not call for ongoing marketing work, tell us about it. We build digital business solutions, and we are good at it. You can reach us anytime at tovnamedia.com.