Opens in a new tab
13 min 18 sec
Available in:English

A/B Testing: The Most Powerful Way to Turn Clicks into Customers

Shows how to stop guessing what makes a website work by testing versions on real visitors, with lessons from the Obama campaigns, Disney, Netflix and IGN on goals, simpler forms, clearer buttons and useful failures.

Table of Content

Most websites are built on opinion: someone senior likes a photo, someone else prefers a button, and nobody knows which one actually brings in customers. In A/B Testing, Optimizely co-founders Dan Siroker and Pete Koomen show how to replace that guesswork with evidence, and their best proof comes from a presidential race. When Barack Obama’s 2008 campaign swapped a crowd photo for a family photo and changed its button to read Learn more, visitors became 40.6 percent more likely to hand over an email address.

The method behind that result is simple. Show different versions of a page to randomly chosen visitors for a short time, then let their behavior pick the winner. The authors’ case is that small, well-planned experiments, judged against a clear goal, beat confident guessing, and that even a test that loses teaches something. Along the way you get a working routine: how to set a goal and a guess, what to cut from a page, which words make people act, and how to bring a skeptical team along.

A presidential campaign was sure its sign-up page looked right. Then it put a handful of alternatives in front of real visitors, and the gap between what the team liked and what worked was worth millions.

A disaster charity added a photo to its donation page, expecting it would raise giving. Donations dropped. What the team did next turned a failed idea into more than a million dollars.

A TV network hoped a new homepage would lift clicks by ten to 20 percent. A clothing brand assumed visitors looked left first. Both learned that testing can do more than polish, and the numbers were startling.

Two small boxes asking for a phone number and a title were costing a charity money on every visit. A retailer and a campaign found their own ways around forms they could not shorten.

The most common label on the web may be one of the weakest. A charity and a software company each changed a few words on a single button, and visitors responded measurably better.

A gaming site moved one link and watched its clicks almost vanish. It looked like a disaster, yet the collapse showed the company that most of its visitors were returning users.

An analyst arrived at a company where nobody had heard of A/B testing. Within a month she had colleagues betting on outcomes, and losing those bets is what finally changed their minds.

The deepest lesson in A/B Testing is about humility. A campaign team, a charity, a TV network and a room full of IGN employees all believed they knew what their visitors wanted, and the data kept surprising them: a family photo over a crowd, a picture beside a form instead of above it, a middle box beating the left one. The authors’ response is to stop relying on guesses and build a habit of asking, with a clear goal, a stated hypothesis and real visitors deciding.

That habit starts small. Pick one page that matters to your business and name the single number that would prove it is working. Then choose one change worth trying, a field to remove or a noun on a button to turn into a verb, and test it against what you have now. If it loses, record what you learned about your visitors, the way IGN discovered how loyal its audience was. If you have no testing tools of your own, the authors point to platforms such as Optimizely that need no team of analysts.

Every button on your site is already a guess. A test turns it into an answer.

About this book

What is this book about?

Every business with a website faces the same quiet problem: the page that brings in customers is usually designed by opinion. Someone likes a photo, someone else prefers a headline, and nobody knows which choice actually turns visitors into buyers, subscribers or donors. Dan Siroker and Pete Koomen, co-founders of the testing platform Optimizely, argue for a simple alternative. Show different versions of a page to randomly chosen visitors, measure what they do, and keep the winner. Their cases range from Barack Obama's 2008 and 2012 campaigns to a charity set up after the Haiti earthquake, a Disney TV network, Netflix, the gaming site IGN and the retailer Cost Plus World Market. Readers learn how to define success before testing, how to form a hypothesis worth checking, why removing form fields and rewording buttons can raise revenue, how tests can justify a full redesign, why a failed experiment still teaches something, and how to win over colleagues who would rather trust their gut.

Book Information

Rating:

Genre:

Marketing & Sales

Topics:

Data & Analytics, Marketing Psychology, Product Strategy

Publisher:

Wiley

Language:

English

Narrator:

Walter

First published:

January 1, 2013

Pages:

208

Length:

13 min 18 sec

Get the book:

Amazon

Who it's for

  • Traffic but few sign-ups

    Founders and marketers whose website gets traffic but not enough sign-ups, sales or donations.

  • Tired of loudest-voice debates

    Product managers and designers tired of settling page debates by whoever argues loudest.

  • Analysts and team leads

    Analysts and team leads trying to persuade a skeptical company to make decisions with data.

Listen to Preview

About the Author

Dan Siroker

Dan Siroker led analytics for Barack Obama's 2008 presidential campaign, the work that gave him the idea for Optimizely. He was earlier a product manager for Google Chrome and AdWords, has lectured at Stanford and Berkeley business schools, and was named to Forbes' 2013 Top 30 under 30 in Technology. Pete Koomen co-founded Optimizely, served as its chief technology officer and speaks publicly about split testing in software products and how to write such tests.

Ratings & Reviews

Ratings at a glance

4.3

Overall score based on 61 ratings.

What people think

Listeners view this book as a solid starting point for A/B testing, offering straightforward content and useful real-world examples. They enjoy the tempo of the material, with one listener mentioning its seamless fit with lean and agile processes, and find the overall tone encouraging. The methodology is well-received, as one listener emphasizes its reliance on a hypothesis-driven framework. Nevertheless, opinions on the level of depth are varied, with one listener commenting that it doesn't provide enough specific detail.

Top reviews

Anawin

The Obama campaign example alone is worth the price of admission for this book. It’s incredibly rare to find a business text that is both this concise and this impactful. Unlike other dry industry manuals, the authors weave a compelling narrative through various case studies like Wikipedia and Lumosity. I was particularly struck by how they translated complex logic into immediate, actionable steps for any organization. It’s much more practical than Lean Analytics, mostly because it focuses on real-world application rather than just theory. The chapter summaries are fantastic for a quick refresh before a big meeting. Frankly, every product manager should have a copy of this on their desk to reference. It really highlights how a hypothesis-driven approach can change your entire company culture for the better.

Show more
Roo

Wow, this was exactly the kickstart I needed to revamp our current testing strategy. The authors have a way of making the logic behind experimentation feel both obvious and revolutionary at the same time. I loved the diverse range of case studies, especially the Wikipedia and Lumosity examples, which provided great context. It’s a much more practical and better-written guide than some of the more academic books on the market. The advice on how to build a testing culture within a team is particularly valuable for any lead. Even though it's a short read, it's packed with insights that are easy to implement right away. I highly recommend this to anyone looking to make their organization more data-driven.

Show more
Kamol

As someone who works in social media and digital strategy, I found this to be a refreshing take on data-driven design. The way the authors explain how testing can convert casual browsers into loyal customers is very logical. One of my favorite parts was the section on how to sell the concept of A/B testing within your own organization. It's often a struggle to get buy-in from stakeholders, and this book provides the right vocabulary to make that happen. The examples from the 2008 Obama campaign and Formstack are fascinating to read. I appreciated the summations at the end of each chapter, as they distill the main points perfectly. It aligns well with the lean and agile methodologies we use daily.

Show more
Tuck

This book moves fast, which is exactly what I want from a modern business guide. It skips the unnecessary filler and gets straight to the core principles of why experimentation matters. I especially liked the focus on the hypothesis-driven mindset rather than just the tools themselves. It shows a clear path from having an idea to proving it with real user data. While some people might complain about the lack of deep math, the clarity of the writing makes it highly accessible. For anyone just starting out with website design, this is a fantastic entry point. It motivated me to start testing small changes on our landing pages immediately. The pacing keeps you engaged from the first page to the last.

Show more
Gun

Picked this up on a recommendation and I was pleasantly surprised by how well it was written. All business books should aim for this level of clarity and brevity. The logic of the authors' thought processes is easy to follow and translates well into practical scenarios. I didn't agree with every single point they made, but the overall value is undeniable. It’s a great resource for anyone needing to understand the 'what, why, and how' of testing without getting bogged down. The inclusion of the Clinton-Bush Haiti case study was a unique touch that showed the versatility of these methods. It’s a motivating read that makes the daunting world of data feel approachable.

Show more
Matteo

Ever wonder why certain websites just click while others feel clunky and confusing? This book breaks down the science of that 'click' through the lens of constant experimentation. The authors provide a great overview of how A/B testing can reveal the hidden preferences of your audience. I was particularly fascinated by the Etsy and IGN examples, which show the impact of even small design shifts. While I would have liked a bit more technical depth, the hypothesis-driven approach is laid out very clearly. It’s an easy read that serves as a perfect introduction for team members who aren't data-oriented. It really helps bridge the gap between creative design and analytical results.

Show more
Maja

Finally got around to reading this one after seeing it on so many 'essential' lists for growth hackers. It serves as a solid foundation for those completely new to the concept of experimentation. Siroker and Koomen do an excellent job explaining the 'why' behind testing, using well-known brands like Etsy and ABC Family to illustrate their points. However, the technical depth is somewhat lacking for anyone who has actually run a test before. It stays at a very high level throughout the chapters. While the pacing is great, I found myself wanting more statistical nuance and less fluff about the benefits. If you need a quick primer to get your team on board, this is perfect. If you want a deep dive into data science, you should probably look elsewhere.

Show more
Varinee

If you are looking for an entry-level primer on website optimization, this book hits the mark. It provides a solid overview of the field using relatable examples from companies like Chrome and Cost Plus World Market. The authors lay the ground rules early and spend the rest of the text reinforcing those points. To be fair, it is a very short read and can be finished in a single sitting. My main gripe is that it feels a bit dated in some areas, and I would have appreciated more detail in Appendix 2. It’s definitely not a comprehensive manual, but it’s a decent starting point for a beginner. It’s a good 101, but don’t expect a masterclass in analytics.

Show more
Samroeng

Look, this is a decent book if you are starting from zero. It covers the basics of A/B testing in a way that is easy to digest and quick to finish. The case studies, ranging from political campaigns to major tech brands, are the highlight of the text. However, once you get past the initial excitement of the examples, the advice starts to feel a bit repetitive. It reinforces the same few points about testing everything without providing much on the 'how' of complex scenarios. I found the chapter summations helpful, but I wish there were more advanced techniques discussed in the later chapters. It's a solid introductory text, but seasoned pros will likely find it too simplistic.

Show more
Felix

Not what I expected given the hype surrounding Optimizely's founders. While the writing style is accessible and the tone is encouraging, the content feels incredibly thin for anyone with basic industry experience. It offers a fifty-thousand-foot view of A/B testing that rarely dives into the actual mechanics of implementation. I was hoping for more 'nitty-gritty' details on experimental design and statistical significance. Instead, the book relies heavily on high-level stories from the Clinton-Bush foundation and IGN. Truth is, you could probably find most of this information in a few well-written blog posts. It’s a fine 101-level guide for a complete novice, but it left me feeling unsatisfied. I really needed more substance and less introductory cheerleading.

Show more
Show all reviews

Questions about this book

The authors argue that the best way to improve a website is to show different versions to randomly chosen visitors and measure which one performs best. Their lead example is Obama’s 2008 campaign, where a family photo and a Learn more button made visitors 40.6 percent more likely to share an email, adding 2.8 million subscribers and $57 million in donations.

Two things: a success metric you can count and a clear hypothesis. A magazine might track shares, comments or return visits rather than clicks, while a store might track purchases or items added to a cart. The Clinton Bush Haiti Fund shows why the hypothesis matters: when a photo lowered donations, the team guessed scrolling was the cause, moved the image beside the form and raised over a million dollars more.

Removing unneeded fields, hiding optional ones and splitting long forms across pages all helped in their examples. Dropping phone number and title fields lifted the Haiti fund’s donated dollars per pageview by 11 percent. Clearer, verb-based button labels also worked: LiveChat’s switch from Free Trial to Try It Free raised its click-through rate by 14.6 percent.

A losing test can rule something out or reveal who your users are. Chrome learned that video did not change sales, so it can use video later without risk. IGN moved its Videos link and saw clicks fall 92.3 percent, which showed that most of its visitors were returning regulars who expected the link in its old place.

Share results from simple tests regularly, with images of each version and the numbers. Scott Zakrajsek at Adidas uses safe bets that produce quick, clear wins. At IGN, Lizzy Allen ran the A/B Master Challenge, asking staff to predict test winners; nearly everyone was wrong, and that shared humility helped make testing official company policy.

Featured in

AUDIO SUMMARY AVAILABLE

Listen to A/B Testing in 15 minutes

Get the key ideas from A/B Testing by Dan Siroker, plus 5,000+ more nonfiction titles.

  • 5,000+ titles
  • Listen as much as you want
  • 7-day free trial
  • Cancel anytime

A/B Testing

Preview · Full summary 13 min 18 sec
Home

Discover

Open app

Log in