Eppo by Datadog

Just a small bit of news this morning: Eppo has been acquired by Datadog! You can read our official announcement in the blog post below, but I wanted to share a personal take on the news and why I'm thrilled: I've been running experiments for over a decade. Somewhere along the way, I decided that just running experiments wasn't enough for me - I wanted to focus on accelerating the overall adoption of experimentation in business. I first got the chance to do that as a consultant at Cro Metrics, working with 5-10 companies at a time and helping them build experimentation programs. I joined Eppo to scale my impact 10-100x. When I placed this bet, Eppo was the only warehouse-native experimentation tool on the market, and I saw a founding team equally dedicated to statistical rigor and widely-accessible UI. I knew this could enable end users at a scale far beyond what other products on the market were capable of, and I was right: today many Eppo customers successfully run thousands of experiments/year across dozens of teams in their organizations. And now it's time for another 100x leap, because now we can take the mission to 30,000+ existing Datadog customers. This is obviously not just a huge win for us, this is a big moment for the experimentation space too. It's the biggest bet yet from a wildly successful, publicly traded company who sees the exact same vision we do: a *lot* of growth and opportunity ahead for experimentation leaders, programs, and companies. There will be a lot more to come over the next few weeks but for now, suffice to say: our focus is still on building industry-leading experimentation capabilities, and it will the same Eppo team building the product for years to come. The acquisition also unlocks huge improvements in scaling feature flags and canary tests (necessary infra underlying all experimentation use cases) and some pretty exciting forward-looking capabilities for Agentic AI. New digs, new teammates, and a way bigger war chest - but the mission remains the same. Let's get everybody experimenting. https://lnkd.in/eX_Cs8jV

I made a controversial podcast appearance a few weeks ago and waited until now to post about it. I've even shared clips from it but never actually linked to the full episode until just now 🤦♂️ So here is my official plug to catch me on From A to B with Shiva Manjunath where I was initially trolled for my belief in "spaghetti testing", and within 35 minutes had turned the tables and steered us into a career growth b̶u̶t̶t̶k̶i̶c̶k̶i̶n̶g̶ coaching session. You can listen here: https://lnkd.in/gPYwUE6u Or if you're looking for something shorter, maybe this quick 5-minute appearance on The Lantern from MGMT Boston - if you've ever wondered why I continue to choose Boston as my homebase for my experimentation evangelism mission (and not, say, SF), I'll break it all down for you real quick: https://lnkd.in/g45wetrV And if you want the *full* briefing on Building Self-Serve Experimentation and Decreasing Cost-Per-Experiment, you've gotta invite me to give a 30min talk to your company. I'll do two of these free of charge for the first folks to get it on the calendar. DM me 😎

Had a fantastic time last night speaking at the Berlin Experimentation Meetup - well over 100 experimentation pros joined us (several flew in from all around Europe!) and packed the room. A few highlights from my talk: 💡 Presented a framework that distills the job of an experimentation leader into 3 main objectives 💡 Made the case for platforms and processes that enable self-service for the end users in our orgs 💡 Shared about a dozen ideas from Eppo and beyond on how to accomplish this by giving users empowerment, guidance, and time back As I said multiple times over the course of the day, I'm not sure that there's a US city where we could get 130 registrants for an experimentation meetup, especially with only light promotion. There's a very real energy and enthusiasm for experimentation here in Berlin and I appreciate everyone in that room for helping lead the charge! It was such a pleasure getting to meet so many of you. Many thanks to Marcel Toben and Patrick Gunia, PhD for the invite and for what you've organized here, and to Nils Stotz on the photo credit and helping host me yesterday. ⏩ Want to hear some hot takes and practical ideas on scaling your experimentation program at your next event? I think this talk is crazy ready for an encore... DM me and let's make it happen :)

We don't show p-values on the main experiment results screen in Eppo. (They're ever-so-slightly hidden in a more advanced "statistical details" tab) The "problem" with p-values, insofar as most businesses have plenty of non-statisticians running experiments, is that they have a very nuanced and precise definition. Academic researchers get the definition wrong all the time. (There are several quite embarrassing studies demonstrating this) And once you are even a little bit incorrect in your definition, assessing the p-value itself is a totally useless exercise at best. Sometimes it's downright harmful. If you want to evaluate which statistical concepts are worth explaining to your coworkers, imagine drawing a "precision-usefulness curve" for each - as Giorgio Martini once suggested to me. I.E., how much usefulness does this [concept/statistic] lose once your definition is no longer 100% correct? The p-value has a quite unforgiving one, in my experience. Confidence intervals, on the other hand, seem far kinder. Yes, they also have a nuanced definition that very few get right. (although they are maybe one step easier since we don't have to explain them being conditioned on the null) But even with a naive interpretation (usually a probablistic one that sounds more like a Bayesian credible interval - e.g. "there is a 95% chance the true value is in this range"), stakeholders can still walk away with reasonable intuition about what their experiment results suggest. Even a simple three-word intuition like "range of uncertainty" or "the experiment's precision" will likely guide sound decision-making. I've been taken with this as a simple little "precision-usefulness" mental model ever since Giorgio mentioned it - a good consideration for folks who care about making experimentation more accessible to teams throughout our orgs. (graphs are illustrative/made up by ChatGPT and not necessarily my exact opinions 😆)

OUTPERFORM is coming! 250+ pages in print of the best insights on experimentation from 25+ expert authors, published by Eppo. Issues ship Summer 2025. Request a free copy here: https://lnkd.in/epyd3fS2 -Experimentation war stories from inside companies who run tens of thousands of experiments a year -In-depth guides on topics like scaling centers of excellence, running office hours, or how gaming companies leverage experimentation while developing new games -Editorial hot takes on statistics and Bayesian priors and NHST too spicy for LinkedIn -Interviews with data leaders, sports statisticians, AI researchers, and many more

This is the story of a 38-year-old business school professor who outbounded his way to become the COO of a $2B company, became CEO 5 years later, then grew it more than 4x during his tenure. Gary Loveman is a PhD economist and was teaching at Harvard Business School in the 90s when he saw an interesting potential test case in Harrah's Entertainment for his ideas on how to identify high-value customers and ensure their loyalty. So he sent a letter to then-Harrah's CEO Phil Satre with some recommendations for the company. A few months later, he found himself with an offer to become Harrah's COO. He wanted to use data and scientific thinking to find the company's biggest levers for growth, so he set out launching the grandaddy of modern Vegas reward schemes: Harrah's "Total Gold" loyalty program. With the reams of customer data that "Total Gold" collected, Loveman and team realized that slots-heavy "AEPs" (for "avid experienced players") were the biggest profit center for Harrah's, not the stereotypical "high roller" many casinos were focused on at that time. Armed with that insight, Harrah's ran "hundreds of tests to see which incentives induced people to stay at the company’s hotels and to what degree. The results revealed that some, such as retail-store discounts, didn’t affect hotel bookings and could be eliminated. Moreover, if the money spent on them was reallocated to effective incentives, such as deeper room discounts, Harrah’s could boost both responsiveness and profits." By 2004, annual revenues were up to $4.6B (from $2B in 1998) and Loveman won his first of four consecutive "Best CEO in Gaming" awards. In 2005, the foundation Loveman had built on experimentation and data-driven strategy paved the way for their single largest expansion: acquiring Caesar's Entertainment for a whopping $10.4B. His instructions to the team? "There are only three ways to get fired here: theft, sexual harassment, and running an experiment without a control group." I first came across Gary's story in Stefan Thomke's "Experimentation Works", which I've been revisiting lately as it approaches it's fifth anniversary. You can ready Thomke and Loveman's co-authored article in HBR "Act Like a Scientist" - where the quotation about "hundreds of tests" comes from - here: https://lnkd.in/e2PU6P_G The pullquote in my image below is from an interview that Michael Schrage conducted with Loveman in 2011: https://lnkd.in/eckR5KtE