jrdorn

Most interactions between tests are really minor and indistinguishable from random noise. Those ones aren't worth worrying about.

There are two ways to deal with the bigger interaction effects. First is to predict which experiments will meaningfully interact ahead of time and split the samples between them (making the tests take longer). Second is to run tests in parallel on all users and look at the data to determine interaction effects after the fact (and potentially need to invalidate some results).

In our experience, a mix of these approaches works best. It's really hard to predict meaningful interaction effects ahead of time, so save that for the really obvious cases (e.g. black text on a black background). For everything else, the benefit of running more experiments usually outweighs the cost of occasionally needing to throw out results because of interaction effects. It's much better to run 10 tests and need to throw out 1 than it is to run 5 tests.

We're big fans of PostHog ourselves. Any tool that encourages people to use data to make product decisions is a huge win in our book.

We are planning on an open core model similar to PostHog. So we would eventually monetize from cloud hosting and license keys for enterprise features. We wanted to use MIT to make it as easy to adopt as possible for all companies.

Thanks! Let us know if you run into any issues integrating and setting up your first feature.

Congrats on the launch! Product looks great and there's so much room for improvement when it comes to onboarding.

Thanks for the suggestion. We added MySQL support so we can work with Matomo now, but we're definitely lacking in documentation and tutorials. There's a similar issue with people using GA4, Snowplow, Segment, and anything else with a pre-defined schema. It should be way easier to use these systems with GrowthBook. Like there's no reason to even write SQL most of the time if we know the Matomo table structure in MySQL. We're hoping to solve that issue in the next few weeks.

Yeah, Statsig is building a really cool product. There's definitely a gap in GrowthBook right now, like you mentioned, between pure feature flags and experiments. We're working on some light-weight event tracking features to try and bridge that gap. So you'll be able to see how many people are using the features, which values they are getting, and easily integrate with tools like DataDog and NewRelic for performance monitoring.

Tools like Flagr typically function as a microservice that handles all of the feature evaluation. That means you need a network request every time you want to get a feature's value. With GrowthBook, there's a single cacheable JSON definition of all of the features and then evaluation happens locally within the SDKs. We found this approach to be much better for performance and scalability.

The other missing part in most feature flagging tools is the analysis side. Most tools let you run an A/B test, but it's up to you to track the data, process it, and run it through a stats engine. We built that part directly into GrowthBook.

Yeah, that's our exact experience as well. I was super surprised to find out how much of a graveyard open source A/B testing was. Intuit (Wasabi), Facebook (PlanOut), and SeatGuru (Sixpack) open sourced their internal platforms years ago, but they are all abandoned now.

Thanks! It's been a crazy 6 months, feels like years ago

Ax is for automated optimization using machine learning. You define the parameters and optimization function and it decides the variations, traffic splitting, and everything else for you.

Growth Book is for hypothesis testing. It let's you define and run a specific controlled experiment and then you can analyze the results and make a decision.

Yep, should be possible once MySQL/MariaDB is done.

Do you mind explaining that a little more? As it's currently designed, a company could use DBT to model their raw data into dedicated metric tables and then Growth Book sits on top of those with a really simple SQL query and some settings (e.g. is the goal to increase or decrease the metric)

I'd love to see an open source standard way to define metrics, but haven't found anything yet.