This. Ron Kohavi 1) has some excellent resources on this 2). There is a lot of noise in data, that is very often misattributed to 'findings' in the context of A/B testing.
Replication of A/B tests should be much more common in the CRO industry, it can lead to surprising yet sobering insights into real effects.
1) https://experimentguide.com/ 2) https://bit.ly/ABTestingIntuitionBusters