I'm called in to consult on a performance problem on a scaled service. Team was load testing their code and seeing low throughput:
Me: so you have an in-memory cache, right?
Them: yes!
Me: what is the TTL?
Them: Oh, it's not set, oops. Here, let's set it to 1 minute. Hey look, the performance went way up!
Me: okay, great. When you say 1 minute, do you mean 60 seconds?
Them: uh...wait...uh....oh, the unit is seconds. Wait, why is the performance so good with a 1 second TTL?
Me: What's your load test?
Them: We crank 1M TPS fetching the same 30 items over and over.
Me: ....
I totally agree about the power of profiling but profiling without understanding would not have helped this team.
Me: so you have an in-memory cache, right?
Them: yes!
Me: what is the TTL?
Them: Oh, it's not set, oops. Here, let's set it to 1 minute. Hey look, the performance went way up!
Me: okay, great. When you say 1 minute, do you mean 60 seconds?
Them: uh...wait...uh....oh, the unit is seconds. Wait, why is the performance so good with a 1 second TTL?
Me: What's your load test?
Them: We crank 1M TPS fetching the same 30 items over and over.
Me: ....
I totally agree about the power of profiling but profiling without understanding would not have helped this team.