athoscouto
Joined 7 years ago
Joined 7 years ago
A couple of services looked like they had a memory leak. Memory was continuously increasing over time. Thanks to Python 3.14, we were able to use memray to understand what was going on. Those services were recreating HTTP clients (aiohttp) for every inbound request, and memory allocated by the downstream SSL lib was growing faster than it was being released.
We ended up rolling back to 3.13, which fixed the issue. I'll try again with 3.14.5.
most tasks I can do better and faster with composer 2
a fellow engineer reported a bug on a code I had written a few months back. I used his report as prompt for composer 2, gpt-5.4-high and claude-4.6-opus-max-thinking. composer found the issue spot on. gpt found another possible vector a couple of minutes later, but a way less likely one and one that would eventually self heal (thus not actually reproducing what we observed on production). claude had barely started when the other two had finished
also, i don't have a budget per se. but it is expected that i over deliver if i'm over spending
I've been trying to use Claude Code seriously for over a month, but every time I do it, I get the impression that it would take me less work to do with Cursor.
I'm on the enterprise plan, so it can get pricey. This is why I used to stick mostly to auto mode.
Now Composer 2 has taken over as my default model. It is not as intelligent as OpenAI's or Anthropic's flagship models, but I feel it has as good as or better intuition. With way better pricing. It can get stuck in more complex tasks though.
Being able to get in the loop, stop and instruct or change models makes all the difference. And that is why I've stayed in the editor mode until now. Let's see if 3.0 changes that.
But I think that is beside the point.
Individuals are not fungible, but team members are - or at least can be, depending on how you structure your teams.
And as your org grows, you want predictability on a team level. Skipping a bunch of reasoning steps, this means having somewhat fungible team members, to give you redundancy.
The engineering parallel here is the tradeoff between resilience and efficiency. You can make a system more reliable by adding redundancy. You make a system more efficient by removing redundancy.
The author also mentions a leader adjudicator, which means there is probably some sort of coordination to pick a leader. This raises the question of how a leader is picked, and if leadership changes based on how hot a key is in a given AZ.
This blog series is a great read. Every day Marc drops excellent content and leaves room for questions, which he ends up answering on the following days. Hope more details come next.
> We’ve learned from building and operating large-scale systems for nearly two decades that coordination and locking get in the way of scalability, latency, and reliability for systems of all sizes. In fact, avoiding unnecessary coordination is the fundamental enabler for scaling in distributed systems
I hope there is a follow-up since the points the author only glossed over are important to understanding the architecture and trade-offs. I would like to know about the cross-adjudicator coordination protocol and how the journal works.
From the information available, it seems that DSQL should be pretty fast as long as you keep writes local. Once you add active-active replication and start writing to the same key in different regions the coordination costs should slow the system down significantly (or not - but if that is the case I want to know how they managed to do it).