kovyrin

I wrote about a workflow I’ve been using to manage AI agents across multiple projects using structured PRDs and task lists. It’s a manual, markdown-based approach that ended up being more reliable than multi-agent automation tools for real-world coding tasks. Covers how I set it up, what worked, and what didn’t.

> I think it can be summarized as "recklessness" (i.e. jumping the gun without proper understanding of/regard to foundational concepts)

Well… I tried to approach it properly. You google or look up in wikipedia a bunch of info on Buddhism, then you find most popular authors writing on the topic. You end up with “Zen mind, beginner’s mind” and a few others. None of them tell you it may hurt you, none of them mention doing it on group settings or with teachers, they recommend you starting alone and doing it daily. One of those foundational books (don’t remember which one), recommended doing an hour a day… How is one supposed to know that it is dangerous?

Great comment! Thanks a lot for sharing this.

Those weren't outages for a specific instance. But from the whole pool of instances we were running 2-3 would have networking issues, random unexplained hangs requiring an instance restart, huge CPU performance drops, IO hangs, etc, etc.

We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us to building are more robust infrastructure able to handle random node outages, we had a business to build and wanted to focus on the product instead of creating a perfectly scalable application for an early stage application).

Honestly, just as it is with many SAAS companies at some scale, we do not need care about any specific customer's traffic anymore. Simply because we get so much traffic already from our existing customers, that none of the new customers could generate enough to cause any significant blip on the radar. If a customer comes to us with some specific requirements (like being able to index 100MM documents with some specific response time guarantees), we build dedicated pieces of infrastructure for them, load-test it all and provide those guarantees. All of the others are placed in their own pools which have enough capacity to handle 3-5x of ALL of our current traffic with no issues, so any single customer would not be able to generate enough load to cause problems.

And, as I mentioned in the article, we could always order new boxes for any of our clusters and get them online within a couple hours, so we are able to scale up pretty quickly if needed.

We get free inter-DC connectivity for free with Softlayer. There really is no difference for us between connecting to a server within a DC or between datacenters (aside from added latency, of cource) – the same private address space, the same access rules (we see all of our machines in a private backend network from any datacenter).

AFAIK Softlayer has hourly-based rental for real hardware as well. Never tried that, so could not comment on it though.

Softlayer manages the network for us. They offer a fully isolated backend network (frontend connection to the world is optional). For frontend connections we simply use iptables.

My personal experience within the last 7 years: * Softlayer – the best option in terms of quality of service, quality of hardware, quality and size of the infrastructure (geo distribution, etc). * Rackspace – nice, until you grow enough to get relatively locked in and then your prices start to go up, provisioning time suffers and their service turns into shit. * Steadfast – provisioning times up to a week, basically a joke in today's world. * Some German/EU providers like Hetzner – dirt cheap option with desktop-like hardware, failing quickly. Service is nowhere near SL level.

I could go on and on about those, but other options were even more painful.

With Softlayer I get multiple locations (transparently routed 100% transparent backend connectivity), get 1-2 hrs provisioning speed, I do not need to worry about networking and hardware. Failed drives (when they fail, which happens rarely since the hardware they use is REALLY good) are replaced within an hour.

Colocation is a very different beast and I certainly would not encourage anybody to do that until a very large scale when rented hardware economics stop working for them.

Colo - that's an option I'll try to stay away from as long as it is humanly possible. All of my experiences with colo hardware caused a lifetime of pain so that I'm happy to be paying SL a premium for their outstanding services (I'm a huge fan of Softlayer as you have probably guessed).

Re: Rackspace and other providers – based on my real-life practical experience with a few of the largest providers in the States, SL quality of services and their provisioning speed are miles away from competitors could offer. So it was a no-brainer to go with SL and I'm happy we did.

Yeah, that (worker nodes, async processing, etc) seems like one of those ideal use-case for clouds that I could definitely support as a viable option for companies that aren't ready for all-in cloud deployment.