mnutt
Joined 16 years ago
Joined 16 years ago
Software is at an inflection point with agents where we have the opportunity to take back control of our apps and data from the large corporations and replace it with software we’ve written ourselves and run ourselves. I’m excited for how Sandstorm can enable this.
Tangent: when you have a lambda returning binary data, it’s pretty painful to have to encode it as base64 just so it can be serialized to json for the runtime. To add insult to injury, the base64 encoding is much more likely put you over the response size limits (6MB normally, 1MB via ALB). The c++ lambda runtime (and maybe rust?) lets you return non-JSON and do whatever you want, as it’s just POSTing to an endpoint within the lambda. So you can return a binary payload and just have your client handle the blob.
We still take some steps to mitigate control plane issues in what I consider a reasonable AWS setup (attempt to lock ASGs to prevent scale-down) but I place the control plane disappearing on the same level as the entire region going dark, and just run multi-region.
I think it was just luck of the draw that the failure happened in this way and not some other way. Even if APIs falling over but EC2 instances remaining up is a slightly more likely failure mode, it means you can't run autoscaling, can't depend on spot instances which in an outage you can lose and can't replace.
This aligns with their “you should be in multiple AZs” sales strategy, because self-hosted and third-party services can’t replicate data between AZs without expensive bandwidth costs, while their own managed services (ElastiCache, RDS, etc) can offer replication between zones for free.
Once you have asterisk set up and running, it becomes easy to also set up all sorts of other extensions like "check the weather" / "tell a joke" / "check the train statuses". I put up some code for it here: https://github.com/mnutt/rotary
[ my public key: https://keybase.io/mnutt; my proof: https://keybase.io/mnutt/sigs/-Fx2q4k7gdzBMPqMmt1yC-a3m-lVGCP3kcxrNhj4LUM ]