emcooke
Joined 17 years ago
Karma
115
Submissions
23
Member since
Aug 5
Recent submissions
Comment · 15 years ago
Twilio has a service-oriented architecture internally. We leverage a variety of application stacks for different services.
Comment · 15 years ago
Several people have asked for additional details. We just posted a quick follow-on:
[UPDATE] A central theme of the recent AWS issues has been the Amazon Elastic Block Storage (EBS) service. We use EBS at Twilio but only for non-critical and non-latency sensitive tasks. We've been a slow adopter of EBS for core parts of our persistence infrastructure because it doesn't satisfy the "unit-of-failure is a single host principle." If EBS were to experience a problem, all dependent service could also experience failures. Instead, we've focuses on utilizing the ephemeral disks present on each EC2 host for persistence. If an ephemeral disk fails, that failure is scoped to that host. We are planning a follow-on post describing how we doing RAID0 stripping across ephemeral disks to improve I/O performance.
Comment · 15 years ago
Our main infrastructure is deployed on AWS but we have capacity at several cloud providers for load-balancing, redundancy, etc.
Comment · 15 years ago
Yup, good idea. We set up an ngnix proxy to cache the page while the blog hosting provider fixes their server.
Comment · 15 years ago
Ironically, this highlights one of the main issues we discuss in the post!
The Twilio Engineering blog is hosted off an external Wordpress site with a single IP that's forwarded from ngnix load balancer pool. Since the load balancers assume that the external service can fail, they won't tied resources blocking access to other parts of the site.
Hope you enjoy the post :)
-Evan Twilio.com
Comment · 15 years ago
We just enabled caching on the ngnix proxy to the external site hosting our Wordpress install for the engineering blog. Hopefully that should help performance.
-Evan Twilio.com
Comment · 16 years ago
Hi Anthony, Twilio service status available via our public status page http://status.twilio.com/ We communicated degraded international SMS service to customers on August 16.
You bring up a good point that the information might not be readily discoverable. We'll work to make the status page more findable and to extend the API (http://status.twilio.com/documentation/rest) with features such as RSS to let customer subscribe up-to-the-minute status information.
Comment · 16 years ago
For those wondering about the technical aspects of this decision take a look at this thread:
http://getsatisfaction.com/twilio/topics/international_sms-1...
When we launched the Twilio SMS Beta we tried hard to support sending SMS messages to both US and International destinations. When there were problems, we worked with our customers to collect forensic data on hundreds of carriers worldwide and pass it to our carriers partners to debug.
At Twilio we are dedicated to working with top quality carriers and technology. After months of working to fix problems, were not able to deliver the reliable International SMS service our customers have come to expect.
We apologize for any problems this has caused for our customer and we'll work to bring back International SMS service after were able to deliver on the quality we do the rest of Twilio services.
Cheers, -Evan
CTO and Co-Founder
Comment · 16 years ago
In a 100% ideal scenario an organization would only need developers as every possible failure/problem would be covered and handled by automation.
Obviously the real world is different. New code is deployed that has bugs, there are unpredicted events, there are complex failure scenarios that are difficult to automate etc.
A DevOps engineer may write automation software in conjunction with developers to automate the operational aspects of business logic. Thus, it's a partnership between the DevOps engineer whose metrics are driven by availability/reliability/scalability/security and the developer who is trying to attain some business objective.
Comment · 16 years ago
Can't someone whose role embodies a certain culture and philosophy not have a title that fits that goal? How would you define DevOps?
Comment · 16 years ago
Thanks for the pointer to OpSec term. DevOpsSec... I sense a tshirt in the making?