b1naryth1ef

We've put a lot of work into our clients and our backend to make sure the impact of being on 1 vs 100 guilds is negligible. It helps that most of the folks building Discord are power users in a bunch of servers (so we feel the pain of poorly optimized paths early). Generally if you don't look at a server often it shouldn't effect the performance of the app / bandwidth usage. I think we have some more blog posts in the works regarding some of these topics so look forward to those!

We won't have support initially but as a Linux user internally I'm definitely hoping to push for it, and I see no reason why it wouldn't get built!

Ah sorry I misunderstood you. We keep the original image around and provide different sizes for different platforms/resolutions/dpis/etc.

As mentioned in the post, one of our core product features is preventing your IP from being shared. Given that requirement, images shared in chat have to be proxied through our infrastructure. When doing this we save a lot of money and improve client performance by reducing image sizes.

vips (Go binding) is included in the benchmarks mentioned in the post, but at the time of running them (~10 months ago) vips pulled 51482954 ns/op on a 1024x1024 test image, where as pillow-simd managed 3324135.3035 ns/op.

GCP has a virtual networking stack to support a bunch of crazy (and awesome) features Google has built. Unfortunately the complexity here seems to hurt power-users like us. In this case it appears that for some unknown reason the node failed to program its network stack when coming up, meaning it was completely unavailable (even the metadata service used internally by google failed).

Yup this, our privacy policy plainly states that we're not in the business of making money from your data. We have various provisions which limit how and when we can share your data.

Thanks, we try our best with these. Past experience has shown they can be very valuable, and help everyone at the company get context on the system and how we handle failures.

Reliability testing is definitely something we're interested in as we spin up more SRE/reliability focused individuals, but also has probably the least amount of cost-benefit for us (compared to engineering effort on improving the things we know need work). Some of the failure in the system we experienced is related to issues we know about, but haven't prioritized (read; had time for) yet.

For the library, we believe the bug is related to hackney and the fact it uses the high priority setting for its pool process. For some reason (this is the part we're not entirely sure on, and still spending some time investigating) this high priority process got stuck and consumed all of the scheduler time (presumably related to the earlier API degradation), breaking the distribution port and the application in a weird way. Oddly enough the systems we run on are SMP, so in theory one rogue process should not be able to have this effect.

Ahhhh, thats actually something we're aware of. Currently direct calls run on an entirely separate set of metals vs. everything else (this was mostly to help us test/measure video & screenshare rollout). Unfortunately some providers seem to be having issues with DDoS filtering over-triggering when they see video traffic, which negatively impacts the whole server. Something we're hopefully fixing in the short term!

Very possible you saw a slight interruption around 11:30PST for around 10 minutes until we found and decommissioned the host that experienced this problem. We generally don't update status until we can verify impact/source, we see tons of limited outages from ISPs misbehaving.

Can you give more explicit examples of the bad audio quality you experience? I'd be happy to forward this onto our native team to look into if there are some concrete things they can look at. Generally 99% of the audio issues we see people experience are due to ISP/peering/DDoS/etc issues, most of which are handled automatically by our servers within a few minutes.