I run a file sharing / content delivery platform called pixeldrain:
https://pixeldrain.comThe system serves 4 PB of data to 60 million visitors per month. I have served 30 PB and 700 million file views since I started tracking usage somewhere in 2018.
I'll go from front to back:
- Most of the frontend is plain HTML, CSS and JS. I have started transitioning some pages to Svelte. I like this framework for its speed and simplicity
- Cloudflare Analytics to get basic info like which pages are popular and where my users are from
- The structure of the website (page wrap, menu, footer, etc) is managed with Go's template system
- Constellix for Geo-DNS. This automatically sends users to the server closest to them by doing Geo-IP lookups on the nameservers
- The user-facing servers are dedicated 10 Gbps Leaseweb servers, stuffed to the brim with SSDs in RAID6 for caching. Each of these servers cost €1200 per month. The storage servers are from Hetzner's SX line.
- The OS is Ubuntu 20.04 server edition. I use Ubuntu over Debian because it ships with TCP BBR
- The API is written in plain Go. The only HTTP libraries I use are httprouter for routing and Gorilla Websockets
- The storage system is custom built to spread files over multiple servers. I call it pixelstore, it's not open source (yet)
- The database is ScyllaDB. I landed on this one after going through multiple other systems with severe bottlenecks. I started with MySQL which was limited to a single location, so other locations had high latency. Then I tried CockroachDB, but it kept hanging under the load no matter how much hardware I threw at it. ScyllaDB is very fast and relatively reliable.
- UptimeRobot for monitoring
- Mailgun for account e-mail verifications
Feel free to ask me more questions :-)
The system serves 4 PB of data to 60 million visitors per month. I have served 30 PB and 700 million file views since I started tracking usage somewhere in 2018.
I'll go from front to back:
- Most of the frontend is plain HTML, CSS and JS. I have started transitioning some pages to Svelte. I like this framework for its speed and simplicity
- Cloudflare Analytics to get basic info like which pages are popular and where my users are from
- The structure of the website (page wrap, menu, footer, etc) is managed with Go's template system
- Constellix for Geo-DNS. This automatically sends users to the server closest to them by doing Geo-IP lookups on the nameservers
- The user-facing servers are dedicated 10 Gbps Leaseweb servers, stuffed to the brim with SSDs in RAID6 for caching. Each of these servers cost €1200 per month. The storage servers are from Hetzner's SX line.
- The OS is Ubuntu 20.04 server edition. I use Ubuntu over Debian because it ships with TCP BBR
- The API is written in plain Go. The only HTTP libraries I use are httprouter for routing and Gorilla Websockets
- The storage system is custom built to spread files over multiple servers. I call it pixelstore, it's not open source (yet)
- The database is ScyllaDB. I landed on this one after going through multiple other systems with severe bottlenecks. I started with MySQL which was limited to a single location, so other locations had high latency. Then I tried CockroachDB, but it kept hanging under the load no matter how much hardware I threw at it. ScyllaDB is very fast and relatively reliable.
- UptimeRobot for monitoring
- Mailgun for account e-mail verifications
Feel free to ask me more questions :-)