daper

Sure, you can. But on advantage of IPv6 is you addresses are globally unique. If you ever need to connect two networks that were created independently (like different companies), unique addresses, even local are very helpful.

I've given a try to IPv6 in a company with few tens on servers in a 2 DCs, an office + additional location, 3 ISPs in total. For me the real challenge is not just different way to write an IP address or doing NAT. The challenge is that IPv6 changes a lot of unexpected things:

- Our ISPs support IPv6 but routing quality is way worse than IPv4 including occasional inability to connect to some networks or greater latency than IPv4. I had to create tickets with such issues understood that most probably they just don't have IPv6 BGP sessions to all their upstream providers they connect.

- How the VPN (an employee / road warrior setup) should be configured since from the routing perspective you don't need a VPN to connect from your home to the office? Assuming both have proper IPv6 connection and all devices in the office and your laptop have a globally addressable IP address. Employee can have IPv4 or dual stack at his home, where is dual stack in the office. Very confusing. Looks like Fortigate also don't have an idea and decided to not support such case.

- You have to be careful with site-to-site VPN since even your internal services like database are now globally addressable. You really need proper firewall rules / routing policies to not leak unencrypted packets over internet.

- SLAAC is cool but doesn't provide DNS configuration. (there is RFC8106 but is it supported by all OSes?). You need DHCPv6 for that. You have to choose: use only DHCPv6 or SLAAC + DHCPv6 or just relay on the vast that DNS will be proviedd by DHCP IPv4 in a dual stack setup.

- The way of providing high availability gateway address in a network is different. You need router advertisement where you can provide priorities. That actually is much better than any other VIP mechanisms (no issue with MAC table updates, etc.) but you need to know that.

- OSPF works a bit differently. For example: there is no authentication in router communication in OSPF itself, you are supposed to use IPSec.

The list is longer unfortunately...

With BGP (especially anycast) you don't have direct influence where the requests lands. You can steer traffic by techniques like AS prepend, some priorities per BGP session (I'm not an expert in BGP) but eventually it's not CDN that decides where the request will be routed. It's decided by routers of the client's ISP and backbone networks, each taking decision themself (shorted AS path from his point of view + BGP policies set by network operators), not by you. You can't for example split traffic between DCs with specific proportion (like 30% here and 70% there). You can't split by something else than network properties like forward request to the region that has better chance to have this content in the cache.

With DNS and dynamic responses you are directing request to specific DC, even server, almost on every request. It may be dedicated for this traffic type (live stream different than static images etc). Your DNS server can take the hostname ("www.google.com") into consideration - BGP doesn't even know the hostname in the URL. If you wanted to do it with BGP you would need to place specific content to a dedicated /24 subnet and that is impossible considering how many IPv4 addresses are available.

BGP doesn't even consider network latency, current network load. CDN knows load on their machines, on their network link, where given content is placed. The bottleneck may be storage, network or CPU processing, different for different sites and content type. They need to direct traffic on request basis considering this and at least the hostname from the URL. That's why DNS is used first.

Varnish (HTTP caching only) also uses one thread per client. I believe worker threads are used to handle reuqests while a dedicated thread handles all the idle connections between requests using epoll(). Also per-threads stack size is lowered so thousands of threads don't occupy massive amount of memory.

Single threaded HTTP servers have their own issues. If the bottleneck is the storage then lack of async open()/stat() and some other calls is problematic. We feel that serving hundreds of millions of files (long tail content) from slow storage using nginx. For that reason you can configure nginx to spawn multiple processes.

Can I ask if your whole content can be stored on a single server so content is simply replicated everywhere or there is some layer above that that directs requests to the specific group of servers storing the requested content? I assume the described machine is not just part of tiered cache setup since I don't think nginx capable for complex caching scenarios.

AFAIK no. The point of DRM is to prevent recording / playing the media on a device without decryption key (authorization). So the goal is different than TLS that is used by the client to ensure the content is authentic, unaltered during transmission and not readable by a man-in-the-middle.

But do we really need such protection for a TV show?

"Metadata" in HLS / DASH is a separate HTTP request which can be served over HTTPS if you wish. Then it can refer to media segments served over HTTP (unless your browser / client doesn't like "mixed content").

I have some experience serving static content and working with CDNs. Here is what I find interesting / unique here:

- They are not using OS page cache or any memory caching for that, every request is served directly from disks. This seems possible only when requests are spread between may NVMe disks since single high-end NVMe like Micron 9300 PRO has max 3.5GB/s read speed (or 28Gbps) - far less than 800Gbps. Looks like it works ok for long-tail content but what about new hot content everybody wants to watch at the day of release? Do they spread the same content over multiple disks for this purpose?

- Async I/O resolves issues with nginx process stalling because of disk read operation but only after you've already opened the file. Depending on FS / number of files / other FS activities, directory structure opening the file can block for significant time and there is no async open() AFAIK. How they resolve that? Are we assuming i-node cache contains all i-nodes and open() time is insignificant? Or are they configuring nginx() with large open file cache?

- TLS for streamed media was necessary because browsers started to complain about non-TLS content. But that makes things sooo complicated as we see in the presentation (kTLS is 50% of CPU usage before moving to encryption offloaded by NIC). One has to remember that the content is most probably already encrypted (DRM), we just add another layer of encryption / authentication. TLS for media segments make so little sens IMO.

- When you relay on encryption or TCP offloading by NIC you are stuck with that is possible with your NIC. I guess no HTTP/3 over UDP or fancy congestion control optimization in TCP until the vendor somehow implements it in the hardware.

I had brief experience managing such IT team after which I've gained big respect for them. I've never seen such low budget to amount of work ratio in my IT career. Everybody expects them to fix the "trivial" problems immediately but reality in my case was:

- 2 "IT guys" for > 500 headcount company - If you look at IT guy instant messaging app you will see many unread notifications all the time. Without tickets it's impossible to properly prioritize and divide work. "Tickets" for everything" was one of my first change. - They are responsible for the broadest set of technologies / vendors I've ever seen. That includes: networking (including cabling, switched, wireless routing, VPN, firewalls), laptops, office apps, conferencing hardware (and software), IT security, accounting and warehouse software (including their MS SQL databases,) windows servers, virtualization, managing email accounts and maaaany more. - without tickets they can't show real amount of work they are doing so justifying adjustment budget and staffing for the need is a big challenge. On the other hand everybody expects to fix trivial problems without forms (so most of their work!). - Upper management don't perceive the as value but only as a cost - trivial problems could be fixed faster but that would require supplies for spare parts. This is bigger than you may think: few generations of laptops in use, office hardware, spare monitors, docking stations etc. That require office space for a small "warehouse" which is not free. How to justify this need considering the previous points? - many people fix "trivial problems" themselves: replacing cable, "borrowing" cable/mouse/monitor from a different desks. That creates more work for the IT guys: more issues because of incomplete office hardware.

Note that they are using 10GbE network when even a single NVMe disk used in the test has few times more bandwidth so the bandwidth results are constrained by the network hardware.

Creating on-premise infrastructure a year ago we went for 2x25Gbps network. This or even 100Gbps seems to be current "mainstream" and 10GbE is definitely not enough for NVMe speeds available today.

I categorize this as bugs caused by data inconsistency because od data duplication. That includes:

- Using asynchronous database replication and reading data from database slaves - Duplicating same data over multiple database tables (possibly for performance reasons) - Having additional system that duplicates some data. For example: in the middle of rewriting some legacy system - a process that was split into phases so functionality between new and old systems overlap for some period of time.

Based on my experience I always assume that inconsistency is unavoidable when the same information is stored in more than one place.

Generally agree, for basic configuration it just works. But I was planning to use IPv6 in some network segments in my company. Providing proper VPN (road warrior style) for employees to access such network isn't simple with IPv6 where every address is routable over internet. Morver Fortigate devices we have don't have such functionality.