cptnntsoobv
Joined 11 years ago
Karma
45
Submissions
18
Member since
Mar 16
Recent submissions
Comment · 9 months ago
Comment · 10 months ago
This abuse of notifications (and the user) is quite prevalent, unfortunately.
In the case of the Uber app you should be able to turn these off (they are conveniently enabled by default, of course!).
On the iphone it's Setting -> Communication -> Marketing Preferences -> Push notifications.
Comment · 1 year ago
For UDP I'd look into GSO/GRO to get an upper bound on what pure kernel can do.
With performance benchmarking, specially in networking there is no end to "oh, but did you think of that?!" :)
Comment · 1 year ago
> You actually can do a full checksum
Indeed! This is what I had in mind when I wrote "cumbersome" :).
It's been a while for me to be able to recall whether the problem was the verifier or me, and things may have improved since, but I recall having the verifier choke on a static size limit too. Have you been able to use this trick successfully?
> Always build the eBPF programs in a container
That should work generally but watch out for any weirdness due to the fact that in a container you are already inside a couple of layers of networking (bridge, netns etc.).
Comment · 1 year ago
> IPtables couldn’t do more than 5Gbps of throughput (TCP!)
Is this for a single connection? IIRC, AWS has a 5gbps limit per connection, does it not? I am guessing since you were able to get to ~10 it must be a multi connection number.
Comment · 1 year ago
XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution:
* The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug
* For the same source code, the code generated by the verifier can change across compiler versions in a breaking way, e.g. because the new compiler version implemented an optimization that broke the verifier (see https://github.com/iovisor/bcc/issues/4612)
* Checksum updating requires extra care. I believe you can only do incremental updates, not just because of better perf as the post suggests but also because the verifier does not allow BPF programs to operate on unbounded buffers (so checksumming a whole packet of unknown size is tricky / cumbersome). This mostly works but you have to be careful with packets that were generated with csum offload, don't have a valid checksum and whose csum can't be incrementally updated.
As the blog post points out, the kernel networking stack does a lot of work that we don't generally think about. Once you start taking things into your own hands you don't have the luxury of ignorance anymore (think not just ARP but also MTU, routing, RP filtering etc.), something any user of userspace networking frameworks like DPDK will tell you.
My general recommendation is to stick with the kernel unless you have a very good justification for chasing better performance and if you do use eBPF save yourself some trouble and try to limit yourself to readonly operations, if your use case allows.
Also, if you are trying to debug packet drops, newer kernels have started logging this information that you can track using bpftrace which gives you better diagnostics.
Example script (might have to adjust based on kernel version):
bpftrace -e '
kprobe:kfree_skb_reason {
$skb = (struct sk_buff *)arg0;
$ipheader = ((struct iphdr *) ($skb->head + $skb->network_header));
printf("reason :%d %s -> %s\n", arg1, ntop($ipheader->saddr), ntop($ipheader->daddr));
}'Comment · 1 year ago
> Saves the full VM state to disk
Does this include the RAM for the VM? For auto-idle systems like this where to park the RAM tends to be a significant concern. If you don't "retire" the RAM too the idling savings are limited to CPU cycles but if you do, the overheads of moving RAM around can easily wreck any latency budget you may have.
Curious how you are dealing with it.
Comment · 1 year ago
The spot termination warnings arrive 120 seconds before termination https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-ins...
Comment · 1 year ago
As the saying at Netflix used to go back in the day: we need to become HBO before HBO becomes Netflix
Comment · 3 years ago
> One cool thing we learned was how quickly you can Hibernate and wake up x86 EC2 instances. That ended up being a game-changer for us.
Could you talk more about it? Are you keeping a cache of hibernated EC2 instances and re-launching them per request? What sort of relaunch latency profile do you see as function of instance memory size?
Comment · 5 years ago
It works on AWS's metal instances, doesn't it?
That is definitely a constraint though, as the smallest available metal instances are pretty large, hence costly on a per instance basis.
I guess you are thinking of iouring being disabled in dockeresque rumtimes. This restriction does not apply to a VM, specially if you can run your own kernel (e.g. EC2).
Experiment away!