_urga
Joined 10 months ago
Joined 10 months ago
1. Add "abstract" to your search query to surface papers.
2. Search for "... reading list". For example, Heidi Howard maintains https://github.com/heidihoward/distributed-consensus-reading...
3. Read blogs like "The Morning Paper" (https://blog.acolyer.org) but skip fields that are outside your scope. You don't have time to follow more than one or two (or three) major fields.
4. Use Google Scholar to find the most cited papers, or to find papers that build on papers you think are good.
5. Keep an eye out for the conferences where these papers were presented. Then read the other papers that were also presented.
6. When you come across an amazing paper, read other papers by the same authors or supervisors.
7. If you're lucky you might also find good "survey" papers that cover and reference the state of the art.
8. Lecture notes from Stanford or MIT or another university can also be a great way to get a big picture of the evolution of techniques for a given data structure or problem. For example, these lecture notes are just brilliant for getting started with stuff around memory hierarchies: https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...
These are a few tricks that I find useful. What else?
1. Pick a data structure (such as a hash table or LSM-Tree) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and implement a 10x faster or more scalable version for the std lib of your favorite language.
2. Pick a fault model (such as storage faults, network faults, cryptography faults) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and write a fault injection or fuzzing harness to break some of the most respected storage/network/cryptography systems (for examples, see the work done by Remzi and Andrea Arpaci-Dusseau on storage faults, Kyle Kingsbury on Jepsen, and Guido Vranken on Cryptofuzz: https://github.com/guidovranken/cryptofuzz).
3. Pick a software field (such as web applications, mobile applications, native applications, file formats such as Office Open XML, or protocols such as SMTP, MIME, HTTP, QUIC) then read as many CVE reports and bug bounty reports as you can find, and then start participating in bug bounty programs within this field. Pick a target and give yourself a goal, e.g. DoS, RCE or read/write access, and do the work to make it happen. Chain as many steps as you can. Automate and enumerate. You'll find a way in if you keep at it. There's nothing like crafting an exploit to change the way you think about programming.
As you gain experience in data structures, storage/networking/cryptography, and security, you'll find this translates well to most software engineering work. You'll gain a speed/safety/security way of thinking, you'll have fun being curious and learning along the way (and hopefully you'll earn a bounty or two and get some CVEs under your name).
[1] Ecclesiastes is a pretty great book to help figure out for yourself what exactly this is. It's not for nothing that the book uses the same word for "meaningless" or "empty" or "vanity" or "vapor" or "breath" as its refrain. It's also a beautiful book in itself as literature: https://en.wikipedia.org/wiki/Ecclesiastes#Influence_on_West...
You'd be surprised how often variations on a formula are all it takes to destroy security. Security is enumeration at the end of the day. If it takes a paper mill to "count all the ways" then I'm grateful.
There are just too many benefits, not least developer velocity and readability.
Zig is also very approachable to anyone who is familiar with TypeScript, in a way that C isn't. This means more of your team can get excited and get involved, at least in terms of reading the code.
And with Zig, where you have to spend a little extra maintenance time paying for the language being pre 1.0, that's still nothing compared to the upfront and long-lived costs associated with C. You can develop so much faster in Zig compared to C, that you're ahead even pre 1.0.
Here is a talk on using Zig in exactly this kind of environment in production:
https://www.youtube.com/watch?v=124wdTckHNY
It also depends on your project's timeline. If your project will be stable in a year or two, that might well coincide with Zig's stability program.
If Zig's design goals suit your project then it might be worth investing for the long term and gaining deep understanding of the language as it grows.
Raft is essentially Multi-Paxos under the hood, with some overstrict invariants that can impact availability and latency. If you need a consensus algorithm, then it's worth doing a survey of the literature beyond Raft.
Other Multi-Paxos variants are Viewstamped Replication Revisited, which has deterministic leader election for lower latency in the common case and thus no issue of split votes, and which I personally find easier to understand than Raft, also making less demands of the underlying hardware (i.e. VRR can do in-memory leader election, so you're safe in the event of disk failure, corruption, lost writes, and misdirected writes, which Raft does not deal with).
For all these Multi-Paxos variants, you can also look at Flexible Paxos, which is a simple technique for making leader election quorums slightly more expensive (since leader elections are infrequent) in order to make replication quorums much cheaper (since they are the common case critical path). You can use Flexible Paxos to save 20% in the number of hardware nodes for the same common case failure tolerance. Flexible Paxos is a game changer.
I find it hard to believe that you can ship a high quality product without caring about the quality of the components going into it, or that this won't cost you down the line. It only takes a few "good enoughs" in terms of probability theory to end up with a multitude of poor product tail results.
It's also not enough to get the job done, you need to maintain what you ship. You don't need to choose the absolute best tool, but choosing quality tools pays dividends over time.
What it all comes down to at the end of the day is "quality".
You need to have an eye for quality, you need to be able to recognize good design decisions when you see them, appreciate their worth and weigh their potential. You need to be able to make up your mind for yourself.
If you can spot quality in a language, framework (assuming you even need one) or database, then "boring" or "proven" become meaningless phrases.
Quality transcends age and hype.
When you can spot quality far off, you can paddle out to the swell, be there as it breaks, and enjoy a long quality ride that's not cut short because of irrelevance. You won't need to see thousands of surfers on a wave to know it's a quality break.
With technology, the earlier you can spot quality, the sooner you can start riding the wave and benefitting from everything that comes with that (long term experience with that technology, deep understanding, the opportunity to shape or introduce key design decisions at a critical stage, learning from new ideas that can compound earlier into your own output, new connections and marketing opportunities). That's how it always works. There's no point catching a wave because everyone else is riding it. By then, you probably need to be paddling out for the next big swell.
To be clear, this is more than "skating to where you think the puck will be". This is looking for quality and having an eye for it, and being sure of it when you find it.
10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?
I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.