It's easy to underestimate just how
slow digital systems are in many real time scenarios compared to analog systems.
Consider it in the context of camera based self-driving cars, it's tangential to this discussion but it's an easy to visualize metaphor:
- A car traveling 60mph is traveling at 88 feet per second
- Assuming a 60hz camera, there would be a 16.67 ms gap between each frame
- The car is traveling 1.5 feet between each frame interval
- A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry
- High framerate cameras often work around this by staggering/interlacing multiple sensors, but doing this implicitly increases the latency of each frame
- A 120hz camera might deliver double the frames per second, but each frame could be arriving 4 frames late in exchange
- 4 frames would be imperceptible to humans, it would be 3 feet for the car
- You haven't even processed any of these frames yet.
- Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time
- The car can drive as fast as 120mph
All digital sensors implicitly have a sampling frequency, and the fundamental disconnect is always high sampling frequency =/= low latency. People constantly make this mistake over and over, and by the time you notice you are already too deep into development to make a change.
Decreasing latency is expensive, and requires specialized knowledge. Often you get expert software engineers who end up bottlenecked by the hardware limitations they can't even comprehend or the reverse, hardware guys bottlenecked by the software they can't introspect. The latency is only truly understood when you get to integration testing.
Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors. Every part of the chain each costing you more latency. It's like it's own ecosystem where almost everyone writes everything from scratch and doesn't share anything. It's gotten better in recent years though.
Full disclosure: I work in medtech and don't actually deal with cars, but it's a very similar problem space. We often use the same hardware/software cars use for this reason.
Consider it in the context of camera based self-driving cars, it's tangential to this discussion but it's an easy to visualize metaphor:
- A car traveling 60mph is traveling at 88 feet per second
- Assuming a 60hz camera, there would be a 16.67 ms gap between each frame
- The car is traveling 1.5 feet between each frame interval
- A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry
- High framerate cameras often work around this by staggering/interlacing multiple sensors, but doing this implicitly increases the latency of each frame
- A 120hz camera might deliver double the frames per second, but each frame could be arriving 4 frames late in exchange
- 4 frames would be imperceptible to humans, it would be 3 feet for the car
- You haven't even processed any of these frames yet.
- Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time
- The car can drive as fast as 120mph
All digital sensors implicitly have a sampling frequency, and the fundamental disconnect is always high sampling frequency =/= low latency. People constantly make this mistake over and over, and by the time you notice you are already too deep into development to make a change.
Decreasing latency is expensive, and requires specialized knowledge. Often you get expert software engineers who end up bottlenecked by the hardware limitations they can't even comprehend or the reverse, hardware guys bottlenecked by the software they can't introspect. The latency is only truly understood when you get to integration testing.
Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors. Every part of the chain each costing you more latency. It's like it's own ecosystem where almost everyone writes everything from scratch and doesn't share anything. It's gotten better in recent years though.
Full disclosure: I work in medtech and don't actually deal with cars, but it's a very similar problem space. We often use the same hardware/software cars use for this reason.