ikhatri

Egocentric means that the sensors frame of reference is the same as the ego (self). The person walking around is wearing the sensors/glasses themselves & their pose is given as “ego pose”.

This dataset is clearly targeted for research on AR/XR/VR applications.

I think I said this already in one of my comments but I'm not a neuroscientist and I don't claim to be. That's why I think it's kind of pointless and silly for me (or any other engineer) to sit here and make arguments about what humans do and don't do in their brains.

IMO it's better for us to focus on what the robots can and cannot do right now, and focus on solving those problems :)

Thanks for the sources though, those papers are definitely neat and I'll be taking a look when I get a chance.

I think you may be over-indexing on the word "selling". I didn't mean it literally as in for sale to you (the customer) directly. That is what Tesla FSD is claiming and I agree with you that we're some indeterminate amount of time away from it.

However Waymo, Cruise and others do exist. If you haven't already, check out JJRicks videos on YouTube. I think you might be changing the number of years in your estimation ;)

Yeah. I mean you're right and wrong at the same time imo. I won't hypothesize about how humans drive. I think for the most part it's a futile exercise and I'll leave that to the people who have better understanding of neuroscience. (I hate when ML/CS people pretend to be experts at everything).

That being said, this idea of a latent space representation of the world is the right tree to be barking up (imo). The problem with "scale it like an LLM" right now is that 3D scene understanding (currently) requires labels. And LLMs scale the way they do because they don't require labels. They structure the problem as next token prediction and can scale up unsupervised (their state space/vocabulary is also much smaller). And without going into too much detail, myself (and others I know in this field) are actively doing research to resolve these issues so perhaps we really will get there someday.

Until then however. Sensors are king, and anyone selling you "self-driving" without them is lying to you :)

As someone who works in the industry (disclaimer these are my own views and don't reflect those of my employer), something about the framing of this article rubs me the wrong way despite the fact that it's mostly on point. Yes it is true that different companies are choosing different sensing solutions based on cost and the ODD in which they must operate. But I think this last sentence just left a sour taste in my mouth "But the verdict is still out as to which is safer".

It is not an open question and I hate it when writers frame it this way. Camera only (specifically, monocular camera only) systems literally cannot be safer than ones with sensor fusion right now. This may change in the future at some point, but it's not a question right now it is a fact.

Setting aside comparisons to humans for a second (will get back to this), monocular cameras can only provide relative depth. You can guess the absolute depth with your neural net but the estimates are pretty garbage. Unfortunately, robots can't work/plan with this input. The way any typical robotics stack works is that it relies on an absolute/measured understanding of the world in order to make its plans.

That isn't to say that one day with sufficiently powerful ML and better representations we would be totally unable to use mono (relative) depth. People argue that humans don't really use our stereoscopic depth past ~10m or so and that's a fair point. But we also don't plan the way robots do. We don't require accurate measurements of distance and size. When you're squeezing your car into a parking spot you don't measure your car and then measure the spot to know if it'll fit. You just know. You just do it. And it's a guesstimate (so sometimes humans make mistakes and we hit stuff). Robots don't work this way (for now), so their sensors cannot work this way either (for now).

You can use the onnx cpu runtime in python or c++ too. It doesn’t have to be rust. And if you want GPU support you can even run models saved in the onnx format on Nvidia GPUs with the TensorRT runtime.

Honestly while ggml is super cool. It started as a hobby project and you probably shouldn’t use it in production. ONNX has been the defacto standard for ML inference for years. What it is missing (compared to ggml) is 2-6bit inference which is helpful for large scale transformers on edge devices (and is what helped ggml gain adoption so fast).

I’m super confused… you’re agreeing that having a radar is superior but you’re disagreeing that it’s going to be a sore spot? How is that possible. Regardless of how you name the configurations (HW4, HW4.5 whatever, the naming is irrelevant) the point is that there will be multiple configurations of sensing suites, some of which will be objectively better than others. That’s the sore spot.

The fact that HW4 and radar are separate configurations by name is not important. HW3 is also included in the mix (you can buy FSD on it) and it has totally different camera placement. So radar aside there’s still a difference in the sensor suite.

People who paid $10k and were promised “FSD” and future hardware upgrades have every right to be pissed off about this.

Yeah that's kind of what I was trying to say. You're right in that we predict the actions of others, but we don't do it in the same way. Even when we execute millimeter level maneuvers, we aren't explicitly measuring anything... Like if you were to ask a driver for instructions on how to repeat that maneuver they wouldn't be able to tell you, they just have a "feel" for it.

Basically humans are really really good at guesstimating with great accuracy (but poor reproducibility) and since we don't use basic measurements in the first place, having better measurement accuracy wouldn't really help us be better drivers on average (it does help for certain scenarios like parking though, where knowing the # of inches remaining to an obstacle can be very useful).

But for everyday driving at speed, we wouldn't even be able to process measurements in real time even if someone was providing them to us. AVs are different and that's basically the gist of what I was trying to say. Because they actually do use, rely on, and process measurements in real time, improving their measurement accuracy (ie. switching from camera based approximate depth, to cm level accurate depth from a LiDAR) can have a meaningful impact on the final performance of the system.