jashmota

These are really really interesting thoughts around the remote teleop interface. A few things you mentioned have been on my mind. Teleop UX is underdeveloped and its affects is underestimated, and it could turn out to be a huge thing for us and humanoid companies if autonomy is harder than it seems now. I don't believe construction equipment operation today is optimal. We just go with it as that's what we have. It's far from intuitive and easy for a newbie to start operating to give some context and requires hours of practice. There would be a bit of training to be done for remote teleop as well. Might as well make it easy to interface and improve on the existing experience, which hasn't really evolved for years and decades!

I'd like to have a chat with you if you're up for it: contact at useflywheel dot ai

I completely agree. If you have worked in the industry, I would like to talk to you and get your feedback on what I might be missing on my understanding in requirements in safety layers. Would be great if we can chat on email: contact at useflywheel dot ai

This is a great idea - I would dig into this. I disagree on AI not winning here. Support crew mostly work on eliminating blindspots for operator. Yes you can eliminate blindspots with cameras and a 7inch TFT screen, but those are small, don't work great in direct sunlight. The current solution not being 10x better is why they still fallback to having those crew members. I genuinely believe we will change that.

I totally agree, what I meant was if we only do CANbus retrofit, we would really limit ourselves in how compatible we are. So we have to focus on doing the right kind of retrofit really well. CANbus retrofit is easier and we've done that many times before. Especially because my cofounder Mahimana used to work in self-driving company and worked on this and single-handedly built out their current ML data and eval pipeline.

I agree - the dumping and digging itself (where you move boom, arm, bucket much more than swing/tracks) should be manual. But swinging to the truck and back to pit (pure swinging motion to center around these areas of interest) do not have to be manual. I agree with your and other comments that the transition has to be smooth and that's something we're working on.

1. We're OEM/machine-agnostic 2. Full compatibility with any existing model and not just specific recent models

I definitely agree with you - recreating the scene in teleop is challenging. In excavators, however, it does make it better. An excavator has huge blindspots on the right (due to arm), to the back and sometime near the bucket. Hence the workers who are hired to stand around (banksmen, spotters, signalmen) and signal to the operator.

It's like driving a Ford F150 without backup camera. You'd add the backup camera upfront, and not display the back view at the back window.

It's definitely challenging and we're far from something that's perfect. We're iterating towards something that's better everyday.

I can relate to how you felt. Even though I developed this myself, when I saw it running remotely with no one sitting inside for the first time, it felt magical and weird. This is going to save lives and look magical and that's too much fun to put off and do something else!

The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision.

So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it.

That's on the vision side. We also stream engine audio, and do haptic feedback.

Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?

I would love to explore that - could you help make this happen (I'm guessing you know him)?

Mining has had autonomous haulage (and to some extent excavation) for some years. They have been very tightly coupled to the individual machines, models and the quarry site.

Construction excavation workflows tend to be different and more varied (hence you would see more variations in the machines/attachments available for this industry). And so far there's no solution which just works or is plug and play. This is an attempt to solve that.

And we believe this is great time to do so due to market demands too. No way Gen Z is going to do these jobs when you can make money being delivery partner/influencer, etc. The high-certified, high-skilled workforce is going to shockingly shrink in some years.