Ask HN: Datasets for computer desktop usage, AI Agents discussion
Are there any datasets, with any labels or no labels, for people using their desktop computers to do mundane tasks like open applications, browse the web, interact with spreadsheets, etc?
I was thinking this type of desktop "screen record" data could be useful for training AI Agents in the future.
It seems the hardest part of getting AI to "just do work" on my laptop is being able to navigate different applications, copy and paste stuff, trouble shoot weird one-off interactions like software updates moving some buttons/features to different places, etc. Rather than trying to programmatically figure everything out from e.g. an excel API to do spreadsheet work.
I was thinking this type of desktop "screen record" data could be useful for training AI Agents in the future.
It seems the hardest part of getting AI to "just do work" on my laptop is being able to navigate different applications, copy and paste stuff, trouble shoot weird one-off interactions like software updates moving some buttons/features to different places, etc. Rather than trying to programmatically figure everything out from e.g. an excel API to do spreadsheet work.