Ep#107: Do As I Do: Dexterous Manipulation Data from Everyday Human Videos

Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from. DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. Bhawna Paliwal, Haritheja Etukuru, Will Liang, and Mahi Shafiullah join us to tell us more. Watch Episode 107 of RoboPapers, with Chris Paxton and Jiafei Duan, now! Abstract How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation. Learn More Project Page: https://do-as-i-do.com/ ArXiV: https://arxiv.org/abs/2606.19333 Github: https://github.com/malik-group/do-as-i-do Original thread on X Transcript Lightly edited for transcription errors, names, technical terminology, punctuation, and readability. Conversational phrasing is otherwise preserved. Editing done automatically with ChatGPT. Chris Paxton (00:05.479) Hey everyone, welcome to another episode of RoboPapers. Today we have some great work from Bhawna, Haritheja, Will, and Mahi: “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos.” Welcome to the show, guys. Could you introduce yourselves quickly before we get started? Bhawna Paliwal (00:24.977) Yeah, hi, I’m Bhawna. I’m a PhD student here at UC Berkeley, advised by Jitendra Malik. Haritheja Etukuru (00:32.692) I’m Haritheja. I’m a second-year PhD student at Berkeley, also advised by Jitendra. William Liang (00:37.929) I’m Will. I’m also a second-year PhD student, advised by Jitendra and Pieter. Mahi Shafiullah (00:42.606) Hey everybody, I’m Mahi. I’m a postdoc currently at Berkeley, and I’m also spending some time at Amazon FAR right now. Chris Paxton (00:50.983) Awesome. Well, yeah, would you guys like to tell us a little bit about this? What have you guys been working on? Bhawna Paliwal (01:01.499) So I’ll just start with a little introduction to the paper. So mainly the motivation behind this work was how can we utilize the massive video data available on internet for training robots. And most of the prior work has been which which has involved learning from human videos, has involved co-training with robot data, which which eventually involves having like hundreds or thousands of hours of teleop data combined with, let’s say, twenty thousand hours of video. So our objective in this paper is to understand. Whether there is some intermediate representation, let’s say in in 3D, where we can extract the right kind of information from human videos and make learning from human videos much more efficient. So essentially every human video where hand-object interaction is going on turns into a separate trajectory on which you can train like a robot model. So prior work has looked into this problem mostly through by collecting their own data and maybe having like specific settings where they’re They have like specific angles and specific robot specific hands hand-object tasks available. But here we look at internet videos in general where there might be a lot of occlusion, motion blur and so on, which are not necessarily recorded for robot learning. Yeah. Chris Paxton (02:21.331) Right. Some of the videos here in the highlights look incredibly good, particularly with the dexterous hand. So how does this work? Bhawna Paliwal (02:35.113) So I think As like broadly there are like two steps. First we have reconstruction where we reconstruct the object in 3D, as shown here. And here and then we retarget it onto a robot hand, in our case the Sharpa hand. And for reconstruction, like the hardest part was object tracking. That is most of the times when we are looking into hand-object videos, the objects are mostly occluded, especially the tiny objects such as this one. There are like just fewer than a hundred pixels where the object is actually visible. And reconstructing the object becomes hard first. And secondly, tracking the object becomes even harder, even if you know the shape. So, in this work, we first we utilize like all the computer vision progress that has been made recently, especially models like SAM 3D, which can just like take a single image and give you the shape and pose of the object in that particular frame. And then we come up with our own method of how do you utilize the SAM 3D kind of methods to also track the object over time. And the motivation behind was just behind that was just this that SAM 3D has been trained on lots of images and artificially constructed images where like some particular object is blurred out or it has been hidden by some other object. And then so it knows when the how the object looks when it’s like heavily occluded. And then what we do is like we predict the shape and pose at one particular frame and then propagate it to next into like further frames. while keeping the constraint that the shape of the object should not change too much and things like that. So I let Will explain the retargeting part. Yeah. So Jiafei Duan (04:18.402) Yeah, I think I have a sorry. I have a question. Yeah, so I actually I read I read the paper quite a few times. and I find it very interesting. So what we are seeing here is an open-loop replay, is that my understanding? Right. So would that be like any limitation that you guys foresee? I mean, maybe I’m jumping ahead of time, but I’m just curious, right? Like for example, certain tasks when you are, you know, doing requires some level of precision or perception, then how will you address that? Or is it just more of the motion that matters at this point? Bhawna Paliwal (04:51.655) Yeah, so these trajectories are right now open loop and the how we foresee this should be changed into a perceptive task, which is definitely necessary, is by training taking this reference, domain randomizing the object position, and then training an RL policy, let’s say using point cloud as input. So this will involve just like since you already have the reference trajectory in simulation, you also know the object shape, and then you can like train an RL policy. The object pose is changing and let’s say like even the your environment is changing, lighting and so on, and then you can do all these kind of domain randomization tricks to kind of make it work for one object. And then once you have lots of such policies, you can probably distill it into one big model with let’s say behavior cloning. yeah. Mahi, maybe you would like to add more things. Mahi Shafiullah (05:46.155) Yeah, I mean I think one of the things that I think about when it comes to this project is really that we’re exploring how to get robot data from kind of arbitrary i… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com