September 25 · 1 hr 2 min
Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from.
DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. Bhawna Paliwal, Haritheja Etukuru, Will Liang, and Mahi Shafiullah join us to tell us more.
Watch Episode 107 of RoboPapers, with Chris Paxton and Jiafei Duan, now!
Abstract
How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.
Learn More
Project Page: https://do-as-i-do.com/
ArXiV: https://arxiv.org/abs/2606.19333
Github: https://github.com/malik-group/do-as-i-do
Original thread on X
Transcript
Lightly edited for transcription errors, names, technical terminology, punctuation, and readability. Conversational phrasing is otherwise preserved. Editing done automatically with ChatGPT.
Chris Paxton (00:05.479)
Hey everyone, welcome to another episode of RoboPapers. Today we have some great work from Bhawna, Haritheja, Will, and Mahi: “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos.” Welcome to the show, guys. Could you introduce yourselves quickly before we get started?
Bhawna Paliwal (00:24.977)
Yeah, hi, I’m Bhawna. I’m a PhD student here at UC Berkeley, advised by Jitendra Malik.
Haritheja Etukuru (00:32.692)
I’m Haritheja. I’m a second-year PhD student at Berkeley, also advised by Jitendra.
William Liang (00:37.929)
I’m Will. I’m also a second-year PhD student, advised by Jitendra and Pieter.
Mahi Shafiullah (00:42.606)
Hey everybody, I’m Mahi. I’m a postdoc currently at Berkeley, and I’m also spending some time at Amazon FAR right now.
Chris Paxton (00:50.983)
Awesome. Well, yeah, would you guys like to tell us a little bit about this? What have you guys been working on?
Bhawna Paliwal (01:01.499)
So I’ll just start with a little introduction to the paper. So mainly the motivation behind this work was how can we utilize the massive video data available on internet for training robots. And most of the prior work has been which which has involved learning from human videos, has involved co-training with robot data, which which eventually involves having like hundreds or thousands of hours of teleop data combined with, let’s say, twenty thousand hours of video. So our objective in this paper is to understand. Whether there is some intermediate representation, let’s say in in 3D, where we can extract the right kind of information from human videos and make learning from human videos much more efficient. So essentially every human video where hand-object interaction is going on turns into a separate trajectory on which you can train like a robot model. So prior work has looked into this problem mostly through by collecting their own data and maybe having like specific settings where they’re They have like specific angles and specific robot specific hands hand-object tasks available. But here we look at internet videos in general where there might be a lot of occlusion, motion blur and so on, which are not necessarily recorded for robot learning. Yeah.
Chris Paxton (02:21.331)
Right. Some of the videos here in the highlights look incredibly good, particularly with the dexterous hand. So how does this work?
Bhawna Paliwal (02:35.113)
So I think As like broadly there are like two steps. First we have reconstruction where we reconstruct the object in 3D, as shown here. And here and then we retarget it onto a robot hand, in our case the Sharpa hand. And for reconstruction, like the hardest part was object tracking. That is most of the times when we are looking into hand-object videos, the objects are mostly occluded, especially the tiny objects such as this one. There are like just fewer than a hundred pixels where the object is actually visible. And reconstructing the object becomes hard first. And secondly, tracking the object becomes even harder, even if you know the shape. So, in this work, we first we utilize like all the computer vision progress that has been made recently, especially models like SAM 3D, which can just like take a single image and give you the shape and pose of the object in that particular frame. And then we come up with our own method of how do you utilize the SAM 3D kind of methods to also track the object over time. And the motivation behind was just behind that was just this that SAM 3D has been trained on lots of images and artificially constructed images where like some particular object is blurred out or it has been hidden by some other object. And then so it knows when the how the object looks when it’s like heavily occluded. And then what we do is like we predict the shape and pose at one particular frame and then propagate it to next into like further frames. while keeping the constraint that the shape of the object should not change too much and things like that. So I let Will explain the retargeting part. Yeah. So
Jiafei Duan (04:18.402)
Yeah, I think I have a sorry. I have a question. Yeah, so I actually I read I read the paper quite a few times. and I find it very interesting. So what we are seeing here is an open-loop replay, is that my understanding? Right. So would that be like any limitation that you guys foresee? I mean, maybe I’m jumping ahead of time, but I’m just curious, right? Like for example, certain tasks when you are, you know, doing requires some level of precision or perception, then how will you address that? Or is it just more of the motion that matters at this point?
Bhawna Paliwal (04:51.655)
Yeah, so these trajectories are right now open loop and the how we foresee this should be changed into a perceptive task, which is definitely necessary, is by training taking this reference, domain randomizing the object position, and then training an RL policy, let’s say using point cloud as input. So this will involve just like since you already have the reference trajectory in simulation, you also know the object shape, and then you can like train an RL policy. The object pose is changing and let’s say like even the your environment is changing, lighting and so on, and then you can do all these kind of domain randomization tricks to kind of make it work for one object. And then once you have lots of such policies, you can probably distill it into one big model with let’s say behavior cloning. yeah. Mahi, maybe you would like to add more things.
Mahi Shafiullah (05:46.155)
Yeah, I mean I think one of the things that I think about when it comes to this project is really that we’re exploring how to get robot data from kind of arbitrary internet videos, right? But then like robot data is not the same as a robot policy. Imagine when you’re doing behavior cloning, you still have to do the same t thing like twenty, twenty-five, two hundred times. And that is because just a single trajectory does not really make a policy. It kind of like gives you what to do in a certain situation, but not what to do in small variations of the same situation. And you have to robustify around that trajectory somehow. multiple ways of doing this, right? Like the behavior cloning people likes to learn a map from observation to actions and the reinforcement learning people likes to train a policy that converts that from observation to action, maybe using some as reward, something as a reward. So What Rabna just described is more of an approach of like, if you wanna, you know, derive a reinforcement learning policy out of it, like a trajectory, how could he do it? But overall, I think the twenty thousand foot view that I personally take is that data is data and you want to figure out like how to make data useful. It could just be that we throw everything into a big policy where the images go in as an input and the actions, the robot actions that we’re deriving from this comes out as an output. And once you have enough scale, then that just learns the general policy for articulating a robot. So it’s pretty much open question, like how do you want to convert a robot trajectory into a robot policy? And I don’t want to like say that like a certain direct approach is better than the other. My personal bias at this point is what kind of Bhawna described, which is maybe we do some sort of sim-to-real. And especially for dexterous hands, I feel like sim-to-real is like a pretty good way to go about it. But You know, there are other papers which have kind of converted this into kind of like a two-finger gripper data and there you can actually do behavior cloning and go a long way. So that’s another viable approach, definitely. But yeah, it’s it’s a pretty important distinction to make between a robot policy and robot data, because robot data is just data and not a policy.
Chris Paxton (08:05.587)
So let’s yeah, let’s talk a little bit about how the about how you actually get this data out then. I guess Will, you’re gonna tell us about retargeting.
William Liang (08:10.939)
Yeah. So Bhawna was talking about how we get how we reconstruct the human hand and object from the video. but that’s just the human hand, so how do we get it into the actual robot hand? so retargeting has been pretty extensive extensively studied recently. this is like there’s a lot of works trying to convert human hands to robot hands. Maybe some earlier ones are trying to simply match fingertips. This is like kinematic retargeting where you don’t actually consider how the hand interacts with the object. So this allows for like collisions or like it doesn’t actually hold the object properly. But what we want is something that’s physically realistic. So there’s a more recent line of work on doing dynamics -aware retargeting, where you actually run the retargeting inside physics simulation and simulate all the contacts and collisions and forces between the hand and object. there’s like two lines of work. There’s like there’s a whole series on doing reinforcement learning. So you directly take you could directly take the reconstructed reference and then use that as a dense reward for reinforcement learning. we didn’t go that route because that would train a single policy for each task, and that might be very expensive. like take at least a few hours per trajectory. So what we did instead was sampling-based planning, or this is kind of like MPC or MPP I, where we roll out a bunch of noisy actions and then optimize the action sequence itself instead of optimizing for a policy. So yeah, there’s like a nice visualization here. We can see this is like ever every like control step is like a bunch of noisy actions that gradually converge towards a sequence that properly follows the reconstructed reference while being physically realistic. So this takes on the order of ten to twenty minutes. This is a lot cheaper than training an RL policy for each task. And then after you do this, you can directly replay on the robot, which is the first set of videos.
Chris Paxton (10:17.907)
Can I ask why does it take 10 to 20 minutes? Is that like I would assume that’s like very parallelizable. So let’s just throw more compute at it, it’d be faster. Okay. Yeah, makes sense.
William Liang (10:23.869)
Yeah. Yeah. I mean, one of the big reasons is we still need the simulator to be very precise. so if you so we’re running this within MuJoCo Warp, this is already paralleliz parallelizable. you could run it a lot faster if you like tweak the simulator setup, but in order for it to deal with like a high diversity of objects, including ones that have like very thin walls, you want the simulator to I guess run a bit slower, but be more precise in those contexts.
Mahi Shafiullah (10:59.586)
So here’s actually a good time to talk about a few of the intricacies of the simulator that actually matters here and things that you want to be careful about when you do this. one of the harder things when you’re kind of trying to simulate dexterous hands is that you want to represent the geometry of the hand as well as you can. Because a lot of times, for example, when you’re doing you know, kind of like a humanoid robot locomotion or you know, like quadruped locomotion. One of the things you can do is you can kind of like simplify the collision geometries a little bit so that you can simulate the whole process a lot faster, right? Because the only contact you’re making is with the ground and there is nothing else that you really have to worry too much about. So as long as you avoid self-collision and avoid as long as you avoid like catastrophic contact with the ground, you’re good. But what happens here is for example, you were trying to interact with arbitrary object. Right. And the problem here is that if you don’t simulate the hand well enough or to a high enough fidelity, what might be happening is there might be some geometric weirdness that the simulator can then take advantage of and do basically reward hacking. Right. So if you’re trying to lift up an object, let’s say this cup that I have in my hand, right? It might be that there is like a weird wedge in the fingers which just latches onto the cup that you have and the object motion is just doing just as well in simulation, it’s not gonna pass in the real world. Right. So one of the big things is that given a kind of complex geometric hand, you have to simulate that to the full fidelity, which slows down each simulation step that you can take. And the same thing you have to do for the object itself, right? Like if you have, you know, like simulated weirdness into the object, then it might be that the simulator can be exploited for the sampling process to just generate the same motion that’s not gonna transfer to the real world. so that’s kind of the balance that we have to draw and because this is kind of focusing on like mass application. We don’t want to go over every trajectory and optimize it ha hand like you know, like by hand. We try to pick kind of a conservative set of parameters, which might be a little slow but is much less exploitable compared to what could be more exploitable but much faster. this is not gonna be a problem if you’re trying to imagine like train a single policy for a single task because then you’re not bounded by the then you’re more bounded by the iteration time and not so much of like
Mahi Shafiullah (13:20.895)
You know, like what’s the level of decimation that you’re doing on the object convex hull?
Chris Paxton (13:25.671)
Presumably RL would also equally be able to exploit any issues in your model or physics, right? Yeah.
Mahi Shafiullah (13:28.96)
Exactly. Exactly. So that’s I guess the difference is for RL, you like train a policy, you look at it and you’re like, something’s being exploited and then you go back, edit something in, and then you run the RL policy again. Right? So the downtime that you have Exactly. Human human usually that’s kind of how the RL policy learning step works, where the human just goes and, you know, like looks at it and being like, that doesn’t look right.
Chris Paxton (13:44.507)
So you’re saying the human would solve it because it’s a human looking at each problem. Okay. Yeah, fair. Yeah, yeah.
Mahi Shafiullah (13:54.659)
But when you’re trying to do like six hundred trajectories or a thousand trajectories or million trajectories, you don’t wanna do that. Right. So you kinda wanna pick like a more conservative setting, let’s say, that still ends up with mostly success cases rather than failures. and that’s kind of the trade-off that we see here once again, because we’re trying to do it on a larger scale, rather than like a single policy.
Chris Paxton (14:16.093)
What does that mean on the side of the objects? So like the object models, cause I because I feel like bad object models are one of the lead things that these things often exploit, right?
Mahi Shafiullah (14:25.548)
Yeah. Will do you have any fun stories?
William Liang (14:28.421)
On the object models. So one of the tricks or one of the problems we ran into early on was for cups like the one you just showed. the walls of the cup are pretty thin. SAM 3D often generates thin walls. and that means the hand can just like ramp into the cup and tunnel through within one discretized time step. so the way you can get around this is by decreasing like having more fine-grained time steps, but also by increasing or like inflating the mesh itself. We do both of them here.
Mahi Shafiullah (14:59.918)
Yeah. But again it’s it’s kind of like yeah, go ahead. Yeah.
Jiafei Duan (15:00.42)
I have a question. So yeah, sorry. Yeah, so I think since we are also on the topic, right? So I find something really interesting after reading the paper, which is that the way success retargeting is measured is by ten centimeter means object position and zero point five radius, that’s like twenty twenty-nine degrees. So isn’t that like a huge margin to define as success?
Bhawna Paliwal (15:22.705)
It is a huge margin. there are two considerations. I think for first of all, this is like taken from like the past retargeting literature, so we try to keep the success definition consistent. The second one is for ours specifically we’re measuring it against the reconstructed reference. And the reconstructed reference might be off or noisy at times. So we don’t actually have like a ground-truth reference to measure proper success retargeting against.
Jiafei Duan (15:30.254)
Yeah. Sure, sure, yeah.
Jiafei Duan (15:48.583)
But what about using like contact s stability or object retention or final state as a way of measuring? ‘Cause I mean like it’s a lot of ha in hand manipulation, right, with like how you hold on to this object. So if the object drops then definitely it fails. Okay. Or like it yeah.
William Liang (16:02.568)
Yeah. Well I mean the object dropping would fail the current success criteria as well. I mean if it like fails the position has pretty few
Jiafei Duan (16:08.2)
Yeah, yeah, sure, sure, sure. But like yeah, if it’s loosely grasped yeah. But yeah. No, I I understand. I think in past literature people used the same thing, but I just find it very interesting why they chose this value. Yeah.
William Liang (16:19.539)
No, I think the success criteria could be a lot better because as you as you like hinted at, like this isn’t penalizing loose grasps or like there’s like there’s like some like failure examples down here where it balances the object on fingertips with like the back of the hand. And these would like unless you just look at the videos themselves, these would be counted as successes if it actually follows the reference. I think something you could do is I mean ultimately we care about whether this helps with policy training. So ultimately I guess you would measure the quality of these like success demos as like how well the policy actually does on them. otherwise you could do like some slight domain randomization to counter whatever like edge cases there are with like loose grasps.
Jiafei Duan (17:03.342)
Makes sense, yeah. Yeah, thank yeah, thanks, thanks.
Mahi Shafiullah (17:05.164)
One of the things I think yeah. I think I think one of the things I think this really like brings in is that it’s very hard to tell just from a simulation like if you’re actually following the intention of the motion itself, right? Like it’s kind of like y every every metric that you can define by hand or like by basic code is always gonna be something that is a proxy for the real motion that you’re trying to achieve. Right. Like even for example here, like you look at the knife and the knife is moving, great, but like the real goal is to cut something. The duster, like the real goal is to wipe something. The blender, the real goal is to blend something. Right? So it’s it’s all a little proxy and we’re kind of looking at it at a very like 20,000-foot view. And I think this is kind of an open problem that really requires like a lot more work. And this is something that we’ve been, you know, encountering in other projects as well, where, you know, like you want to define if a human-like action on Dexter’s hand is actually achieving the functionality that intention like initially was intended. And maybe the robot does something that kind of sort of looks like that, but is it really doing it?
Jiafei Duan (18:10.394)
Yeah, I totally agree. I mean like I totally agree with what Mahi was saying. I mean like also I think like the the notion of like you know being like dexterous aware requires us to identify object dynamics sorry, dynamics-aware requires us to identify object dynamics. It means that you can’t just assume that every object that you generate is having the same mass. Right. So I mean there’s a lot of these problems. System ID. Yeah. Yeah, yeah.
Chris Paxton (18:10.835)
So how do you
Mahi Shafiullah (18:29.058)
Mm-hmm. Yeah. Or like same density, I think in this case. I think that’s what we’re kind of doing here. but yeah, I mean I think I think this is a this is a pretty important problem to work on, right? Like it’s kind of like how do you look at interaction between a robot and the world and decide that if that this is actually s successfully satisfying the constraints that you want it to follow? especially automatically.
Chris Paxton (18:51.155)
Yeah, so I
Bhawna Paliwal (18:51.283)
This is probably the one component in pipeline which needs like automation as in when you start from any given video, the only human interference that you need in this pipeline is just like validating whether the final trajectory you got actually completed the task or not. And other components can potentially be scaled with like more videos as in more data and more compute. We did try asking VLMs but they weren’t good enough yet. Yeah, we didn’t say it weren’t good enough yet.
Chris Paxton (19:22.899)
That’s always getting better. So you try with Astra. that’s did you so I guess so you’re saying Will that one of the big goals was to produce data that’s good enough for training. So do you I guess it’s not maybe this is moving a little bit beyond the paper, but do you guys have any any signs of that? Like how much data were you guys able to generate?
William Liang (19:26.835)
Mm-hmm.
William Liang (19:41.694)
Yeah, so we generated five hundred of these like dynamically consistent trajectories. Bhawna can talk a bit more about how we train policies with them.
Bhawna Paliwal (19:48)
As part of this work, we tried training like some residual RL policies on top of these trajectories. One benefit of having these physically consistent trajectories is that the RL part becomes a lot easier. Like instead of h involving let’s say hours of compute, you can just do it in let’s say less than an hour, even with RL, once you have like a good enough physically consistent reference. And after that, the problem just becomes sort of like domain randomization and other things that you want to learn. Let’s say when there is an arm attached to the hand. In some of the cases, the wrist might be moving too fast compared to the acceleration limits on your arm, or compared to let’s say some Kp or Kd parameters in your hand which are not able to follow the reference or some force limits, for instance. These kind of these kind of hardware limitations can be can probably be imposed when you’re trying to learn an RL policy on top of these generated references. And then, as Mahi said, another route is to just take log. of those trajectories and try to distill them into a behavior cloning policy as and not go through a simulated route and treat each of this trajectory as like a teleop reference.
Chris Paxton (21:10.781)
Yeah.
Jiafei Duan (21:11.078)
Well does the pipeline, so like the whole going from video, you know, to the retargeting, how well does that work on like generated videos? Have you guys tried?
Bhawna Paliwal (21:21.48)
Yeah. So we are looking into generated videos as well and generated videos are a lot more scalable than internet videos. In fact one comparison is that if you ca if you like it takes roughly a dollar to find, let’s say, one good video through Gemini when you have to go through a lot more internet videos. And then you can also generate like a very high-resolution video in just like a dollar from a very good video model. So like cost-wise it does make sense to scale this pipeline through generated videos and then you can potentially generate lot more data and also have more control over your data. For instance, if you want to learn a particular task done in a certain way, you can just prompt your way out of generating the right kind of videos and then also get robot data corresponding to those videos. Yeah, this is like one of the examples where the video was
Jiafei Duan (22:22.326)
This is generated video okay. okay.
Bhawna Paliwal (22:28.137)
Then is what’s it? Yeah. So we just like get the segmentation mask and then at least like run the whole reconstruction pipeline through it and then we are able to just get these and object interactions and finally retarget them.
Jiafei Duan (22:47.888)
Do you find like a difference between egocentric versus exocentric? Like which one gives a better intrusive part?
Bhawna Paliwal (22:55.017)
So the If you have ex extrinsics for the camera, as in let’s say you’re recording them with Aria glasses or any other hardware in which you know how the camera is moving, then both of them are pretty equivalent in terms of quality or like the reconstructed trajectories that you can get out of them. In fact, I would say egocentric videos are slightly better because you would also have some idea of the depth. As in if like your hand is moving a certain way, it cannot be it cannot be this much far away from your eyes and like this is how much but when you are in like exocentric world there are a lot more things that can happen. For instance like random camera cuts or and so on. So if I think it does make a lot of sense to like collect egocentric data and scale like the data pipeline so that kind of videos.
Haritheja Etukuru (23:56.845)
So I think one thing that we found important was like we could use internet video, we can use egocentric video, we can use generated video. But there are like certain criteria that we felt like should be met if we wanted some video to be high quality and able to go through our pipeline. Like among The criteria is like you want hand and object interaction of course, but then you also want the hand and the object to be inside of the boundaries of the video. you don’t want like super heavy occlusions. So we found in a lot of the exocentric video data there was It was prone to a lot more occlusion than in like exocentric generated and internet video data. you’d also want objects to be rigid in our case because SAM 3D can only reconstruct rigid objects and it’s also easy us easier for us to simulate the rigid objects. and a lot of the exocentric data did have deformed polar articulated objects. And I think
Bhawna Paliwal (24:58.201)
Actually have some videos for this. I can probably share the screen and how they can walk through this.
Chris Paxton (25:05.959)
Great.
Chris Paxton (25:10.503)
Yeah, ‘cause I guess my my next question was actually going to be about like how many videos how what your accept rate for videos was, how often you could use a video. But so we’ll go. Yeah. Yeah.
Haritheja Etukuru (25:16.406)
Yeah. So then we
Bhawna Paliwal (25:19.773)
We have some numbers that highlight I can probably talk about. just yeah.
Haritheja Etukuru (25:25.086)
So we were curious about this ourselves. So we like took a couple thousand clips from 100 Days of Hands and yeah, we took yeah, two thousand clips from 100 Days of Hands and manually went through them just to see how many would like actually end up going through our pipeline and we found only four percent of them would. so in the end what we ended up doing was like passing thousands of clips through Gemini and filtering that way. But To break down like why some clips failed, I think we have some examples. So in a lot of these clips from like 100 Days of Hands, which is an already curated data
Bhawna Paliwal (26:12.669)
This is an example. how are you getting audio as well?
Jiafei Duan (26:19.728)
Yeah, yeah, I can hear that.
Mahi Shafiullah (26:20.268)
Yeah, I’m getting audio.
Bhawna Paliwal (26:24.965)
Natural flavor. That could mean anything. But there is no meaning. It doesn’t happen. Natural flavor.
Haritheja Etukuru (26:31.722)
I think we can escape and don’t play audio, just go through the slides. So in the first slide, in slide fifty eight
Haritheja Etukuru (26:44.384)
So like in all of these there’s like hands that are being detected in this data set. like even the thumbs up icon counts as like a hand being detected. So of course here there’s no hand like even though there’s a hand there’s no object and there’s no interaction. So that’s one thing we want to look out for. and we want there to be some meaningful task being performed as well.
Haritheja Etukuru (27:10.556)
And yeah, oftentimes the hand and the object can be outside of the boundaries, which is not ideal when you’re trying to reconstruct it, since works best if objects are inside the boundary. I mentioned this earlier that sometimes if the objects are fully occluded then we have trouble in reconstructing it as well.
Chris Paxton (27:33.427)
It’s part of the issue there just that it’s it’s still single-frame reconstruction plus tracking, right? Because presumably like you could reconstruct there is probably an ideal frame where that is visible, right?
Haritheja Etukuru (27:44.586)
Yeah, so we can re reconstruct it for some frames, but then when it is like going frame by frame, and so when the object is invisible and it’s completely lost, it’s hard to get it back.
Bhawna Paliwal (27:51.773)
Yeah. After you’ve stirred a little bit.
Haritheja Etukuru (27:59.979)
Yeah, and a lot of these clips on YouTube have camera cuts and camera motion, which we struggle with if we don’t have the ground-truth intrinsics.
Bhawna Paliwal (28:03.358)
Jamie’s a little technical.
Haritheja Etukuru (28:13.14)
And here are examples of camera cuts.
Bhawna Paliwal (28:16.015)
And then an out.
Chris Paxton (28:17.597)
So are these admitted videos or are these rejected videos that you’re showing us? Okay.
Haritheja Etukuru (28:19.862)
These are all rejected videos and these are like things that we don’t want from our videos.
Chris Paxton (28:26.195)
So I’m curious because like so some of this like w I think that you have this really cool video of the hammering, which is r like there is really good. But the other hand’s holding a nail. And that one looks like that wouldn’t work. Is that right? Or is that so that sounds that seems really hard because he’s holding this small object that is in yeah.
Haritheja Etukuru (28:40.992)
I think for this work we primarily d wanted to deal with the hand that like single hand-object interaction and didn’t really focus on the bimanual tasks and especially objects that are like super small, like a nail we didn’t bother reconstructing. So we wanted the object to be large enough, just because yeah, the smaller objects would not make it through our object tracking button.
Bhawna Paliwal (29:05.001)
I think around the smallest we have is like a pen.
Mahi Shafiullah (29:10.306)
Yeah, I think one of the things philosophically here to understand is that like hand-object interaction itself, like if if you’re c familiar with like computer vision and they’re like the very famous I think it’s John Barron whose talk is it that was like, we’re like trying to solve a bunch of fake problems and these are like the real problems. So for example, if you think about like object detection as being a fake problem and you know, like VQA being a real problem, if you subscribe to that philosophy, then you should think that like also hand-object reconstruction itself is kind of a fake problem. Right? Like the real thing that you want to understand is kind of like a human and world interaction, right? Like how’s a human interacting with the world? What are the things that are changing or what’s really like the underlying mechanism behind it? And that’s a much harder problem, I would definitely say, but that is really like the problem that we eventually want to solve. And object interaction you can think of as kind of like an intermediate checkpoint, trying to get to that. But the thing is it’s kind of like it’s like applies on one way where if you can’t solve the hand-object interaction problem, you definitely cannot solve the problem of like a human and world interaction. And so robot-world interaction is kind of a harder problem on top of that, because you need to figure out what the robot does with the world. so yeah, so that’s kind of like the way we’re thinking about it. and yeah, as you as we say, right, like on the way we make a bunch of simplifying assumptions that you just kind of have to get there. But there are some challenges that we pick, some Battles that we fight, that you know necessarily you won’t have to fight to get there. One of the things, for example, is the problem of depth. we’re assuming that you kinda have like a binocular video and you don’t really have any depth of the world. which is true if you’re trying to just scrape the internet for data, which is not true if you’re going to collect your own data. Right? So if you’re gonna go collect your own data, my recommendation would be just, you know, like instead of RGB camera, just strap on a depth camera. Maybe get a you know, iPhone Pro, like get the LiDAR. Like that itself will solve like so many of those problems, just having reliable depth because monocular depth isn’t quite there as we need it to be. So that’s one of the big issues. you know, if you’re getting an iPhone, maybe you get like camera motion for free. You don’t have to worry about like reconstructing the camera motion from SfM or something. That will get rid of another bunch of problems. If you’re collecting your own data, maybe you’re gonna be able to like frame the video in a way so that your hand doesn’t go out of frame, right? Like things like that I think
Mahi Shafiullah (31:33.322)
Are a lot of things that you can do via operations, which is easier than to like solve it through a research. approach, right? Like by the time you were able to like reconstruct the hand, even though it’s going out of frame, maybe you were just able to get 20 times more data by just strapping a camera on your own head. So this is really for us to understand, right? Like what are the easy problems and what are the hard problems? And when we say that like here are the problems that we see come up very often. We have run some of this analysis on you know like egocentric data sets that people are putting out open source. A lot of them still come on, right? Like for example motion blur is still there, hand Going out of frame, object going out of frame still there. you know, depth not being consistent, tracking not being consistent, all of these problems are still like largely there. So even if you had like this kind of perfect algorithms of the world, unless you’re using an algorithm that is kind of like building an inner model of the world, almost kind of like you know, like a world model with a way of extracting those latents into a simulator, then You know, like just being more conscious of what data you’re collecting is pretty much the best thing you can do. And that’s kind of the story that I want people to take away from it, which is, yeah, don’t don’t go too easy on the data collection side, right? Like be very vigilant, be very aware of it.
Chris Paxton (32:54.119)
So if I’m understanding so part of your argument here is that like I mean with a four percent accept rate, maybe it’s better just to collect data yourself because then you can frame everything correctly and use depth cameras or iPhones, whatever it is, and then you just don’t can circumvent a lot of the issues. Is that
Mahi Shafiullah (33:09.162)
Yeah, pretty much. I mean I think having more control over the data that you’re like getting through. And I think the video model itself is kind of like a pretty interesting direction there as well, right? ‘Cause you can clean up a lot of those issues by just prompting better, right? Like which kind of gives you a pretty nice out. and it’s kind of you can think of that as kind of like a video models is a distillation of all of internet videos. So that’s another direction that I think is kind of an interesting mental model to have, I must say.
Chris Paxton (33:39.155)
So you would say then that the like the future of this sort of set of methods then is like or if you wanted to continue this, you might use like infinite data from just from purely generated, not even do the directly used thing.
Mahi Shafiullah (33:50.348)
Yeah. Yeah. I mean I think I think that’s probably like the cool thing to try.
Haritheja Etukuru (33:57.772)
Yeah, one issue that
Jiafei Duan (33:58.084)
Yeah, I have a question. I think just to add on to this, right? I think it’s very interesting taken and I so totally agree, right? Like always people are like, how much human video do you actually need? I think you just need one human video for every task. Correct. So I mean I mean, I my my my understanding is that the whole idea of this paper is to demonstrate data, how you can generate data. But I wanted to ask, have you guys tried to use the data to see whether how well does that train?
Mahi Shafiullah (34:25.486)
So we have done some like single task policy stuff and you know, like under a reasonable control setting it’s fine. Right. I think the real question is of generalization, right? Like in the thing is you never generalize from a single example. Because a single example is very specific and you just don’t know like what are the covariates that you should care about and what are the covariates that you shouldn’t care about. So the real question is really like how do you generate data at a large enough scale so that you’re able to generalize from this? that’s still an open question. I would I would love to say more about it, but so far, right, it’s kind of like a pretty tough problem because generalization itself is really like the challenge. Here again, like the rollouts that you’re seeing, right? Like the reason why we call them rollouts and bot policies is because they don’t have like that necessary generalization. If you change the scene a little bit, then like this wouldn’t work. And that’s fine, right? Because it’s a rollout and it’s not a policy. But I think this is really like philosophically how I think about it is that it’s very mute to talk about a policy without talking about the level of generalization that you can get out of that policy. Right? Like if you have a policy that just only works in a certain space in a certain under like certain lighting conditions under certain spatial arrangements, right? Like Okay, that’s like pretty close to an open loop rollout, versus a policy that you just can like, you know, bring in your home and like run it on a robot. That’s significantly different than that. But even then it’s still a spectrum because you can find scenes where it doesn’t work. Right. And it’s not that hard to find it. You can still like mess around with things and figure out a configuration where things don’t work because the data doesn’t cover it. so yeah, so I think I think to summarize this is that there is just no magic trick, right? Like the real question is like okay, how do you scale it?
Chris Paxton (36:25.683)
Alright, so I I guess what’s next then? What what if you were if you were gonna continue working on this, is that is what you do? Scale it, use internet use generative video models like use WAN or Seed Dance or something, generate in generate infinite videos.
Mahi Shafiullah (36:42.22)
I think that’d be pretty cool. Yeah. I mean I think I think one of the things that I’ve noticed I played around with this a little bit and I think one of the biggest shortcomings that I’ve seen with this is that video models today are where language models were like two years ago. Like for I don’t know if if either of you have tried interacting with Llama 3, right? Like the base model. And like you can give it like a prompt and you can give it like some descriptions and you’ll be like, can you do this or can you tell me this? And like there is a kind of fifty fifty chance of like this doing what you want it to do versus like another fifty percent of chance is like it’s just gonna keep rambling on and do something like or like say something really weird or nonsensical or non sequitur. video models are in like that era today, right, where controllability is hard, right? Like controllability is practically the key challenge for robotics if you want to get video models to like really scale for the robotic tasks. For Things that are in domain, right? Like it’s super nice. If if you’re trying to do something common, if you’re like, yeah, you know, hammer and nail with the you know, like hammer a nail with a hammer, okay, yeah, that’s probably seen like thousands of videos of this. Okay, great. Like super nice clip comes out straight away. but then if you’re trying to do something a little bit more complicated, right? Like imagine you put that hammer in a way that’s like the handle is like facing away from you and be okay, how do I pick it up? Right. It would do something weird, like, well, the hand will go there and then, you know, it will like cover up the hammer for like one second and then it will like f pick it up and then it’s phased upside down. Right? Like something weird like that. so what I wanna say is that like they work very well in the modes that are covered and may not work as well in the modes that are not covered. With the caveat that I’m mostly talking about, like the open source models that I’ve or like open weight models that I’ve have been able to play around with. You know, like maybe Google internally has a much better model that is much more controllable. I don’t know. so that’s I think one of the things that is pretty like it’s probably gonna be one of the biggest challenge that people who want to use video models for robotics will need to figure out how to do. So that’s one. I think the second problem is kind of like the problem of like physics grounding, right? Like physics grounding could be like on a very simple or a very difficult level. let’s let’s talk about the simple level first. Like even knowing that like okay, an object
Mahi Shafiullah (39:00.482)
Doesn’t change size, the object doesn’t change shape, the object doesn’t, you know, like has a consistency. If you like pick up a hammer and you like rotate it around like 720 degrees, right, you’re gonna see the same thing twice. That’s kind of like an easy level physics consistency that a lot of times these video models tend to fail, in my experience, right? Like in the experiments that I’ve done for this. Another problem that I’ve seen happen quite often is really like kind of understanding the physical interactions between things, right? Let’s just say that like okay again going back to the example of a hammer, you make it like stand upright on its handle, right? And then you let go, right? And you start generating your video from there. If you generate a video like this, you’ll see like some very bizarre interaction with the ground when it drops. Right? Like it would not bounce the same way you imagine a hammer bouncing. It’s kind of like gonna have like constant velocity and it would fall and then it’d like slowly bounce and maybe roll around in a weird way.
Chris Paxton (39:54.269)
That’s video game physics. Yeah.
Mahi Shafiullah (39:56.247)
Yeah, it just it just doesn’t know physics or physical grounding very well. And this is a very simple example of like something dropping on the ground, right? Like this is not even like the hardest thing to imagine. Now imagine like the actual physical grounding that you need to do like context rich tasks, including like tas that you do with a dexterous hand. Right? Like a lot of these things just doesn’t quite have the right level of physical grounding that we have and maybe in like you know, even like the right order of magnitude. And so Right now I think you need like so for example what we do here and do as I do is we do this as a physics-grounding step, right? Like we lift the entire scene into a physical simulator and then we say that like whatever you saw in the video is a rough reference. However, you’re allowed to deviate from that video reference by a certain s number of centimeters. And that is okay as long as you’re roughly doing the same thing and it respects physics. Right? ‘Cause the physics is a hard constraint in the simulator. It’s not optional the way it is in the video model. And so I think this is a direction that I personally feel very bullish in. Exactly, right? Like I think that’s also like an open question of like how do you even for for simpler tasks, for simple atomic things like you know, pick, rotate, move, things like that, it’s pretty easy to quantify, but for something that is much harder to say, right? Like for example, hammering a nail, wiping a board, right? Like things like that are much harder to define. So I think those are the parts
Chris Paxton (40:58.717)
How do you quantify doing the same thing?
Mahi Shafiullah (41:21.068)
That I think we need to focus on more. I think video models are covering like a very important axis of imagination, right? Which is kind of figuring out what to do. But there is another axis of imagination, which is how to do it. And I think that doesn’t get covered very well, right? Like, or how do how do you ground that motion into like a physical world? That doesn’t get covered very well with a video model. And in this way, I kind of think of You know, like simulators that you see with like MuJoCo and MuJoCo Warp and Isaac Sim and Newton and everything, man, you know, like meta’s superdecks, whatever just came out and Genesis. Everything has this kind of like very nice physics grounding properties with the caveat that it cannot or you cannot just immediately lift a scene into a simulator immediately, right? And the video models on the other side has this very nice property which is you take a picture, immediately you have like lifted that world into the simulator. Right, a simulator, I’m calling a world model a simulator here, loosely speaking. and then you can like just start interacting with it. But you don’t have very good physics grounding, right? So we have kind of like these two different directions where we either have like either good physics grounding and not very good, you know, like following of the visual semantics of the world, or we have like a very good visual semantics of the world and really poor following of the physics. and You know, like eventually I think these two paths will merge and we’ll get something that is pretty close to the real world in terms of you know, like in terms of both like physical fidelity as well as the semantics and the visual nature of it. And I think that’s kind of when you can really start doing crazy things with the robots. Because when you have when you want to do something crazy with the robot, you can just roll out five hundred thousand times in your imaginations and be like, okay, this worked well, this did not work well. And then you go and fail in the real world, maybe a few times, but that is also data that can let you improve everything. So, you know, like when I think about what, you know, future robe robots, robots in the wild looks like, I think this is a direction that I find pretty promising that is pretty underexplored right now. people either bet on one or the other, but I think both of them needs to be cultivated a lot more before we can get there.
Chris Paxton (43:38.843)
Makes sense.
Chris Paxton (43:42.365)
Great. Well, I think Haritheja, you were gonna say something earlier and I think we cut you off. Was there something you wanted to add to all everything?
Haritheja Etukuru (43:49.623)
Yeah, I was going to add on like the path for generated videos. I think even if the generative video models get very good at prompt following one failure case we one major failure case that we find is just in the depth estimation. since we’re dealing with monocular videos, if you don’t have very accurate depth estimation, it doesn’t have to be super accurate, but you want the relative depth between the hand and the object to be sort of accurate because if the relative depth is too large then there’s not really much you can do on the reconstruction side. so the good thing with egocentric videos is that you do have ground-truth depth. So if you want to scale that way you’ll usually you have a higher probability I would say of getting hand-object reconstructions that are well aligned.
Chris Paxton (44:42.547)
What what kinds of videos or how how often do you think that shows up in the videos? I guess I guess that’s relatively rare still. Do you think that’d be good for sorry.
Haritheja Etukuru (44:51.168)
I guess if you’re referring to just failure cases in monocular adaptation, I think we find it to be pretty common. The good thing is with our spider modified retargeting, we’re able to deal with a lot of the noise in the hand-object alignment, but we still find that it’s pretty important for the initial frame for the hand-object alignment to be accurate, because we want the grasp the object to be grasped at the initial frame and after that we can sort of deal with it if the hand-object alignment isn’t perfect. but of course as like these monocular depth estimation models continue to improve, this will be less of an issue in the future. And in general, like I think our pipeline is pretty modular. So like as different components of it improve over time, our overall pipeline success rate will also improve.
Chris Paxton (45:49.523)
I’m curious though that video that was just playing a second ago has a deformable object. This was another thing that I wanted to ask a little bit about because like how do you like the physics because we talked a lot about the physics. You’re getting 3D models from from Sam 3D. Like how does how does to how do deformable objects fit into that?
Mahi Shafiullah (46:08.75)
That’s good question. I think for now what we’re doing is we’re kind of like approximating the object. So once it gets there in like a second or Bhawna can like pull up the video a little faster. let’s see. Or we can maybe scroll down, down. yeah, that one, right? The whitebreak the duster video. So we’re just kind of approximating where’s the duster?
Chris Paxton (46:27.731)
It was so. It was the one after this. No.
Haritheja Etukuru (46:30.59)
Okay.
Chris Paxton (46:34.675)
I don’t know. There, I think it was this one. That’s
Haritheja Etukuru (46:35.948)
The second one.
Mahi Shafiullah (46:38.614)
It’s okay. We’ll we’ll we’ll figure it out. so I think one of the yeah. there. Yes. so if you look at the object reconstruction itself, it’s actually not the deformable. You’re kind of reconstructing it as a kind of stick with the blobby bit at the end. And that’s the part that we’re kind of not focusing on too much. one of the things I think on the retargeting side that needs to be done or needs to happen more carefully is thinking not just about the objects and the kinematics and a little bit of the dynamics of the world, but also thinking a level deeper into like the forces that you’re applying with your hand and with the tools that you have with the world. Right? Because like okay, think about this as like a like different levels that you have of this. The level one problem of it is like okay, how do you grab grab hold of an object that has a certain weight and physical properties and move it through space in a certain way? And here what you’re trying to control is really like, you know, position at some Time steps, right? And that could that translates to, for example, like you know, if you discretize it at some level, like you’re trying to control the object’s like position and velocity over time. but what you really do care about is not even like objects position and velocity, but you probably do care about the forces the object is applying into the real world. But also you maybe do not care about the forces in every direction, you only care about the forces that is being applied, like straight onto the surface with the object that you have. Right? So as you start thinking about this problem, it becomes more and more complex and there are layers to it. And you can of course solve this at a different level of abstraction. What we’re showing here is really like solving the problem at the first layer of abstraction and we’re hoping that, you know, given the object is compliant itself, when you move it in a certain height above the surface, then you get the same kind of wiping behavior that you saw in this video. But it may not be true for everything, right? Like for example, for hammer like hammering in a nail. What you really do care about is like the impact that you make at the moment of contact. Right? That is not even that’s that’s pretty much like the third derivative of the position that you’re caring about, not like anything below that. And so eventually Exactly, exactly. Exactly. And so the thing that you want to really think about is how do we involve this higher order terms?
Chris Paxton (48:53.095)
Do need to be pretty precise for that too, actually, right? Otherwise you’re gonna hammer your wall.
Mahi Shafiullah (49:03.828)
Into the process and right now I want to say that this is on the verge of being possible especially given that there are you know like very good I don’t know if you’ve seen like the GPT-6 Astra release videos but one thing I was looking at and was I found it very fascinating. It created kind of like the like one of the very popular buildings in SF I’m blanking out on the name right now. It’s like the Palace of Fine Arts, exactly thank you Will so it just created like a replica of it in Blender.
William Liang (49:25.619)
Palace of Fine Arts.
Mahi Shafiullah (49:32.118)
Right, like I find it super fantastic. but the thing is you know, like this kind of large models, they they probably have a pretty good idea in the scene of like, you know, you’re seeing a little, you know, like planter and you’re looking at a bunch of books in the background and you have like you have a shelf on top, like on top of which everything s sits. You probably or like these big models probably have a pretty good idea of like what those physical parameters look like and if not exactly what they look like, what what kind of ranges would they remain in?
Chris Paxton (49:35.619)
Yeah.
Mahi Shafiullah (50:02.061)
Right, a book’s probably not a hundred kilograms. It’s probably not ten grams, right? Like it has a reasonable range that you can guess from just looking at the scene. And so one of the things I personally predict will happen and kind of become very big is really kind of this real to sim methods, which are using these agents to estimate like what the scene parameters are. just to rebuild the world in a simulator. this has traditionally been something that humans had to hand code, but I think that era is like pretty close to being done as this VLMs kind of get better. and another thing you can think about is that this could really go into kind of like this recursive improvement loop. Where you can be like, here’s the environment, here is where you are, here is a video of a human doing the same thing. Now figure out how to do the same thing. And, you know, you create a simulator, you’d like try to do this task in the simulator. When you’re certain enough, when you’re good enough, you try to do it in the real robot. And, you know, you do it in a few loops. I can totally imagine a world where that the whole process can be done reasonably automatically where you don’t really need to intervene much. just because of how good this whole process is becoming and kind of scary too, right? You’re kind of like automating grad students at the point, which is fun. but, you know, like what’s what’s what’s humanity if not just like finding more and more abstractions to automate away?
Chris Paxton (51:16.147)
Yeah.
Chris Paxton (51:33.757)
Yeah, that’ll be interesting to see. I feel like the there’s definitely like a ton of potential in like closing the loop around experiments like this. Like I was I was thinking that while you’re doing this, like the you do have a physics engine, so you can presumably estimate physics pretty well, at least for the rigid objects. Like the so should be a loop you could build that would let you do this. Makes sense. Cool. Yeah.
Mahi Shafiullah (51:44.109)
Mm-hmm. Yep. Yep.
Mahi Shafiullah (51:52.652)
Yeah, yeah.
Bhawna Paliwal (51:52.86)
It’s it’s also about like sometimes the physics can also help you. For instance if your reconstruction has some normal noise like your hand and object are moving this way, then you know that it ha the object has to be grasped by the hand for the object to move a certain way. And then but then there are small some small ambiguities that cannot be resolved should the object be s grasped this way or this way. And that is why you need like some human interference slash VLM interference to actually fix the reference and then get the right kind of behaviors out of it. Yeah.
Chris Paxton (52:23.987)
Also feels like there’s a good amount of lift that you could gain just by reasoning over multiple frames here. Because you still are like the I think that a lot of those things like lifting, like your phone example, right, Bhawna, like that seems like one that would be disambiguated if you were watching the whole video, right? There’s only so many ways I can hold it if I’m moving it. Right? Or is that okay.
Bhawna Paliwal (52:41.437)
Yeah.
Mahi Shafiullah (52:41.602)
Yeah. Yeah. Yeah. I mean I think I think it really requires kind of like a little bit more I hate to use this word and overuse this word definitely, like agenti capabilities where you can like look at multiple frames and reason that this is the same object being held and moved in different frames. This is something that I think is kind of like appearing more recently. And I’m sure like, you know, like people are training on the like this kind of data more. but I think this is critical. I think this is critical to get this kind of capabilities into the into the models and it’s happening. I think it’s pretty cool. I do wanna say though that there is like one gap which is kind of hard to close, which is one that the robot hands don’t don’t look anything like human hands, right? And s and that’s okay. Like maybe you can in paint it away, it’s fine. but the other thing is that the robot hands don’t move the same way as a human hand. And that’s one of the things that you really have to worry about a lot. let me give you a very simple example, right? Like okay, imagine I have this cup. This cup is super light. As you can see, it’s kind of like wobbling in my hand. And I can close my eyes and I can pick this cup up and it’s fine. Okay? What did I do? My hands have super high precision sensors and it’s super light compared to the sharpa hands that you’re seeing here. the sharpa hands fully made of metal. It probably has, you know, like at least five times the mass that my own hand has. And what comes with the five times of the mass, it comes with the five times the momentum. Right? So if you are trying to make the same motion, when you’re grasping the object, you are kind of transferring five times the energy or like more than five times the energy into the object. And also like if you think about conservation of momentum, like if I just hit it, it’s gonna fly away five times faster because the velocity of this object mass is the same. Right? So a lot of times just kind of like imitating the human blindly is not gonna get you there. Un like if you have a hand that is physically very different, right? If the mass is 5x, that’s not gonna get you to the same place. So you’re gonna have to adapt the behavior a little bit. the other problem is that the robot hands a lot of times are just more powerful than a human hand, right? Like for example, it can do some like crazy things like this, where this the object that I’m holding might just be super heavy, and a human could never do it, but the robot hand’s made out of metal and it does not care.
Mahi Shafiullah (55:03.18)
Right. So in the simulator and in the real world, it is a viable solution that makes the object move in that certain way. and the only complaint that you might have could be of two different aspects. One could be that you’re trying to do actually something that is much longer term and you’re not able to do that with that particular grasp or with that particular hand-object interaction. And the two is just purely aesthetic, right? Like yes the robot does the thing, but it looks ugly. And the thing is as humans, when we look at a robot with a humanoid hand, we expect it to do humanoid things. Right? And if it does something weirdly non-human, then it’s just off and you don’t like that robot. so both of those things kind of require injection of priors that are not purely physically describable, right? Like then you kind of have to for lack of a better word, you have to post-train your hand model to do something more human-like. And those kind of again open questions. We don’t really know what is a good way to do it. How do you even like inject this kind of human preferences into a human hand? I’ve seen some very cool work, I’m forgetting the name of it, but they’re at work which kind of create a biomechanical simulator of a human hand and restrict the robot hand to the motion that it can make w with like a biomechanical hand. And I think that’s kind of cool, right? Like restricting into the human action manifold. but Again, you know, like this the simulators they’re having like very early days, none of this is settled and how do you even like finalize this into like a real human space without having done the kind of long horizon things that we’re doing in the real world? Yeah, like I think I think it’s all like pretty open and up to research and researchers to figure out.
Jiafei Duan (56:44.752)
How how good is Shapahe?
Mahi Shafiullah (56:47.366)
That’s a good question. I think it’s pretty much the best we’ve tried so far. yeah, I t I one of the things that kinda like go you know, like isn’t quite obvious on the surface is that this whole paper would be pretty much impossible to do let’s say like three years ago. Right? Like even if you had like all the model component, if you had like vision part and everything without like a good hand, this just wouldn’t work because
Jiafei Duan (56:52.408)
Yeah.
Mahi Shafiullah (57:13.134)
Two years ago, if you look at the hands, they’re all like significantly bigger than a human hand. And that means that the retargeting step just kind of like trivially fails or does something very different. Right? the Sharpa hand, the biggest advantage that we found with this is that it’s like the shape is significantly closer to a human hand compared to previous hands that we’ve tried. because of which you can actually do human tasks in a very human-like fashion. this wouldn’t this wouldn’t work, this wouldn’t exist in like a few years ago. And in terms of things that we still would like to see, I think one of the things, be very basic things, is just like more human-like compliance, right? again this is a very heavy hand and you ha you have to make a human sized hand means you have to like really gear up the little motors that you put in your hands, which really means that now you have to make the finger and every joint basically very, very stiff. Right. but that kind of like creates a problem of if you’re trying to do something more force-aware, if you’re trying to do something that is a little bit less, you know, like position-controlled, that makes it a hard problem to solve. And so I think overall I’d say that like Sharpa’s practically like the best hand that we’ve tried and we’re very glad that we had it for this project. but overall I would say that like I don’t think this is where the humanoid hand march stops. In fact, I mean I would be very, very surprised if doesn’t like innovate very quickly and basically like creates a newer version of a hand like every three months which has like significantly better properties in every axis. Yeah. Yeah. Yeah. So so from this, like really like the takeaway for a researcher in my opinion should be that like don’t really like don’t try to like build methods that are tied to a particular hand.
Chris Paxton (58:49.479)
Yeah, and this new hands coming out every week, it practically, so
Mahi Shafiullah (59:03.852)
Which which again, like, is a side commentary on like why I think teleoperation is not gonna be a very stable solution for hands yet. Just because you’re gonna have a new hand every three months. And either you’re like collecting a lot of data and throwing it out and refreshing it every three months, or right, you’re stuck with a hand that is kind of like significantly behind state of the art.
Chris Paxton (59:27.443)
Okay. It feels like a lot of your questions here are someone’s gonna fix the pro someone else is gonna fix this, my like someone’s gonna build you a better hand, someone’s gonna GPT-6 Astra agents are gonna
Jiafei Duan (59:29.498)
I wanted to
Mahi Shafiullah (59:33.88)
Sorry.
Mahi Shafiullah (59:38.455)
It’s a it’s a good question and I think the answer to that is that you kinda have to accept the fact that the world is moving at a pace where, you know, like you’re not gonna be on top of the field at everything. So you really have to think about like what you’re the best at and do that instead of like trying to improve on every axis at the same time. so for example, like right, like there was a there was a time, you know, like during the summer where I was like, Hey, okay, the hands are great, but none of the hands have the compliance that I that I want and so like maybe I should get into like building hand hardware.
Chris Paxton (59:46.931)
Yeah, that’s fair.
Mahi Shafiullah (01:00:08.538)
Big problem with that, of course, is that I don’t speak Chinese very well or at all, right? and so I’m there are people who are much better f suited for this problem than I am. But what what I can do is make a lot of noise about the complaints that I have so that people who are building this know what kind of issues that come up. but again, like this is kind of the fun part of doing research is that we’re all in this together, right, as a community and not just kind of like You know, like and I’m and I love like doing open source research. I love being able to talk about the problems that I’m working on and problems I’m facing and why things work and why things don’t work. I think that’s really like the power of having a community with you, that you can you can talk about this, you can share feedback with others and you can get feedback from others of like, hey, I tried your thing and it works here and it doesn’t work here. but yeah, like I don’t think I don’t think anyone’s gonna get too far in isolation. I think now’s the time to collaborate. Yeah.
Chris Paxton (01:01:01.843)
That’s why we’re here, right? Yeah. Jeffy, did you have some question to wrap up?
Jiafei Duan (01:01:08.676)
Yeah. A quick question. So I think I saw this tweet recently from Jitendra, which is that you when Skill released their S1, and it was like, you know, time to use Texas hand. So I think the question I have is that you know, like the way a lot of tools are designed was designed for human hand. But that doesn’t really mean that you know human hand is the most optimum form factor. Because just because the tools are naturally designed for human hand. So I mean like if if you sort of like look a bit, I mean this is more philosophical, like you know, going forward and when you know robots sort of up. human labor, then do we want to design more efficient tools that could potentially use for parallel gripper? I mean of or you know like there should be some form of transferability between parallel grippers to dexterous hands. You know, what what’s your take on this?
Mahi Shafiullah (01:01:55.576)
Does anyone else want to go first? Because I’ve been speaking for a while.
Bhawna Paliwal (01:02:01.162)
I can go. I like generalist take, which is you don’t need to have a hand to hold the tool, your tool becomes your hand and you just have an adapter that connects it to the arm. I feel like that would be like the more robotic solution.
Jiafei Duan (01:02:20.656)
So you don’t you don’t agree with Jitendra’s text just hand narrative?
Bhawna Paliwal (01:02:24.517)
I agree in that you need a hand is the most versatile tool you have to be able to interact with any pre existing tool, but if you can design your own tool then you should design it for the robot. But of course most of the tools we have right now are not designed for the robot. So if you need a something for the robot that can interact with something humans do then then you still need a hand.
Jiafei Duan (01:02:36.676)
Mm.
Chris Paxton (01:02:44.731)
I mean, v it’s very common for industrial robots to have tool changers, for what it’s worth. Like it’s very normal. Yeah.
Jiafei Duan (01:02:47.962)
Yeah, exactly, yeah, yeah. Yeah, and it’s it’s a matter of how how optimized is the you know, usage of tool, right? Like if human with our hands we do this task so badly versus like if you can design a custom tool or custom gripper or whatever, then robots could efficiently do the task, then why not? Correct Yeah. But maybe I don’t know, Mahi might design his own hand, so that might change.
Mahi Shafiullah (01:03:09.914)
Might be fun. Yeah. I think I think in in here, right, like philosophically you also have to think about like what exactly that you’re going for, like what’s the application that you want to see it, right? Like ‘cause you can you can find this, you know, like just go on Twitter and like search for humanoid and then you can find like infinite iterations on this debate on like do we need humanoids or do we not need humanoids? And the group who likes humanoids will say that like you can put it anywhere a human goes and then they’ll just do the human thing. And the group that doesn’t like humanoids be like, but we have such special specialized robots to do such things so much faster, right? Like why use a hand-driven screwdriver if you can just use an electric screwdriver? And why use an electric screwdriver if you can just use your like mold your hand into a screwdriver? Right? And I think at the end of the day the real answer is that there is just gonna be as many robots as you see, like robot varieties as you see like dog breeds. And I really like the exact express example of dog breeds because dogs, right, are bred for human uses. Right? Like you have a different application. You’re kind of like, okay, this dog is a guard dog and this dog is a hunting dog and this dog is, you know, like a herding dog and so on and so forth. And you know, like two hundred years later you just have so many different varieties and breeds of dogs. And some dogs just only have one application, which is to be cute. Right? And they’re they’re all dogs, they’re all valid. None of them are, you know, like better than the other. They’re just like different applications. And so I don’t think there is gonna be like one robot form factor that ends up dominating over every other robot form factor. I think eventually we’re gonna have many humanoids and eventually we’re gonna have many non-humanoids and some of them are gonna be in factories and some of them are gonna be in homes and you know, like and some of them are just gonna be a low
Mahi Shafiullah (01:04:52.13)
Robot ball or like you know the micro duck if you guys have seen that’s also very cute and I really love that robot right it’s it’s probably like not very useful for any chore that you want to do and definitely not very useful in a factory but it’s a cute robot. And so I think overall as far as things go I think I think that’s kind of the answer that like everyone’s correct and I love everyone’s opinions on this because it’s a it’s a different way of thinking about the problem. Personally for me, I think it’s an interesting problem to think about because when you’re working on two-fingered gripper land, and I’ve done this for basically the whole of my PhD, right? Like with the little two-fingered gripper and an iPhone on top, and actually and like the shelf behind Chris, I actually saw a version of it sitting on top. Yep, right over there. Exactly. So, you know, like so I think my opinion on this is that like you can actually afford to ignore a lot of tough questions about contact. with the world when you’re thinking about a problem of like two-finger grippers that you really have to face when you’re thinking about a multi-fingered hand. And will we be using multi-fingered handed robots in 50 years? Maybe it’s just gonna be like, you know, like octopus-like tentacles and everything’s gonna be done with that. Right? But I think whatever insights that we’re gonna gain from working with robots that really make a lot of contacts with the world. Is going to transfer over to whatever robot that we end up using in whatever form factor that we end up doing. Because at the end of the day, the job of a robot is to make contact with the world, right? And I think that’s really the big problem that we’re all trying to solve. Working on hands is just a better playground for us to understand, like, okay, what really happens when you make a lot of contact with the world. Maybe it’s not application-ready, like from day one, but that’s probably also not the goal that I’m like. personally focusing on. I think my goal really is to understand what happens when you have to be precise, when you have to make contact with the world, when you really have to control the output forces of your robot into the world, and do it in like a fine way. Do it fast. Do it, you know, like in summary dexterously. I think that’s kind of the fun research problem to think about. But you know, like if you if you were to ask me like, you know, like hey, okay, like you have to maybe bring up a factory tomorrow.
Mahi Shafiullah (01:07:11.372)
That’s gonna do like t-shirt folding at like 90% success rate in a month. What are you gonna do? Like, yeah, like two-fingered gripper, great, fine, just do it.
Chris Paxton (01:07:19.603)
So you’re saying we should do five-finger hands because they’re more fun.
Mahi Shafiullah (01:07:22.658)
Definitely so. I think I think it’s a lot more fun because the problems are a lot harder, and you know, like the solutions are a lot less clear, I would say. And so I think I’m having a lot more fun working on this. I think it’s yeah, it’s and again, you know, like it’s i if if you don’t have like a if you don’t have a mandate of like, hey, I need to start making money within like the next three months, then I think that’s definitely what you should do. If you have a mandate on what to do, like if you have to make money within the next three months, I would definitely say like don’t do dexterous hands. It’s just
Chris Paxton (01:07:35.919)
It’s a good reason to setting.
Mahi Shafiullah (01:07:52.502)
Yeah. So it’s like nowhere near solved enough that you’re gonna be able to do that.
Chris Paxton (01:07:59.464)
Interesting. Any opinions from from the rest of guys? Bhawna or Haritheja?
Bhawna Paliwal (01:08:07.294)
Yeah, so I think I would also agree with Mahi in the sense that one, they’re more fun. Also I think that the fact that they are all multiple degrees of freedom, they all it also makes the problem more broad in the sense that whenever you have lots of joints with it which are making contact with the world all at the same time, the problem comes back to whatever problem you saw in solving the dexterous hands. So as you move from, let’s say, two-finger grippers to another form factor where you have like multiple grippers Coming out of maybe the same arm, then you are back to the same problem where you’re controlling like lots of multi like degrees of freedom, and you have like you’re making multiple contacts at the same time, you have to manage your own momentum, you have to see how the world is changing as all these multiple finger grippers are changing the world simultaneously. So the problem is basically the same. It’s just like one particular form factor of multiple finger hands, which we are trying to solve by making it human-like.
Haritheja Etukuru (01:09:04.972)
Yeah. I think it’s cool to have like one robot embodiment that’s able to interact with any of the o everyday objects that humans us humans interact with. and in general I just find hands to be cooler. So that’s why I’m interested in working on
Chris Paxton (01:09:27.795)
Well, so thanks, thanks for so usually to wrap up, we ask one last question, which is basically what’s the coolest thing you’ve seen recently? Coolest paper or like dem, whatever you’d like. so anything you’d like to give a shout out to? start with maybe maybe we can start with Will ‘cause it’s been a minute.
William Liang (01:09:48.65)
I think along the veins of dexterous manipulation, we haven’t seen that many we haven’t seen too much evidence of sim-to-real for stuff like Sharpa hand, but I think Tyler has a lot of good work on this. sim-to-real and I probably the second one was called. Sorry Tyler. Yes, that one. I think those are very cool.
Chris Paxton (01:10:12.115)
With yeah, with some tool on the paper on the podcast earlier.
William Liang (01:10:19.751)
I think I was thinking of like play to perfect work, but I was I would say that. apart from sim-to-real I would also like to give a shout out to like another work from Berkeley which is more on like two-finger grippers and just like the like the only open source stack, probably for teleop, which is like the ABC work recently. So that is also pretty cool. Yeah, because you can like build on top of that.
Chris Paxton (01:10:44.883)
Very useful for people too. Very great.
Haritheja Etukuru (01:10:51.126)
Think sim-to-real is promising and in this work we didn’t create any policies but there are like lots of works that are learning to make simulation training for dexterous manipulation more general and also have done sim-to-real so a couple of papers are adept adept from Nvidia and also another paper today from Stanford called Demo Mimic. I think these are both like expanding the capabilities for dexterous hands or
Chris Paxton (01:11:12.603)
Don’t
Haritheja Etukuru (01:11:19.5)
I think I think they’re in an interesting direction.
Mahi Shafiullah (01:11:23.55)
I’ll I’ll give two shout outs. One is for people who has a gazillion dollars and are spending it on and another for people who are, you know, like working on a modest budget like me. so the first one goes to I don’t know if you’ve guys seen the hug from I mean not not not like gazillion dollars, maybe like a bazillion.
Chris Paxton (01:11:37.553)
Amazon FAR is modest now? What’s sorry, they not they’re not giving you enough
Mahi Shafiullah (01:11:45.772)
Yeah. but yeah, so one of the things I would say is have you seen the hug paper from my old lab at NYU? it’s Human Universal Grasping, from Kevin. I think that was a pretty cool work, right? Like I think what they’re doing is using smart glasses and grasping data to learn kind of like a grasping model for arbitrary objects.
Chris Paxton (01:12:05.288)
I think that one will that we did record that episode of them. Actually, I think that will air before this one. So if it hasn’t heard. So yeah. Yeah, yeah.
Mahi Shafiullah (01:12:10.31)
Wonderful. Okay, great. So it’s kind of like a reverse shout out. That’s great. so yeah, so if if people are listening to this one I think they’ll enjoy that one as well. so I I really love that one, right? Like I think conceptually it’s super simple and really like kind of following up on the same kind of like handled or devices for data collection and transferring it into a robot embodiment. I think all of this is just fantastic ideas and we should really like, you know, double triple down on this. So that’s one. And I think the second one I already talked about a little bit, but the newer models And the ability that they have of like creating or recreating the world in code. I think I find that it’s a fantastic ability because this is kind of the ability or the direction that the world models are not covering as much or as well. I think if we need to get robots into the world, we really have to be careful about the physics. And this is to me kind of like the complementary direction of be building a world model that is general and visually introduced. interesting and simulatable. But I think once we cover both of these axes, it should be just fantastic to be able to do simulation of arbitrary scenes. Just drop a robot into a new home, have it wander around and you know like ground it into the real data eventually. So I think I think those are directions that I personally find super cool and super interesting.
Chris Paxton (01:13:31.891)
So that’s your shout out to OpenAI Astra if you haven’t heard of that. So
Mahi Shafiullah (01:13:34.582)
And also the Claude 5.1 I think the five point one model, the Fable 5.1 had like a real home scene recreation stuff. But it’s it’s funny because everyone’s kind of like working on the same ish problems. so you know that these problems are important. But I think as a roboticist as well, I just don’t wanna like handwrite simulated scenes. That’s just not a thing that we will be doing in like two like two months.
Bhawna Paliwal (01:13:39.421)
Baby.
Bhawna Paliwal (01:13:55.457)
And also I guess World Labs as well, like
Mahi Shafiullah (01:14:00.406)
Yeah, yeah, yeah. I think that also came out. That’s also very cool. But I think that’s kind of like falls in my head more like a like a video model but four D or like you know, like spatially camera controllable video model world. So yeah. Yeah, that’s also very cool.
Chris Paxton (01:14:16.891)
Very cool. Yeah. All right. Yeah. Good shout out to OpenAI and Anthropic if you haven’t heard of them. Check those out. Really cool companies. So yeah, thanks. I agree though. Yeah. All right. Awesome. Yeah, well, thank guys for coming on. I think this is fantastic work. It’s been a really interesting discussion. And I guess Geoffrey had to drop early. So but thank you. Thanks again. See you all later.
Bhawna Paliwal (01:14:24.505)
Okay.
Mahi Shafiullah (01:14:37.848)
Yeah.
Bhawna Paliwal (01:14:40.083)
Thank you so much for us. Thanks.
Mahi Shafiullah (01:14:40.397)
Yeah. Absolutely. Good seeing you again, Chris. Alright.
Haritheja Etukuru (01:14:40.886)
Thanks for having us.
Chris Paxton (01:14:44.476)
Good to see you.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
