Ep#112: SlipSense and Touch2Trace

Touch is crucial for humans when they want to perform precise tasks to a high degree of reliability, and yet robots almost universally use vision — they’re often completely blind to this whole modality. Two works from Tong Jian, Tao Yu, Saleh Nabi, and Matteo Grimaldi show us how important this sense is. In SlipSense, they look at how to detect objects slipping in grippers using a compact multimodal tactile sensor, which generalizes across unseen objects and to different robots. They then talk about how tactile sensing can be used to enable high real-world performance with a challenging cable-tracing task, where a cable must be fed through the hand with a repeated pinch-and-curl motion. To learn more, watch Episode 112 of RoboPapers today, with Michael Cho and Ruijie He! SlipSense Abstract Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a 32×32 piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal attention with causal temporal prediction at 240 Hz. Experiments on a dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining. Touch2Trace Abstract Dexterous manipulation of deformable objects demands continuous fingertip-level regulation of pressure, friction, and incipient slip. We study one of the most challenging cases: dexterous cable tracing, feeding a cable through the hand with repeated pinch-and-curl motions of the thumb and index finger. We introduce Touch2Trace, a tactile-driven imitation-learning system for this task, and provide, to our knowledge, the first systematic real-world characterization of how encoder pretraining, control rate, temporal context, and spatial resolution each shape policy performance. The winning learning recipe combines a tactile encoder pretrained for a custom 32 x 32 piezoresistive sensor (TacV5) via self-supervised learning with a lightweight transformer policy trained on teleoperated demonstrations via behavior cloning, deployed at 60 Hz on a Tesollo DG-5F hand. Tactile feedback without vision or explicit cable-state estimation significantly improves tracing performance versus a proprioception-only baseline: from 0.2 cm to 20.1 cm mean distance and 0% to 93% success rate, with zero-shot transfer to unseen cables and routing conditions. The results quantify the influence of key parameters in tactile-driven systems for reliable dexterous deformable object manipulation. Learn More SlipSense Project Page: https://tjian-ai.github.io/corl26-slipsense/ SlipSense on ArXiV: https://arxiv.org/abs/2609.15910 Touch2Trace on Project Page: https://grmmtt.github.io/corl26-touch2trace/ Touch2Trace on ArXiV: https://arxiv.org/abs/2609.15921 Transcript Introduction Michael Cho: Hello, everyone. Welcome to another episode of RoboPapers. Today we’re really happy to have the crew from Analog Devices, the Dexterous AI Group: Tao Yu, Tong Jian, Saleh, and Matteo. They’re coauthors of two papers that they’ll walk us through. One is Touch2Trace: Tactile-Driven Imitation Learning for Dexterous Cable Tracing, and the other is SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection. Thank you so much for coming, guys. Maybe we can start with everyone introducing themselves, and then go through the papers. Tao, do you want to start? Tao Yu: Yeah, great. Thanks. My name is Tao Yu. I lead the DexAI team, short for Dexterous AI Group, at Analog Devices. The team was set up about two years ago, and our mission is to unlock humanoid dexterity. That’s how we started looking into tactile sensing, which we believe is one of the missing pieces in dexterous manipulation. With me are the other team members. Maybe Tom can go next. Tong Jian: Yeah, I’m Tom. I’ve been with the DexAI team for two years, working on applied machine learning and, more recently, robot learning. Saleh: Matteo, do you want to go next? Matteo Grimaldi: Hello, everyone. I’m Matteo. I’ve been working with the DexAI team for two years now, mostly on robot learning. Saleh: And I’m Saleh. I also joined Dexterous AI about two years ago. When I met Tao, we were still prototyping a lot of things on the sensor side. The two works you’ll see today are the result of a lot of interesting internal research, both on the hardware side and on the algorithm side. Tao Yu: I also want to add that I personally believe this isn’t just a hardware problem or an algorithm problem. It’s a combination of both. One question we’ll touch on in these papers is what hardware capabilities can unlock learning, and vice versa: what can learning tell us about the hardware requirements? We believe the path forward is a loop: create new hardware, integrate it with a robot, train a model, demonstrate whether the new hardware capability improves performance, and then use what we’ve learned to design the next generation of hardware. Michael Cho: Very cool. I’m really looking forward to these two papers. Which one do you want to start with? Tong Jian: Maybe we can start with SlipSense. Let me share my screen. Michael Cho: Okay, cool. SlipSense: Detecting Slip Quickly and Reliably Tong Jian: Can you see the screen? Great. We’ll be talking about two works accepted at CoRL 2026, both built on our TacV5 tactile sensor. They demonstrate tasks where vision alone isn’t enough. I’ll begin with SlipSense, and Matteo will follow with Touch2Trace. As the title suggests, SlipSense studies slip detection. When an object starts to slip from a robot gripper or fingers, we want a model that can detect it quickly and reliably. Slip detection has a long history. Researchers have been working on it for decades, and the core problem is well understood. But two things remain relatively underexplored. The first is detection latency. A lot of prior work reports accuracy without really characterizing latency. To emphasize, detection latency is not the same as inference speed. It’s the time between when slip actually begins—the slip onset—and when the model first detects it. That’s the interval between these two lines on the plot. A model might run in two milliseconds, but if it needs 100 milliseconds of evidence to make a correct prediction, that’s still 100 milliseconds of detection latency. By the time the model detects slip, the object may already be gone. That’s why we care so much about detection latency. It depends both on the sensor’s capabilities and on the model’s capabilities. RJ He: Can I jump in here? This might be a naive question, but it sounds like you’re saying that different materials and different objects can actually produce different slip-detection latencies. Is that correct? Tong Jian: Yes. It depends on many things: the material, the object, the actual slip speed, how quickly the sensor can capture the relevant slip features, and how quickly the model can interpret those features and make the correct detection. All of those things matter. There’s not just one dominant factor. RJ He: I see. And my assumption is that this is a learned detector, not just a property of the sensor? Tong Jian: Yes, exactly. It’s learned. And that brings us to another important aspect: detection latency naturally trades off against false alarms. It’s a sensitivity-versus-specificity trade-off. If you want to detect slip faster, you may make more mistakes and trigger more false alarms. Prior work rarely evaluates that trade-off directly, and that’s where we started. Before going into the details, let’s look at some demos. Here, SlipSense is running in real time on UMI parallel-jaw grippers. The orange component on the fingertip in all three videos is our TacV5 sensor. It’s a multimodal tactile sensor developed by our team. It measures normal force using a piezoresistive array, as well as high-frequency vibrations using an accelerometer sampled at 8 kHz. This sensor wasn’t originally designed for slip detection. It’s a general-purpose tactile sensor, but we found that it works very well for this task. In all three videos, you can see the model’s predictions updating live. In the left video, when the operator pulls on the object, SlipSense detects the slip almost immediately. In the middle video, it can also detect rotational slip. Michael Cho: I’m curious: physically, where exactly is the three-axis accelerometer? You have this TacV5 device with a 32-by-32 array of taxels. Is the accelerometer at a particular taxel, or somewhere else? I imagine the position would affect the vibrations it senses. Tao Yu: The module is really like a fingertip. The sensor array is on the surface, but inside there’s a PCB that reads out the sensor, and the accelerometer is on that same PCB. Michael Cho: I see. So somewhere inside the module. Tao Yu: Exactly, and rigidly coupled to the surface. Michael Cho: When you’re measuring vibration, you’re using all three axes of the accelerometer? Tong Jian: Yes. We use X, Y, and Z, and we also try to compensate for gravity and motion-induced acceleration so that we can focus on friction-induced vibrations, which are the slip features we’re interested in. Tao Yu: Just to add some background on why we did this research: a lot of the questions we get when talking about our piezoresistive tactile sensor are about its limitations. One drawback of a piezoresistive array is that it only directly measures normal force. People ask, “How do you measure 3D force? How do you measure shear force?” Magnetic, vision-based, and capacitive tactile sensors can offer those capabilities. My next question is always, “Why do you need shear force?” And often the answer is, “I want to detect slip.” We believed that, with a learning-based model, we could do that without directly measuring shear force. That’s why we built SlipSense: to show that we can reliably detect slip even without shear-force measurements. Tong Jian: To summarize the numbers: with SlipSense and the TacV5 sensor together, we achieve both accurate and fast detection. Seventy-six percent of slip events are detected within about 23 milliseconds. That’s fast enough to support downstream tasks, such as slip prevention. We also show that the model, trained entirely on data collected with the UMI gripper, transfers zero-shot to the Tesollo dexterous hand. There’s no retraining and no fine-tuning required for slip detection on the new platform. Achieving this is really a system-level effort across three pillars: the sensor, the data, and the model. The sensor modalities provide the slip features. Precise labels let us train a supervised model. And the model architecture helps address the latency trade-off and generalization. You need all three pieces. RJ He: Just to make sure I understand: when you say zero-shot transfer, that’s specifically for slip detection, right? You’re not claiming transfer of some broader manipulation capability? Tong Jian: Exactly. Just slip detection. That’s important to clarify. The TacV5 Sensor and Tactile Hardware Tong Jian: We’ve already discussed the sensor a little, but I’ll give some more detail. TacV5 is a compact multimodal tactile sensor that packs multiple sensing modalities into a fingertip. For slip detection, we use two of them: a 32-by-32 piezoresistive array running at 240 Hz, with about 1.55 taxels per square millimeter, and an accelerometer running at 8 kHz. The two modalities respond to slip through different physical mechanisms. The piezoresistive array captures pressure redistribution, while the accelerometer captures friction-induced vibrations. The sensor can be integrated across different platforms. Here it’s on the UMI gripper, and here it is on three fingers of the Tesollo hand. RJ He: Since we’re talking about the sensor, maybe we could take a step back. Some of our audience may not know much about tactile sensing. They know that there are different options—piezoresistive, piezoelectric, capacitive, vision-based, and so on. Could you describe the landscape and explain why you chose piezoresistive sensing? Tao Yu: I don’t think we have a plot ready, but I can talk through it. First, there’s an important distinction between piezoresistive and piezoelectric sensing. The latter can also be used as a vibration sensor. We started with piezoresistive sensing because it has two big advantages. The first is scalability. We can make the sensor very large or very small, and we can make the sensing elements quite dense. In fact, this isn’t the densest array we could make. We could make it much denser, but that introduces other challenges, like how to read out so many taxels. It’s also flexible, so it can conform to different surfaces without too much difficulty. There are downsides, of course: temperature drift, other nonidealities, and the fact that it only measures normal force directly. The second advantage is manufacturability. We’re using an existing kind of reel-to-reel manufacturing process. We’ve developed our own custom version of it, but fundamentally it’s a process that can scale. For those two reasons, we think piezoresistive sensing has a lot of potential for commercialization and widespread adoption. We’re also working on solutions to address some of the limitations of the sensing technology itself. Other popular approaches include vision-based tactile sensing. Sharpa’s recent developments, for example, and the papers around the sensors on their Wave hand have attracted a lot of interest. Vision-based tactile sensing has real advantages, including high resolution and the ability to capture fine-grained features. But those sensors are generally bulkier than ours and more expensive. At this point, it’s hard to say whether the field will converge on one sensing technology, or whether we’ll end up with two or more approaches. It’s a little like autonomous driving, where there are still different camps around sensing. RJ He: Awesome. Thank you, I appreciate that overview. Michael Cho: One last question on that. These two papers involve mounting sensors on robotic grippers or a Tesollo hand. But suppose someone wanted to collect human data, using tactile gloves. Am I right that piezoresistive sensing is particularly well suited to that? I imagine vision-based tactile sensing would be much harder to put into a glove. Tao Yu: Yes, the vision-based systems are bulkier. I really like how flexible the piezoresistive sensor is, and that’s a good point. We’re also working on tactile gloves. That’s one of our important next steps, and hopefully you’ll see some new work in the coming months. By the way, if anyone in the audience is coming to IROS, we’ll be demonstrating the next version of this sensor. It will look nicer than this one, with the bare sensor exposed on the surface. RJ He: This episode probably won’t come out before IROS, unfortunately! Saleh: One thing I’d add is that, beyond the hardware capabilities, piezoresistive sensing has another interesting advantage: ease of simulation. We discussed this recently with a university collaborator. Some other tactile sensing technologies, including magnetic and vision-based ones, give you measurements that aren’t directly force readings. With piezoresistive sensing, the relationship to contact force can make it easier to bridge the gap between simulation and reality. That’s useful if you want to build sim-to-real pipelines. Michael Cho: Do you think that’s because piezoresistive sensing is limited to normal force, so it’s relatively straightforward to simulate? Is that the reason? Saleh: There are efforts to add shear-force simulation as well. You’re right that simulating shear force is an interesting challenge in its own right, but researchers are working on it. I think that’s becoming more tractable. SlipSense: Data Collection, Architecture, and Results Tong Jian: Let’s move on to data collection. This is a supervised learning problem with three labels: no contact, contact without slip, and slip. We can’t use episode-level labels, because the transition to slip can happen so quickly. Instead, we label at the level of individual piezoresistive frames. The sensor runs at 240 Hz, so that’s 240 labels per second. The most important labels are near transitions—especially the transition from contact without slip to slip. Those are the moments we need the model to predict correctly. To generate labels, we use a motorized test stand. There’s a crosshead that moves vertically. The object is clamped at one end, and the other end is held against the sensor. When the crosshead moves, it causes the object to slip across the sensor. We read the crosshead position to identify when motion starts, and use that as the slip-onset label. There’s one constraint here: during data collection, the objects need to be effectively nonstretchable, or else the crosshead motion wouldn’t precisely correspond to slip at the sensor. But this constraint only applies to collecting labeled training data; it isn’t a restriction on the inference model. We collected about 1.4 million frames across 37 objects, including cables, connectors, different materials, and tools. We used 28 objects for training and held out the other nine for testing. The model combines convolutional encoders with attention. Its inputs are the piezoresistive array and the accelerometer, along with a short history of observations. We predict the contact state at the latest piezoresistive frame, because this is a real-time detector. Slip may have happened at some point in the history, but we care about the state right now. The history provides context. We use modality-specific encoders: a convolution-based encoder for the piezoresistive maps, and a spectrogram-based encoder for the accelerometer signal. We first convert the accelerometer data into a log-mel spectrogram, then encode it with an audio-spectrogram-based transformer. We also perform pooling across sensors, which makes the architecture agnostic to how many sensors are present. Whether you have one finger, two fingers, or five, the model can handle the inputs. Then we use cross-attention to let the model learn relationships between spatial pressure and vibration. We compared cross-attention to simpler concatenation and late-fusion approaches and found that cross-attention worked best. Finally, a prediction head produces the frame-level contact label. The whole system runs at 240 Hz. The biggest finding, I think, is that the two modalities are not just useful individually; they’re crucially complementary. Looking at our evaluation results, a piezoresistive-only model gets a macro F1 score of roughly 82%. An accelerometer-only model gets roughly 81%. But when we fuse the two, performance rises to around 95% on that comparison. Why? Their failure modes are different. In this first example, the operator repeatedly grasps and releases an object, without any real slip. The piezoresistive-only model can confuse pressure redistribution during grasping with slip and generate false alarms. The accelerometer-only model is sensitive to vibrations, including vibrations caused by moving the gripper even when there’s no friction-induced sliding at the contact surface. So it can make its own mistakes. When we combine them, the pressure array helps confirm contact, and the accelerometer helps identify relative motion at the surface. The two signals compensate for each other’s failure modes, producing a much more robust prediction. RJ He: I was wondering whether you could do an ablation using just the accelerometer, without the tactile pressure array, to isolate the value of the tactile sensor itself. Tong Jian: Yes, that’s exactly the accelerometer-only row in this table. There’s no piezoresistive input. As you can see, it confuses some of those other vibrations with friction-induced slip and makes mistakes. RJ He: I see. Sorry, I missed that. Michael Cho: I’m curious about the observation history in the architecture. How far back does it look? How many frames do you use? Tong Jian: We did an ablation study on the history length. We varied it from six piezoresistive frames up to 144 frames. To our surprise, performance largely saturates at around 60 frames, or 250 milliseconds. After that, accuracy still improves slightly, but not much. I’d expect that number to depend partly on slip speed. If the slipping happens faster, a shorter observation history may be enough. We also study history length in Touch2Trace, and Matteo will discuss that in more detail. Another interesting finding is how much spatial resolution matters. Our default is a 32-by-32 array, but we also tested lower resolutions: 8-by-8 and 16-by-16. The improvements were almost monotonic, from around 83% at 8-by-8 up to around 96% at 32-by-32 in that experiment. Higher resolution captures finer contact patterns that coarser arrays may miss entirely. We may not have hit the ceiling yet. RJ He: It should eventually taper off, though, right? You can’t get that much closer to 100%. Tong Jian: Probably. RJ He: Could you actually build and test a denser sensor? Tao mentioned that was possible. Tao Yu: It’s totally possible. We just haven’t built one denser than the current version yet. SlipSense: Transfer Across Robots and Sensor Units Tong Jian: The last result I want to share is the transfer from UMI to the Tesollo hand. This is perhaps our most surprising result. We train exclusively on data collected with UMI and then deploy the model on the Tesollo hand, which is a very different platform. Despite that, the model still performs reasonably well and detects slip effectively. The video on the left shows SlipSense running on the Tesollo hand. I think two things make this transfer possible. First, the sensor and model work together to learn signatures of slip at the contact surface: the pressure distribution and the friction-induced vibration. Those signatures are intrinsic to the physics of sliding contact, rather than being tied to a particular robot. But the transfer does introduce noise. For example, the Tesollo hand has direct joint actuation, which changes the contact stiffness and produces different vibration signatures in the accelerometer. Those weren’t present in our UMI training data. We address this with accelerometer augmentation, which turns out to be critical. We have an ablation in the paper: without this augmentation, transfer performance drops significantly. On the right, we’ve also added a simple slip-prevention behavior on top of the detector. When SlipSense detects slip, the fingers squeeze to prevent the object from falling. It’s rule-based, but it works pretty well. To wrap up, SlipSense is a joint effort across sensing, data collection, and learning. We have high-density, high-frequency piezoresistive measurements and accelerometer signals; a system for obtaining precise labels; and a model using cross-modal attention and augmentation. Together, they give us a low-latency, accurate, generalizable slip detector. SlipSense is the first capability we’ve demonstrated with TacV5. Matteo will now talk about Touch2Trace, which uses the same sensor for a downstream dexterous manipulation task learned through imitation learning. Matteo Grimaldi: Thank you. Let me share my screen. Michael Cho: While Matteo sets up, I have one more hardware question. Am I right that TacV5 also has a MEMS microphone? Tao Yu: Yes. It’s actually an overprovisioned design, because we weren’t sure which sensors would be most useful. For vibration, we included an acoustic MEMS microphone, a bone-conduction microphone, and an accelerometer. In this study, we only showed the accelerometer, but we’ve heard about other use cases where people would like to use the acoustic microphone to detect particular events. Michael Cho: That’s really interesting. It might complement the accelerometer because it gives you a different signal and frequency profile. Even with an 8 kHz accelerometer, audio is still a different measurement. Tao Yu: Exactly. RJ He: On that note, we’ve been talking about transfer across robots. How transferable are these models across different types of tactile sensors, or even different sensors within the piezoresistive category? One of the challenges with tactile sensing is that, unlike vision, we haven’t converged on a mature sensing technology that everybody uses for data collection. There’s a chicken-and-egg problem: which sensors should people build datasets around, and how do you reuse data collected with different ones? Does your model architecture help with that transfer? Tong Jian: I can start from the experimental side, and Tao can discuss the sensor side. In SlipSense, our data collection spanned four months, which gives us some experience with drift and aging. The UMI and Tesollo experiments also used different physical TacV5 units. So it’s not just cross-platform transfer; it’s also transfer between sensor units. Performance was still pretty good, so we’re not particularly worried about unit-to-unit variation in these experiments. Tao Yu: I think it’s a very good observation. Even within one sensing technology, we’re still exploring how best to represent the data. Right now, we essentially treat the two-dimensional tactile array like an image, as you can see in the paper. But I’m not convinced that’s the right—or the best—representation. Force has a spatial distribution. If you mount the same sensing technology on different embodiments, like a gripper and a finger, or a larger and smaller finger, their 2D arrays could look the same while representing physically different contacts. We’re still trying to understand how best to account for that. At the end of the day, these sensors are designed to be used by a model. As the technology matures and becomes more widely adopted, I think we’ll understand the representation we want, and then design sensors—using the same or different technologies—to produce data compatible with it. RJ He: That’s interesting. In a previous episode, some researchers working on depth were making the argument that image-like representations are beneficial because you can take advantage of learned visual encoders. It will be interesting to see how that trade-off plays out in tactile sensing. Saleh: One quick comment before we move on: I think the community is increasingly recognizing that there are different tactile sensing technologies, different data formats, and different kinds of information. Recognizing that problem is the first step toward solving it. Hopefully we’ll find ways to unify those representations and make tactile sensing a first-class modality. RJ He: Absolutely. All right, let’s let Matteo get to his presentation. Touch2Trace: Dexterous Cable Tracing Without Vision Matteo Grimaldi: Thank you. We’d now like to talk about another paper from this year: Touch2Trace: Tactile-Driven Imitation Learning for Dexterous Cable Tracing. It’s easiest to start with the video so you can understand the task. Cable tracing is a challenging problem in dexterous manipulation. Here, a robotic hand repeatedly performs pinch-and-curl motions to feed a cable through its fingers. It has to do that over and over again. This is difficult for several reasons. First, a cable is a deformable linear object, so it has complex dynamics. Second, we’re doing the task with a multi-degree-of-freedom robotic hand, which hasn’t been explored as much for this particular problem. And third, we don’t use vision. The policy has access only to tactile sensing and proprioception. You can imagine it as a person trying to trace a cable without looking at it. Our central research question is whether we can learn to perform this task without vision or explicit estimation of the cable’s state. Let’s start with the system. We use a Tesollo DG-5F hand, equipped with our TacV5 sensors on the fingertips. For this work, we control eight degrees of freedom across the first three fingers. We also use MANUS MetaGloves for teleoperation, which I’ll explain in a moment. For evaluation, we use an external rotary encoder to measure how far the cable has been traced. That measurement is completely separate from the policy; the robot isn’t using the encoder as input. Data collection is based on teleoperation. Even when a human operator is controlling the robot, the task is quite difficult. In the video, you can see the operator moving their hands with the MANUS gloves and the robot trying to reproduce those movements. We collected just ten minutes of demonstration data, and from that we trained an imitation-learning policy. The policy observes proprioception and tactile sensing. Proprioception captures the eight controlled degrees of freedom. For tactile sensing, we use the pressure maps from the thumb and index fingertips. Those go through a pretrained tactile encoder. We then combine the encoded tactile information with proprioception and train a policy to predict eight-dimensional joint targets. Here you can see the tactile maps during the three main phases of the motion: opening, pinching, and curling. The activation patterns are different in each phase. As Tong mentioned, each fingertip has a 32-by-32 piezoresistive array. That tactile information, combined with proprioception, is enough to learn cable tracing without vision. The central result is shown in this plot. With tactile sensing and proprioception, the policy traces about 20 centimeters of cable, compared with essentially no progress using proprioception alone. The horizontal axis is time, and the vertical axis is distance traced. The videos make the difference particularly clear. With proprioception alone, the hand has no feedback about where the cable is or how it’s moving. It cannot effectively trace the cable. With tactile sensing, it can perform the repeated pinch-and-curl action and keep the cable moving through its fingers. RJ He: I noticed you have another slide about zero-shot transfer to different cables. What kinds of cables and wires does this model work with? And where does it break? Is the important factor diameter, material, weight, flexibility, or something else? Matteo Grimaldi: That’s a good question. We did study generalization, although of course we only tested a subset of the cables that are out there. We trained on one cable, the USB cable shown on the left, and then tested zero-shot on other cables and configurations. You can see that the tracing distance and success rate change across those conditions. Material makes a difference, particularly how deformable or rigid the cable is. Training on more diverse cables would probably make the policy more robust. But we deliberately trained on just one to see how much transfer we could get, especially with only ten minutes of demonstrations. We also chose a cable that was relatively easy to manipulate during teleoperation. Even teleoperating the task was challenging. So there’s definitely a lot of room to improve this result with more data. Michael Cho: Just to clarify, you didn’t use the accelerometer for this work, right? It’s the tactile pressure maps and proprioception? Matteo Grimaldi: Correct. We only used the tactile array and proprioception. Adding the accelerometer is a possible future extension. RJ He: Going back to the range of cables, even theoretically, where do you expect tactile-only control to break down? For example, a very springy cable might be much more difficult to handle. Matteo Grimaldi: I think a large difference in the required grip force between training and deployment could be a problem. Material and stiffness matter, as does how soft the cable is. Diameter also matters. The diameters in our experiments aren’t dramatically different. If you tried a very thin cable, that could be quite difficult, especially because the Tesollo fingertips are larger than human fingertips. The relationship between the finger size and cable diameter becomes important. Those are limitations of the current study. We’d like to extend the range of conditions, especially with more data. Deformable linear objects have become a popular research area in recent years, and there’s still a lot to explore. Saleh: I want to add some context about why we isolated tactile sensing from vision here. We actually have some results using vision as well, but focusing on tactile-only control gave us a way to understand which properties of tactile sensing matter for policy learning. In many visuotactile manipulation papers, vision is so dominant that adding tactile sensing might take success from, say, 70% to 75%. That’s useful, but it doesn’t necessarily tell you which tactile parameters made the difference. We wanted to study that directly. So we investigated things like the tactile observation history, the sensing and control frequency, and other properties of the tactile input. That gave us a clearer picture of what we need from tactile sensing before combining it with vision. There’s another difference from some previous approaches. A lot of work uses tactile measurements to estimate an object’s pose and then feeds that estimate into model-predictive control or some other model-based controller. Here, we learn end to end, from tactile signals directly to actions. We compared that approach with low-dimensional intermediate representations, and the end-to-end model performed better in our experiments. The idea is that a learned policy can take advantage of information in the raw tactile signal that might otherwise be discarded when you compress everything down to an estimated pose. Tao Yu: I’d also add a significant failure mode. Sometimes the cable moves outside the reachable workspace of the fingers. The hand has many degrees of freedom, but we’re only controlling eight here, and there are physical limits to what the fingers can reach. A stiffer cable, for example, might bend away until it’s outside the hand’s reach. That’s one reason Saleh is looking at combining vision and touch: visual feedback can detect those larger unexpected changes and help the robot recover. Saleh: Exactly. In this demonstration, I’m perturbing the cable from side to side, but I’m not pulling it out of the fingers’ workspace. The policy can bring it back toward the center between the fingers, which is an interesting stabilizing behavior. That wasn’t explicitly programmed; it seems to have come from the demonstrations. Another thing worth noticing is the role of the middle finger. The thumb and index do much of the tracing, but the middle finger also moves up and down to adjust the cable’s height. We didn’t handcraft that either. It came from the human demonstration data. And the skill of the demonstrator matters. I tried teleoperating this task myself and wasn’t nearly as good at it! The person who provided the demonstrations was very dexterous, and I think that made a difference. What Makes a Good Tactile Learning Recipe? Matteo Grimaldi: Yes, exactly. We wanted to demonstrate that cable tracing could be done with tactile sensing and proprioception alone, but we also wanted to understand why it works and which design choices matter most. We studied four aspects of the system: encoder pretraining, control frequency, temporal context, and spatial resolution. The numbers we found aren’t a universal recipe for every robot, task, and sensor. But they offer a way to think systematically about designing tactile-learning systems. Let’s look at spatial resolution first. Just as Tong found for SlipSense, increasing the tactile resolution from 8-by-8 toward 32-by-32 improved performance substantially. The plot shows both the success rate at ten centimeters and the distance traced, and both increased with higher resolution. We also varied the control frequency, looking at 15, 30, and 60 Hz. Higher frequency improved performance in our experiments. Temporal context is more interesting, because there’s a trade-off. In our experiments, 15 stacked frames at 60 Hz, or about 250 milliseconds of history, worked best. Increasing the context to 20 frames, about 333 milliseconds, actually reduced performance. So there’s an optimal region for the amount of context, rather than simply ‘more history is better.’ For another task or hardware platform, the optimum might be different. Tong Jian: I want to jump in on that temporal-context result. It’s a remarkable coincidence: for SlipSense, we also found that performance started to saturate around 250 milliseconds of context. For cable tracing, 15 frames at 60 Hz gave the best result. We need to explore whether there’s a deeper reason for that similarity. Michael Cho: I was wondering about that, especially because the second task uses imitation learning from human teleoperation. Could the 250 milliseconds have something to do with the human operator’s reaction time to visual feedback? A very good operator might still be reacting on roughly that timescale. Of course, SlipSense doesn’t involve a human in the same way. There, the physical slip event could be caused by gravity or something else. So perhaps the similarity is a coincidence, but it’s an interesting question. Saleh: That’s a good point. We’re also exploring reinforcement learning, where the policy can learn through trial and error rather than relying only on human demonstrations. Some of these events happen so fast that it may make sense to let the learning algorithm discover responses that are difficult for a human to demonstrate. I also want to emphasize the 60 Hz policy rate. A lot of robot-learning work runs policies at 5, 15, or 30 Hz, partly because those rates are natural when you’re relying on cameras. The sensing modality often ends up dictating the policy frequency. But a tactile sensor provides very fast temporal information. If we design algorithms that can use it, we can get fast, reactive corrections that would be difficult to obtain from vision alone. We were trying to push in that direction with this paper, and it’s an important area for future work. Matteo Grimaldi: Exactly. We tested several policy models and control frequencies. The best performer for this task was a transformer-based policy with a Gaussian mixture model (GMM) head. It’s small and fast enough to operate at 60 Hz, which seems to be important here. We also tried diffusion policies. The diffusion-policy models performed worse than the transformer with a GMM head in these experiments. We’re not saying diffusion policies are generally worse. The point is that for a task requiring very fast, compact control, there can be a significant trade-off between model expressiveness and execution speed. In the appendix, we also include results with vision. We collected another dataset with a camera operating at a lower frequency, around 20 Hz, and showed that vision and tactile sensing can be complementary. The purpose of this paper, though, was to characterize tactile-specific design choices. The other important aspect is pretraining the tactile encoder. With such a small demonstration dataset—just ten minutes—we found it helpful to keep the pretrained encoder frozen during policy learning. Fine-tuning the encoder didn’t improve our results and could easily reduce performance. That suggests that encoder fine-tuning is delicate when you have very little policy-training data. So the full recipe we arrived at combines decisions about tactile representation and pretraining with control frequency, context length, and spatial resolution. Again, this is a recipe for cable tracing on our system, not a prescription for all tactile manipulation tasks. But we believe studying these axes can help other groups design better tactile policies, and we’ve seen related patterns in some other tasks as well. Where Tactile Robot Learning Goes Next Michael Cho: Thank you, Matteo. To bring the two papers together, Tao, how do you see SlipSense and Touch2Trace fitting into the broader direction of the Dexterous AI Group? Tao Yu: I would say these two papers are just the beginning of the journey. They grew out of our curiosity about what new hardware can bring to robot learning. We’ve been doing relatively isolated experiments to understand what the sensor enables. But I think a truly disruptive solution will involve something more like reflexes. Think about how a human grasps a cable. It may start to slip, or I may want to manipulate it. I don’t need to consciously think about every movement. My hands just respond. These behaviors are integrated and almost instinctive. How do we build learning-based systems with those kinds of capabilities? We need multimodal sensing with high spatial and temporal resolution, and architectures that combine those signals. Vision should play a role, too, but without slowing down the fast tactile reactions. There is interesting work along these lines, such as T-Rex, which is starting to think about tactile sensing differently from vision and develop architectures for foundational reactive manipulation capabilities. But how well these systems work will depend on both algorithms and hardware. That’s why we believe in iteratively designing sensors, integrating them with robots, training models, and using the results to inform the next hardware design. RJ He: I’ll add one more question. These are really nice, in-depth studies of specific capabilities like slip detection and cable tracing. But how soon do we get to a more general foundation-model approach for tactile sensing—a single model that supports lots of different tasks? Tao Yu: Right. And I don’t think that should replace a vision-language-action model or a world-action model. It should complement those systems. A hierarchical architecture seems like a natural direction. The question is: what belongs at the lowest level of that hierarchy? Michael Cho: That’s a great way to put it. I personally really like this line of tactile research. I don’t know if you know this, but we were helping organize a robotic origami competition with the Sharpa team. Part of the reason I helped start it is that I really like origami myself. Last year, we invited a grandmaster from the Japanese origami association to CoRL for a demonstration—a kind of human benchmark. I stood there for ten minutes watching this man, probably in his sixties or seventies, making paper cranes while talking to people. He wasn’t even looking at the paper most of the time. His hands were just working. That really struck me. So much of what he was doing seemed to rely on touch. And origami involves lots of occlusion: even when you’re looking, the important contacts aren’t necessarily visible. For very fast reactions, you also can’t depend on vision alone. That’s why I like seeing papers that deliberately remove vision and explore tactile sensing as a modality in its own right. It’s also great to see a hardware-centric research approach, because the sensing hardware matters so much when you’re operating at these control frequencies. Saleh: Origami is also a good example because paper is deformable. That makes the dynamics much more complicated, which connects it to the cable work. Michael Cho: Absolutely. I don’t think you’re going to solve robotic origami reliably with vision alone. You need touch. Papers and Research Worth Following Michael Cho: To wrap up, we always ask our guests this: have you come across any interesting papers recently that you’d like to give a shout-out to? They don’t have to be about tactile sensing—anything in robotics is fine. Matteo Grimaldi: One that comes to mind is the recent NVIDIA work on Sonic. We discussed it in our journal club, and the video was very impressive. Tao Yu: We’ve actually been doing journal clubs internally, so there are quite a few. Maybe Tom, Saleh, and Matteo can each mention some of the work we’ve been looking at. Tong Jian: I’ve also been looking at Robot TDT, particularly its approach to long-term memory. I thought that was really interesting. Saleh: I’d like to mention some work from Yun Zhu Li’s group at Columbia University. They have a series of interesting tactile-sensing and simulation papers, including 3D-ViTac, VT-Refine, and Touch in the Wild, which also used UMI. More recently, there has been work on interactive world simulation. Those are all exciting directions. I’m also interested in the papers we haven’t seen yet: how to use tactile information efficiently in foundation models. T-Rex is interesting in that regard, but I think there are still many approaches to explore. If language provides instructions for robots, and images are the language of vision, you might say that contact is the language of physics. That’s where tactile sensing becomes a first-class modality. For physical AI and dexterous manipulation, I think tactile sensing is essential, and the community is increasingly recognizing that. I’m excited to see how researchers integrate tactile information into foundation models. Data is another problem, though. There’s a real bottleneck in collecting tactile data at scale. We have some ideas about that, and I expect we’ll see more research on it. Tao Yu: Speaking of data, Michael, you mentioned human egocentric data earlier. I think that’s one of the most scalable paths for collecting manipulation data. I recently came across the MINT paper, which I believe was a collaboration involving UJI and a university. That work is interesting because egocentric data processing and annotation pipelines still aren’t very standardized. Every company or research group seems to have its own approach. Understanding what makes human egocentric data useful for robot learning is going to be crucial. Michael Cho: Hopefully we’ll get some of those authors on RoboPapers soon. Thank you all again for coming on the podcast. I found both papers really interesting, especially the way hardware, data, and model architecture are all treated as central parts of the solution. It sounds like you have a lot of follow-up work in progress, and we’re looking forward to seeing it. Please come back and join us again. Tao Yu: Thanks for having us. Saleh: Thank you. Matteo Grimaldi: Thank you very much. Bye-bye. Tong Jian: Thank you. RJ He: Thanks so much, everyone. Michael Cho: Thanks for coming on. See you in the next one. Bye! This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com