{"id":664554,"date":"2025-04-23T00:58:15","date_gmt":"2025-04-22T21:58:15","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/system-learns-after-watching-how-to-videos\/"},"modified":"2025-04-23T00:58:15","modified_gmt":"2025-04-22T21:58:15","slug":"system-learns-after-watching-how-to-videos","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/system-learns-after-watching-how-to-videos\/","title":{"rendered":"System learns after watching how-to videos"},"content":{"rendered":"
\n
\n
\n
\n \"Robot
\n Kushal Kedia (left), a doctoral student in the field of computer science<\/a>, and Prithwish Dan are members of the development team behind RHyME, a system that allows robots to learn tasks by watching a single how-to video. Credit: Louis DiPietro
\n <\/figcaption><\/figure>\n<\/p><\/div>\n<\/div>\n

Cornell University researchers have developed a new robotic framework powered by artificial intelligence\u2014called RHyME (Retrieval for Hybrid Imitation under Mismatched Execution)\u2014that allows robots to learn tasks by watching a single how-to video.<\/p>\n

Robots can be finicky learners. Historically, they’ve required precise, step-by-step directions to complete basic tasks, and tend to call it quits when things go off-script<\/a>, like after dropping a tool or losing a screw. RHyME, however, could fast-track the development and deployment of robotic systems by significantly reducing the time, energy and money needed to train them, the researchers said.<\/p>\n

“One of the annoying things about working with robots is collecting so much data on the robot doing different tasks,” said Kushal Kedia, a doctoral student in the field of computer science. “That’s not how humans do tasks. We look at other people as inspiration.”<\/p>\n

Kedia will present a paper titled “One-Shot Imitation under Mismatched Execution,” in May at the Institute of Electrical and Electronics Engineers’ International Conference on Robotics and Automation, in Atlanta. The work is also available<\/a> on the arXiv<\/i> preprint server.<\/p>\n

Home robot assistants are still a long way off because they lack the wits to navigate the physical world and its countless contingencies. To get robots up to speed, researchers like Kedia are training them with what amounts to how-to videos\u2014human demonstrations of various tasks in a lab setting. The hope of this approach, a branch of machine learning called “imitation learning,” is that robots will learn a sequence of tasks faster and be able to adapt to real-world environments.<\/p>\n

\n
\n
\n
\n
\n
\n
\n
Credit: Cornell University<\/figcaption><\/figure>\n

“Our work is like translating French to English\u2014we’re translating any given task from human to robot,” said senior author Sanjiban Choudhury, assistant professor of computer science.<\/p>\n

This translation task still faces a broader challenge, however: Humans move too fluidly for a robot to track and mimic, and training robots with video requires gobs of it. Further, video demonstrations\u2014of, say, picking up a napkin or stacking dinner plates\u2014must be performed slowly and flawlessly, since any mismatch in actions between the video and the robot has historically spelled doom for robot learning, the researchers said.<\/p>\n

<\/p>\n

“If a human moves in a way that’s any different from how a robot moves, the method immedia<\/a>tely falls apart,” Choudhury said. “Our thinking was, ‘Can we find a principled way to deal with this mismatch between how humans and robots do tasks?'”<\/p>\n

RHyME is the team’s answer\u2014a scalable approach that makes robots less finicky and more adaptive. It supercharges a robotic system to use its own memory and connect the dots when performing tasks it has viewed only once by drawing on videos it has seen. For example, a RHyME-equipped robot shown a video of a human fetching a mug from the counter and placing it in a nearby sink will comb its bank of videos and draw inspiration from similar actions\u2014like grasping a cup and lowering a utensil.<\/p>\n

RHyME paves the way for robots to learn multiple-step sequences while significantly lowering the amount of robot data needed for training, the researchers said. RHyME requires just 30 minutes of robot data; in a lab setting, robots trained using the system achieved a more than 50% increase in task success compared to previous methods, the researchers said.<\/p>\n

“This work is a departure from how robots are programmed today. The status quo of programming robots is thousands of hours of tele-operation to teach the robot how to do tasks. That’s just impossible,” Choudhury said. “With RHyME, we’re moving away from that and learning to train robots in a more scalable way.”<\/p>\n

Along with Kedia and Choudhury, the paper’s authors are Prithwish Dan, Angela Chao, and Maximus Pace.<\/p>\n

\n

More information:<\/strong>
\n\t\t\t\t\t\t\t\t\t\t\t\tKushal Kedia et al, One-Shot Imitation under Mismatched Execution, arXiv<\/i> (2024). DOI: 10.48550\/arxiv.2409.06615<\/a><\/p>\n

\n\t\t\t\t\t\t\t\t\t\t\t\t\tJournal information:<\/strong>
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tarXiv<\/cite>
\n
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/svg>
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n<\/p><\/div>\n
\n Provided by
\n Cornell University
\n \t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/svg>
\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/a><\/p><\/div>\n

<\/p>\n

\n

\n Citation<\/strong>:
\n Robot see, robot do: System learns after watching how-to videos (2025, April 22)
\n retrieved 22 April 2025
\n from https:\/\/techxplore.com\/news<\/a>\/2025-04-robot-videos.html\n <\/p>\n

\n This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
\n part may be reproduced without the written permission. The content is provided for information purposes only.\n <\/p>\n<\/p><\/div>\n<\/p><\/div>\n