In this episode of Visions: A Machine Vision and Automation Solutions Podcast, VSD's Head of Content Sharon Spielman interviews Dr. Jimmy Li of Vention’s Physical AI Lab about using real factory vision data, foundation models, and the GRIP modular pipeline to build robots that can see, understand, and learn in messy manufacturing environments.

Visions: A Machine Vision and Automation Solutions Podcast, is the podcast for engineers, designers, integrators, and end users who want to keep an informed eye on the imaging and machine vision industry. Every Tuesday we will explore the latest in imaging trends, developments and solutions. Here you will find interesting, useful insights and observations from expert interviews, solo episodes, even the occasional panel discussion, all of which aim to expand your knowledge on imaging and machine vision.

Transcript

Well, hello and welcome to Visions: A Machine Vision and Automation Solutions Podcast. I'm your host, Jim Tatum, senior editor of Vision Systems design and visions is an endeavor business media production from your friends at Vision Systems Design. Here you'll find the latest on everything from end user machine vision solutions to trends, developments, and perspectives on all things machine vision and imaging. Whether you've been working in the industry for a while or you're just starting to take a closer look at it, this podcast is designed to grow your knowledge and bring greater focus to your understanding of the imaging and machine vision industry. And now on to our show.

Hello. Thank you for joining us today in manufacturing, robots aren't just performing fixed tasks anymore. They're becoming smarter, more adaptable partners on the factory floor. But how do you move from promising AI breakthroughs in the lab to machines that can reliably learn and operate alongside humans in messy, unpredictable environments. I'm Sharon Spielman, the head of content at Vision Systems Design, and today I am excited to chat with Doctor Jimmy Li, who leads Vention's physical AI lab in Montreal. In fact, it was a press release that I received about this new lab that was the impetus for this conversation today. Doctor Li and his team are building exactly that next generation of living robots, ones that see, understand and learn from the real world around them. All right. Thank you so much, Jimmy, for being here today.

Thank you for having me.

Absolutely. All right. So I'm just going to jump right in to start. Can you share how Ventions physical AI lab uses real manufacturing vision data from actual factories to shape and improve your AI systems? And how important is that ongoing feedback loop from production lines.

Yeah. Um, this is one of the amazing things about working at, uh, before I joined Vention, I was working in academia. And when we're doing research, we are always trying to, uh, think about what problems would be useful to solve. And if you go to academic conferences, you'll see people trying to solve problems where a robot is cleaning a kitchen or cleaning a bedroom. But I get to see the applications that are actually useful on real factory floors. And that's quite special. And I think having that to guide the development of physical AI is very important. Right. It makes sure that we solve problems that actually matter in the world. And we don't actually have to be in a factory necessarily to get useful learnings because we often ask our potential clients to send their parts to us, to our research facility. And just with that, we can already learn a lot, right? We see the kinds of tasks that need to be completed by a robot, and that goes a long way towards guiding our research. Having said that, we also are putting the infrastructure that's necessary to collect a robot execution data on the factory floor. But you know, this will require permission from our client, right? Not every factory will allow us to collect the data, but we have the infrastructure in place to collect data either from the factory floor or from our own R&D lab. And this data is very valuable because it will allow us to potentially train foundation models that have a lot of manufacturing knowledge baked in. So that's when we have a new task that we need to solve. It will have a good starting point.

Okay. so you blend classical computer vision with newer Vision Foundation models in your work. In practical terms, how do you decide which tools or techniques to use for different robotic tasks like detecting objects or planning a robot's grasp?

Yeah. The goal typically is to minimize the effort required to onboard a task while also maximizing performance. Um, and we use whatever tool is best suited to achieve those two objectives. And sometimes those two objectives conflict with each other. And I'll give some concrete examples. So recently we did a project where we needed to pick up these cylindrical tubes. That's a situation where a classical method works very, very well. Right. So the way we did that is we first use a very powerful foundation model to estimate the depth map. Given a stereo pair of images. But after that we use a classical method. We use a RANSAC based method to fit a cylinder to the point cloud. And that works very, very well. So in that case, I wouldn't use some fancy new foundation model to detect the pose of that cylinder. But there are cases where a foundation model is very useful for reducing the onboarding effort. So if, for example, if we're picking from a bin of parts that have a very complex shape, right, using a foundation model to detect them and estimate their pose is very powerful. These models can be zero shot. So we just have to prompt it. There's no need to go through a very expensive training process. Right. Um, and they're very robust to lighting conditions because these foundation models are trained on a massive amount of data. But the downside to using foundation models is that sometimes they're quite heavy in terms of processing, right? So before there were foundation models, usually when we need to detect something, we would train a model that is very specific to detecting that part, right. Like whether you call that classical vision or not, that's up to you. But it is like an older style of technology, right? So that style of technology will run very, very fast. They don't have a lot of compute requirements. However, it requires a more expensive onboarding process, right? So in that case, there is kind of this trade off to, uh, when you want to both minimize the cost of onboarding and maximize the execution speed. And so it really depends on the requirements of the application.

Okay. So in that press release, there was something mentioned about the grip pipeline that you've developed, and it sounds like a really powerful modular system. So can you walk us through how it connects the dots from capturing the scene to planning safe, collision free motions, especially in the unpredictable factory types of environments?

Yeah. So the modular pipeline that we have built is an approach to robot control, where we break down a task into many sub tasks. Some notable components include depth estimation where we use a foundation model for that. There is image segmentation. We use also a foundation model called Segment Anything. And then there's pose estimation where we use a foundation model called foundation pose. And that would output object poses. And then we give that to a motion planner to then plan the robot trajectory to go and grab the object. So that works quite well. And we're starting to be able to deploy some robust systems using that kind of grip system. Right. However, this, uh, modular design has some limitations specifically around the level of dexterity that it can achieve. So that's why we're starting to now look at what we call sensor to action policies. So for sensor to action policies, instead of breaking down the perception and motion planning into two separate components, we're starting to train a single model that's kind of maps from perceptual inputs directly to low level robot actions. And to train these methods, we often can use human demonstrations. So the human can demonstrate a very dexterous behavior to the robots. And the robot can learn from that. Um, so like that kind of new approach is a little bit less modular than the one we are deploying now. But I do foresee that for real applications, it's very likely it will be a mix of both approaches where, you know, if you're thinking about like a bin bin picking process, right, we might utilize this sensor to action policy specifically for the most difficult parts, like going down for the final approach to pick the object or the insertion into the receptacle. But, you know, for moving above the bin, you know, maybe moving close to the object, moving near the receptacle, transporting, like all of that can still be taken care of by a more traditional approach.

Okay. Thank you. All right. So manufacturing floors, as we know, can be incredibly busy and unstructured. Um, what are some of the biggest perception challenges that your AI systems need to overcome to work reliably day in and day out.

Yeah. So perception on static images is somewhat easy. Now with all the advancements that we've made in computer vision. What's hard is the interaction between the robot and the physical world. And by that I mean the robot going down to touch a part to grab it, or trying to insert the part into a receptacle that's still very, very hard. If you think about trying to pick out of a deep bin, right? When you get to the bottom of the bin, the bin walls impose a lot of constraints on how your robot arm can move. And when you have interlocking parts that are tightly packed together in a pile, if you pick one part, everything in the bin will start to move. You have to like wedge your robot fingers between the two parts, right? So that's all very, very challenging. And being able to insert accurately. It's also quite challenging. And beyond that, if you want to do an assembly, right, that's also very, very challenging. So yeah, so I think that's where some of the biggest challenges are. And how are you overcoming those challenges? Yeah. So, um, with the modular pipeline, there are certain things that we can do. So for example, if you want to pick out of a deep bin, um, like the pool of candidate grasps become quite important, right? You need to have a good set of grasps that will allow you to grab a part no matter what. Um, and then you have to also in order to place, sometimes you have to pick them in a specific way, right? So, so when you're thinking about, okay, should I pick the part this way? Then you have to kind of think ahead and plan for the placement. And beyond that, there are things we can do on the hardware side to help. So, for example, we can agitate the bin, we can shake the bin when it's difficult to find a pick at the very bottom right. But like I mentioned earlier, we are now starting to investigate the sensor to action policies where the robot can learn from a human right. So, you know, if there's like a part that's wedged at the bottom corner of the bin, maybe like a human can show the robot to tip the part over and then grab it.

All right, so with access to so much varied industrial data and clients across different sectors, how does that scale help you keep your models robust and adaptable to the new challenges?

Yeah. So this is where foundation models really shine because foundation models, they're not trained on a specific task. They're trained across like a wide variety of data, right? So for example, let's say foundation pose, which is a pose estimation method that we use extensively, right? It's not only trained on detecting the pose of a specific object, it's kind of trained on many, many objects. Right? So what that means is that now we can throw it a new object that it has never seen before, and it's able to detect the pose of that object just given the CAD model. Right. And same thing with segmentation. We. So the segmentation method that we use segment anything is really trained on a vast amount of data. So what we found is that our system is actually very robust to lighting changes. For example. Um, like there are some days where we leave a run on overnight and then at some point the janitor just turns off the lights, but our system just keeps on working, right? It doesn't care about those changes because it's trained on so much data.

Okay. So as someone who works closely with engineering teams, deploying embedded machine vision systems, what lessons or advice can you share about moving from promising prototypes to dependable physical AI solutions in the real world?

Mhm. Um, I think that, uh, being open to adapting the existing process a little bit can sometimes go a long way. So I'll give one example. Uh, we did this project where we had to pick apart from a bin and then insert it into a receptacle, which was initially kind of placed inside a very tight enclosure. Right. Um, but if we just move that receptacle out of that enclosure, it suddenly gives the robot so much more room to move around. And that had a big impact on the performance of the system. And it doesn't really prevent the human from operating there. Right. You know, the human can still reach the receptacle, so nothing was lost. It's just making some small tweaks like that. It really helps, right? Um, if you're open to agitating the bin, like I said, it can really open up, uh, the opportunity for the robot to find more grasps. Um, for a gripper finger design I find also, uh, has a big impact, right? If you can have fingers that better conform to the shape of the part that you want to pick to give you a more stable grasp than that really helps with the cycle time. Because if the grasp is not stable, you might have to drop the part and then pick it up. Or, you know, if you're moving around very, very fast and then the part shifts in your fingers, then that will make the placement more difficult. Okay. So yeah, so being able like being open to making those hardware changes to simplify the vision problem is very helpful.

Okay, so looking ahead, what do you see as the most exciting breakthroughs coming in vision, AI and robotics control that could actually really change how factories operate?

Yeah. So we're we are investigating learning from demonstration. And right now, um, so there's been a lot of cool demos built around learning from demonstration. So this is often where like human might teleoperated robot to show you how to do something, but there are a lot of ways to provide, um, demonstrations and teleoperation. But like this method has been shown to work quite well and can enable very dexterous behaviors on robots. But one problem is that this style of methods right now requires a lot of training data and a lot of manual data collection. So finding a way to reduce the amount of manual demonstrations required in order to train a dexterous learning based policy requires some work. And I think if we can make a breakthrough there, then that will open up like that will make deploying these types of policies much more feasible for more manufacturers. And reinforcement learning could potentially play a key role here. Right. So reinforcement learning or RL, it will allow a robot to explore on its own to figure out how to achieve a dexterous behavior. Um, but RL can be difficult to train to bootstrap. Um, because, you know, if you let a robot just start trying random things, then it can take a long time until it converges. So like finding a way to have a system that is well bootstrapped, maybe with learning from demonstration and then later have RL take over to tune it further. I think it's a promising direction. Another thing that my team at venture has been doing is using AI agents to tune a physical AI cell. Um, so earlier we spoke about the modular pipeline. There are many parameters in the modular pipeline. And what we've found is that we can actually have an AI agent automatically tune those parameters, right? So we can have a robot cell run for a while with our modular pipeline. It generates some trace, right? You can look at the cycle time over many, many picks. And then an AI agent like Claude can actually look at that trace and then propose us some specific changes to the configuration or the Python code. And that could actually give us more performance. And another very exciting thing that's happening these days is around the release of GPT six Astra. We're starting to see these large language model or LLM agents being able to directly process the sensor inputs, like images and output actions for the robot to execute. Um, it's not completely clear to me yet how that technology will fit into manufacturing, but I'm very excited to see what will come out in the next few months.

Okay, so for our VSD audience, those are the engineers and the integrators for those who are tuning in and they're building AI vision systems today. What best practices have you found? Really the most critical for delivering reliability, scalability, and confidence in these AI powered robots?

Yeah. So since I joined vision for the past two years, we've had this hypothesis that it is possible to build an AI powered robot cell that is, uh, fully, uh, zero shot. As in no training is required. By utilizing these foundation models. Right. And I think recently we were able to validate that hypothesis. So we have been successful at deploying several customer cells now that utilize foundation models. So there's no training. You just prompt it, right. So the prompt usually would look like a piece of text that describes the thing you want to pick, uh, like a CAD model of the part. Um, and just with that, right, there's no training. We're able to build a cell that can do pick and place. And the really cool thing is that now we're able to do proof of concept in a matter of hours, right? Um, because there's no training, there's no data collection. So we have a grip SDK, right? That we released recently that you can use with the hardware platform to program these pick and place applications. And yeah, and being able to do them so quickly, I think is very, very nice. And we have found that by using these foundation models, we're very robust to environmental conditions. Like I'm not worried about lighting at all. Right. We can deal with transparent objects no problem. We can deal with reflective objects. So that's been working quite well. Um, another thing we found, like I mentioned earlier, taking advantage of hardware design wherever possible to make the vision problem easier. I think it's, uh, very helpful. Right. So, you know, be open to hardware tweaks. And also what we've found recently is that leveraging a genetic AI during the development, debugging and the performance tuning of a system is actually very, very powerful. Right? So now we can, you know, tell Claude, okay, I'm going to run my cell. I'm going to run my robot cell overnight. Keep an eye on it. If the run dies, just restart it. Look at the trace. Fix what you need to fix and just keep tweaking the cell. And we found that's been very helpful in improving the performance of our robot cell systems.

Okay. You know, some companies and some people are very standoffish when it comes to AI and how just pervasive that it is. How do you deal with the pushback that you might receive from customers who are like, no, I don't want I don't want to deal with to I don't want to go into that realm. What do you tell them? How do you deal with that tension?

Yeah. So there are different ways you can use AI, right? Like when we deploy a robot cell right now at a customer site. It's not an AI that's driving that cell, right? Like the way we use AI today is that we use an AI during development to optimize our Python code. And this Python code is human readable. The human can go look at it, make sure it's not doing crazy things. And what gets deployed is the Python code, not the AI agent, right? So the kind of things that it will do is very predictable. Um, yeah. And most of the Vision Foundation models that we are using that we would deploy, they're all for very specific tasks like segmentation or pose estimation, right? You know, they're not generally intelligent creatures that will do very creative things, right. So for me right now, I think that's not a huge concern for me, at least when I see these cells being deployed at customer sites. Okay, great, I appreciate that. Uh, is there anything else that you want to talk about? Uh, like I said, the impetus for this whole thing was the press release I received about the new lab. Do you want to give a quick overview of of your lab? Um, yeah, sure. So in my lab, the size of the number of people that contribute to physical intervention, um, is around fifteen people, right? And there are about eight people that directly report to me. And, you know, my team is a split. Like half of it will work on the modular pipeline as well as doing customer projects. And the other half are more research focused and they're working more on the sensor to action policies. Um, the reinforcement learning and the learning from demonstration. We recently got a lot of new hardware. So my team has access to, uh, five robot cells that we can use to do our R&D work. And we are hiring. Yeah. So I'm constantly trying to find talent, especially if you have a lot of deep experience in a related area, like learning from demonstration or reinforcement learning. You know, I would love to talk to you.

Well, that's a wrap for this episode of Visions produced by Endeavor Business Media, a division of endeavor B2B. Thanks very much for tuning in. If you enjoyed today's show, be sure to subscribe to the podcast and share this episode with a colleague who would find it helpful. Until our next episode, you can find us at vision dash systems dot com or on LinkedIn, Facebook or X for more insights, updates, and breaking news to keep you in the know. Thanks for tuning in. Until next time, stay focused on your visions.