In this episode of Visions: A Machine Vision and  Automation Solutions Podcast, host Jim Tatum talks with Dr. Stefan Hrabar, Co-Founder/Chief Strategy Officer for Emesent, about autonomous navigation in GPS denied or deficient areas with Hovermap, Emesent's solution that utilizes multiple technologies, including SLAM-based LiDAR. They discuss real-world deployments in mines and urban canyons, sensor challenges (dust, smoke, featureless spaces), and the future of multimodal perception and autonomy.

Visions: A Machine Vision and Automation Solutions Podcast, is the podcast for engineers, designers, integrators, and end users who want to keep an informed eye on the imaging and machine vision industry. Every Tuesday we will explore the latest in imaging trends, developments and solutions. Here you will find interesting, useful insights and observations from expert interviews, solo episodes, even the occasional panel discussion, all of which aim to expand your knowledge on imaging and machine vision.

Related: Drones That See

Transcript

Hello everyone, and welcome to visions. Today we are pleased to welcome Doctor Stefan Hrabar, co-founder and chief strategy officer for Emesent, an Australian robotics and mapping technology company that specializes in autonomous navigation, simultaneous localization and mapping, also known as SLAM, LiDAR mapping and data analytics for challenging environments. Welcome Doctor Hrabar, and thanks very much for joining us on Visions. First, if you would just give us a little background about yourself and what you guys have been doing in this area.

Sure. Thanks, James. Great to be here. Um, so a bit of background by myself. I, did my undergrad in mechanical engineering, in South Africa and then earned a Masters in Mechanical engineering, started getting interested in the connection between computers and the physical world. So I wanted to get into robotics. So I moved over to the US and did a PhD in robotics at USC. And, um, my thesis was vision based navigation in urban environments for UAVs. So I started working on drones back in around 2001 Um, completed the yeah, I know this is sort of in the days before you could just go and buy a drone off the shelf, we had to sort of hand build, uh, convert sort of gas powered helicopters. And, um, yeah, so I started working with putting sort of cameras on drones back then to do stereo vision and optic flow based navigation. And then ended up moving to Australia to join, uh, the CSIRO, which is a national research lab in Australia and continued working on drone autonomy, eventually adding lidar and sometimes radar to drones for doing perception and navigation. Um, and then eventually, yeah, kind of combined the use of lidar with slam on a drone and saw commercial applications. So decided to co-found Emesent and spin out the technology and, uh, yeah, kind of have gone from there eight years ago.

Okay, great. well, it looks like, what we'll be talking today is, working with them. Your work in GPS denied areas. Uh, and this, I'm sure, sounds basic, but just to start off, what exactly is GPS denial? And just how widespread of a problem is it? And how far reaching is it? And why is it so important to you to get around that?

Yeah, I mean, GPS denied or I guess deprived as well as any area where you you don't get a good GPS signal. So, um, the obvious ones are underground in a mine or inside in a building. But then even outdoors if you're, underneath in a forest under below canopy, you know, trees can impact the GPS signal. And in downtown areas, you can get multi-path. So you can get the GPS signal kind of bouncing off between buildings. So you don't get a good GPS signal. It doesn't resolve well, so you don't get a very accurate position, estimation from your GPS. And, these days it's yeah, becoming more and more, I guess, widespread and common in conflict zones as well. So it's not just about, um, things that interfere with GPS naturally, like trees and mountains. It's can be actively jammed or spoofed. So that's very common in different conflicts these days to actively interfere with GPS. So then again, you can't rely on GPS to figure out where you are and how you are moving because GPS kind of provides both position and velocity. And if you don't know how you're moving in terms of velocity, you can't control yourself. If you're if your robot, you can't control your velocity if you don't know how you're moving. So yeah, it becomes very important for, for robotics and autonomy to, to understand where you are and how you're moving.

Okay. Well, um, with that in mind, things like machine vision systems, for example, have become very good at identifying objects, which I assume extrapolates to vision for autonomous vehicles. Um, what's required to move from recognizing things to truly understanding the 3D world around them?

 Um, yeah. I mean, being able to just recognize something in the world. And obviously, you know, when I'm talking about perception, our, what we do and what I'm mostly involved in these days is lidar for real time perception and navigation. Um, at Emesent, we do capture imagery as well, but it's usually as a not used in the real time control. We capture imagery which is then used post-process to add sort of um, RGB data to the lidar point cloud or to detect features in it if you're trying to find things in the environment. Um, but yeah, so to understand the scene, um, I think, and sort of really have an understanding of what's going on around you if you're an autonomous system, it's, it's obviously semantics. So understanding what things are in the environment is that a chair is at a table. Is it a, is it a person? And then it kind of helps to understand relationships between objects and how they typically are related. So, you know, usually you would see a chair near a table or something would be placed on a table. So those relationships and how they interact becomes important. Um, and then the context of, what's happening around you, helps as well. And ultimately even like the intent, if you detect something and then in an environment, like a person, you know, is that person intending to cross the street in front of me or are they on the sidewalk? Um, so, I don't need to worry about, them being in my way. So yeah, there's a lot of things that I think go into full understanding rather than just being able to identify a person in a scene. It's really about, you know, what's their intent, what's the context? Um, how they relate to other objects around in the environment as well.

Okay. Um, well, with, for example, Hovermap, that's more LIDAR than SLAM or a combination thereof.

Yeah. SLAM-based LiDAR. So we use the LiDAR data. Um, in real time. We run the slam algorithm on the LiDAR data. And that gives us um position and orientation without GPS. So we know exactly how our Hovermap device is positioned oriented and moving through the environment. Once you know that you can take all the lidar range measurements that come from the LiDAR device, reproject them into the known coordinate or common coordinate frame and generate a 3D point cloud. So, you know, that's when you have your perception, you've got your 3D point cloud being created in real time, you know, your position and orientation. And then yeah, you can start detecting features in the point cloud. Um, to understand the scene. Um, that's not something we do yet. We're not doing semantic labeling in real time of, of the scene for our kind of navigation. We simply need to know is something an obstacle or not? Can I move through the space or do I need to go around it? That's kind of the level of understanding our autonomy requires to operate in the environments we're operating today.

Okay. Okay. Well, then what are some common environmental challenges you're running into with this?

Um, so for SLAM in particular, um, I mean, the way SLAM works is by recognizing features in the environment again and again over time. So, um, imagine you see a wall in front of you three meters away. A second later, you recognize the same wall that's two meters away. So you know, you've moved one meter in that direction. So what it's doing is identifying thousands of recognizable 3D patches or features in the environment and tracking those over time. So if there's an environment where everything looks the same or there's nothing to track, then, you know, SLAM is going to have challenges. So if you're, for example, in a perfectly smooth steel or concrete lined pipe, everything looks the same, um, around you in that direction. So it's very hard to estimate how far down the, um, the pipe you've moved. Or if you're flying over a very flat terrain, like a, I guess a runway or some big, large parking structure where there's no features, everything looks the same, then you won't be able to estimate your sort of your, your x, y orientation or position. Um, or if you don't see anything at all, if you're flying too high and on a drone and all the terrain is beyond the range of the lidar and you don't see anything, then you've got no way of estimating your, your motion or um, if you've got something that's blocking the LiDAR signal. So if you, if you fly into, um, I guess very thick dust or smoke or very thick, or dense water vapor. So water absorbs LiDAR. Ah. So if you fly into, you know, a very, I guess, misty or cloudy environment where the lidar data is just being absorbed. Then again, you've got no way of perceiving the world around you and running the SLAM algorithm.

That was my next question. How do you how do you get around those challenges like that?

Yeah. I mean, that can be built into the autonomy. Um, so our autonomy adapts to some of those conditions. For example, if we're flying in an underground mine, you do get conditions which create kind of an inversion layer or basically a cloud underground. Um, as you fly into a void that, um, there's a lot, it can be a lot of moisture that's trapped underneath. It gets hot and humid. So that condenses. So imagine the drones flying into a void and starts flying up into basically a cloud. So we, we can sense whether the lidar data is, is, uh, returning. And if we start seeing those points drop off and we're not getting any returns, we can detect that this is a situation like I described. So basically avoid that area, uh, abort the mission and return to home before you get into a situation where you're losing all the points, um, similar for dust, you know, we've, we've got fail safes designed to handle very sort of thick dust environments. If you've flown down a mine tunnel, kicking up dust from the drone's props and then need to return to home and suddenly everything's very dusty and you get blinded, we eventually go into a mode which basically backtracks the way we've come, assuming that there aren't any obstacles because we didn't encounter as long as we can retrace our path. Um, if there wasn't an obstacle on the way in, we assume there's not an obstacle on the way out. So we basically, you know, fly blind to get out of that dusty area, um, and back into the clear space.

Okay. Interesting. Sounds like it kind of begets more, uh, more potential, um, challenges to solve. But that's very interesting how that works. Um, what was the most surprising thing you came across?

Uh, when it comes to moving SLAM technology from research prototypes into commercial challenges, um, I think, yeah, one of the things is just the variety of applications or potential applications there was for SLASM. I mean, we came to SLAM from a robotics perspective. Um, you know, it was developed for autonomous capabilities. if you're sending a robot into an environment it's never seen before, it needs to build a map. and then it needs to know where it is in that map. And it's kind of a chicken and egg problem. If it doesn't have a map, it doesn't know where it is and vice versa. So that's, that's why SLAM was developed is to solve that robotics problem. But then getting into other commercial applications, not just on robots, but just for people to map environments. Yeah. the variety of applications is, immense. I mean, there's so many environments where you might not have GPS or good GPS signal, but still want to map an area or, operate autonomously. So yeah, I think that was one of the, pleasant surprises. And then on the flip side, um, for those like in this, if you're using it as a surveying tool, surveyors are used to existing techniques and technologies that they've known and trusted and they can understand. I mean, if you're using a laser measurement tool or a tape measure or something that you really understand, and then somebody comes along with a SLAM-based system, which kind of looks like black magic. And it's, not always easy to understand. So yeah, definitely some skepticism, um, in the early days, but, that's changed dramatically over the last number of years as SLAM has become, pretty much mainstream and surveyors are relying on SLAM based systems every day. Now they, you know, it's now something that is understood And obviously the quality of of land based solutions have improved. So, um, yeah, but early on it, yeah, people didn't know what SLAM was. So, you know, a lot of education from outside.

Surveyors have certainly come a long way from laying chains and shooting lines, I guess. Yeah. Wow. Interesting. how much of modern autonomy consists of perception challenges versus planning and control challenges? You were just talking about flying backwards. 

Yeah. Um, I think, I mean, there's still definitely challenges in both areas. I'm my background is more on the perception side of things. So I still think there are, more challenges to overcome than if I, I'm not a controls expert, but if I see what, you know, robotic systems are doing these days, you know, you have humanoids doing backflips and gymnastics. And so it seems like, you know, control, is obviously very well advanced and so is perception. But I mean, there still will be environments where it's difficult to perceive. I mean, I mentioned some SLAM challenges. Uh, if you're going into, you know, very dark or dusty environments, there's not always one way of perceiving or a sensor that's going to solve all challenges. So, SLAM is great because it's an active sensor and a lidar is great because it's an active sensor and it illuminates the scene versus cameras, which require external lighting. But then both of those can be disrupted by thick smoke. So then you might want to use radar. But radar has its own challenges. So I think, yeah, from from my understanding, the perception side of things still has challenges. And if you can't perceive the world, then it doesn't matter how good your control is. Um, you know, it doesn't help having great control if you don't know what's around you.

Sure. Okay. if you were designing a perceptions stacked for an autonomous robot from scratch today, what would you think it would look like?

Um, well, I, yeah, I guess I alluded to some of the points before. there's not one sensor that is suitable for every kind of environment. So definitely some multimodal sensing capability having a combination of cameras, LiDAR, radar acoustics. I mean, obviously it really depends on whether you can afford to carry those sensors weight wise and compute wise. If you're operating on a drone, then obviously, you know, weight is very important as so is power consumption. But if you're on an autonomous vehicle, where those are not such an issue, then definitely make sense to use a combination of different sensors. So I think designing that perception stack, you'd first look at, you know, the intended platform and use case. And then where possible, try to use as many different types of sensors as possible to give redundancy and, overcome some of the challenges that one particular sensor would have by itself.

All right. well as LiDAR equipped robots such as Hovermap move beyond mapping into autonomous inspection and decision making, where do you see the boundary between the imaging, the robotics and spatial AI disappearing? Or is there, uh, might not be the right word? Maybe. Maybe meld come together?

Yeah, it's an interesting question. I mean, because sometimes you'd have a sensor or capability that's just being used for the navigation or autonomy. And sometimes you'd have a sensor that's just being used for the inspection task. Um, I mean, often on a, if you're trying to inspect a bridge with a drone, then um, you know, a small lightweight vision autonomy collision avoidance sensor might not give you the resolution that you need to detect very small cracks or corrosion. So those you can't just use one camera system for both applications. Um, likewise with lidar sometimes, you know, LiDAR, like something like off map is great for providing the autonomy. Um, and it gives you, you know, very detailed 3D point cloud, but you might not get the resolution or precision in the point cloud to find, again, small, small cracks or corrosion. so where, where there are applications where the same sensor can be used for both autonomy and the mapping or the inspection task. Obviously, it's a great outcome because then you don't have to double up on sensors. so yeah, we have many customers that, you know, where Hovermap makes sense because it's the data that's generated is good enough for the, inspection or measurement side of things. And it's providing the autonomy. You don't have to double up on sensors. Um, so yeah, if, if again, if you're, if you're designing a solution that can have one sensor that gives you everything you need for autonomy for scene understanding and perception, that's the best outcome.

What's the next step for something like, for example, what's the next great thing you want to accomplish with it?

Um, yeah. I mean, we're always improving our Capabilities from the autonomy side. So being able to handle more challenging types of environments and situations, getting into smaller spaces or, handling some of the environments that I mentioned before. and then, yeah, I mean, it would definitely help, to be able to do more real time scene understanding and semantics, coupled with the autonomy. for example, I mentioned, you know, when using slam, you need to be able to see the environment around you as you're navigating. If you fly over a large lake or over the ocean, you're not going to get a result. So it would make sense if you're flying autonomously from A to B, and you come across a big body of water to then hug the shoreline so that you're not just flying out over the middle. So, those kind of real time decisions in the autonomy to make sure that, the slam solution remains stable and you still achieve the goal or things we build into our autonomy stack as well as integrating, additional kinds of sensing and sensing capability, either for the real time perception or for post-processing overlay, sensor fusion of, you know, if you care about measuring gas levels in an environment or radiation levels, we can bring in data from gas or radiation sensors, display that data real time overlaid on our point cloud. But you know, now we can start doing autonomies based on that data. So if you're trying to find the source of radiation, you know, a radiation source in it, you know, you'd be able to react in real time and kind of create a, I guess, a digital radiation sniffer, which goes and locates the strongest source. Um, yeah. So for us, there's always infinite, possibilities that we could do in terms of improvements and new use cases. But, you know, we generally listen to our customers. Um, understand, you know, what their needs are and then try to work with them to solve problems. we're never going to run out of problems to solve. that's the great thing about working in this space is something new every day.