In this episode of Visions: A Machine Vision and Automation Solutions Podcast, host Jim Tatum interviews Christopher Shelton about Penn State Behrend's VAR lab project to create a research-grade digital twin of the Liberty Bell using structured light, laser metrology, LIDAR, photogrammetry and edge computing. The team captured hundreds of gigabytes of data under tight time constraints to preserve and share high-precision 3D models for research, visualization, and education.

The conversation covers equipment and workflows, the challenges of scanning a reflective, historic artifact without touching it, and future applications including acoustic simulation, 3D printing for outreach, and public access to the raw datasets.

Related: Visualizing American History

Visions: A Machine Vision and Automation Solutions Podcast, is the podcast for engineers, designers, integrators, and end users who want to keep an informed eye on the imaging and machine vision industry. Every Tuesday we will explore the latest in imaging trends, developments and solutions. Here you will find interesting, useful insights and observations from expert interviews, solo episodes, even the occasional panel discussion, all of which aim to expand your knowledge on imaging and machine vision.

Transcript

Well, hello and welcome to visions, a machine vision and Automation Solutions podcast. I'm your host, Jim Tatum, senior editor of Vision Systems Design and Visions is an Endeavor Business Media production from your friends at Vision Systems Design. Here you'll find the latest on everything from end user machine vision solutions to trends, developments, and perspectives on all things machine vision and imaging. Whether you've been working in the industry for a while or you're just starting to take a closer look at it, this podcast is designed to grow your knowledge and bring greater focus to your understanding of the imaging and machine vision industry. And now on to our show.

Well, hello everybody, and welcome to Visions. Today we're going to take a closer look at an absolutely fascinating and important project in an area we don't get to cover that often: Historic preservation. Indeed, for machine vision engineers, it's hard to imagine a more challenging scanning target than the Liberty Bell. Yes, that Liberty Bell, the one in Philadelphia. The one with the great big crack in it, that Liberty Bell. Think about these conditions for a moment. You're trying to get a complete, detailed scan of a priceless national treasure you can't put your hands on. You have limited access windows, complex geometry, reflective metal surfaces, and zero margin for error. How do you do this? Well, we recently had the pleasure of talking with Chris Shelton, a psychology professor at Penn State Behrend, who heads up the school's VAR lab about their recent and remarkable effort to create one of the most detailed digital twins ever produced of the Liberty Bell. Using a combination of structured light scanning, laser metrology, lidar, photogrammetry, edge computing, and hundreds of gigabytes of captured data. Shelton and his team set out not only to preserve a piece of American history, but also to open new possibilities for 3D metrology, digital preservation, simulation, and immersive visualization. From scanning at micron level, precision under intense time constraints to making the resulting data sets freely available to researchers worldwide. This project offers fascinating lessons for anyone working in machine vision imaging and advanced measurement technology.

Welcome, Chris, and thanks very much for joining us today. First, please give us a little background on the VAR lab and tell us a little bit about how you got involved with this project.

It's an emerging technologies hub, and I built it to work for everybody. So all the students and faculty across all the different schools and colleges and whatnot. So probably seventy percent of what we do has nothing to do with my line of work, which makes it really interesting because it means I get to dabble in everything, right. And it just so happened our friends over at the, uh, applied research lab, AR lab at Penn State wanted a scan of -- we have a replica here of the Liberty Bell, and they wanted a scan of the replica that they could then slice down the middle, do a profile of, and look at and try and determine the thickness at the various points of the bell so that they could use that for acoustic simulations that they were doing. And so we said, you know, we're not sure based on the metal, how well this is going to work. Give us a little bit. We'll come back. And that afternoon we went out and scanned. It worked really well. Uh, but it got me thinking, you know what's out there for the actual bell? And I'll be honest, I was a bit surprised to find that there wasn't a whole lot. In fact, uh, to my knowledge, I know that there's been a scan, I want to say over a decade ago, but to my knowledge, there's no publicly available data set for the Liberty Bell, whether it's structured light, light or whatnot. So our hope was to change that. And so we reached out to the National Park Service to gauge their interest. And at first they thought they had a model that worked. Turns out they didn't. And so they reached out and they're like, hey, actually, can we do this? So for a while, I didn't think it was going to happen. We reached out early in the year, and when we got back and set it all up, we had six days from the day we agreed on a date that they could have staff there. To the day that we actually showed up, there was a time span of six days, so we had a very short time frame to work with it.

Sure. Well, with that in mind, um, tell me a little bit about the system you use. I'm understanding you use several different mediums, uh, RGB structured light scanning lidar works. I'm not exactly sure how each piece of it was set up and if it goes in order or if they work separately. I also understand you have the computer at the edge doing all the analysis.

Absolutely. The system was really important to us, so we were working with a lot of our tech partners to try and make sure that system was the best possible for the limited time we had. We had nine hours spread across a day and a half, so we could only be in outside of closing hours. So we came in a couple hours prior, came in, came back a couple hours after, and then the next morning, a couple hours prior. That was our hour time frame that we had to work within. So our process was essentially to go in and hit it with everything we had. And hope when we leave that we have enough data to make something work, because we knew that there wasn't going to be a whole lot of second tries. It was a lot to get to get in there to begin with. So we reached out to our partners at Lenovo, who hooked us up with a Legion 9i so we could do on site processing, really heavy duty system. We reached out to partners at FARO, who brought out their Quantum Arm and a Reform system. And then we brought a bunch of tech that we had in the lab. So we had Artec Space Spiders and EVAs. We had the Matterport Pro3, we had the Insta Pro 2, the 11K camera version of it, and then the Insta360 X5 as well as the, uh, Kandao Cam 3 with the VR 180 Mod. And then we used the Canon EOS R5, Mark 2. So we were trying to essentially get it with as many scans and as many different types as we could. So we were capturing it for photogrammetry with the Canon, with the Kandao. We were trying to capture it with as like a VR walk through of the center that has depth in it. And same thing with the Pro two. And then the X5 would give us the 360. And so they all offered a little bit something different. And we went in and just tried to capture as much as we can. The other thing we brought, but we didn't end up using was the FARO Focus S70, which in hindsight, I'm a little upset at myself for not finding the time to work that in because I think that would have worked well. So each one of these, each one of these pieces were were run separately, or they all ran at the same time or basically run separately. Yeah. To ensure that we had good field of view, that other stuff wasn't in the way interrupting anything. We ran the Artec and the Space Spider, uh, back to back. Uh, but then we had to clear everything out to bring the FARO in and then clear everything out to bring the Matterport in and, and whatnot. So it was a bit of a process. Not to mention all of the equipment, including ourselves, had to be scanned and, and, uh, thoroughly checked with police coming in every day. They don't want anything, you know, near it that can harm it. So it's a lot of work lugging all that stuff back and forth.

Wow. Yeah. So with each scan, that information goes into the lens of a and it is. What's it doing? Yeah. So we were using. Oh, yeah. Sorry. Go ahead. No, I was just saying, um. Yeah. I guess we're trying to figure out what unique information each particular technology contributed and how it all blends together, which is essentially what I just said.

Yeah. So, uh, the Lenovo kind of pulled it all together for us and allowed us to run a lot of it off of the one machine, which was great for on site. Uh, we paired that with poly works and given the specs of the Lenovo, it had a ton of RAM. It had a 5080 in it. So for a laptop, this thing was a beast. It was able to do a lot of on site processing, which allowed us to capture a ton of data because the process time on our tech or on any type of scan, once you're doing post process can take a long time unless your system is quite capable. So essentially, the Lenovo allowed us to do a lot of on site processing and didn't slow us. It allowed us to not be slowed down by the tech. It recorded and captured as quick as FARO could throw the LiDAR, or in this case, laser line scans at it and as quick as Polyworks could pick it up. So we left with scans that were millions and millions of data points, tens of millions of data points in some cases. Uh, the photogrammetry, uh, same thing. We're looking at pictures that are thirty megs each and thousands of them. And so it's allowing us to process that quickly. I think all told across the different cameras and the different systems. And we're still offloading, believe it or not, but I think we're just under four hundred gigabytes total of raw data from the various methods. So it's a ton of raw data that was captured and had to be captured quickly so that we didn't introduce error into it, and so that we could see a lot of it. We got a ton, but we weren't able to do full post-processing there on site at the moment. There just wasn't enough time. It wasn't an equipment thing, there wasn't enough time to do it. But with our equipment that we had with the system we had set up, we were able to see a lot of what we got so that we could fill in those blanks. Still, while we were out there, to a degree we wouldn't have been able to otherwise, if that makes sense.

Yeah, yeah. Which leads into the next thing. When you're getting into resolutions as fine as you were, was it ten to twenty five microns without even touching the bell? Yeah. What are you having to overcome other than. Yeah, sure. Get the stuff in and out, you know, in nine hours or else.

Yeah. You know what? I kind of assumed with all our different systems that it would be easier than it turned out to be, it really was a big challenge, especially the inside of the bell. Even with the quantum arm and the maneuverability of it. You know, there's the spider inside the bell that essentially it's a metal framework that the bell sits on so that it's not actually holding the weight on the bell itself anymore. And the wooden yoke doesn't hold the weight either. There's a hidden metal bar inside of it, because neither the yoke or the bell are all that intrinsically strong. They have a lot of weak points. And so working around what they call the spider on the inside was really challenging because as, as I'm guessing your listeners know, we can only scan what we can see, right? The scan has to be able to touch all the different parts in the back, you know, the backs, the sides, tops, bottoms. And so that was really difficult. Um, did a couple of things, you know, uh, probably not the best way we could. Um, thankfully everything worked out, but yeah, we took the FARO off of the Quantum Arm at one point and sat it underneath. So, so still on the Quantum Arm. It just wasn't on the tripod that it's built for. So we had to sit it underneath, build sandbags around it to steady it and then try and get up there without, uh, we had to take the trigger guard off of the the XS and XR probes that we were using to get in there. That's why I was a little bummed about the FARO Focus, not being able to find time to use that, because the Matterport Pro 3 with the LiDAR, uh, worked quite well. And I think the Focus would have worked as well, um, to be able to get in and just shoot the laser beams at it, um, from further off because it's hard to, to be close without touching it. Sure. There's that fear there. They're sitting there, uh, as they should be the park staff, but they, they would move around the bell as we would move around the bell. So they were never six feet away from us at any time. And wow, there's that pressure. Knowing that, you know, this is one of the most unique and highly recognizable pieces of Americana. So there's that pressure sitting on you. Don't touch this thing.

Don't touch it. Don't, you know, um, do not even breathe on it. Yeah. Yeah. Uh, did you need any special lighting or anything like that. I know LiDAR doesn't necessarily need it, but did you have to do any other kind of metallic stuff is very difficult to image, I understand.

Um, yeah. So, uh, we we didn't need special lighting. We brought lighting rigs with us just to capture the actual event and make sure that we could show behind the scenes because we wanted people and educational institutions to see what was actually done. So we have time lapses of it. Um, but in terms of, I guess the one part of special lighting that we did use was polarized lighting for the photogrammetry. So we were using polarized lighting to try and remove all of the built in or all of the light that's artificially on it so that it can be relit in, uh, unreal or unity without having baked in lighting to the best degree we could. It's kind of, I believe it's called the void method, essentially. So removing all of the natural light as much as possible.

Okay, well, at the risk of repeating myself, um, with a nine hour window to get stuff done, and it sounds like a split shift, no less. How do you balance speed, accuracy, and risk management to get what you want?

Yeah. Uh, I think we started moving a little bit quicker towards the end when we realized the time limit was coming up quicker than we had anticipated at first. We managed it. Uh, everything was put into, you know, being very risk averse. And we stayed risk averse throughout, but we definitely loosened up a little bit and personally took risks. Nothing that would risk the bell. But, you know, we probably leaned further off of the the ladders than we should have, uh, you know, stuff like that versus getting down, removing and then climbing back up because we were trying to, we literally were walking out five minutes after our lap or after our time. Um, and that was a big deal. I mean, they had, they were waiting and holding up the front doors for us to get out. So like we were running up to the very last second.

Um, yeah. The extent of the whole setup sounds just really challenging to say the least.

But yeah, it definitely is. So it's now inside the Liberty Center. They have this big space to one end of the Liberty Center that's dedicated just to it. But on the outskirts of the bell is a little, probably three foot tall barrier fence that wraps around most of it. And then the parts that doesn't wrap around have leather belts that go across it that meet the two bars. So we took those down, but we also had those in our way. So it kind of trying to work in between those. Um, so the depth and field of view was challenging because we had those things artificially. It wasn't just the bell and nothing surrounding it. Our bell here at Behrend is lower off the ground than that was, which we were worried was going to be a big challenge. It was a challenge here at Behrend. Thankfully, the bell is lifted higher up off the ground, so it was easier for us to get inside and underneath of it, And I think that's one of the coolest things about the project because. Even though you can get close, the fencing is probably about four to six feet away from the bell, so it doesn't seem all that far. But you could lay on the floor and still not really be able to get a great view underneath. So, so we are able to provide that to kids, to students, to the public, and be able to, you know, let them see it from anywhere, any angle.

And again, going back a little bit to the Lenovo computer, um, had you not had that, what would have changed? I mean, it sounds like that was a big driver in getting everything you wanted to get so quickly.

Yeah. So we would have gone out there. We had another Lenovo that we use off site, but it's a one of the students personal laptops. It's nowhere near as capable. So what it would have changed essentially is it would have slowed our work down. We would have left with probably half to two thirds of the amount of data that we were able to capture having something as. I mean, it had the fifty eighty series in it, so it had the capabilities to capture as much as we could throw at it, which our other system wouldn't have had, which meant we would have had to slow down for grabbing pictures and checking to see if they're good for photogrammetry and scrolling through, you know, three to four thousand of them. That would have slowed us down considerably, even using Lightroom and stuff like that. So it allowed us to get a quick look at almost all of our data that needed to be run and processed on local hardware. Some of the stuff was processed on cloud hardware like the Matterport, but outside of that, everything was local, which meant we needed something as fast as possible.

Sure. Okay. So yeah. Okay. So ultimately, you've been able to build a research grade digital twin. Um, or you're in the process of that, I guess, uh, beyond preservation, what kind of opportunities is it going to create for researchers and educators, historians, even industrial types?

Yeah, I think there's a lot of possibilities. In fact, I was talking with a gentleman yesterday who gave me a call and wanted to know if we could map out certain parts of the bell, because they're looking to try and figure out how the bell cracked in the way it cracked, and what parts of history align with some of those cracks on the bell. Um, for acoustic simulation, I think it would be really cool. They've simulated another lab. Back in nineteen ninety nine, another Penn State lab simulated what the bell would sound like if it was whole and didn't have the crack, but we don't actually know what it would sound like with the crack beyond a couple of tappings that they've done, they won't actually bang the clapper or the big metal piece inside against it. For obvious reasons. It's beyond the big crack. There's thousands of hairline fractures. Mhm. And so with this, similar to simulating some of the acoustic sounds of the replica. Now we have a good model of the actual one that should allow us to simulate what it would sound like if we did swing the clapper against it. And so we have a number of different things that we want to do with it. We have the research grade model is available, and the non-research heritage grade model, the one that we filled in the holes had to interpret some of that. So it's not accurate to the degree that the FARO is. Both of those are available on Sketchfab, including the raw non decimated models are included in the additional files. For anyone that wants it, they can download it for free. But what we see in addition to that is we'd like to then use those models to print up like 3D one to one 3D prints of it, and mount speakers in it that we can send around to outreach or school districts around PA or around the country, showing what it would sound like with the crack without the crack, letting students and members of the community feel the bell because the bell itself, you can't touch it. Certain members, they. For accessibility, if you're, uh, if you have vision impairments, for instance, they'll let you put gloves on if you're there in person and touch it. But most other people can't do that. Um, and for obvious reasons, the, uh, oils in our hands are cracking some of those metals and the lettering is flaking off now. So being able to physically handle a replication of the bell, being able to hear what it would sound like with the crack and without, I think it's something that a lot of us don't have that opportunity very often, even many of us in Pennsylvania, accessibility to the bell is really important.

Yeah, that's that's incredible. What do you think could be, uh, extrapolated to industry or to, um, say inspections or cultural heritage preservation? Yeah, 3D metrology, any of that.

So I think in terms of like historic preservation and in terms of industry, I've also talked to carry over at the National Park Service. Who wanted to know, could we look at and. And yes, we can. I'm not the expert to do it, so I have to put her in touch with our metallurgy folks here at Penn State. But using those models to look at like stress points of where the bell might be stressed in how it's currently housed and the way it's sitting. It's been a couple of weeks since I was looking at this email, but their question was essentially, you know, can we look at and see if there is more stress being placed on, I believe, the north side of the bell based on its housing in my rudimentary talks, this is not my area of expertise, but in my rudimentary talks with our ace and metal folks and other folks, I believe all of that is possible. It's work that still needs to be done. We haven't had a chance to do it yet, but that's why we're trying to make as much of this available to the public as possible so that any expert, not just for folks at Behrend or at Penn State, but anyone, can access the raw data and do their own analyses. So we're working right now with Open Heritage 3D as well as potentially the Smithsonian 3D. There's talks about them looking into potentially assessing the data sets to ensure that in perpetuity, these data sets remain free and widely available to as many folks as possible. That way, any expert from any field that wants it, if you're trying to acoustically simulate it, if you're looking at some of the wear patterns, the way the metal is cracking. So if you're a metallurgist and you want to look at how the lettering is flaking off in certain areas, all of that is there at a scale that hasn't been available to the public before. Like we talked about, the FARO is between ten to twenty five microns. Human hair is is seventy, give or take. So for a sense of scale, it picked up some really fine detail on the the non decimated models.

Wow. That's incredible. That's just great. Okay. Do you have anything like this coming up in the future? What's next for this project.

So what's next for this project? There's a couple of things we still want to do. I want to move forward with the 3D prints, but I also want to make more versions of it available. So we have Matterport tours, we have the 3D models, and with Matterport we have tours of the entire Liberty Center as well as the bell. But I'd like to also work it within a number of different immersive technologies. So I'd like to utilize our data sets and make available photogrammetry data set for anybody that wants to visualize it that way, um, as well as using Gaussian splatting to try and create a very lightweight model that people can explore, uh, from within their browsers. So there's a number of new immersive tech that's always coming out on the scene that I'd like to try and make it available on as many of those as possible. We'll also be working to post-process and publish the the VR 180 walkthrough so that you have that POV view that'll be published on on YouTube in DeoVR. So we have a lot more work and a lot more post-processing to do in terms of the future though. We love doing this. A big passion of mine personally, outside of the clinical work I do, is history. 

Well, that's a wrap for this episode of visions produced by Endeavor Business Media, a division of endeavor B2B. Thanks very much for tuning in. If you enjoyed today's show, be sure to subscribe to the podcast and share this episode with a colleague who would find it helpful. Until our next episode, you can find us at vision dash systems dot com or on LinkedIn, Facebook, or for more insights, updates, and breaking news to keep you in the know. Thanks for tuning in. Until next time, stay focused on your visions.