Underwater Human-Robot Interaction #ICRA2022 – Robohub

How do folks talk when they’re underwater? With physique language, in fact.

Marine environments current a novel set of challenges that render a number of applied sciences that had been developed for land functions utterly ineffective. Communicating utilizing sound, or no less than as folks use sound to speak, is one in every of them.

Michael Fulton tackles this problem together with his presentation at ICRA 2022 by utilizing physique language to speak with an AUV underwater. Tune in for extra.

His poster will be considered right here.

Michael Fulton

Michael Fulton is a Ph.D. Candidate on the University of Minnesota Twin Cities. His analysis focuses totally on underwater robotics with a give attention to functions the place robots work with people. Specifically, human-robot interplay and robotic notion utilizing pc imaginative and prescient and deep studying, with the intent of making methods that may work collaboratively with people in difficult environments.

transcript

Abate: [00:00:00] So inform me a little bit bit about your presentation earlier at this time.

Michael Fulton: Yeah, so I used to be presenting at this time, my collaborative work with Jungseok Hong and my advisor Junaed Sattar on diver method. So mainly the issue of when you will have an AUV and a diver working collectively underwater it’s essential that they be shut collectively after they wanna talk, whether or not it’s for, , doing gestures to the AUV, to inform it, , Go do that process, go have a look at this space.

Or if it’s the AUV speaking to the diver, perhaps they’re telling it, Hey, I discovered this cool factor over right here. You ought to come test it out in both of these conditions, it is advisable to be shut collectively, proper? Mm-hmm nonetheless, for AUVs to be helpful underwater, they should depart the diver. They must go do looking out and, , carrying merchandise or, or instruments and supplies and stuff like that.

Uh, so that is the issue that we’ve, proper. We must be shut to speak, however we must be distant to, to do stuff. So to repair this, we’d like a functionality for diver method. We want to have the ability to seek for the diver, discover them and method them to an acceptable distance and orientation for communication.

So our algorithm is known as ADROC autonomous diver relative operator configuration. And it’s this monocular imaginative and prescient primarily based methodology of doing this the place we we do that diver method primarily based on solely monocular imaginative and prescient. Yeah. Because we wished to maintain it as low-cost as attainable, ? No, no sonar, no stereovision and, and as minimal sensing as we may, we may handle this with and mainly the best way the algorithm works is as an alternative of making an attempt to do monocular depth estimation, which is, , you will get first rate accuracy on it, however you generally want excessive computational energy.

Mm-hmm . Instead of doing that, we realized, okay, what we really must know is, is the space that the divers is at the moment at “Good sufficient”? Is it shut sufficient for, for us to work with the communication a part of issues.

Abate: So you want a tough estimate?

Michael Fulton: Yeah. You want, you want a really tough common estimate. I don’t care if the, if the robotic’s, , one meter away or 1.1, , 0.9 0.7.

It doesn’t actually matter to me so long as it’s shut sufficient. Yeah. Rough sufficient. So the best way that we did that is by utilizing shoulder width as a previous piece of knowledge, as a result of we all know from biomedical literature that there’s a spread that human shoulder widths are available in. We know the typical of that vary.

We know, , the place most individuals’s shoulder widths are fairly near. From that we will calculate the anticipated pixel width between shoulders for a detailed sufficient tough estimate, distance for communication. Yeah. And then we simply evaluate: is the diver shoulder width smaller than that? Okay. We want to return nearer.

Is it, is it bigger than that? Okay. We must again up. And the best way we do the the precise calculation of the shoulder width is a two-step course of. We both use a diver detector, which takes a picture of a, of the scene and finds. Diver attracts a bounding field round them. We can use the width of that as sort of a proxy for shoulder width.

Mm-hmm however it’s not tremendous correct, proper? The diver may very well be sort of on their facet. Yeah. Uh, there’s numerous issues that may change the bounding field width with out altering shoulder width. So that will get us a really, very tough estimate. And if we simply approached primarily based on that, the, the AUV can be manner off on distance as a result of the bounding field adjustments loads.

What doesn’t change loads is the precise shoulder width that is still. So we additionally use the diver pose estimation algorithm to get key factors on the shoulders and calculate the space between them. Yeah. And so it’s this cascaded method the place mainly what finally ends up taking place is from distant, the detector works.

We’ve really run this so far as 15 meters away. Um, and that permits you to heart the diver within the picture and begin getting nearer to them. And then as you get nearer inside the vary of, I’d say most likely about six to seven meters is the efficient vary. Uh, you’ll be able to really begin detecting the important thing factors for the shoulders and you then get correct distance.

Not distance estimation, however distance ratio calculation, we name this the pseudo distance. Yeah. Cause it’s probably not distance, however it features at it. Yeah.

Abate: So I imply, one of many good issues that you just mentioned in your presentation is that even in several poses and orientations, the area between your shoulders stays comparatively the identical.

But on the flip facet, say my shoulders and your shoulders are totally different lengths.

Michael Fulton: They are totally different. But if you have a look at the magnitude of the distinction in comparison with the magnitude of the scene, it’s really very small. Right. Like, I’d say simply on a tough guess, I’d say the distinction between our shoulder width is just a few centimeters mm-hmm proper.

And if you had been utilizing this, I can’t bear in mind my actual shoulder width. It was one thing like 40 one thing centimeters. I, I don’t bear in mind after we’re utilizing that as our, as our, mainly our sign for the space a distinction of a few centimeters does make a distinction, however it doesn’t wreck issues.

Yeah. We can nonetheless work with it. And, and like I mentioned, within the, within the presentation earlier, we will run it off of the typical diver shoulder width. But in case you are happening with an AUV and , you’re gonna work [00:05:00] with it, you may additionally calibrate it to your actual shoulder width. We did this just a few occasions and it really works.

The algorithm works regardless in the event you calibrate it to your actual shoulder width, you will get very nice distance like last distance for method. It works actually properly in the event you calibrate it to the precise shoulder width, however it works usually on the typical as effectively. Is there any distinction

Abate: between say taking these these measurements and pictures above floor versus underwater. Does water distort that measurement?

Michael Fulton: Yeah, so completely underwater imaginative and prescient usually. there’s distortion of coloration. There’s distortion of turbidity particulate matter and bubbles, numerous issues. So, so this facet of underwater imaginative and prescient is sort of it’s the manner it’s.

Mm-hmm all underwater imaginative and prescient stuff suffers from this. There is a, a extremely energetic thread of labor on underwater picture. Enhance. Which principally makes an attempt to take care of like mild or coloration altering coloration. Yeah. Yeah. Um, so that truly, it helps a bit, however doesn’t assist a ton with this. Um, the opposite massive factor. So, in order that’s from the visible facet of issues.

When we’re speaking extra concerning the I don’t know fairly find out how to say this. The, the, the educational facet of issues, our diver detector is skilled on photos of divers, so it is aware of what they appear like. It approaches them simple. The physique pose that we use is TRT pose from nvidia IOT. it’s skilled on terrestrial imagery. So the factor about that’s that in these conventional photos, individuals are standing or sitting, no person is sideways, proper? Cuz we, we will’t go sideways, however within the water we will, individuals are sideways on a regular basis.

They’re swimming, they’re floating. And so this really causes issues with ADROC. Um, If, if any individual is in a, a vastly totally different orientation it, it, it’s loads more durable, which is why, , in the event you learn the paper, you’ll see, we, we made a few simplifying assumptions. One of them was that there’s just one diver within the scene as a result of whereas we’re wanting into discriminating between divers proper now, the algorithm doesn’t do this.

So, and it’ll method whichever one, it sees first . Um, the opposite simplifying assumption that we made was that the diver is mostly upright. You know, we didn’t inform folks, it’s important to keep one hundred percent straight up and down, however we mentioned, , keep principally upright. Yeah. And after we tried it on folks, , sideways, it nonetheless does work, however not as effectively.

Abate: Yeah. So that is an space that’s like, you’ll be able to positively see a path to enchancment.

Michael Fulton: Absolutely

Abate: probably not a problem. It’s only a matter of getting the info and becoming it to yeah.

Michael Fulton: With underwater robotics, brown reality is all the time an enormous, enormous hassle. And for labeling one thing like pose. That is a few actually it’s, it’s not a lot that it’s like tough work, however the labeling is gonna take months for that.

But I really, I imply, it, this is the reason ICRA is nice. Like I used to be speaking with any individual on Monday night time or no sat Sunday night time. Um, they usually had been telling me about some pose community I ought to attempt. So I’m gonna go house and take a look at, attempt it for our knowledge and see if it really works any higher.

Abate: Yeah.

Michael Fulton: Um, I believe the 2 most important areas of enchancment, three, three areas of enchancment, pose estimation, we already talked about.

Yeah. Second massive one is search conduct. Our search conduct for this was actually easy. If you don’t see the diver flip mm-hmm proper, however there’s, there’s some apparent enhancements that may be made there. Things like if we lose observe of the diver, we must always flip within the course that we final noticed them.

Right. Or if we’re making an attempt to cowl a big area, perhaps turning isn’t gonna be sufficient. You know, I, I mentioned earlier, we, we ran this from 15 meters away. I’d guess… I don’t have knowledge. I’d guess that previous 30 meters it’s not gonna work as a result of we simply can’t see something. So for an area that’s like 30 meters or bigger, which open water underwater environments are you’re gonna want to have the ability to do extra than simply turning.

It’s gonna want to love search the area by some means. Yeah. That I believe is the entire massive factor by itself. Um, after which the opposite massive factor by itself is what I mentioned earlier about diver discrimination. Yeah. Being capable of inform the distinction between diver a and diver B, , I don’t, I don’t actually care if it’s, , this man versus that man versus that woman.

It doesn’t matter who particularly, however I do need the algorithm to have the ability to handle a number of divers within the scene, figuring out which one it’s … approached earlier than. And, and after we really first got here up with this concept, the concept was we’re gonna activate the robotic and it’s gonna like go as much as all people and ask, Hey, are you my operator?

I actually wish to do that also. So if we get the diver discriminator working effectively sufficient,

Abate: And that shall be by way of gestures, they’ll say like, …

Michael Fulton: yeah. So, so it’ll come as much as the diver and it’ll do like a, so I I’ve carried out this work with movement primarily based communication, robotic communication by way of. um, and it, so the di the robotic’s gonna come up and it’s gonna sort of do like a, you ever seen like a canine ask to play fetch with you?

Yeah. It’s gonna sort of go like, Hey, Hey, Hey, Hey, Hey, are you, are you? Yeah. And then the diver will say sure or, or no, I’m not your, I’m not your operator. And then it’ll go, okay, I’ll cross you off the listing seek for the following [00:10:00] particular person. Yeah. That’s the place this work hopefully goes sooner or later. Um, , my, my work usually, my thesis work is about robotic communication and interplay underwater.

Uh, I believe I discussed this briefly within the speak, , underwater human robotic collaboration is a model new discipline. Yeah. Like this didn’t exist earlier than the early two 1000’s. Um, partially as a result of the AUVs which might be cheap to, to work with underwater are like, since 2000’s,

Abate: they had been, they had been created within the 2000’s.

Michael Fulton: Yes.

Abate: And that was the impetus for why now working with a robotic, proper. Underwater is even an idea that we’re speaking about.

Michael Fulton: Yes. Cause the primary AUV’s are in just like the sixties, and these are these massive ocean going submarine, issues which might be for oceanography, nice work, , actually essential stuff, however they’re greater than you and I are.

Yeah. And you’ll be able to, you’ll be able to work together with that, however it’s probably not what they’re for due to this fact doing these lengthy deployments that people can’t do. We’re now in, in underwater robotics, seeing the, the arrival, the approaching of collaborative AUV’s. It is, it’s a new factor that’s developing and you may see it within the work, , underwater HRI papers weren’t written 20 years in the past.

Um, perhaps any individual wrote one 20 years in the past that I don’t find out about they usually’re gonna get mad at me, however I’ve solely seen ones relationship again to early two 1000’s. Um, and now there’s, there’s just a few right here and there. I’ve offered a few ICRA now, and whereas we’re not but on the level the place the AUVs and the individuals are really working collectively , I, I, I don’t know of anyone who’s really doing collaborative work with AUVs for like an organization.

Um, however it’s coming. Yeah, it’s coming quickly. And, and particularly, for me, I’m actually fascinated by like environmental conservation and organic remediation. So like trash cleanup, oil spills uh, observing invasive or so it’s both eradicating invasive species or preserving endangered species.

Yeah. This sort of factor the place what’s taking place proper now could be around the globe. Some scientist is diving, , they’re diving with all these undergrads for hours lengthy a day. I need to have the ability to give them robots which might be low-cost and, and overtly out there. And , my massive a part of it’s robots that they’ll talk with in a manner that’s not onerous for them to study.

Yeah. I don’t need these scientists to need to study Python or need to study C++ or ROS and discover ways to program these robots. I need them to have the ability to use my communication frameworks, and my process administration frameworks in order that they’ll process these AUVs with totally different items. Work go discover me this, this sort of Marine life.

Go discover me this trash. Tell me the place to go choose up this trash. Uh, convey me instruments, carry samples for me. Yeah. This sort of stuff I believe may be very a lot inside the realm of chance and the work that I, and the opposite nice Ph.D. college students and grasp college students and undergrad college students and our advisor of the interactive robotics and imaginative and prescient lab do is actively transferring us in direction of that.

Yeah. We’re getting, , notion, capabilities, and navigation mapping. Capabilities you noticed within the Marine, robotics talks, all these various things. You know, the acoustic localization, the GoProfessional-based imaginative and prescient for mapping all these items. It’s all items of the puzzle. And the piece that I’m most fascinated by is the human-robot interplay half as a result of it’s, it’s such an fascinating, difficult surroundings.

There’s so many assumptions that you just make terrestrially that simply aren’t there. Like the massive, the. Know, in the event you’re speaking with a robotic, you sort of anticipate to speak to it and have it speak again. You can’t do this underwater. You gotta,

Abate: yeah. There’s no voice.

Michael Fulton: There’s no voice. There’s a respiration equipment in your mouth.

Yeah. And you’ll be able to hear, however probably not effectively. Yeah. So I’ve developed, , movement, light-based communication. I’m making an attempt sound, however nonverbal sounds so like tones as an alternative of phrases.

Abate: Yeah. And what’s fascinating too, is like as in there are plenty of trade examples like offshore wind and like offshore buildings which might be being constructed the place The divers are usually not gonna get changed.

Michael Fulton: No, no. In no time quickly.

Abate: Yeah. They have such an extremely tough job to automate. Yes. That, and due to that, they’re additionally there, a few of onerous to search out yep. Must be costly. Yep. Um,

Michael Fulton: it’s harmful too

Abate: and harmful.

Michael Fulton: Yeah. People die yearly.

Abate: So you don’t, you, we wish to do every part you’ll be able to to make that dive probably the most environment friendly model of themselves attainable.

Michael Fulton: And secure and, and simpler. Yeah. You know, it’s, it’s, it’s onerous, work. It, such as you mentioned, it’s onerous to search out individuals who do that as a result of there’s numerous scuba dive licensed folks, proper?

It’s a, it’s a typical pastime, however technical diving and diving for, for business functions. There’s not too a lot of them on the market. There’s. I imply, [00:15:00] in, in, within the grand scheme of issues, , it’s, it’s, it’s a rarer discipline and a lot essential work is, is in there. Uh, there’s this quote, I actually. um, it’s a, I, I, I don’t know if it’s really, it’s attributed to Leonardo DaVinci water is the driving pressure of all life on our planet.

Mm-hmm I actually consider that. Like, clearly there’s the, the scientific causes, , photosynthesis, local weather local weather stuff, but additionally identical to a lot commerce depends upon ocean environments, the web. I imply, we’ve cables below sea, all of these items. You want AUVs. There are some locations the place we wanna substitute divers with AUV’s.

But we actually wanna increase the divers who’re at the moment doing work underwater with AUVs, with these collaborative AUVs, partially since you’re proper. It’s gonna be a very long time earlier than they’re changed if ever it’s such a difficult discipline, but additionally personally, I’m, I, I actually like the concept of robots making folks’s lives higher.

Mm-hmm and generally changing them in jobs is the best way in direction of that. There are some jobs. So harmful, so uninteresting, so, so soiled that you just don’t need anyone to do them, however there’s plenty of jobs the place like, folks depend upon this for his or her livelihood. I don’t wanna substitute these folks. I wanna make their lives simpler.

I wanna make their lives simpler and I wanna make it attainable for them to do extra fascinating work. You know, there’s we take into consideration, we consider ourselves as such a complicated society, proper? Like we go to area, we go to Mars, a ridiculous quantity of our ocean is unexplored. We don’t know the way a lot of the life that exists in our ocean is. We don’t, we there’s a lot primary science there that’s undone as a result of the surroundings is so inhospitable.

You want air tanks, there’s strain concerns. There’s a most restrict you’ll be able to dive to. So something that you just’re doing underwater is mechanically 100 occasions more durable, 100 occasions extra expensive, extra effortful.

And that is the place AUVs, my advisor mentioned this actually, rather well within the session. So we wish to improve underwater divers by having underwater divers do the issues, AUVs can’t and having AUVs do the issues underwater divers can’t. Yeah, I believe that’s an ideal summation of the place this discipline is headed.

Awesome. Thank you. Yeah, no downside. Thank you for asking me.

transcript

tags: bio-inspired, c-Research-Innovation, cx-Research-Innovation, human-robot interplay, podcast, Service Professional Underwater, software program, video