ATC voice-to-text conversion

[iWantWings]

My tailwheel CFI (who is also an A&P) has difficulty hearing and relies mostly on reading lips to communicate with somone like myself (he says he is deaf).

His speech is as good as mine and in the training conditions we seldom had difficulty communicating (also true of other current students).

But there are limits/dificulties for him with being able to hear and make sense of the broadcast spoken ATC comm (as if I don't - ha!)

This got me thinking and I started searching for commercial off-the-shelf system that is designed to convert spoken voice into written text, with empahsys on ATC comm.

Maybe I didn't search well enough, but I am surprised I didn't find any such product. I mean, there are some surprisingly good voice-to-text conversion apps for mobile platforms for "free speech", but there aren't any systems dedicated for converting ATC spoken Comm to text (if we consider that ATC comm is generally structured and has a reduced vocabulary compared to common speach, then ATC comm could be converted into text with significantly better accuracy, one would think).

I'm not out to solve anything, but does anyone know if such a system exists, that average dude could buy?

I'm guessing that a system like this would (1.) be fed the broadcast audio on the frequency (2.) use an algorithm to magically convert the audio of spoken ATC into equivalent words in digital format and (3.) display to text (4.) and then whenever the tail number (or tail substring of it) is matched, the user can be given a vibrating buzz to alert them that their tail number might have been spoken by ATC and they should read the text on a screen.

Easy-peasy.

For real though, anyone know if simething like this exists?

 

[Gucci Pilot]

Yeah, controller-pilot data link communication, but it is text to text and only really used by airliners during oceanic crossings.

 

[upstateny]

How about feeding the audio into your smart phone and using one of the many voice-to-text apps?

 

[mcmanigle]

Voice-to-text technology is getting very good, and I wouldn't be surprised if at some point in the next few years something like this could happen, but the problem at the moment is that radio signal quality isn't good enough. Parts of conversations (including tail numbers) get stepped on, transmissions get cut off or individual words get garbled, etc. Usually this isn't such a big deal because 1) it's more of a problem on CTAF rather than ATC frequencies, 2) transmissions will be repeated if they get stepped on too badly or not acknowledged, and 3) the human brain is really good at deriving meaning when missing parts of words, whole words, bits of tail number, etc.

The problem, I think, is that right now #3 is a big part of the equation. You might be able to get 90% accuracy by rigging something up with your favorite text-to-speech device (though the most accurate ones currently use off-site computing ie internet, which is its own can of worms), but a commercial enterprise isn't going to launch a product with 90% accuracy on ATC communications and say "this will help deaf pilots fly" with all of the liability that implies.

tl/dr: Yes, you could rig something up for this that will work pretty well, but it won't come particularly close to 100% with today's technology, and because of that (and liability) you're not likely to find a commercial solution.

 

[JOhnH]

I'd love to see it because I am partially deaf. But I think the best solution would be one incorporated into a comm device with a substantial display. But since it would have to be FAA sanctioned, and it would have a rather limited market, I suspect it would be quite expensive. I think that for someone that can hear normally, it would be more distracting, but for someone like myself, it would be worthwhile, even if I only use it when I am not sure what I heard.

 

[iWantWings]

Yeah, controller-pilot data link communication, but it is text to text and only really used by airliners during oceanic crossings.

When doing the web search for ATC voice-to-text, i came across what the same thing; it think it was abbreviated as CPDL(C). Kind of interesting; read the it is used in the European airspace as well.

It's okay. Soon commercial planes will have full autonomy from the heart-beating pilot and no need for voice-to-text, or pilots:

"This is your Captain, HAL, version 2.01. Welcome aboard Libery Air - 'where everyone gets a parachute'".

 

[iWantWings]

How about feeding the audio into your smart phone and using one of the many voice-to-text apps?

i tried that briefly; ingested SoCal Approach audio (from hand-held) to iphone. The text conversion was pretty funny.

i don't think i was using the right cable and the exterior mike on the iphone was probably not disabled; i'm ordering a dedicated cable and see what that does, just for fun.

 

[iWantWings]

Voice-to-text technology is getting very good, and I wouldn't be surprised if at some point in the next few years something like this could happen, but the problem at the moment is that radio signal quality isn't good enough. Parts of conversations (including tail numbers) get stepped on, transmissions get cut off or individual words get garbled, etc. Usually this isn't such a big deal because 1) it's more of a problem on CTAF rather than ATC frequencies, 2) transmissions will be repeated if they get stepped on too badly or not acknowledged, and 3) the human brain is really good at deriving meaning when missing parts of words, whole words, bits of tail number, etc.

The problem, I think, is that right now #3 is a big part of the equation. You might be able to get 90% accuracy by rigging something up with your favorite text-to-speech device (though the most accurate ones currently use off-site computing ie internet, which is its own can of worms), but a commercial enterprise isn't going to launch a product with 90% accuracy on ATC communications and say "this will help deaf pilots fly" with all of the liability that implies.

tl/dr: Yes, you could rig something up for this that will work pretty well, but it won't come particularly close to 100% with today's technology, and because of that (and liability) you're not likely to find a commercial solution.

Interesting and your explanation makes sense. Some of the worst commercial aviation accidents had some sort of "communication failure" as a main factor, with pilots and ATC who were surely not hard of hearing. I can see how a "certified" system for converting voice-to-text will not work reliably with the current infrastructure. A bit of a bummer.

But, I'll play around with what's available (and promise no one will break any FAR(t)s. )

Thank you for the info.

 

[iWantWings]

I'd love to see it because I am partially deaf. But I think the best solution would be one incorporated into a comm device with a substantial display. But since it would have to be FAA sanctioned, and it would have a rather limited market, I suspect it would be quite expensive. I think that for someone that can hear normally, it would be more distracting, but for someone like myself, it would be worthwhile, even if I only use it when I am not sure what I heard.

You know, i'm thinking that because other non-certified devices can be used in an aircraft to "enhance situational awareness" - like all the portable devices used for moving maps, and stuff, then the same concept could be applied to an ATC voice-to-text? It could be directed as "for trainining purposes only".

Sure, you're right, it could be a distraction as well.

I've emailed the developers of x-plane and Xavion and their reply was not "no, never".

 

[NoHeat]

I think your idea is a good one, iWantWings. It makes sense to exploit the standard phraseology that ATC is expected to use. As is often mentioned there is no standard phraseology for pilots specified in regulations, but there is for controllers. So if the goal is merely to make sense of a controller, it might work well enough to help a deaf pilot, despite some problems that were mentioned above that would prevent 100% success.

 

[iWantWings]

I think your idea is a good one, iWantWings. It makes sense to exploit the standard phraseology that ATC is expected to use. As is often mentioned there is no standard phraseology for pilots specified in regulations, but there is for controllers. So if the goal is merely to make sense of a controller, it might work well enough to help a deaf pilot, despite some problems that were mentioned above that would prevent 100% success.

see the text i bolded in your response: glad you took notice of that because the "standard phraseology" is a key factor.

The ATC "vocabulary" is a minute subset of "English language" - that is a big advantage.

Secondly, the usually standard ATC phraseology can be used by algorithms to predict certain values, and than can help conversion with junk audio makes it through.

No, these won't make ATC voice-to-text anywhere near a correct translation 100% of the time, but would surely make it much more successful than converting voice-to-text for normal English spoken language.

But challensges like the ones posted by mcmanigle still remain.

I can easily see a comment of the sort: combine the flaky ATC voice-to-text converstion, along with the "funky" Airbus "modes" of automation and you've got a disaster (ha). But that wouldn't be the scope of this system.