This winter vacation, a group of deaf universities have generated remote volunteers for the Winter Olympics. They are from the deaf artificial College of Tianjin University of technology. These volunteers convert the contents of TV news columns into sign language sequential texts to provide large-scale corpus support for AI (Artificial Intelligence) sign language digital people needed in the broadcasting of the Winter Olympic Games. Yuan Tiantian, vice president of deaf artificial College of Tianjin University of technology, is a teacher of these hearing-impaired college students and a member of the barrier free intelligent technology R & D center. She said: "the barrier free intelligent technology R & D center will provide AI sign language digital human sign language corpus support for the live broadcast of the Winter Olympic Games. My students and I have been busy for more than two months. We are full of honor to serve the Winter Olympic Games." Teach AI cognitive sign language In 2021, a "sign language corner" attracted many people to stop at the 30th anniversary of the founding of China's higher engineering special education for the deaf. As long as you type sign language in front of the camera, a semantic and coherent text translation will appear on the display screen behind you. Yuan Tiantian demonstrated the project "whale can speak - Multimodal continuous sign language automatic labeling and recognition system" (hereinafter referred to as "whale can speak"), which is called "Chinese sign language real-time translation system in complex scenes". It is reported that in a well lit and stable room, the system can recognize sign language up to 95%, and "synchronous translation" can be realized in some scenes. The purpose of developing this system is to make it easy for every healthy listener to understand sign language. The automatic labeling and recognition system, "whale language", won the gold medal of the seventh China International Internet plus innovation and entrepreneurship competition. Wang Jianyuan is a member of the R & D team. He is a deaf child who grew up in a deaf family. He has serious hearing loss and it is very difficult to speak. Sign language is the first and most effective way for him to deal with the world. Over the past 22 years, the identity of deaf people has not made him feel inferior; He could not speak, nor did he delay his becoming a proper "Xueba"; But if you ask him what inconvenience he has encountered from childhood to childhood because of hearing impairment, he will calmly type four words on his mobile phone: all aspects - because too few people know sign language. A typical example is seeing a doctor. If there is no sign language interpreter, even if it is just a headache, it is difficult for "sign language families" like Wang Jianyuan to see a doctor alone - most doctors can't sign language. From September 11, 1991, the first batch of six deaf students entered the school to today, the teachers of the College of the deaf of Tianjin University of technology have accumulated a lot of experience in communicating with the deaf. Yuan Tiantian majored in computer science. As soon as she graduated from graduate school in 2006, she worked in the Department of computer science and technology of artificial College for the deaf. On her first day at work, her challenge was to learn sign language. By reading words and lips, students can understand the teacher, but the teacher can't understand the students quickly. He stumbled to learn sign language. Yuan Tiantian couldn't remember how many times he accompanied the students to see a doctor at one or two o'clock in the morning to help do sign language translation. There are 27.8 million deaf people in China. How can we rely on the power of science and technology to let healthy people enter their world? Yuan Tiantian, Wang Jianyuan and others started the research and development of "whale language" sign language real-time translation system. The real-time translation of "Jingke language" sign language is similar to the sign language version of "voice input method". But the latter is to input voice, machine recognition and convert into text, while the former is to input sign language actions and expressions, machine recognition. To train artificial intelligence into a qualified sign language translator, we have to convert gestures, expressions and big body movements from video to data, and then "teach" them to the computer. Therefore, hearing impaired students who can fully understand and show the beauty of sign language are the most suitable "teachers" for artificial intelligence. The sign language data sets established by many foreign teams can not support sign language translation in complex scenes. A direct reason is that they can not find enough people to collect sign language in a natural state. "Our team is backed by the deaf artificial college. Many members are sign language users and have advantages in the collection of natural sign language." Yuan Tiantian said. It is more important for healthy people to "understand" hearing impaired people In the Winter Olympics, the AI sign language digital people served by Yuan Tiantian's team can help the deaf better understand the expression of the deaf, and the "whale can speak" sign language recognition and translation help the deaf better understand the expression of the deaf with the help of science and technology. What is the difference between the two? To be an AI sign language digital human is to let the deaf understand the expression of healthy people in a familiar way. The core R & D point is sign language generation, while the core R & D point of "whale can language" sign language recognition and translation is sign language recognition. The technology of "whale can language" is more difficult. "Whale can speak" requires two technologies: computer vision and natural language recognition and processing: deaf sign language, including expression, hand movement and limb movement, requires computer fusion recognition, which is computer recognition technology; It is a natural language recognition and processing technology to use computer programs to turn the information recognized by the computer into a language pattern familiar to healthy listeners. AI sign language digital human, to some extent, only involves the process of natural language recognition and processing technology. From the perspective of technology transformation, the industrial landing time of AI sign language digital people will be shorter, the investment in data is relatively small, and it is easier to market the results. "Why do we have to do sign language translation? Deaf people can't hear. Now there is speech recognition! Some deaf people can't speak. Can't they just type with their mobile phone?" Some enterprises once learned from Yuan Tiantian about the sign language real-time translation system under development and wanted to talk about investment. But after talking, they questioned the necessity of the project and felt that sign language translation had no meaning other than "doing good deeds". Why should we devote ourselves to the R & D and transformation of "whale language"? "In the process of communicating with students, I know that the best concern for the disabled is equality. They are not disabled, but different from us." Yuan Tiantian said. Communication is always two-way. We teach the hearing-impaired to write, read and lip talk. They are trying to integrate into the world of healthy people; We healthy listeners should also make some efforts to understand and accept the deaf. Yuan Tiantian said that until today, sign language is still the most natural and efficient expression for many hearing-impaired people, "faster than writing and typing". However, in order to promote barrier free communication, it is obviously unrealistic to require healthy listeners to learn sign language. The "whale can speak" sign language real-time translation system uses high technology to make healthy people try to get closer to the hearing-impaired. (Xinhua News Agency)