The reporter learned from the Macau University of Science and Technology on the 13th that the research team composed of researchers from the Medical College of the University of Science and Technology of Macao, West China Hospital of Sichuan University, the University of Hong Kong and other institutions took the lead in proposing an artificial intelligence based multimodal fusion lung image medical diagnosis model "IRENE", which is a medical auxiliary diagnosis method that uses a unified artificial intelligence model to conduct overall Feature learning on multimodal clinical information at the same time. Relevant research reports have been published in the latest issue of Nature Biomedical engineering. In medical clinical practice, doctors usually need to consider a variety of different modes of medical information to make diagnosis, such as patient medical records, Blood test, image reports, etc. This comprehensive analysis ability requires doctors to have rich medical expertise and long-term clinical experience. If artificial intelligence can be utilized to obtain this comprehensive analysis ability and assist doctors in diagnosis, it will greatly improve medical efficiency and alleviate the shortage of medical resources. Medical image diagnosis based on artificial intelligence has made great progress in recent years, but how to enable computers to integrate and interpret medical images and related clinical information remains a major challenge. To solve the above problems, the team led by Professor Zhang Kang of Macau University of Science and Technology developed "IRENE". This model includes a unified data input processing module and a bidirectional cross modal attention mechanism module, aiming to make decisions by jointly learning the overall features and correlations between different information. This model can effectively integrate medical images, unstructured medical record information and laboratory test data, and use a unified cross modal analysis process to comprehensively process different data, so as to make more accurate judgments. According to Zhang Kang, the team applied the unified model to identify lung diseases and predict COVID-19's adverse clinical symptoms. Compared with the image only model and the Unconformity multimodal diagnosis model, the accuracy rate in identifying lung diseases increased by 12% and 9%, respectively, and in predicting the adverse clinical results of COVID-19 patients increased by 29% and 7%. Zhang Kang stated that this newly developed medical assisted diagnostic method provides a powerful tool for alleviating the shortage of medical resources, and also provides new ideas for integrating any multimodal information with medical artificial intelligence in the future. (Outlook New Era Network)