Description du sujet de thèse

Medical emergency call triage aims to assess the level of emergency and direct the appropriate response. Given the significant impact of this response on the patient's health trajectory, improving the triage process is of utmost importance. Rapid decision-making with limited access to comprehensive medical information is required [1]. Predictive models based on large volumes of call data are believed to significantly enhance the safety and accuracy of decisions, e.g., in the case of cardiac arrest [2]. In parallel, learning deep patient representations from healthcare data has rapidly followed up trends in NLP [3], even through reaching application in emergency medicine [4]. However, the evolution and impact towards large language models (LLMs) has not yet been evaluated in the context of Emergency Medical Services (EMS) and call triage.

The objective of this PhD is to adapt an open, general-purpose LLM such as Llama2 [5] or Falcon [6] to the medical emergency call domain so as to help EMS physicians. Fine-tuning the LLM on in-domain data [7] is useful but not sufficient, due to the small amount of such data to adapt the LLM to both domain and task [8]. Instead, we will seek to incorporate structured medical knowledge in the form of medical ontologies such as the Unified Medical Language System (UMLS) [9] or memory modules such as in Memory Augmented Neural Networks [10]. We will also augment the LLM using semi-structured medical knowledge [2,3], e.g., using an in-house emergency medical note dataset which reports the physician's observations of a patient's medical state in textual form using sections such as “Comorbidities”, “Treatment”, and “Clinical examination”, as well as non-textual data including lab results, a severity score, the ICD-10 (International Classification of Diseases) diagnostics billing code, and the clinical outcome. The overall challenge will be to align the representations of medical emergency calls and these other knowledge sources. Different knowledge sources may translate into different data augmentation and/or fine-tuning approaches to improve the LLM while avoiding catastrophic forgetting.

Results will be evaluated on the SimSAMU dataset, a collection of acted medical emergency calls with diarization, transcription, dialog act, and slot filling annotations. The duration of the calls is 1 to 8 min, with a total duration of 3 hours. Models will be evaluated in terms of perplexity and other metrics such as Slot Error Rate, Sentence Level Semantic Accuracy, F1-scores on subtasks (e.g., dialog act detection), and BLEU/METEOR for surface quality. The quality of the latent representation learned by the model will also be assessed by predicting the severity score, the ICD-10 diagnostics billing code and the outcome. AP-HP experts, led by Dr. Ivan Lerner, will manually review the quality of the system's response.
[1] K. Bohm and L. Kurland, “The accuracy of medical dispatch — A systematic review”, Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 26:94, 2018.
[2] M. L. Scholz, H. Collatz-Christensen, S. N. F. Blomberg, S. Boebel, J. Verhoeven, and T. Krafft, “Artificial intelligence in Emergency Medical Services dispatching: assessing the potential impact of an automatic speech recognition software on stroke detection taking the Capital Region of Denmark as case in point”, Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 30(1):36, 2022.
[3] X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, et al., “A large language model for electronic health records”, Digital Medicine 5:194, 2022.
[4] J. S. Obeid, E. R. Weeda, A. J. Matuskowitz, K. Gagnon, T. Crawford et al., “Automated detection of altered mental status in emergency department clinical notes: a deep learning approach”, BMC Medical Informatics and Decision Making 19:164, 2019.
[5] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi et al., “Llama 2: Open foundation and fine-tuned chat models”, arXiv preprint arXiv:2307.09288, 2023.
[6] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli et al., “The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only”, arXiv preprint arXiv:2306.01116, 2023.
[7] I. A. Sheikh, E. Vincent, I. Illina, “Training RNN language models on uncertain ASR hypotheses in limited data scenarios”, Computer Speech and Language, pp.101555, 2023.
[8] G. Guibon, M. Labeau, L. Lefeuvre, and C. Clavel, “An adaptive layer to leverage both domain and task specific information from scarce data”, in AAAI Conference on Artificial Intelligence, 37(6), 2023.
[9] I. Lerner, N. Paris, and X. Tannier, “Terminologies augmented recurrent neural network model for clinical named entity recognition”, Journal of Biomedical Informatics 102: 103356, 2020.
[10] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, “Meta-learning with memory-augmented neural networks”, in International Conference on Machine Learning (ICML), pp. 1842–1850, 2016.

Contexte de travail

This PhD is part of the ANR LLM4All project (2023-2027), which aims to design continual learning and footprint reduction mechanisms for LLMs, and to exploit them in challenging spoken dialog scenarios, including medical emergency calls. It will be co-supervised by Gaël Guibon (https://gguibon.github.io/), Dr. Ivan Lerner (https://scholar.google.fr/citations?user=1TglQmsAAAAJ), and Emmanuel Vincent (https://members.loria.fr/EVincent/). . The PhD student will have the opportunity to spend time in both the Synalp (https://synalp.gitlabpages.inria.fr/synalp-website/) and Multispeech (https://team.inria.fr/multispeech/) teams at LORIA and the PRIME team at Assistance Publique - Hôpitaux de Paris (AP-HP) (https://www.aphp.fr/), and to benefit from the hands-on expertise of Dr. Gustave Toury, an EMS physician at SAMU 92 — the French EMS.