PB1132 - Enhancing adverse drug event surveillance through natural language processing: Focus on hemorragic events’ identification from clinical narratives

Tuesday, June 25, 2024

13:45 – 14:45 ICT

Room: Exhibition Hall

Presenting Author(s)

Frederic Gaspar, PhD

Postdoctoral researcher
School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland
Lausanne, Vaud, Switzerland

Co-Author(s)

MZ

Mehdi Zayerne

Data Scientist, Master Student
École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
Lausanne, Vaud, Switzerland
Coumau Claire, PharmD

PhD Student
School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland
Lausanne, Vaud, Switzerland
ZS

Zachary Schillaci

Head of AI
Effixis SA, Lausanne, Switzerland
Lausanne, Vaud, Switzerland
MB

Marie Bettex

Medical intern in public health
Unisanté Center for Primary Care and Public Health, Department of Epidemiology and Health Systems, University of Lausanne, Lausanne, Switzerland
Lausanne, Vaud, Switzerland
EB

Elliott Bertrand

Data Science Director
Effixis SA, Lausanne, Switzerland
Lausanne, Vaud, Switzerland
ML

Marie-Annick Le Pogam

Associate Chief of Clinic
Unisanté Center for Primary Care and Public Health, Department of Epidemiology and Health Systems, University of Lausanne, Lausanne, Switzerland
Lausanne, Vaud, Switzerland
CC

Chantal Csajka

Full Professor
School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland
Lausanne, Vaud, Switzerland

Background: Hemorrhagic incidents significantly affect patient outcomes and healthcare costs but are often under-reported in Electronic Health Records (EHRs). Traditional detection methods that rely on rule-based strategies and structured data have proven ineffective, highlighting the need for more efficient detection approaches.

Aims: The aim of this study was to develop and evaluate a natural language processing (NLP) model to accurately detect hemorrhagic events from patient discharge letters and categorize them in terms of time of occurrence (before or during stay) and severity and compare the effectiveness of the NLP model with Classification of Diseases, 10the revision (ICD-10) coding.

Methods: Firstly, manual annotations were performed by clinicians to classify documents and sentences as ‘no hemorrhage', 'clinically significant hemorrhage', 'severe hemorrhage', or 'history hemorrhage (prior to stay)', to form development and validation sets for model testing. Secondly, two methods were developed: a machine learning NLP model using both binary logistic regression (BOW-LOG) and support vector machine (BOW-SVM) classifiers and a rule-based model using specific hemorrhage ICD-10 codes. Models were assessed for accuracy, precision, recall, F1 score, and, for the NLP model, Receiver Operating Characteristic (ROC) curves by comparison with manual annotations.

Results: Analysis of 400 discharge letters containing 65,706 sentences demonstrated that the NLP model outperformed its rule-based counterpart, particularly in the detection of 'clinically significant' and 'severe haemorrhage', with macro-averages of 0.81 for accuracy and 0.80 for F1 score (Table 1). It also discriminated accurately between current and past haemorrhage. The ROC curve for the NLP model (Figure 1) shows a high true positive rate with minimal false positives, indicating its strong diagnostic ability.

Conclusion(s): This research marks a significant advancement in artificial intelligence applications in healthcare, presenting a robust NLP model that outperforms traditional ICD-10 coding in detecting bleeding events reported within clinical notes.