Extensive Evaluation of Transformer-based Architectures for Adverse Drug Events Extraction

by   Simone Scaboro, et al.

Adverse Event (ADE) extraction is one of the core tasks in digital pharmacovigilance, especially when applied to informal texts. This task has been addressed by the Natural Language Processing community using large pre-trained language models, such as BERT. Despite the great number of Transformer-based architectures used in the literature, it is unclear which of them has better performances and why. Therefore, in this paper we perform an extensive evaluation and analysis of 19 Transformer-based models for ADE extraction on informal texts. We compare the performance of all the considered models on two datasets with increasing levels of informality (forums posts and tweets). We also combine the purely Transformer-based models with two commonly-used additional processing layers (CRF and LSTM), and analyze their effect on the models performance. Furthermore, we use a well-established feature importance technique (SHAP) to correlate the performance of the models with a set of features that describe them: model category (AutoEncoding, AutoRegressive, Text-to-Text), pretraining domain, training from scratch, and model size in number of parameters. At the end of our analyses, we identify a list of take-home messages that can be derived from the experimental data.


page 1

page 2

page 3

page 4


Improving Adverse Drug Event Extraction with SpanBERT on Different Text Typologies

In recent years, Internet users are reporting Adverse Drug Events (ADE) ...

Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical text

Detecting protein-protein interactions (PPIs) is crucial for understandi...

Language Modelling for Source Code with Transformer-XL

It has been found that software, like natural language texts, exhibits "...

Interpreting Language Models Through Knowledge Graph Extraction

Transformer-based language models trained on large text corpora have enj...

FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems

This paper presents the FlowTransformer framework, a novel approach for ...

ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them

This paper presents the winning system for the propaganda Technique Clas...

Evaluating BERT-based Pre-training Language Models for Detecting Misinformation

It is challenging to control the quality of online information due to th...

Please sign up or login with your details

Forgot password? Click here to reset