Microbial named entity recognition using BERT models

Bacteria are critical subjects of microbiological research that span many rapidly-growing fields of study.The development and widespread application of high-throughput sequencing has led to more microbial data being collected in recent years than ever before.This study investigates the capabilities of the popular Natural Language Processing (NLP) model Bidirectional Encoder Representations from Transformers (BERT) on the relatively understudied text mining domain of microbiology.This is done by fine-tuning a variety of BERT models (BERT, DistilBERT, SciBERT, BioBERT, PubMedBERT) on the Bacteria Biotope 2019 Open Shared Task (BB2019-OST) corpus of annotated microbial research text and evaluating the best performing models on the Named Entity Recognition (NER) task.Following this, an in-depth error analysis was conducted to gain insights into BERT's entity recognition capabilities.Finally, to investigate performance capabilities further, learning rate and batch size hyperparameters were tuned to increase F1-score.The best BERT model in the comparison was BioBERT, earning an F1-score of 73.82 (±1.04) with default hyperparameters, and 75.35 (±0.62) with tuned hyperparameters.BioBERT had better F1-scores and entity-level statistics, despite PubMedBERT ranking high in biomedical NLP benchmarks.This suggests that the generality of the pretraining corpora of BERT models is particularly important for text mining in the microbial domain.