Exploration of fine-tuning and inference time of large pre-trained language models in NLP

In this master's thesis, we perform an analysis based on the language models BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), DistilBERT (Sanh et al., 2020), ALBERT (Lan et al., 2019) and XLNET (Yang et al., 2019) whose architectures are based on the Transformer encoder which relies solely on an attention mechanism.We incorporate a measuring of the fine-tuning and inference time in the Python module transformers which was implemented by huggingface (Wolf et al., 2019).In the scope of this master's thesis, the models are fine-tuned on the General Language Understanding Evaluation tasks (Wang et al., 2019), a collection of nine sentenceor sentence-pair language understanding tasks.To monitor our analyses, we used the tool "Weights & Biases" (wandb) (Biewald, 2020) where a logging of hyperparameters and output metrics is possible.The goal of this master's thesis is to develop a diverse benchmark setting and evaluate the respective fine-tuning as well as inference times across different hyperparameters, language models and tasks.More precisely, a hyperparameter sweep with the maximum sequence length and the batch size are regarded, taking on the values {128, 256, 512} and {8, 16, 32}, respectively.We show that the maximum sequence length, the task size as well as the model have an effect on the fine-tuning time, whereas the batch size seems to be of secondary importance.Moreover, the maximum sequence length, the batch size and the choice of the model affect the inference time, whereas the task itself does not show such results.