Study of transfer learning and domain adaptation for language modeling in end-to-end speech recognition

Automatic Speech Recognition (ASR) has become a technical foundation for a natural human-machine interface: speech input.Recently, there have been various daily applications based on ASR: 1) transcription in dictation as smartphone input method, in automatically generated subtitles, as upstream task of speech translation (ST); 2) speech command recognition in a personal assistant like Siri, Alex, Cortana or intent recognition: booking flight, etc; and 3) QA systems like auto attendant phone systems.Deep neural network-based end-to-end (E2E) ASR systems have gained popularity in recent years due to the fast development of deep learning technologies since they can leverage gigantic speech training data with high performance and high efficiency which outperformed human for domains with sufficient data.For conventional ASR systems, independent modules can be trained on audio signals and textual sequences which gives the ASR system the capability to easily be adapted to different scenarios by only changing one of those modules.Compared to conventional ASR systems, E2E ASR systems usually have a single indivisible module for transforming speech to text.If adaptation is necessary, sufficient data from target domain should be prepared or an external language model can be integrated into the system.However, labeled speech data can always be insufficient for a specific domain while text data are much more easily obtained from web pages, books, journals, etc.Because of these reasons, Language models (LM) have been essential in enhancing the performance of end-to-end (E2E) automatic speech recognition (ASR) systems.There are two dimensions of LM integrations: 1) space dimension, how deep LM should be integrated into an E2E ASR model (combining outputs or neural networks); 2) time dimension, when to integrate LM (training stage; decoding stage; or both).This thesis starts from introducing crucial methodologies for modern E2E, then present several contributions considering above dimensions.