Non-Autoregressive Neural Machine Translation

In recent years, a number of mehtods for improving the decoding speed of neural machine translation systems have emerged. One of the approaches that pro- poses fundamental changes to the model architecture are non-autoregressive models. In standard autoregressive models, the output token distributions are conditioned on the previously decoded outputs. The conditional dependence al- lows the model to keep track of the state of the decoding process, which improves the fluency of the output. On the other hand, it requires the neural network computation to be run sequentially, and thus it cannot be parallelized. Non- autoregressive models impose conditional independence on the output distri- butions, which means that the decoding process is parallelizable and hence the decoding speed improves. A major drawback of this approach is lower trans- lation quality compared to the autoregressive models. The goal of the non- autoregressive translation research is to find methods that improve the trans- lation quality, while retaining high decoding speed. In this thesis, we explore the research progress so far and identify flaws in the generally accepted eval- uation methodology. We experiement with non-autoregressive models trained with connectionist temporal classification. We find that even though our models...