Feedforward Neural Networks and the Transformer model

This thesis presents the mathematical framework underlying two central architectures in modern deep learning: the feedforward neural network and the transformer model. Beginning from first principles, the report introduces the perceptron model and builds toward the multi-layer perceptron by incorporating activation functions, layer structure, and matrix notation. The learning process is examined through parameter initialization, the cost function, the backpropagation algorithm, stochastic gradient descent, and regularization techniques including early stopping and dropout. Hyperparameter tuning via grid search is also discussed. The thesis then transitions to the transformer model, introduced in the 2017 paper "Attention Is All You Need," covering the mathematical formulation of scaled dot-product self-attention, multi-head attention, tokenization and positional encoding. The encoder and decoder modules are derived in full matrix form, detailing their respective sublayers, residual connections and layer normalization. Finally, the optimization and regularization choices of the original transformer are discussed. The aim throughout is to bridge the gap between intuitive descriptions and mathematical exposition.