Exploring Resource-Efficient Methods on Inhibitor Attention-Based Transformers

In this thesis, quantization and knowledge distillation are applied to a transformer model with the inhibitor attention mechanism and the results are compared to that of a scaled dot-product attention (SDPA) model. The two main questions that are explored in this thesis are (1) whether an inhibitor model can be trained with knowledge distillation to match the performance of a SDPA model and (2) if inhibitor models are more robust than SDPA at low bit-width quantization. For the knowledge distillation, previous work trained an inhibitor model with DistilBERT as the teacher, yielding a GLUE score of 74.3. This work used a similar distillation method but replaced the teacher model with BERT, which resulted in a score of 75.0 on GLUE. This is not a significant increase, so using a bigger teacher model does not yield better results in this case. Quantization on the inhibitor model was also investigated and results of equivalent (number of parameters) inhibitor and dot-product based models were compared. 8-bit integer quantization showed little performance degradation (compared to floating-point models) for both quantization-aware training as well as post-training quantization for both model types. In the case of 4-bit quantization-aware training, the performance degradation was greater for the dot-product based model compared to the inhibitor model, where the inhibitor model scored 64.2 on GLUE while the dot-product model scored 59.3, down from 74.3 and 77.5 respectively. At this bit-width, quantization-aware training outperformed post-training quantization with the latter scoring 36.6 and 45.5 (GLUE score) for dot-product and inhibitor models respectively. With these results it can be concluded that inhibitor attention outperforms scaled dot-product attention at low bit-width quantization (specifically 4-bit in this case). It was found that inhibitor attention brought some challenges for integer-only calculations but one potential solution for computing inhibitor attention with integer-only arithmetic was found. It is not entirely efficient however, as many rescaling operations are required. It is therefore unclear how an efficient integer-only inhibitor model can be implemented using standard quantization schemes.