Inhibition: Efficient Attention for Transformers
Summary
The Inhibitor: ReLU and Addition-Based Attention for Computationally Efficient Quantized Transformers. Quantized Transformers without dot-product and Softmax. Replace with ReLU activation. Reduced precision for efficiency.
Themes
Keywords
efficient transformers, quantization, ReLU attention, Inhibitor, homomorphic encryption
Poster
Click image to open full size