Abstract

Vision Transformer models, such as ViT, Swin Transformer, and Transformer-in-Transformer, have recently gained significant traction in computer vision tasks due to their ability to capture the global relation between features which leads to superior performance. However, they are compute-heavy and difficult to deploy in resource-constrained edge devices. Existing hardware accelerators, including those for the closely-related BERT transformer models, do not target highly resource-constrained environments. In this paper, we address this gap and propose ViTA - a configurable hardware accelerator for inference of vision transformer models, targeting resource-constrained edge computing devices and avoiding repeated off-chip memory accesses. We employ a head-level pipeline and inter-layer MLP optimizations, and can support several commonly used vision transformer models with changes solely in our control …

Metadata

publication

2023 IEEE International Symposium on Circuits and Systems (ISCAS), 1-5, 2023

year

2023

publication date

2023/5/21

authors

Shashank Nag, Gourav Datta, Souvik Kundu, Nitin Chandrachoodan, Peter A Beerel

link

https://ieeexplore.ieee.org/abstract/document/10181988/

resource_link

https://arxiv.org/pdf/2302.09108

conference

2023 IEEE International Symposium on Circuits and Systems (ISCAS)

pages

1-5

publisher

IEEE