Tinjauan Arsitektur Transformer Dan Penerapannya Pada Pemrosesan Bahasa Alami

Abstract

The rapid development in the field of natural language processing (NLP) cannot be separated from the emergence of the Transformer architecture first introduced in 2017. This architecture relies on a self-attention mechanism that allows the model to process all tokens at once, unlike the sequential approach in RNN or LSTM. This article presents a literature review of the Transformer architecture, its main components such as multi-head attention, positional encoding, and feed-forward network, as well as its various applications in NLP tasks such as machine translation, sentiment analysis, and text generation. In addition, the advantages and limitations of the Transformer compared to previous architectures are discussed, along with challenges such as high computational requirements and large memory consumption. This review is expected to provide a comprehensive basic understanding of the Transformer for researchers and practitioners in the field of machine learning.