References & Citations
Computer Science > Machine Learning
Title: Towards smaller, faster decoder-only transformers: Architectural variants and their implications
(Submitted on 22 Apr 2024 (v1), last revised 24 Apr 2024 (this version, v2))
Abstract: Research on Large Language Models (LLMs) has recently seen exponential growth, largely focused on transformer-based architectures, as introduced by [1] and further advanced by the decoder-only variations in [2]. Contemporary studies typically aim to improve model capabilities by increasing both the architecture's complexity and the volume of training data. However, research exploring how to reduce model sizes while maintaining performance is limited. This study introduces three modifications to the decoder-only transformer architecture: ParallelGPT (p-gpt), LinearlyCompressedGPT (lc-gpt), and ConvCompressedGPT (cc-gpt). These variants achieve comparable performance to conventional architectures in code generation tasks while benefiting from reduced model sizes and faster training times. We open-source the model weights and codebase to support future research and development in this domain.
Submission history
From: Sathya Krishnan Suresh [view email][v1] Mon, 22 Apr 2024 06:19:46 GMT (832kb,D)
[v2] Wed, 24 Apr 2024 03:52:49 GMT (833kb,D)
Link back to: arXiv, form interface, contact.