References & Citations
Computer Science > Software Engineering
Title: Challenges of Using Pre-trained Models: the Practitioners' Perspective
(Submitted on 23 Apr 2024 (v1), last revised 1 May 2024 (this version, v2))
Abstract: The challenges associated with using pre-trained models (PTMs) have not been specifically investigated, which hampers their effective utilization. To address this knowledge gap, we collected and analyzed a dataset of 5,896 PTM-related questions on Stack Overflow. We first analyze the popularity and difficulty trends of PTM-related questions. We find that PTM-related questions are becoming more and more popular over time. However, it is noteworthy that PTM-related questions not only have a lower response rate but also exhibit a longer response time compared to many well-researched topics in software engineering. This observation emphasizes the significant difficulty and complexity associated with the practical application of PTMs. To delve into the specific challenges, we manually annotate 430 PTM-related questions, categorizing them into a hierarchical taxonomy of 42 codes (i.e., leaf nodes) and three categories. This taxonomy encompasses many PTM prominent challenges such as fine-tuning, output understanding, and prompt customization, which reflects the gaps between current techniques and practical needs. We discuss the implications of our study for PTM practitioners, vendors, and educators, and suggest possible directions and solutions for future research.
Submission history
From: Taichuan Li [view email][v1] Tue, 23 Apr 2024 03:39:14 GMT (1253kb,D)
[v2] Wed, 1 May 2024 04:53:55 GMT (1253kb,D)
Link back to: arXiv, form interface, contact.