Improving Evidence-Based Practice Through the Development and Validation of Key Words to Identify Intervention Articles in Communication Disorders

Purpose: Although evidence-based practice (EBP) promotes better clinical practice, implementing it in speech-language pathology is challenging. A limited number of intervention studies and inconsistent use of key words in abstracts complicates the implementation of EBP. Given time constraints in clinical practice, identifying relevant research from study abstracts would provide an efficient method for identifying intervention studies. In this project, we compare the classification accuracy of an artificial intelligence (AI)-created set of key words and a set of evidencebased key words that were developed through an analysis of commonly used words in intervention abstracts from American Speech-Language-Hearing Association (ASHA) journals. Method: Abstracts from three major ASHA journals were crawled using Selenium and WebDriver Manager, creating a large database of abstracts for analysis. In Study 1, a random sample of 180 abstracts was annotated as reporting on intervention or nonintervention studies to develop a set of evidence-based key words. In Study 2, classification accuracy was calculated to validate these key words by comparing them with a set of AI-generated (ChatGPT-4.0) key words. Results: The results suggested that 12%–15% of studies report on an intervention study. Evidence-based key words had higher sensitivity (85%) and a higher positive predictive value (27%) but a lower specificity rate (67%). Also, the likelihood ratio suggested that evidence-based key words had a moderate capacity to identify nonintervention studies (true negatives) accurately. Conclusions: Evidence-based key words have the potential to accurately classify intervention studies, addressing a key barrier to EBP implementation in clinical practice. Future research should focus on refining key words by integrating AI and promoting standardized journal reporting by incorporating uniform key words in abstracts.