Computing & Software 23-087

Abstract

Machine learning (ML) is a method that acquires the ability to identify patterns and tendencies within data. Nevertheless, the standard of reporting ML in research frequently falls below optimal levels, resulting in flawed conclusions and impeding advancements in the field. This is particularly concerning when such inaccuracies are disseminated in literature reviews, which are pivotal for offering researchers insights into a field, identifying current knowledge gaps, and suggesting future directions. Despite the existence of numerous tools for evaluating the quality and risk of bias in studies, there is presently no universal tool designed specifically for assessing the reporting standards of ML in literature.

Researchers at McMaster University have developed a novel screening tool named STAR-ML (Screening Tool for Assessing Reporting of Machine Learning), along with a comprehensive user guide. This tool can be utilized during full-text screening in reviews, providing a systematic approach to evaluating the quality of ML research. To examine the efficacy of this tool, a preliminary scoping review was conducted focusing on machine learning applications in chronic pain. The study investigated the time required for paper screening and the impact of varying threshold selections on the papers included. STAR-ML offers researchers a dependable and methodical approach to assess the reporting quality of machine learning studies, aiding in informed decisions regarding study inclusion in scoping or systematic reviews. Additionally, this research furnishes authors with guidance on selecting inclusion thresholds and proficiently utilizing the tool. Ultimately, STAR-ML can function as a valuable checklist for researchers aiming to develop or implement machine learning techniques with precision.

Applications

• Machine Learning Screening tool

Advantages

• Efficient Screening: Using the tool, screening takes 4-5 minutes on average, enabling rapid assessment and covering more literature.
• Improved Reporting: STAR-ML helps researchers self-assess and enhance reporting quality in ML work, guiding best practices for more accurate models.
• Promoting Best Practices: STAR-ML ensures high-quality ML papers in reviews, leading to precise conclusions and knowledge dissemination.
• Increased Study Quality: A score of 7 or 8 on STAR-ML correlates with higher quality studies, aiding reviewers in selecting reliable research.
• New Researcher Support: The tool aids new researchers, providing a standardized checklist to improve reporting and research quality

References

• Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research
• STAR-ML: A Rapid Screening Tool for Assessing Reporting of Machine Learning in Research