MATES: A tool for appraising the completeness with which a meta-analysis has been reported

Kyle Morrison; Patrice Pottier; Pietro Pollo; Lorenzo Ricolfi; Coralie Williams; Yefeng Yang; Damien Beillouin; Simone Jaqueline Cardoso; Verónica Ferreira; Brian Gallagher; Jelaine L. Gan; Guang Hao; Mojtaba Keikha; Betina Kozlowsky-Suzuki; T. M. Kiran Kumara; Francesco Latterini; Alexandro B. Leverkus; Erin L. Macartney; Silvina Magdalena Manrique; April Robin Martinig; Ayumi Mizuno; Shanika Nanayakkara; Evangelia Ntzani; Dakis Yaoba Ouédraogo; Edward Pursell; Zachary Simpson; Harriet Sleight; Kok Sin Woon; Ziqian Xia; Mona Ghannad; Eliza Grames; Emily Alden Hennessy; Joanna IntHout; David Moher; Rose E. O’Dea; Matthew J. Page; Paul Whaley; Malgorzata Lagisz; Shinichi Nakagawa

doi:10.1016/j.envint.2025.109935

MATES: A tool for appraising the completeness with which a meta-analysis has been reported

Kyle Morrison
, Patrice Pottier
, Pietro Pollo
, Lorenzo Ricolfi
, Coralie Williams
, Yefeng Yang
, Damien Beillouin
, Simone Jaqueline Cardoso
, Verónica Ferreira
, Brian Gallagher
, Jelaine L. Gan
, Guang Hao
, Mojtaba Keikha
, Betina Kozlowsky-Suzuki
, T. M. Kiran Kumara
, Francesco Latterini
, Alexandro B. Leverkus
, Erin L. Macartney
, Silvina Magdalena Manrique
, April Robin Martinig

Ayumi Mizuno, Shanika Nanayakkara, Evangelia Ntzani, Dakis Yaoba Ouédraogo, Edward Pursell, Zachary Simpson, Harriet Sleight, Kok Sin Woon, Ziqian Xia, Mona Ghannad, Eliza Grames, Emily Alden Hennessy, Joanna IntHout, David Moher, Rose E. O’Dea, Matthew J. Page, Paul Whaley, Malgorzata Lagisz, Shinichi Nakagawa

Biology

Binghamton University

University of New South Wales
Australian National University
Zhejiang University
Université de Montpellier
Universidade Federal de Juiz de Fora
University of Coimbra
Concordia University
Newcastle University
University of the Philippines
Guangdong Pharmaceutical University
Kerman University of Medical Sciences
Universidade Federal do Estado do Rio de Janeiro
ICAR - National Institute of Agricultural Economics and Policy Research, New Delhi
Institute of Dendrology of the Polish Academy of Sciences
University of Granada
Inter-University Institute for Earth System Research in Andalusia (IISTA)
The University of Sydney
University of Alcalá
University of British Columbia
University of Alberta
Westmead Hospital
Brown University
University of Ioannina
Independent Researcher
Anglia Ruskin University
United States Department of Agriculture
University of York
Hong Kong University of Science and Technology
Xiamen University
Stanford University
Massachusetts General Hospital
Harvard University
Radboud University Nijmegen
University of Ottawa
University of Melbourne
Monash University
Lancaster University

Research output: Contribution to journal › Article › peer-review

1 Scopus citations

Abstract

Meta-analysis is commonly a core component of systematic reviews and has become an important method to reconcile conflicting findings, increase statistical power, and chart new research directions. However, poor reporting practices make it challenging to evaluate the validity of meta-analyses. Despite the existence of reporting checklists, a specifically designed tool has yet to be developed to appraise the completeness with which a meta-analysis has been reported. To bridge this gap, we introduce the Meta-analysis Appraisal Tool for Environmental Sciences (MATES). To develop MATES, we adapted a Delphi process involving experts in meta-analysis methodologies, researchers with experience in guideline/appraisal tool development, and editors of relevant journals. The Delphi process had five steps, including three workshops (11–16 participants), a survey (193 participants), and a validation task (30 participants). This iterative development process resulted in a 14-item appraisal tool that reflects the environmental science and research syntheses community’s consensus on essential elements to appraise the completeness with which a meta-analysis has been reported. Validation across 50 meta-analyses demonstrated that the tool is repeatable (average agreement rate: 88.97 %) and time-efficient to implement (17.00 ± 12.23 min). We also outline guidance for interpreting MATES results, describe its potential applications, and reflect on the development process. The authors provide practical implementation guidance for each MATES item, illustrated with real examples in the supplementary material. We also report an extended development methodology to support reproducibility. Finally, we built created a ShinyApp that includes both a training module and an application tool to enhance the usability of MATES ( https://kylemorrisonisshiny99.shinyapps.io/MATES_shiny/ ). Overall, MATES provides authors, readers, stakeholders, and editors with a reliable and accessible tool for appraising the completeness with which a meta-analysis in environmental sciences has been reported.

Original language	English
Article number	109935
Journal	Environment International
Volume	207
DOIs	https://doi.org/10.1016/j.envint.2025.109935
State	Published - Jan 2026

Keywords

Reliability
Reproducibility
Risk of bias
Transparency
Triage

Access to Document

10.1016/j.envint.2025.109935

Cite this

Morrison, K., Pottier, P., Pollo, P., Ricolfi, L., Williams, C., Yang, Y., Beillouin, D., Cardoso, S. J., Ferreira, V., Gallagher, B., Gan, J. L., Hao, G., Keikha, M., Kozlowsky-Suzuki, B., Kiran Kumara, T. M., Latterini, F., Leverkus, A. B., Macartney, E. L., Manrique, S. M., ... Nakagawa, S. (2026). MATES: A tool for appraising the completeness with which a meta-analysis has been reported. Environment International, 207, Article 109935. https://doi.org/10.1016/j.envint.2025.109935

@article{33d6e3d4ce9f416eacb861a87868276c,

title = "MATES: A tool for appraising the completeness with which a meta-analysis has been reported",

abstract = "Meta-analysis is commonly a core component of systematic reviews and has become an important method to reconcile conflicting findings, increase statistical power, and chart new research directions. However, poor reporting practices make it challenging to evaluate the validity of meta-analyses. Despite the existence of reporting checklists, a specifically designed tool has yet to be developed to appraise the completeness with which a meta-analysis has been reported. To bridge this gap, we introduce the Meta-analysis Appraisal Tool for Environmental Sciences (MATES). To develop MATES, we adapted a Delphi process involving experts in meta-analysis methodologies, researchers with experience in guideline/appraisal tool development, and editors of relevant journals. The Delphi process had five steps, including three workshops (11–16 participants), a survey (193 participants), and a validation task (30 participants). This iterative development process resulted in a 14-item appraisal tool that reflects the environmental science and research syntheses community{\textquoteright}s consensus on essential elements to appraise the completeness with which a meta-analysis has been reported. Validation across 50 meta-analyses demonstrated that the tool is repeatable (average agreement rate: 88.97 \%) and time-efficient to implement (17.00 ± 12.23 min). We also outline guidance for interpreting MATES results, describe its potential applications, and reflect on the development process. The authors provide practical implementation guidance for each MATES item, illustrated with real examples in the supplementary material. We also report an extended development methodology to support reproducibility. Finally, we built created a ShinyApp that includes both a training module and an application tool to enhance the usability of MATES ( https://kylemorrisonisshiny99.shinyapps.io/MATES\_shiny/ ). Overall, MATES provides authors, readers, stakeholders, and editors with a reliable and accessible tool for appraising the completeness with which a meta-analysis in environmental sciences has been reported.",

keywords = "Reliability, Reproducibility, Risk of bias, Transparency, Triage",

author = "Kyle Morrison and Patrice Pottier and Pietro Pollo and Lorenzo Ricolfi and Coralie Williams and Yefeng Yang and Damien Beillouin and Cardoso, \{Simone Jaqueline\} and Ver{\'o}nica Ferreira and Brian Gallagher and Gan, \{Jelaine L.\} and Guang Hao and Mojtaba Keikha and Betina Kozlowsky-Suzuki and \{Kiran Kumara\}, \{T. M.\} and Francesco Latterini and Leverkus, \{Alexandro B.\} and Macartney, \{Erin L.\} and Manrique, \{Silvina Magdalena\} and Martinig, \{April Robin\} and Ayumi Mizuno and Shanika Nanayakkara and Evangelia Ntzani and Ou{\'e}draogo, \{Dakis Yaoba\} and Edward Pursell and Zachary Simpson and Harriet Sleight and Woon, \{Kok Sin\} and Ziqian Xia and Mona Ghannad and Eliza Grames and Hennessy, \{Emily Alden\} and Joanna IntHout and David Moher and O{\textquoteright}Dea, \{Rose E.\} and Page, \{Matthew J.\} and Paul Whaley and Malgorzata Lagisz and Shinichi Nakagawa",

note = "Publisher Copyright: {\textcopyright} 2025 The Author(s).",

year = "2026",

month = jan,

doi = "10.1016/j.envint.2025.109935",

language = "English",

volume = "207",

journal = "Environment International",

issn = "0160-4120",

publisher = "Elsevier Ltd.",

}

Morrison, K, Pottier, P, Pollo, P, Ricolfi, L, Williams, C, Yang, Y, Beillouin, D, Cardoso, SJ, Ferreira, V, Gallagher, B, Gan, JL, Hao, G, Keikha, M, Kozlowsky-Suzuki, B, Kiran Kumara, TM, Latterini, F, Leverkus, AB, Macartney, EL, Manrique, SM, Martinig, AR, Mizuno, A, Nanayakkara, S, Ntzani, E, Ouédraogo, DY, Pursell, E, Simpson, Z, Sleight, H, Woon, KS, Xia, Z, Ghannad, M, Grames, E, Hennessy, EA, IntHout, J, Moher, D, O’Dea, RE, Page, MJ, Whaley, P, Lagisz, M & Nakagawa, S 2026, 'MATES: A tool for appraising the completeness with which a meta-analysis has been reported', Environment International, vol. 207, 109935. https://doi.org/10.1016/j.envint.2025.109935

TY - JOUR

T1 - MATES

T2 - A tool for appraising the completeness with which a meta-analysis has been reported

AU - Morrison, Kyle

AU - Pottier, Patrice

AU - Pollo, Pietro

AU - Ricolfi, Lorenzo

AU - Williams, Coralie

AU - Yang, Yefeng

AU - Beillouin, Damien

AU - Cardoso, Simone Jaqueline

AU - Ferreira, Verónica

AU - Gallagher, Brian

AU - Gan, Jelaine L.

AU - Hao, Guang

AU - Keikha, Mojtaba

AU - Kozlowsky-Suzuki, Betina

AU - Kiran Kumara, T. M.

AU - Latterini, Francesco

AU - Leverkus, Alexandro B.

AU - Macartney, Erin L.

AU - Manrique, Silvina Magdalena

AU - Martinig, April Robin

AU - Mizuno, Ayumi

AU - Nanayakkara, Shanika

AU - Ntzani, Evangelia

AU - Ouédraogo, Dakis Yaoba

AU - Pursell, Edward

AU - Simpson, Zachary

AU - Sleight, Harriet

AU - Woon, Kok Sin

AU - Xia, Ziqian

AU - Ghannad, Mona

AU - Grames, Eliza

AU - Hennessy, Emily Alden

AU - IntHout, Joanna

AU - Moher, David

AU - O’Dea, Rose E.

AU - Page, Matthew J.

AU - Whaley, Paul

AU - Lagisz, Malgorzata

AU - Nakagawa, Shinichi

PY - 2026/1

Y1 - 2026/1

N2 - Meta-analysis is commonly a core component of systematic reviews and has become an important method to reconcile conflicting findings, increase statistical power, and chart new research directions. However, poor reporting practices make it challenging to evaluate the validity of meta-analyses. Despite the existence of reporting checklists, a specifically designed tool has yet to be developed to appraise the completeness with which a meta-analysis has been reported. To bridge this gap, we introduce the Meta-analysis Appraisal Tool for Environmental Sciences (MATES). To develop MATES, we adapted a Delphi process involving experts in meta-analysis methodologies, researchers with experience in guideline/appraisal tool development, and editors of relevant journals. The Delphi process had five steps, including three workshops (11–16 participants), a survey (193 participants), and a validation task (30 participants). This iterative development process resulted in a 14-item appraisal tool that reflects the environmental science and research syntheses community’s consensus on essential elements to appraise the completeness with which a meta-analysis has been reported. Validation across 50 meta-analyses demonstrated that the tool is repeatable (average agreement rate: 88.97 %) and time-efficient to implement (17.00 ± 12.23 min). We also outline guidance for interpreting MATES results, describe its potential applications, and reflect on the development process. The authors provide practical implementation guidance for each MATES item, illustrated with real examples in the supplementary material. We also report an extended development methodology to support reproducibility. Finally, we built created a ShinyApp that includes both a training module and an application tool to enhance the usability of MATES ( https://kylemorrisonisshiny99.shinyapps.io/MATES_shiny/ ). Overall, MATES provides authors, readers, stakeholders, and editors with a reliable and accessible tool for appraising the completeness with which a meta-analysis in environmental sciences has been reported.

AB - Meta-analysis is commonly a core component of systematic reviews and has become an important method to reconcile conflicting findings, increase statistical power, and chart new research directions. However, poor reporting practices make it challenging to evaluate the validity of meta-analyses. Despite the existence of reporting checklists, a specifically designed tool has yet to be developed to appraise the completeness with which a meta-analysis has been reported. To bridge this gap, we introduce the Meta-analysis Appraisal Tool for Environmental Sciences (MATES). To develop MATES, we adapted a Delphi process involving experts in meta-analysis methodologies, researchers with experience in guideline/appraisal tool development, and editors of relevant journals. The Delphi process had five steps, including three workshops (11–16 participants), a survey (193 participants), and a validation task (30 participants). This iterative development process resulted in a 14-item appraisal tool that reflects the environmental science and research syntheses community’s consensus on essential elements to appraise the completeness with which a meta-analysis has been reported. Validation across 50 meta-analyses demonstrated that the tool is repeatable (average agreement rate: 88.97 %) and time-efficient to implement (17.00 ± 12.23 min). We also outline guidance for interpreting MATES results, describe its potential applications, and reflect on the development process. The authors provide practical implementation guidance for each MATES item, illustrated with real examples in the supplementary material. We also report an extended development methodology to support reproducibility. Finally, we built created a ShinyApp that includes both a training module and an application tool to enhance the usability of MATES ( https://kylemorrisonisshiny99.shinyapps.io/MATES_shiny/ ). Overall, MATES provides authors, readers, stakeholders, and editors with a reliable and accessible tool for appraising the completeness with which a meta-analysis in environmental sciences has been reported.

KW - Reliability

KW - Reproducibility

KW - Risk of bias

KW - Transparency

KW - Triage

UR - https://www.scopus.com/pages/publications/105025029116

U2 - 10.1016/j.envint.2025.109935

DO - 10.1016/j.envint.2025.109935

M3 - Article

C2 - 41411971

SN - 0160-4120

VL - 207

JO - Environment International

JF - Environment International

M1 - 109935

ER -

MATES: A tool for appraising the completeness with which a meta-analysis has been reported

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this