Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity

Kevin Volf; Liam Kelly; Enrique García Bengoechea; Bláthín Casey; Anna Gobis; Jeroen Lakerveld; Joanna Zukowska; Peter Gelius; Sven Messing; Sarah Forberger; Catherine B. Woods

doi:10.12688/hrbopenres.13089.1

Home Browse Policy Evaluation Network (PEN): Protocol for systematic literature...

ALL Metrics

Views

159

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity

[version 1; peer review: 1 approved with reservations]

Kevin Volf¹, Liam Kelly¹, Enrique García Bengoechea¹, [...] Bláthín Casey¹, Anna Gobis², Jeroen Lakerveld³, Joanna Zukowska², Peter Gelius⁴, Sven Messing⁴, Sarah Forberger⁵, Catherine B. Woods¹

Kevin Volf¹, Liam Kelly¹, [...] Enrique García Bengoechea¹, Bláthín Casey¹, Anna Gobis², Jeroen Lakerveld³, Joanna Zukowska², Peter Gelius⁴, Sven Messing⁴, Sarah Forberger⁵, Catherine B. Woods¹

PUBLISHED 04 Sep 2020

Author details Author details

¹ Physical Activity for Health, Health Research Institute, Department of Physical Education and Sport Sciences, University of Limerick, Limerick, Ireland
² Faculty of Civil and Environmental Engineering, Gdansk University of Technology, Gdansk, Poland
³ Amsterdam University Medical Center, Amsterdam, The Netherlands
⁴ Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
⁵ Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany

Kevin Volf
Roles: Conceptualization, Methodology, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Liam Kelly
Roles: Methodology, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Enrique García Bengoechea
Roles: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Bláthín Casey
Roles: Writing – Review & Editing

Anna Gobis
Roles: Writing – Review & Editing

Jeroen Lakerveld
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Joanna Zukowska
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Peter Gelius
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Sven Messing
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Sarah Forberger
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Catherine B. Woods
Roles: Conceptualization, Methodology, Project Administration, Supervision, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Introduction: Over 40 million deaths annually are due to noncommunicable diseases, 15 million of these are premature deaths and physical inactivity attributes an estimated 9% to this figure. Global responses have included the sustainable development goals and the global action plan on physical activity. Both point to policy action in physical activity (PA) to address change, yet the impact of policy is unknown. The protocol described outlines a systematic literature review that will be undertaken by the Policy Evaluation Network to address this knowledge gap.
Protocol: This review of school PA policies is the first of seven planned reviews. The seven best investments for promotion of population PA identified in the Toronto Charter (whole-of-school programmes, transport policy, urban design policy, primary health care policy, public education policy, community programmes and sport programmes) will form the basis of these reviews. Seven individual scientific literature searches across six electronic databases, using key concepts of policy, PA, evaluation and a distinct concept for each area will be conducted. This will be supplemented with a search of the reference list of included articles. Methodological quality will be assessed and overall effectiveness for each included study will be described according to pre-determined categories.
Conclusions: The review will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined impact on PA directly or indirectly. By collating the evidence, and demonstrating the depth of the science base which informs these policy recommendations, this review will provide guidance to policymakers to use evidence-based or evidence-informed policies to achieve the 15% relative reduction in physical inactivity as defined by the ‘Global Action Plan on Physical Activity’.
Registration: PROSPERO CRD42020156630 (10/07/2020).

Keywords

physical activity, policy, protocol, systematic review, evaluation

Corresponding author: Catherine B. Woods

Competing interests: No competing interests were disclosed.

Grant information: Health Research Board [JPI-HDHL-PEN-2017-1].
The PEN project is funded by the Joint Programming Initiative (JPI) “A Healthy Diet for a Healthy Life”, a research and innovation initiative of EU member states and associated countries. The funding agencies supporting this work are (in alphabetical order of participating countries): Germany: Federal Ministry of Education and Research (BMBF); Ireland: Health Research Board (HRB); Italy: Ministry of Education, University and Research (MIUR); The Netherlands: The Netherlands Organisation for Health Research and Development (ZonMw); New Zealand: The University of Auckland, School of Population Health; Norway: The Research Council of Norway (RCN); Poland: The National Centre for Research and Development (NCBR). Additionally, the French partners acknowledge the support through the Institute National de la Recherche Agronomique (INRA).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2020 Volf K et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Volf K, Kelly L, García Bengoechea E et al. Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity [version 1; peer review: 1 approved with reservations]. HRB Open Res 2020, 3:62 (https://doi.org/10.12688/hrbopenres.13089.1) First published: 04 Sep 2020, 3:62 (https://doi.org/10.12688/hrbopenres.13089.1) Latest published: 17 Jan 2022, 3:62 (https://doi.org/10.12688/hrbopenres.13089.4)

Introduction

Physical activity (PA) is defined as “any bodily movement produced by skeletal muscles that requires energy expenditure” (Caspersen et al., 1985). The relationship between PA levels and health outcomes is well established (Rutten et al., 2016). Insufficient PA has been identified by the World Health Organisation (WHO) as the fourth leading risk factor for mortality worldwide (WHO, 2009) and in 2012 it was estimated that 9% (range 5.1 – 12.5%) of global premature mortality can be attributed to physical inactivity (Lee et al., 2012). The European region has been strongly affected by the costs of inactivity, absorbing 16.9% of the substantial disability that inactivity causes through its contribution to morbidity from coronary heart disease (CHD), cancer, stroke and diabetes and 21.8% of the healthcare cost (Ding et al., 2016; Lee et al., 2012).

This epidemiological evidence reveals inactivity to be a substantial public health issue and advocacy by public health specialists and the academic community has demanded policy responses to this issue. For the purposes of this document, policy should be understood as “decisions, plans, and actions that are implemented by national or regional governments to achieve specific health promotion goals within a society” (Lakerveld et al., 2020). As indicated by the WHO (WHO Regional Office for Europe, 2010), policy can give support, coherence and visibility at the political level, while making it possible for the organisations involved at national, regional, and local levels – e.g., national government sectors, regional or local authorities, stakeholders, and the private sector – to be logical and consistent in their actions to achieve a shared goal. This definition refers to food and PA environments, systems and behaviours (WHO Regional Office for Europe, 2010). In order to reflect the complexity of the policies that may affect the PA policy environment, a distinction is made between “direct” policies, which refers to policies where improving the PA environment and increasing participation is the primary aim, and “indirect” policies, where the primary aim of the policy is not to increase PA but this may occur as a co-benefit of successful implementation. The International Society for Physical Activity and Health (ISPAH) was established in 2009 (Kohl et al., 2012) and numerous articles and editorials in leading academic and medical journals have pointed out the need to address physical inactivity (Bull & Bauman, 2011; Das & Horton, 2012; Kohl et al., 2012; Woods & Mutrie, 2012).

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These seven domains provide a structure for systematic literature review search.

Over recent years there has been an acceleration in the production of policy responses to the epidemics of PA and sedentary behaviour (Klepac Pogrmilovic et al., 2018). The Global Observatory for Physical Activity (GoPA) reports that by 2013, 139 countries were members of its PA advocacy alliance and 26.6% of these countries had already published a stand-alone PA plan (Ramirez Varela et al., 2016). Furthermore, in 2013, the WHO published a document which recognised PA as a part of the global effort to combat non-communicable diseases (NCDs) (WHO, 2013).

A significant development occurred in 2017 when, in response to demands for direction on the problem of physical inactivity, the WHO committed to publishing a stand-alone action plan on this issue. This commitment was realised in 2019 when the WHO published the Global Action Plan on Physical Activity (GAPPA), which targeted an even more ambitious PA target than the previous NCD plan (WHO, 2019).

The recent rise in the number of national PA policies allows research into the question of which of these policies are effective in increasing PA. A scoping review published in 2016 provided evidence that research into policy effectiveness lagged behind research that links PA to health and research that links PA interventions to behaviour (Rutten et al., 2016). However, with the increase in the number of PA policies there may have been a concomitant rise in research examining the effectiveness of these policies. Furthermore, to the best available knowledge, no project has linked existing policy statements with research that corroborates or discredits the effectiveness of these statements.

As part of the Joint Programming Initiative on a Healthy Diet for a Healthy Life, researchers from 28 institutes in seven European countries (France, Germany, Ireland, Italy, Norway, Poland, and the Netherlands) and New Zealand combine their expertise to form a Policy Evaluation Network (PEN) (Lakerveld et al., 2020). PEN’s vision is to provide Europe with tools to identify, evaluate and benchmark policies designed to directly or indirectly address unhealthy lifestyle behaviours which contribute to overweight and obesity, while accounting for existing health inequities. Using structured evaluation principles and methods, PEN will examine the content, implementation and impact of lifestyle policies across Europe and will build on existing knowledge. PEN will provide an overview of the ‘best’ public policies most likely to sustainably support more favourable health behaviours. This protocol paper outlines the methodology for the first of several complementary systematic literature reviews designed to determine the impact of policy on physical activity outcomes based on the different policy domains identified in the “seven best investments” document (ISPAH, 2012). This review will provide evidence supporting the development of a tool named the Physical Activity Policy Environment Index (PA EPI), based on similar principles to the food environment policy index (Food EPI) (Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve physical activity outcomes across these settings.

Protocol

Original material examining the evidence of what works in terms of direct and indirect policies on PA will be identified in the following ways:

(1) A search, with no date restrictions, of the following electronic databases: four specialized sport science or biomedical databases, MEDLINE (Ebsco), SportDiscus, Cinahl, and Cochrane library, and two general social science databases, Web of Science and Scopus. Search results will be limited to articles that are identified through searching the titles and abstracts.
(2) Manual reference checks of identified original studies.
(3) Publicly available English-language resources and documents of major national and international stakeholders will be searched to identify existing reviews and position papers discussing the evidence of what works in terms of direct and indirect policies on PA, e.g. the WHOs European database on Nutrition, Obesity and Physical Activity (NOPA), Global Action Plan on Physical Activity (GAPPA), the European Physical Activity Strategy (EPAS) (WHO Regional Office for Europe, 2015) and the European Physical Activity Guidelines (EPAG) (European Commission, 2008).

A content analysis was performed on the Toronto Charter supplementary document (ISPAH, 2012). The document declares that the policy interventions should prioritise the following key areas for intervention: whole-of-school programmes, transport policy, urban design policy, primary health care policy, public education policy, community programmes and sport programmes. This document was searched for key words to be included in the search syntax. Researchers consulted with librarians and other research staff for suggestions on search terms.

The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy with seven distinct search concepts. The basic search strategy will consist of three search concepts (Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword of “physical activity”; and search concept three (C3), which will do the same for the keyword of “impact”. The three search terms will be combined with the Boolean operator “AND” (Table 1).

Table 1. Search terms.

Keyword	Synonyms
“Policy”	(MH "Policy") OR (MH "Public Policy") OR (MH "Policy Making") OR (“policy”) OR (“policies”) OR (“national policy”) OR (“national framework”) OR (“policy framework”) OR (“policy action”) OR (“legislation”) OR (“strategy”) OR (“policy making”)
“Physical Activity”	(MH "Exercise") OR (MH “Sedentary Behavior”) OR (“physical activit”) OR (“physical inactivity”) OR (“play”)OR (“physical education”) OR (“sedentar”) OR (“sitting”) OR (“healthy lifestyle”) OR (“health initiative”)
“Impact”	(“evaluat”) OR (“impact”) OR (“appraisal”) OR (“effect”) OR (“assessment”)
“Whole of School Approach”	“Whole-of-school” OR “Whole School” OR “Whole of School” OR WSCC OR “school intervention” OR “school based intervention” OR “school initiative” OR “school based initiative” OR “school program*” OR “School health” OR “Wellness” OR “well-being”

Abbreviations: 'MH' = MeSH Heading.

Each of the seven searches will further be combined with a search term constructed to replicate one of the seven best investments declared in the document Non Communicable Disease Prevention: Investments that Work for Physical Activity (ISPAH, 2012) (Table 2). It is proposed that individual systematic literature reviews will be performed on each of the seven best investments, with this initial review focusing on the ‘whole-of-school approach’ category, and ‘transport policy’ and ‘public education’ categories to follow. The keywords were selected based on an iterative process which combined preliminary searches, solicitation of professional opinion and content analysis of the seven best investments document.

The following criteria will be applied for searches in databases: language will be limited to English language only.

Table 2. Search terms based on the seven best investments document.

1. “Whole of School Approach”	“Whole-of-school” OR “Whole School” OR “Whole of School” OR WSCC OR “school intervention” OR “school based intervention” OR “school initiative” OR “school based initiative” OR “school program*” OR “School health” OR “Wellness” OR “well-being”
2. “Transport Policy”	“active transport” OR “walk” OR “cyclist” OR “bik” OR “bicycl” OR “cyclist” OR “cycling” OR "active travel" OR “commute” OR "transport mode" OR "transportation mode" OR "travel mode" OR “pedestrian” OR "traffic volume" OR "traffic count" OR “transport plan*” OR “road safety” OR “public transport” OR “transport systems”
3. “Urban Design”	MH “Environment Design” OR MH “Environment” OR MH “Environment and Public Health” OR “urban design” OR “urban environment” OR “built environment” OR “urban environment” OR “mixed-use development” OR “footpaths” OR “bikeways” OR “street network” OR “green spac” OR “green areas” OR “green network” OR “recreational spac” OR “urban plan” OR “public amenit*” OR “network infrastructure”
4. “Primary Health care systems”	“primary health” OR “health care” OR “health system”
5. “Public Education”	“public education” OR “mass media” OR “mass communication” OR “social marketing” OR broadcast* OR MH “Communications Media” OR MH “Social Media” OR “media” OR “health campaigns” OR “public education”
6. “Community Programmes”	“Whole-of-community” OR “Community-wide programs” OR "capacity building" OR "community development" OR "community empowerment" OR "community network" OR "coalition building" OR "community capacit" OR "community”
7. “Sport Programmes”	“health promoting clubs” OR “sport*” OR “athletics”

Abbreviations: 'MH' = MeSH Heading.

Eligibility criteria

In order to answer our research question some eligibility criteria were developed to screen out irrelevant articles. Studies will be included based on the following criteria for; 1) type of study, 2) participants/population, 3) exposure/intervention, and 4) outcomes.

General “mandatory” eligibility criteria were formulated as well as specific “preferred” eligibility criteria for each of the seven searches. Publications that do not meet the “mandatory” eligibility criteria will be excluded from review. Publications that do not meet the “preferred” eligibility criteria will be set aside and possibly reassigned to a different search category if they are not duplicates of any publication already included in that search category.

Types of study to be included/excluded. No limitations regarding study type will be placed as long as the study design allows the research questions to be addressed. In addition, reviews using a comprehensive search strategy (including systematic, scoping and realist reviews) and analysing original research on the evidence of what works, and how, in terms of direct and indirect policies on PA; and reviews and policy analysis documents issued by major national and international stakeholder organisations addressing recommendations referring to the same evidence will be eligible for inclusion. Studies will be excluded based on the following criteria; a direct or indirect form of policy intervention is not identifiable, no information is provided regarding the effects of the policy under consideration on the desired outcomes, and published in a language other than English.

Condition or domain being studied. Review examining the evidence of what works in terms of direct and indirect policies on PA, and how.

Participants/population. Eligibility criteria relating to population characteristics are described in Table 3.

Table 3. Population related inclusion criteria.

General criteria	School	Transport	Urban design	Primary health care	Public education	Community programmes	Sport programmes
The study targets the general population or parts of it that are relevant for the respective review	The study targets children and adolescents in the school setting. Preferred: Whole-of- school and environ- mental approaches	The study targets the general population or parts of it and their way of transport. Preferred: The study targets the general population.	The study targets the general population or parts of it and is related to urban design. Preferred: The study targets the general population.	The study targets the general population or parts of it and is related to the primary health care setting. Preferred: The study targets the general population.	The study targets the general population or parts of it and is related to public education. Preferred: The study targets the general population.	The study targets the general population or parts of it and is related to the community setting. Preferred: The study targets the general population.	The study targets the general population or parts of it and is related to the sport setting. Preferred: The study targets the general population and mentions sport participation rates or sport club membership.

Exposure(s), intervention(s). Policies that aim to have a direct or indirect effect on PA behaviour of target groups and populations and on the PA environment that support the behaviour under consideration. “Direct” policy refers to policies where the primary aim is improving the PA environment and increasing PA participation. “Indirect” policy refers to policies where the primary aim is not to increase PA levels but this may occur as a co-benefit of successful implementation.

Grey literature/Other: Similar to the empirical studies, included grey literature will need to make reference to the impact of PA policy in the school setting.

Context. This systematic review is performed as a task of the Policy Evaluation Network (PEN). PEN’s vision is to provide Europe with tools to identify, evaluate and benchmark policies designed to directly or indirectly address physical inactivity. The initial review targets children and adolescents in the school setting. The study must include; a change in PA (or proxy), assessed by means of self-report or wearable devices (e.g., accelerometer) and/or; a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention.

Main outcome(s). All study designs (reviews, empirical evidence and grey literature/other) must include the following outcome(s); a changes in PA (or proxy, e.g. fitness), assessed by means of self-report or wearable devices (e.g., accelerometer); a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention.

Study selection and data extraction

Download of title and abstract records. Titles and abstracts identified by the search will be downloaded as “Endnote import” (extension.enw) files. They will be uploaded to Endnote X9, a citation management software, and Rayyan (Ouzzani et al., 2016), online software dedicated to managing reviews. A freely available alternative is Mendeley reference manager. Once records have been uploaded to Rayyan, the software will identify duplicate articles and one of the two identical articles will be removed. The remaining articles will undergo the first round of screening by two researchers (KV and LK) in a shared Rayyan, account.

Title and abstract review. Title and abstract review will be performed by at least one reviewer (KV) and checked by another reviewer (LK). Checking will involve reviewing title and abstracts decisions to establish whether the second reviewer concurs with the screening decision. Discrepancies will be resolved by discussion to reach consensus, in consultation with a third researcher (EG) when necessary. The screening process will involve comparing the information presented in the title and abstract to the eligibility criteria. Titles and abstracts that appear to conform to the eligibility criteria will be deemed eligible for full text review while those that do not will be discarded from the next stage of the data extraction process.

Download of full articles. Full text articles for all title and abstract recorded deemed eligible for full text review will downloaded using the resources provided by their Institution. If a full text record cannot be acquired using these resources, researchers will investigate whether they can be located through use of other libraries to which the research team has access. If a full text article cannot be located through any of these library resources, the authors will be contacted through whichever channels can be identified from the information in the title and abstract.

Full text review. Full text review will be performed by at least one reviewer (KV) and checked by another reviewer (LK) using a template. Discrepancies will be resolved by discussion to reach consensus, in consultation with a third researcher (EG) when necessary. The following information will be extracted: first author, year of publication, country, study design, data collection method, sample, recruitment/setting, sample size, and response rate.

Risk of bias (quality) assessment

Risk of bias will be assessed by at least one reviewer (EG) and checked by another reviewer (BC). Discrepancies will be resolved by discussion to reach consensus, in consultation with a third researcher when necessary. The results of the quality assessment will be narratively incorporated into the synthesis process. A descriptive summary using the criteria described below will be presented at study level and discussed in the review. Furthermore, the methodological quality will be narratively summarized at review level.

The quality of the included quantitative studies, inclusive of randomised, non-randomised and observational studies (encompassing both longitudinal and cross-sectional studies) will be assessed by means of an adapted ‘Downs and Black’ checklist tool (Downs & Black, 1998). This tool is apt to assess common biases in a range of study types as noted. The checklist will be modified to meet the aims of this review with some items deemed non applicable and subsequently removed.

The AMSTAR measurement tool will be used for the assessment of systematic reviews and comprehensive reviews with a rigorous search strategy, including reviews of reviews. This tool consists of 11 items and has good face and content validity for measuring the methodological quality of systematic reviews (Shea et al., 2007). Not all items are applicable to every type of review being assessed and quality ratings will take account of this circumstance. Similar to Messing et al. (2019), to assess the quality of included studies, we will calculate percentage values for each study. These values will represent the percentages of criteria met per study, based on the criteria specific to each of the assessment tools we use that are applicable in light of the type of study design considered.

Strategy for data synthesis

A narrative synthesis will be used to interpret and analyse the data. Outcome data extracted will be tabulated to determine the impact on policy areas and policy actions (policy indicators). Evidence on the effectiveness of policy actions will be described using a method described by Panter and colleagues (2019); policy actions will be coded as “significantly positive” (+), “significantly negative” (-), “no significance test” (?) or “inconclusive” (0). Lastly, for the included mixed methods studies, reviews and policy analysis documents, the main findings stated in the discussion and conclusions section will be extracted. Main findings of the articles will be copied into a single table along with a reference to the article itself, and details of the overall risk of bias of the study from which the information is extracted. The synthesised data will be presented in a six-column table with the different columns presenting information on the reference, study description, study type, main findings or outcomes, risk of bias and category of evidence, respectively.

Dissemination

Study findings will be presented at professional networking events such as the World Congress on Public Health. The PEN consortium intends to hold a five-day symposium at the biennial conference of the International Society of Physical Activity and Health inviting experts and policymakers to discuss and providing them with the evidence of from the systematic literature reviews.

Study status

The submission of the first of seven intended reviews is being finalised. A further three reviews are being formulated.

Conclusion

The aim of this project is to assist policymakers to achieve the GAPPA target of a 15% relative reduction in the prevalence of insufficient PA (WHO, 2019). The proposed review will attempt to determine what is the impact of policy interventions on PA outcomes in the domain of whole-of-school programmes. By providing this evidence, this review will support the development of the PA EPI. The PA EPI in turn will support policy makers by facilitating the development and benchmarking of policies which will work towards achieving this target. Achieving this target will provide health benefits such as reduced premature mortality as well as substantial co-benefits such as contributing to a sustainable environment and quality education (WHO, 2019).

Data availability

Underlying data

No underlying data are associated with this article.

Reporting guidelines

Open Science Framework: PRISMA-P checklist for “Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity”. https://doi.org/10.17605/OSF.IO/26QYF (Volf, 2020).

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

The authors thank Dr Sarah Taylor and Michael Lawlor for their assistance in developing the search terms used in the study.

Faculty Opinions recommended

References

Bull FC, Bauman AE: Physical inactivity: the "Cinderella" risk factor for noncommunicable disease prevention. J Health Commun. 2011; 16(Suppl 2): 13–26. PubMed Abstract | Publisher Full Text
Bull F, Gauvin L, Bauman A, et al.: The Toronto Charter for Physical Activity: A Global Call for Action. In: 3rd International Congress for Physical Activity and Health. Toronto, Canada, 2010; 7(4): 421–422. Publisher Full Text
Caspersen CJ, Powell KE, Christenson GM: Physical activity, exercise, and physical fitness: definitions and distinctions for health-related research. Public Health Rep. 1985; 100(2): 126–31. PubMed Abstract | Free Full Text
Das P, Horton R: Rethinking our approach to physical activity. Lancet. 2012; 380(9838): 189–190. PubMed Abstract | Publisher Full Text
Ding D, Lawson KD, Kolbe-Alexander TL, et al.: The economic burden of physical inactivity: a global analysis of major non-communicable diseases. Lancet. 2016; 388(10051): 1311–1324. PubMed Abstract | Publisher Full Text
Downs SH, Black N: The feasibility of creating a checklist for theassessment of the methodological quality both ofrandomised and non-randomised studies of healthcare interventions. J Epidemiol Community Health. 1998; 52(6): 377–384. PubMed Abstract | Publisher Full Text | Free Full Text
European Commission: EU Physical Activity Guidelines: Recommended Policy Actions in Support of Health-Enhancing Physical Activity. Brussels: European Commisssion, 2008. Reference Source
International Society for Physical Activity and Health: Investments that Work for Physical Activity. Br J Sports Med. 2012; 46(10): 709–712. Publisher Full Text
Klepac Pogrmilovic B, O'Sullivan G, Milton K, et al.: A global systematic scoping review of studies analysing indicators, development, and content of national-level physical activity and sedentary behaviour policies. Int J Behav Nutr Phys Act. 2018; 15(1): 123. PubMed Abstract | Publisher Full Text | Free Full Text
Kohl HW, Craig CL, Lambert EV, et al.: The pandemic of physical inactivity: global action for public health. Lancet. 2012; 380(9838): 294–305. PubMed Abstract | Publisher Full Text
Lakerveld J, Woods C, Hebestreit A, et al.: Advancing the evidence base for public policies impacting on dietary behaviour, physical activity and sedentary behaviour in Europe: The Policy Evaluation Network promoting a multidisciplinary approach. Food Policy. 2020. Publisher Full Text
Lee IM, Shiroma EJ, Lobelo F, et al.: Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy. Lancet. 2012; 380(9838): 219–229. PubMed Abstract | Publisher Full Text | Free Full Text
Messing S, Rütten A, Abu-Omar K, et al.: How Can Physical Activity Be Promoted Among Children and Adolescents? A Systematic Reviewof Reviews Across Settings. Front Public Health. 2019; 7: 55. PubMed Abstract | Publisher Full Text | Free Full Text
Ouzzani M, Hammady H, Fedorowicz Z, et al.: Rayyan — a web and mobile app for systematic reviews. Syst Rev. 2016; 5(1): 210. PubMed Abstract | Publisher Full Text | Free Full Text
Panter J, Guell C, Humphreys D, et al.: Can changing the physical environment promote walking and cycling? A systematic review of what works and how. Health Place. 2019; 58: 102161. PubMed Abstract | Publisher Full Text | Free Full Text
Ramirez Varela A, Pratt M, Borges C, et al.: 1st Physical Activity Almanac: the Global Observatory for Physical Activity-GoPA. The Global Observatory for Physical Activity. 2016. Reference Source
Rutten A, Schow D, Breda J, et al.: Three types of scientific evidence to inform physical activity policy: results from a comparative scoping review. Int J Public Health. 2016; 61(5): 553–63. PubMed Abstract | Publisher Full Text | Free Full Text
Shea BJ, Grimshaw JM, Wells GA, et al.: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007; 7: 10. PubMed Abstract | Publisher Full Text | Free Full Text
Swinburn B, Sacks G, Vandevijvere S, et al.: INFORMAS (International Network for Food and Obesity/non-communicable diseases Research, Monitoring and Action Support): overview and key principles'. Obes Rev. 2013; 14(Suppl 1): 157–64. PubMed Abstract | Publisher Full Text
Volf K: Checklist for protocol for systematic literature review examining the evidence for impact of school policies on physical activity. 2020. http://www.doi.org/10.17605/OSF.IO/26QYF
Woods CB, Mutrie N: Putting Physical Activity on the Policy Agenda. Quest. 2012; 64(2): 92–104. Publisher Full Text
World Health Organisation: Global Health Risks: Mortality and burden of disease attributable to selected major risks. Geneva: World Health Organisation. 2009. Reference Source
World Health Organisation Regional Office for Europe: Physical Activity Strategy for the WHO European Region 2016 - 2025. Vilnius: World Health Organisation, 2015. Reference Source
World Health Organization: Global action plan for the prevention and control of noncommunicable diseases 2013-2020. 2013. Reference Source
World Health Organization: Global action plan on physical activity 2018-2030: more active people for a healthier world. World Health Organization. 2019. Reference Source
World Health Organization Regional Office for Europe: Review of physical activity promotion policy development and legislation in European Union Member States. Copenhagen: WHO Regional Office for Europe. 2010. Reference Source

Comments on this article Comments (0)

Version 4

VERSION 4 PUBLISHED 04 Sep 2020

Author details Author details

Kevin Volf
Roles: Conceptualization, Methodology, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Liam Kelly
Roles: Methodology, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Enrique García Bengoechea
Roles: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Bláthín Casey
Roles: Writing – Review & Editing

Anna Gobis
Roles: Writing – Review & Editing

Jeroen Lakerveld
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Joanna Zukowska
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Peter Gelius
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Sven Messing
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Sarah Forberger
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Catherine B. Woods
Roles: Conceptualization, Methodology, Project Administration, Supervision, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

Health Research Board [JPI-HDHL-PEN-2017-1].
The PEN project is funded by the Joint Programming Initiative (JPI) “A Healthy Diet for a Healthy Life”, a research and innovation initiative of EU member states and associated countries. The funding agencies supporting this work are (in alphabetical order of participating countries): Germany: Federal Ministry of Education and Research (BMBF); Ireland: Health Research Board (HRB); Italy: Ministry of Education, University and Research (MIUR); The Netherlands: The Netherlands Organisation for Health Research and Development (ZonMw); New Zealand: The University of Auckland, School of Population Health; Norway: The Research Council of Norway (RCN); Poland: The National Centre for Research and Development (NCBR). Additionally, the French partners acknowledge the support through the Institute National de la Recherche Agronomique (INRA).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (4)

version 4

Revised

Published: 17 Jan 2022, 3:62

https://doi.org/10.12688/hrbopenres.13089.4

version 3

Revised

Published: 29 Mar 2021, 3:62

https://doi.org/10.12688/hrbopenres.13089.3

version 2

Revised

Published: 01 Dec 2020, 3:62

https://doi.org/10.12688/hrbopenres.13089.2

version 1

Published: 04 Sep 2020, 3:62

https://doi.org/10.12688/hrbopenres.13089.1

© 2020 Volf K et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

VIEWS

1715

downloads

159

Citations

SEE MORE DETAILS

CITE

how to cite this article

Volf K, Kelly L, García Bengoechea E et al. Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity [version 1; peer review: 1 approved with reservations]. HRB Open Res 2020, 3:62 (https://doi.org/10.12688/hrbopenres.13089.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 04 Sep 2020

Views

Reviewer Report 05 Nov 2020

Melinda Craike, Institute for Health and Sport, Victoria University, Melbourne, Vic, Australia; Mitchell Institute, Victoria University, Melbourne, Australia

Approved with Reservations

https://doi.org/10.21956/hrbopenres.14190.r28217

Thank you for the invitation to review this paper. This protocol paper outlines the details of the first of several planned systematic literature reviews to assess the impact of policy on physical activity outcomes based on the “seven best investments” document (ISPAH, 2012). This is a very ambitious review series and I commend the authors on undertaking this important work. While the methods to be employed is generally clear, there are several areas that require more detailed explanation/rationale. I believe that doing so will strengthen this paper and ultimately, the review.

Introduction
It would be helpful to include specific aims of the systematic review.

P 4- Eligibility criteria
The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.

I was confused by the distinction between the “mandatory” eligibility criteria and specific “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?

p. 5 Main Outcomes
The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).

p. 6 Full text review
Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies.

Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.

Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?

p. 6. Strategy for data synthesis
I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?

I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.

I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.

Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?

Is the rationale for, and objectives of, the study clearly described?

Partly
Is the study design appropriate for the research question?

Partly
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: I have previously worked with one of the co-authors of this paper. This has not affected my ability to review impartially.

Reviewer Expertise: Physical activity behaviour change; systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 01 Dec 2020

Kevin Volf, University of Limerick, Ireland

01 Dec 2020

Author Response

Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would ... Continue reading Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would especially like to thank you for the timely manner in which your review was conducted. We have received your feedback and have made changes to the manuscript accordingly, and believe this has strengthened the quality of the paper.

Please find below a point-by-point response to the issues raised in your review, with changes from the previous version highlighted in red, for your consideration.

Yours sincerely,
Kevin Volf, B.Sc., MM.Sc. PhD Candidate

Comment 1:
Introduction
It would be helpful to include specific aims of the systematic review.
--
We agree with the comment and the aims have added to the final paragraph of the Introduction section. To clarify the focus of this paper is to provide information on the protocol being used for seven reviews, this was unclear in the original manuscript. The text has been edited to make this point clearer throughout. Please see amendments below, text in bold is ‘new’ and has been added for clarity:

[Introduction page 6, paragraph 2]

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

[Introduction page 7 paragraph 1]
This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are schools, transport, urban design, healthcare, public education, the community and sport. These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Comment 2:
P 4- Eligibility criteria
The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.
--
The reference to ‘whole of school approach’ is directly related to the ‘Seven Investments’ paper. The use of ‘Whole of school’ within the search strategy as part of the schools domain, is not for the purpose of evaluating the impact of ‘whole of school programmes’, rather to review policies pertinent to the school setting, to determine if any evidence of policy impact exists. The text has been modified as follows, text in bold is ‘new’ and has been added for clarity;

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’

It is proposed that individual systematic literature reviews will be performed on each of the seven best investment domains, with this initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.”

Comment 3:
I was confused by the distinction between the “mandatory” eligibility criteria and “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?
--
In essence, the “mandatory” and “preferred” nomenclature refer to “elements that are common to all reviews” and “elements that only relate to a particular review” respectively. While this was explained in the search details, to make this clearer we have edited the mandatory and preferred labels to “general” (applies to all mooted reviews) and “specific” (only applies to a particular review) as follows, text in bold is ‘new’ and has been added for clarity:

[Methods, p7 paragraph 5]:

“The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e., general eligibility criteria) with seven distinct search concepts (i.e., specific eligibility criteria for each domain). The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).
Each of the seven searches will further be combined with a specific search concept constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2011) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.

Comment 4:
p. 5 Main Outcomes
The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).
--
We agree with the inconsistency and the text has been reworded in the following location, text in bold is ‘new’ and has been added for clarity:

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011).

Comment 5a:
p. 6 Full text review
Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies. […]
--
While we agree that level of implementation of policies is important, and believe that ‘impact on PA’ will be reflected in the degree of implementation presented, it is beyond the scope of this study to carry out a detailed examination of type of implementation beyond what is reported as evidence of impact. This very important topic may indeed by the subject of a follow up review.

Comment 5b:
[…] Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.
--
Detailed information on any outcome measures used to assess PA will be provided in the data extraction tables and presented in the corresponding reviews.

Comment 6:
Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?
--
This links back to Comment 2, the focus of the systematic review is not to evaluate ‘whole of school approach’ but rather the available evidence on impact of direct or indirect policies within specific settings, e.g., the school setting, on PA outcomes. Therefore, the reviewer is quite correct, we envisage that policy action areas will be broad, e.g., physical education or extra-curricular sport, but the specific policy actions, e.g., require a minimum PE duration or require a PE curriculum may be more defined. Until the reviews are carried out, the level of evidence is unknown.

Comment 7:
p. 6. Strategy for data synthesis
I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?
--
We agree with the reviewer that this is not clear. In order to improve clarity this sentence has been removed and the third paragraph on page 12 has been restructured as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

“A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter and colleagues (2019) […]”

Comment 8:
I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.
--
Detail has been added to the second half of the first paragraph on page 12 to describe the coding system inspired by Panter et al. It now reads as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

[…] Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0).

Comment 9:
I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.
--
This may indeed be challenging, but until we examine the main results and conclusions of any systematic reviews that have reference to policy impact in the targeted domains, we cannot be sure of whether their evidence is strong enough and clearly defined in order to assess impact. While we agree it may be difficult, in the search strategy we will include these documents in the initial search and depending on what we find include or exclude in the final data synthesis.

Comment 10:
Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?
--
Until data extraction is complete, it is difficult to identify any specific stratification variables that would be pertinent to the presentation of the results. However, the data extraction tables will be devised to take into account any relevant stratification variables for each review as pertinent to the topic and domain.

As the reviewer’s comment applies to the systematic review corresponding to the school setting only evidence of effectiveness will be presented in a manner that makes it clear which policy actions were shown to be effective at different school levels, across gender and SES where appropriate. The following sentence has been added to the strategy for data synthesis section to clarify this, text in bold is ‘new’ and has been added for clarity.

[Strategy for data synthesis, page 12 paragraph 3]

“In addition, data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools review , data extraction columns will be included to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio economic status where appropriate.
Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would especially like to thank you for the timely manner in which your review was conducted. We have received your feedback and have made changes to the manuscript accordingly, and believe this has strengthened the quality of the paper.

Please find below a point-by-point response to the issues raised in your review, with changes from the previous version highlighted in red, for your consideration.

Yours sincerely,
Kevin Volf, B.Sc., MM.Sc. PhD Candidate

Comment 1:
Introduction
It would be helpful to include specific aims of the systematic review.
--
We agree with the comment and the aims have added to the final paragraph of the Introduction section. To clarify the focus of this paper is to provide information on the protocol being used for seven reviews, this was unclear in the original manuscript. The text has been edited to make this point clearer throughout. Please see amendments below, text in bold is ‘new’ and has been added for clarity:

[Introduction page 6, paragraph 2]

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

[Introduction page 7 paragraph 1]
This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are schools, transport, urban design, healthcare, public education, the community and sport. These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Comment 2:
P 4- Eligibility criteria
The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.
--
The reference to ‘whole of school approach’ is directly related to the ‘Seven Investments’ paper. The use of ‘Whole of school’ within the search strategy as part of the schools domain, is not for the purpose of evaluating the impact of ‘whole of school programmes’, rather to review policies pertinent to the school setting, to determine if any evidence of policy impact exists. The text has been modified as follows, text in bold is ‘new’ and has been added for clarity;

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’

It is proposed that individual systematic literature reviews will be performed on each of the seven best investment domains, with this initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.”

Comment 3:
I was confused by the distinction between the “mandatory” eligibility criteria and “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?
--
In essence, the “mandatory” and “preferred” nomenclature refer to “elements that are common to all reviews” and “elements that only relate to a particular review” respectively. While this was explained in the search details, to make this clearer we have edited the mandatory and preferred labels to “general” (applies to all mooted reviews) and “specific” (only applies to a particular review) as follows, text in bold is ‘new’ and has been added for clarity:

[Methods, p7 paragraph 5]:

“The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e., general eligibility criteria) with seven distinct search concepts (i.e., specific eligibility criteria for each domain). The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).
Each of the seven searches will further be combined with a specific search concept constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2011) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.

Comment 4:
p. 5 Main Outcomes
The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).
--
We agree with the inconsistency and the text has been reworded in the following location, text in bold is ‘new’ and has been added for clarity:

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011).

Comment 5a:
p. 6 Full text review
Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies. […]
--
While we agree that level of implementation of policies is important, and believe that ‘impact on PA’ will be reflected in the degree of implementation presented, it is beyond the scope of this study to carry out a detailed examination of type of implementation beyond what is reported as evidence of impact. This very important topic may indeed by the subject of a follow up review.

Comment 5b:
[…] Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.
--
Detailed information on any outcome measures used to assess PA will be provided in the data extraction tables and presented in the corresponding reviews.

Comment 6:
Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?
--
This links back to Comment 2, the focus of the systematic review is not to evaluate ‘whole of school approach’ but rather the available evidence on impact of direct or indirect policies within specific settings, e.g., the school setting, on PA outcomes. Therefore, the reviewer is quite correct, we envisage that policy action areas will be broad, e.g., physical education or extra-curricular sport, but the specific policy actions, e.g., require a minimum PE duration or require a PE curriculum may be more defined. Until the reviews are carried out, the level of evidence is unknown.

Comment 7:
p. 6. Strategy for data synthesis
I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?
--
We agree with the reviewer that this is not clear. In order to improve clarity this sentence has been removed and the third paragraph on page 12 has been restructured as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

“A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter and colleagues (2019) […]”

Comment 8:
I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.
--
Detail has been added to the second half of the first paragraph on page 12 to describe the coding system inspired by Panter et al. It now reads as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

[…] Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0).

Comment 9:
I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.
--
This may indeed be challenging, but until we examine the main results and conclusions of any systematic reviews that have reference to policy impact in the targeted domains, we cannot be sure of whether their evidence is strong enough and clearly defined in order to assess impact. While we agree it may be difficult, in the search strategy we will include these documents in the initial search and depending on what we find include or exclude in the final data synthesis.

Comment 10:
Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?
--
Until data extraction is complete, it is difficult to identify any specific stratification variables that would be pertinent to the presentation of the results. However, the data extraction tables will be devised to take into account any relevant stratification variables for each review as pertinent to the topic and domain.

As the reviewer’s comment applies to the systematic review corresponding to the school setting only evidence of effectiveness will be presented in a manner that makes it clear which policy actions were shown to be effective at different school levels, across gender and SES where appropriate. The following sentence has been added to the strategy for data synthesis section to clarify this, text in bold is ‘new’ and has been added for clarity.

[Strategy for data synthesis, page 12 paragraph 3]

“In addition, data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools review , data extraction columns will be included to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio economic status where appropriate.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 01 Dec 2020

Kevin Volf, University of Limerick, Ireland

01 Dec 2020

Author Response

Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would ... Continue reading Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would especially like to thank you for the timely manner in which your review was conducted. We have received your feedback and have made changes to the manuscript accordingly, and believe this has strengthened the quality of the paper.

Please find below a point-by-point response to the issues raised in your review, with changes from the previous version highlighted in red, for your consideration.

Yours sincerely,
Kevin Volf, B.Sc., MM.Sc. PhD Candidate

Comment 1:
Introduction
It would be helpful to include specific aims of the systematic review.
--
We agree with the comment and the aims have added to the final paragraph of the Introduction section. To clarify the focus of this paper is to provide information on the protocol being used for seven reviews, this was unclear in the original manuscript. The text has been edited to make this point clearer throughout. Please see amendments below, text in bold is ‘new’ and has been added for clarity:

[Introduction page 6, paragraph 2]

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

[Introduction page 7 paragraph 1]
This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are schools, transport, urban design, healthcare, public education, the community and sport. These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Comment 2:
P 4- Eligibility criteria
The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.
--
The reference to ‘whole of school approach’ is directly related to the ‘Seven Investments’ paper. The use of ‘Whole of school’ within the search strategy as part of the schools domain, is not for the purpose of evaluating the impact of ‘whole of school programmes’, rather to review policies pertinent to the school setting, to determine if any evidence of policy impact exists. The text has been modified as follows, text in bold is ‘new’ and has been added for clarity;

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’

It is proposed that individual systematic literature reviews will be performed on each of the seven best investment domains, with this initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.”

Comment 3:
I was confused by the distinction between the “mandatory” eligibility criteria and “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?
--
In essence, the “mandatory” and “preferred” nomenclature refer to “elements that are common to all reviews” and “elements that only relate to a particular review” respectively. While this was explained in the search details, to make this clearer we have edited the mandatory and preferred labels to “general” (applies to all mooted reviews) and “specific” (only applies to a particular review) as follows, text in bold is ‘new’ and has been added for clarity:

[Methods, p7 paragraph 5]:

“The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e., general eligibility criteria) with seven distinct search concepts (i.e., specific eligibility criteria for each domain). The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).
Each of the seven searches will further be combined with a specific search concept constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2011) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.

Comment 4:
p. 5 Main Outcomes
The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).
--
We agree with the inconsistency and the text has been reworded in the following location, text in bold is ‘new’ and has been added for clarity:

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011).

Comment 5a:
p. 6 Full text review
Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies. […]
--
While we agree that level of implementation of policies is important, and believe that ‘impact on PA’ will be reflected in the degree of implementation presented, it is beyond the scope of this study to carry out a detailed examination of type of implementation beyond what is reported as evidence of impact. This very important topic may indeed by the subject of a follow up review.

Comment 5b:
[…] Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.
--
Detailed information on any outcome measures used to assess PA will be provided in the data extraction tables and presented in the corresponding reviews.

Comment 6:
Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?
--
This links back to Comment 2, the focus of the systematic review is not to evaluate ‘whole of school approach’ but rather the available evidence on impact of direct or indirect policies within specific settings, e.g., the school setting, on PA outcomes. Therefore, the reviewer is quite correct, we envisage that policy action areas will be broad, e.g., physical education or extra-curricular sport, but the specific policy actions, e.g., require a minimum PE duration or require a PE curriculum may be more defined. Until the reviews are carried out, the level of evidence is unknown.

Comment 7:
p. 6. Strategy for data synthesis
I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?
--
We agree with the reviewer that this is not clear. In order to improve clarity this sentence has been removed and the third paragraph on page 12 has been restructured as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

“A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter and colleagues (2019) […]”

Comment 8:
I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.
--
Detail has been added to the second half of the first paragraph on page 12 to describe the coding system inspired by Panter et al. It now reads as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

[…] Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0).

Comment 9:
I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.
--
This may indeed be challenging, but until we examine the main results and conclusions of any systematic reviews that have reference to policy impact in the targeted domains, we cannot be sure of whether their evidence is strong enough and clearly defined in order to assess impact. While we agree it may be difficult, in the search strategy we will include these documents in the initial search and depending on what we find include or exclude in the final data synthesis.

Comment 10:
Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?
--
Until data extraction is complete, it is difficult to identify any specific stratification variables that would be pertinent to the presentation of the results. However, the data extraction tables will be devised to take into account any relevant stratification variables for each review as pertinent to the topic and domain.

As the reviewer’s comment applies to the systematic review corresponding to the school setting only evidence of effectiveness will be presented in a manner that makes it clear which policy actions were shown to be effective at different school levels, across gender and SES where appropriate. The following sentence has been added to the strategy for data synthesis section to clarify this, text in bold is ‘new’ and has been added for clarity.

[Strategy for data synthesis, page 12 paragraph 3]

“In addition, data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools review , data extraction columns will be included to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio economic status where appropriate.
Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would especially like to thank you for the timely manner in which your review was conducted. We have received your feedback and have made changes to the manuscript accordingly, and believe this has strengthened the quality of the paper.

Please find below a point-by-point response to the issues raised in your review, with changes from the previous version highlighted in red, for your consideration.

Yours sincerely,
Kevin Volf, B.Sc., MM.Sc. PhD Candidate

Comment 1:
Introduction
It would be helpful to include specific aims of the systematic review.
--
We agree with the comment and the aims have added to the final paragraph of the Introduction section. To clarify the focus of this paper is to provide information on the protocol being used for seven reviews, this was unclear in the original manuscript. The text has been edited to make this point clearer throughout. Please see amendments below, text in bold is ‘new’ and has been added for clarity:

[Introduction page 6, paragraph 2]

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

[Introduction page 7 paragraph 1]
This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are schools, transport, urban design, healthcare, public education, the community and sport. These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Comment 2:
P 4- Eligibility criteria
The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.
--
The reference to ‘whole of school approach’ is directly related to the ‘Seven Investments’ paper. The use of ‘Whole of school’ within the search strategy as part of the schools domain, is not for the purpose of evaluating the impact of ‘whole of school programmes’, rather to review policies pertinent to the school setting, to determine if any evidence of policy impact exists. The text has been modified as follows, text in bold is ‘new’ and has been added for clarity;

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’

It is proposed that individual systematic literature reviews will be performed on each of the seven best investment domains, with this initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.”

Comment 3:
I was confused by the distinction between the “mandatory” eligibility criteria and “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?
--
In essence, the “mandatory” and “preferred” nomenclature refer to “elements that are common to all reviews” and “elements that only relate to a particular review” respectively. While this was explained in the search details, to make this clearer we have edited the mandatory and preferred labels to “general” (applies to all mooted reviews) and “specific” (only applies to a particular review) as follows, text in bold is ‘new’ and has been added for clarity:

[Methods, p7 paragraph 5]:

“The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e., general eligibility criteria) with seven distinct search concepts (i.e., specific eligibility criteria for each domain). The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).
Each of the seven searches will further be combined with a specific search concept constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2011) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.

Comment 4:
p. 5 Main Outcomes
The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).
--
We agree with the inconsistency and the text has been reworded in the following location, text in bold is ‘new’ and has been added for clarity:

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011).

Comment 5a:
p. 6 Full text review
Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies. […]
--
While we agree that level of implementation of policies is important, and believe that ‘impact on PA’ will be reflected in the degree of implementation presented, it is beyond the scope of this study to carry out a detailed examination of type of implementation beyond what is reported as evidence of impact. This very important topic may indeed by the subject of a follow up review.

Comment 5b:
[…] Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.
--
Detailed information on any outcome measures used to assess PA will be provided in the data extraction tables and presented in the corresponding reviews.

Comment 6:
Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?
--
This links back to Comment 2, the focus of the systematic review is not to evaluate ‘whole of school approach’ but rather the available evidence on impact of direct or indirect policies within specific settings, e.g., the school setting, on PA outcomes. Therefore, the reviewer is quite correct, we envisage that policy action areas will be broad, e.g., physical education or extra-curricular sport, but the specific policy actions, e.g., require a minimum PE duration or require a PE curriculum may be more defined. Until the reviews are carried out, the level of evidence is unknown.

Comment 7:
p. 6. Strategy for data synthesis
I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?
--
We agree with the reviewer that this is not clear. In order to improve clarity this sentence has been removed and the third paragraph on page 12 has been restructured as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

“A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter and colleagues (2019) […]”

Comment 8:
I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.
--
Detail has been added to the second half of the first paragraph on page 12 to describe the coding system inspired by Panter et al. It now reads as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

[…] Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0).

Comment 9:
I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.
--
This may indeed be challenging, but until we examine the main results and conclusions of any systematic reviews that have reference to policy impact in the targeted domains, we cannot be sure of whether their evidence is strong enough and clearly defined in order to assess impact. While we agree it may be difficult, in the search strategy we will include these documents in the initial search and depending on what we find include or exclude in the final data synthesis.

Comment 10:
Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?
--
Until data extraction is complete, it is difficult to identify any specific stratification variables that would be pertinent to the presentation of the results. However, the data extraction tables will be devised to take into account any relevant stratification variables for each review as pertinent to the topic and domain.

As the reviewer’s comment applies to the systematic review corresponding to the school setting only evidence of effectiveness will be presented in a manner that makes it clear which policy actions were shown to be effective at different school levels, across gender and SES where appropriate. The following sentence has been added to the strategy for data synthesis section to clarify this, text in bold is ‘new’ and has been added for clarity.

[Strategy for data synthesis, page 12 paragraph 3]

“In addition, data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools review , data extraction columns will be included to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio economic status where appropriate.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 4

VERSION 4 PUBLISHED 04 Sep 2020

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 4 (revision) 17 Jan 22			read
Version 3 (revision) 29 Mar 21	read	read	read
Version 2 (revision) 01 Dec 20	read
Version 1 04 Sep 20	read

Melinda Craike, Victoria University, Melbourne, Australia; Victoria University, Melbourne, Australia
Petru Sandu, Babes-Bolyai University, Cluj-Napoca, Romania
Eric Breton, École des hautes études en santé publique, Rennes, France; Arènes research unit (UMR CNRS 6051: team INSERM U1309), Rennes, France

Comments on this article

All Comments(0)

Add a comment

Back to all reports

Reviewer Report

23 Views

26 Jan 2022 | for Version 4

Eric Breton, Social and Human Sciences Department, École des hautes études en santé publique, Rennes, France; Arènes research unit (UMR CNRS 6051: team INSERM U1309), Rennes, France

23 Views Cite this report Responses(0)

Approved

I am satisfied with the answers given to the points I raised in my first report.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Community health, prevention strategies, health promotion, capacity building strategies for the prevention of chronic diseases.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

40 Views

17 Nov 2021 | for Version 3

Eric Breton, Social and Human Sciences Department, École des hautes études en santé publique, Rennes, France; Arènes research unit (UMR CNRS 6051: team INSERM U1309), Rennes, France

40 Views Cite this report Responses(1)

Approved With Reservations

Dear authors,

Informing policy-makers on the best options available to improve the level of physical activity across whole populations is definitely the next step after years of research looking at the determinants of physical activity. This project is therefore likely to generate valuable evidence that will not only pinpoint the best policies but also facilitate advocacy for policy change.

I was surprised not seeing any reference to the Physical Activity Policy Analysis Tool (HEPA PAT) that allows for a diagnosis of national policy responses to physical inactivity. I think it should be mentioned as part of the current efforts to improve policies.

The general methods is appropriate but is likely going to be adapted as work on the 7 domains will be carried out. Other search and inclusion/exclusion criteria are likely to be stated when facing the large body of papers published in some of the domains.

In the following lines, I address some of the specific issues I identified.

Second paragraph of the introduction: the last sentence seems at odd with the topic covered by the paragraph (“The International Society for Physical Activity and Health (ISPAH) was established in 2009 (Kohl et al., 2012) and numerous articles and editorials in leading academic and medical journals have pointed out the need to address physical inactivity (Bull & Bauman, 2011; Das & Horton, 2012; Kohl et al., 2012; Woods & Mutrie, 2012).”) - It is not fitting well there.
In the fourth paragraph, you wrote: “Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour” - Seems odd talking about the epidemics of physical activity.
In table 1 on the search terms for policy: there is a fair bit of overlap. Searching “policy” as a single term will automatically yield hits that include “national policy”. Therefore no need to enter the latter. I assume the terms that will not be searched in the MH will be searched in the title/abstract as mentioned in the text. In the “impact” search string, shouldn’t “outcome” be included?
In Table 2: in the “transport policy” search string, “cyclist” appears twice and “Commute*” should be entered as “commut*”.
In the “urban design” one, “urban environment” appears twice as well. If I am not wrong in the literature, there are mentions along the green spaces to the blue spaces. Lakes and waterways can be settings for physical activity.
For the “Primary Health care systems” search string, I would add “primary care”.
For the “Community Programmes” string, I do think the term “capacity building” will generate many hits that are not relevant. I would remove it and add “community building” and “community strengthening”.
In table 3, under the “Urban design specific criteria” we read that: “The study intervention targets the residents of urban areas”. I did not understand that this idea of using the leverage of actions on urban design was only for urban areas. If that is the case, it would definitely leave behind a significant share of populations that are known to present a poorer health status.

This is all I can see reading this paper.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Community health, prevention strategies, health promotion, capacity building strategies for the prevention of chronic diseases.

Respond to this report

Responses (1)

Author Response

17 Jan 2022

Kevin Volf, University of Limerick, Ireland

Dear Dr Breton,

Thank you for your feedback on our paper. By our reading, you have highlighted 9 different points for improvement in the paper. We quote each point and provide our response below.

“I was surprised not seeing any reference to the Physical Activity Policy Analysis Tool (HEPA PAT) that allows for a diagnosis of national policy responses to physical inactivity. I think it should be mentioned as part of the current efforts to improve policies.”

The following sentence has been added to the introduction (lines 45 to 47) to add reference to the HEPA PAT: “Another development promoted by the WHO in the field of PA policy is the proliferation of audits of policy responses to inactivity using a tool entitled the Health-Enhancing Physical Activity Policy Audit Tool (HEPA PAT).”

“Second paragraph of the introduction: the last sentence seems at odd with the topic covered by the paragraph (“The International Society for Physical Activity and Health (ISPAH) was established in 2009 (Kohl et al., 2012) and numerous articles and editorials in leading academic and medical journals have pointed out the need to address physical inactivity (Bull & Bauman, 2011; Das & Horton, 2012; Kohl et al., 2012; Woods & Mutrie, 2012).”) - It is not fitting well there.”

We agree that this seems out of place. We have moved the first clause and deleted the second, this influences the reference list which we have also updated accordingly. Paragraph two (lines 12 to 27) now reads: “This epidemiological evidence reveals inactivity to be a substantial public health issue and advocacy by public health specialists and the academic community has demanded policy responses to this issue. For the purposes of this document, policy should be understood as “decisions, plans, and actions that are implemented by national or regional governments to achieve specific health promotion goals within a society” ( Lakerveld et al., 2020). As indicated by the WHO ( WHO Regional Office for Europe, 2010), policy can give support, coherence and visibility at the political level, while making it possible for the organisations involved at national, regional, and local levels – e.g., national government sectors, regional or local authorities, stakeholders, and the private sector – to be logical and consistent in their actions to achieve a shared goal. This applies to food and PA environments, systems and behaviours ( WHO Regional Office for Europe, 2010). In order to reflect the complexity of the policies that may affect the PA policy environment, Lakerveld & colleagues (2020) distinguish between “direct” policies, which refers to policies where improving the PA environment and increasing participation is the primary aim, and “indirect” policies, where the primary aim of the policy is not to increase PA but this may occur as a co-benefit of successful implementation.”

“In the fourth paragraph, you wrote: “Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour” - Seems odd talking about the epidemics of physical activity.”

We have amended this sentence to read "...epidemics of inactivity and sedentary behaviour” [line 40].

“In table 1 on the search terms for policy: there is a fair bit of overlap. Searching “policy” as a single term will automatically yield hits that include “national policy”. Therefore, no need to enter the latter. I assume the terms that will not be searched in the MH will be searched in the title/abstract as mentioned in the text. In the “impact” search string, shouldn’t “outcome” be included?”

The assumption that terms that are not searched in the MH will be searched by title and abstract is correct. As some of the reviews have been commenced and submitted, we cannot amend the general search string. However, other suggestions made below have been incorporated.

“In Table 2: in the “transport policy” search string, “cyclist” appears twice and “Commute*” should be entered as “commut*”.”

Since the transport review has been completed and is currently under review by an academic journal, the transport policy search string cannot be revised.

“In the “urban design” one, “urban environment” appears twice as well. If I am not wrong in the literature, there are mentions along the green spaces to the blue spaces. Lakes and waterways can be settings for physical activity.”

We have amended the search terms so that "Blue spac*" is added, and the second "urban environment" removed in the urban design search string.

“For the “Primary Health care systems” search string, I would add “primary care”.”

We have added the search term "primary care *" to the primary health care search string.

“For the “Community Programmes” string, I do think the term “capacity building” will generate many hits that are not relevant. I would remove it and add “community building” and “community strengthening”.”

We have amended the search terms so that “community building” and “community strengthening” are added, and “community strengthening” is removed from the community programmes search string.

“In table 3, under the “Urban design specific criteria” we read that: “The study intervention targets the residents of urban areas”. I did not understand that this idea of using the leverage of actions on urban design was only for urban areas. If that is the case, it would definitely leave behind a significant share of populations that are known to present a poorer health status.”

It is correct that the urban design review will not reach rural populations. It is expected that policy actions with evidence for effectiveness will include actions such as mixed urban land use. However, rural communities will be reached through the other reviews.

Kind regards,
Kevin Volf

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

48 Views

03 Nov 2021 | for Version 3

Petru Sandu, Faculty of Political, Administrative and Communication Sciences, Department of Public Health, Babes-Bolyai University, Cluj-Napoca, Romania

48 Views Cite this report Responses(1)

Approved

This article presents very clearly the process undertaken for conducting 7 SLRs looking at the impact of PA policies in 7 priority settings towards established PA related outcomes.

The introduction is very clear and comprehensive, gives a good overview of the research in this area and the need for the current project.

The methods are also clear, concise but comprehensive. I only have one question: the two researchers conducting title and abstract, full-text review and risk of bias assessment are the same to researchers of there are 3 sets of 2 researchers? It is not obvious from the text. Also, who is the 3r researcher in each case? I guess details about this will be described in the following papers, but I think something should be also mentioned in this paper.

The strategy for data synthesis is also clear but I think the specific data included in the tables for each of the 7 reviews should be more clearly defined. Again, probably these will be detailed in future papers.

The conclusion seems more like a impact paragraph to me. It is necessary and useful, but not as conclusions. Conclusions should refer to the study aim, methods put in place and envisioned results.

Also, some phrasing related feedback:

Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour - do you mean physical inactivity and sedentary behaviour?
Instead of: Original material examining the evidence of what works in terms of direct and indirect policies on PA will be identified in the following ways - Original material examining the evidence of what works in terms of direct and indirect policies to increase PA will be identified in the following ways.

In conclusion, I think this paper is a good description of the methods put in place by PEN network in order to conduct a series of SLRs aimed at exploring the impact of PA policy on selected PA related outcome, with the final aim to assist decision makers in making evidence informed investments.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Physical activity policy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

17 Jan 2022

Kevin Volf, University of Limerick, Ireland

Dear Dr Sandu,

Thank you for your feedback on our review protocol. This will contribute significantly to the improvement of this paper. By our reading, you have highlighted 5 different points for improvement in the paper. We quote each point and provide our response below.

“The methods are also clear, concise but comprehensive. I only have one question: the two researchers conducting title and abstract, full-text review and risk of bias assessment are the same to researchers of there are 3 sets of 2 researchers? It is not obvious from the text. Also, who is the 3r researcher in each case? I guess details about this will be described in the following papers, but I think something should be also mentioned in this paper.”

As there are several reviews planned, and responsibility for these tasks will differ between reviews, it is not possible to assign authors initials to each of the roles specified in this publication. However, we have added the following to the paragraph entitled ‘full text reviews’ to provide greater clarity: “The authors involved in screening and risk of bias will differ between the different reviews; however, the authors involved in full text screening will be the same authors with responsibility for title and abstract screening” [lines 193 to 195].

The strategy for data synthesis is also clear but I think the specific data included in the tables for each of the 7 reviews should be more clearly defined. Again, probably these will be detailed in future papers. “To provide further clarity on how the data synthesis table can be visualised the following was added to the second paragraph of the subsection entitled ‘Strategy for data synthesis’: “In summary, the data synthesis table will display three types of data: brief policy statements, codes which help visualise the number and direction of effects found in the literature supporting those statements, and the supporting references” [lines 230 to 233].

The data extraction tables will display descriptive data on the papers including general data in of the papers (first author, date, country) descriptions of the population (sample size, demographics), description of the policy itself and reported effects.

“The conclusion seems more like a impact paragraph to me. It is necessary and useful, but not as conclusions. Conclusions should refer to the study aim, methods put in place and envisioned results.”

To provide reference to the study aim the following sentence has been added: “The aim of the planned work is to determine the level and type of evidence reported in the international scientific literature for policies that contribute directly or indirectly to increasing PA within 7 priority domains identified as best investments for PA” [lines 255 to 258].
To refer to the methods the following line has been added “this will be achieved through searches electronic databases and extensive snowballing techniques” [lines 258 to 259].
To address the lack of envisioned results the following sentence has been added to the conclusion: “We anticipate that the recommendations will mirror and expand upon some of the prescriptions made in GAPPA, the European Physical Activity Strategy or by experts in the PA community.” [lines 264 to 266].

“Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour - do you mean physical inactivity and sedentary behaviour?”

We have amended this sentence to read "...epidemics of inactivity and sedentary behaviour” [line 40].

“Instead of: Original material examining the evidence of what works in terms of direct and indirect policies on PA will be identified in the following ways - Original material examining the evidence of what works in terms of direct and indirect policies to increase PA will be identified in the following ways.”

We have amended this sentence so that "On PA" is replaced with "to increase PA" [line 88].

Kind regards
Kevin Volf

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

31 Views

31 Mar 2021 | for Version 3

Melinda Craike, Institute for Health and Sport, Victoria University, Melbourne, Vic, Australia; Mitchell Institute, Victoria University, Melbourne, Australia

31 Views Cite this report Responses(0)

Approved

No new comments.

Competing Interests

I have previously worked with one of the co-authors of this paper. This has not affected my ability to review impartially.

Reviewer Expertise

Physical activity behaviour change; systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

42 Views

17 Dec 2020 | for Version 2

Melinda Craike, Institute for Health and Sport, Victoria University, Melbourne, Vic, Australia; Mitchell Institute, Victoria University, Melbourne, Australia

42 Views Cite this report Responses(0)

Approved

I thank the Authors for their thorough responses to my comments and proposed revisions. I am happy with the revisions and have no further comments. I wish the Authors all the best with this series of reviews.

Competing Interests

I have previously worked with one of the co-authors of this paper. This has not affected my ability to review impartially.

Reviewer Expertise

Physical activity behaviour change; systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

58 Views

05 Nov 2020 | for Version 1

Melinda Craike, Institute for Health and Sport, Victoria University, Melbourne, Vic, Australia; Mitchell Institute, Victoria University, Melbourne, Australia

58 Views Cite this report Responses(1)

Approved With Reservations

Is the rationale for, and objectives of, the study clearly described?

Partly
Is the study design appropriate for the research question?

Partly
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

I have previously worked with one of the co-authors of this paper. This has not affected my ability to review impartially.

Reviewer Expertise

Physical activity behaviour change; systematic reviews.

Respond to this report

Responses (1)

Author Response

01 Dec 2020

Kevin Volf, University of Limerick, Ireland

Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would especially like to thank you for the timely manner in which your review was conducted. We have received your feedback and have made changes to the manuscript accordingly, and believe this has strengthened the quality of the paper.

Please find below a point-by-point response to the issues raised in your review, with changes from the previous version highlighted in red, for your consideration.

Yours sincerely,
Kevin Volf, B.Sc., MM.Sc. PhD Candidate

Comment 1:
Introduction
It would be helpful to include specific aims of the systematic review.
--
We agree with the comment and the aims have added to the final paragraph of the Introduction section. To clarify the focus of this paper is to provide information on the protocol being used for seven reviews, this was unclear in the original manuscript. The text has been edited to make this point clearer throughout. Please see amendments below, text in bold is ‘new’ and has been added for clarity:

[Introduction page 6, paragraph 2]

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

[Introduction page 7 paragraph 1]
This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are schools, transport, urban design, healthcare, public education, the community and sport. These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Comment 2:
P 4- Eligibility criteria
The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.
--
The reference to ‘whole of school approach’ is directly related to the ‘Seven Investments’ paper. The use of ‘Whole of school’ within the search strategy as part of the schools domain, is not for the purpose of evaluating the impact of ‘whole of school programmes’, rather to review policies pertinent to the school setting, to determine if any evidence of policy impact exists. The text has been modified as follows, text in bold is ‘new’ and has been added for clarity;

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’

It is proposed that individual systematic literature reviews will be performed on each of the seven best investment domains, with this initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.”

Comment 3:
I was confused by the distinction between the “mandatory” eligibility criteria and “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?
--
In essence, the “mandatory” and “preferred” nomenclature refer to “elements that are common to all reviews” and “elements that only relate to a particular review” respectively. While this was explained in the search details, to make this clearer we have edited the mandatory and preferred labels to “general” (applies to all mooted reviews) and “specific” (only applies to a particular review) as follows, text in bold is ‘new’ and has been added for clarity:

[Methods, p7 paragraph 5]:

“The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e., general eligibility criteria) with seven distinct search concepts (i.e., specific eligibility criteria for each domain). The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).
Each of the seven searches will further be combined with a specific search concept constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2011) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.

Comment 4:
p. 5 Main Outcomes
The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).
--
We agree with the inconsistency and the text has been reworded in the following location, text in bold is ‘new’ and has been added for clarity:

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011).

Comment 5a:
p. 6 Full text review
Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies. […]
--
While we agree that level of implementation of policies is important, and believe that ‘impact on PA’ will be reflected in the degree of implementation presented, it is beyond the scope of this study to carry out a detailed examination of type of implementation beyond what is reported as evidence of impact. This very important topic may indeed by the subject of a follow up review.

Comment 5b:
[…] Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.
--
Detailed information on any outcome measures used to assess PA will be provided in the data extraction tables and presented in the corresponding reviews.

Comment 6:
Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?
--
This links back to Comment 2, the focus of the systematic review is not to evaluate ‘whole of school approach’ but rather the available evidence on impact of direct or indirect policies within specific settings, e.g., the school setting, on PA outcomes. Therefore, the reviewer is quite correct, we envisage that policy action areas will be broad, e.g., physical education or extra-curricular sport, but the specific policy actions, e.g., require a minimum PE duration or require a PE curriculum may be more defined. Until the reviews are carried out, the level of evidence is unknown.

Comment 7:
p. 6. Strategy for data synthesis
I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?
--
We agree with the reviewer that this is not clear. In order to improve clarity this sentence has been removed and the third paragraph on page 12 has been restructured as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

“A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter and colleagues (2019) […]”

Comment 8:
I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.
--
Detail has been added to the second half of the first paragraph on page 12 to describe the coding system inspired by Panter et al. It now reads as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

[…] Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0).

Comment 9:
I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.
--
This may indeed be challenging, but until we examine the main results and conclusions of any systematic reviews that have reference to policy impact in the targeted domains, we cannot be sure of whether their evidence is strong enough and clearly defined in order to assess impact. While we agree it may be difficult, in the search strategy we will include these documents in the initial search and depending on what we find include or exclude in the final data synthesis.

Comment 10:
Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?
--
Until data extraction is complete, it is difficult to identify any specific stratification variables that would be pertinent to the presentation of the results. However, the data extraction tables will be devised to take into account any relevant stratification variables for each review as pertinent to the topic and domain.

As the reviewer’s comment applies to the systematic review corresponding to the school setting only evidence of effectiveness will be presented in a manner that makes it clear which policy actions were shown to be effective at different school levels, across gender and SES where appropriate. The following sentence has been added to the strategy for data synthesis section to clarify this, text in bold is ‘new’ and has been added for clarity.

[Strategy for data synthesis, page 12 paragraph 3]

“In addition, data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools review , data extraction columns will be included to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio economic status where appropriate.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] Bull FC, Bauman AE: Physical inactivity: the "Cinderella" risk factor for noncommunicable disease prevention. J Health Commun. 2011; 16(Suppl 2): 13–26. PubMed Abstract | Publisher Full Text

[2] Bull F, Gauvin L, Bauman A, et al.: The Toronto Charter for Physical Activity: A Global Call for Action. In: 3rd International Congress for Physical Activity and Health. Toronto, Canada, 2010; 7(4): 421–422. Publisher Full Text

[3] Caspersen CJ, Powell KE, Christenson GM: Physical activity, exercise, and physical fitness: definitions and distinctions for health-related research. Public Health Rep. 1985; 100(2): 126–31. PubMed Abstract | Free Full Text

[4] Das P, Horton R: Rethinking our approach to physical activity. Lancet. 2012; 380(9838): 189–190. PubMed Abstract | Publisher Full Text

[5] Ding D, Lawson KD, Kolbe-Alexander TL, et al.: The economic burden of physical inactivity: a global analysis of major non-communicable diseases. Lancet. 2016; 388(10051): 1311–1324. PubMed Abstract | Publisher Full Text

[6] Downs SH, Black N: The feasibility of creating a checklist for theassessment of the methodological quality both ofrandomised and non-randomised studies of healthcare interventions. J Epidemiol Community Health. 1998; 52(6): 377–384. PubMed Abstract | Publisher Full Text | Free Full Text

[7] European Commission: EU Physical Activity Guidelines: Recommended Policy Actions in Support of Health-Enhancing Physical Activity. Brussels: European Commisssion, 2008. Reference Source

[8] International Society for Physical Activity and Health: Investments that Work for Physical Activity. Br J Sports Med. 2012; 46(10): 709–712. Publisher Full Text

[9] Klepac Pogrmilovic B, O'Sullivan G, Milton K, et al.: A global systematic scoping review of studies analysing indicators, development, and content of national-level physical activity and sedentary behaviour policies. Int J Behav Nutr Phys Act. 2018; 15(1): 123. PubMed Abstract | Publisher Full Text | Free Full Text

[10] Kohl HW, Craig CL, Lambert EV, et al.: The pandemic of physical inactivity: global action for public health. Lancet. 2012; 380(9838): 294–305. PubMed Abstract | Publisher Full Text

[11] Lakerveld J, Woods C, Hebestreit A, et al.: Advancing the evidence base for public policies impacting on dietary behaviour, physical activity and sedentary behaviour in Europe: The Policy Evaluation Network promoting a multidisciplinary approach. Food Policy. 2020. Publisher Full Text

[12] Lee IM, Shiroma EJ, Lobelo F, et al.: Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy. Lancet. 2012; 380(9838): 219–229. PubMed Abstract | Publisher Full Text | Free Full Text

[13] Messing S, Rütten A, Abu-Omar K, et al.: How Can Physical Activity Be Promoted Among Children and Adolescents? A Systematic Reviewof Reviews Across Settings. Front Public Health. 2019; 7: 55. PubMed Abstract | Publisher Full Text | Free Full Text

[14] Ouzzani M, Hammady H, Fedorowicz Z, et al.: Rayyan — a web and mobile app for systematic reviews. Syst Rev. 2016; 5(1): 210. PubMed Abstract | Publisher Full Text | Free Full Text

[15] Panter J, Guell C, Humphreys D, et al.: Can changing the physical environment promote walking and cycling? A systematic review of what works and how. Health Place. 2019; 58: 102161. PubMed Abstract | Publisher Full Text | Free Full Text

[16] Ramirez Varela A, Pratt M, Borges C, et al.: 1st Physical Activity Almanac: the Global Observatory for Physical Activity-GoPA. The Global Observatory for Physical Activity. 2016. Reference Source

[17] Rutten A, Schow D, Breda J, et al.: Three types of scientific evidence to inform physical activity policy: results from a comparative scoping review. Int J Public Health. 2016; 61(5): 553–63. PubMed Abstract | Publisher Full Text | Free Full Text

[18] Shea BJ, Grimshaw JM, Wells GA, et al.: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007; 7: 10. PubMed Abstract | Publisher Full Text | Free Full Text

[19] Swinburn B, Sacks G, Vandevijvere S, et al.: INFORMAS (International Network for Food and Obesity/non-communicable diseases Research, Monitoring and Action Support): overview and key principles'. Obes Rev. 2013; 14(Suppl 1): 157–64. PubMed Abstract | Publisher Full Text

[20] Volf K: Checklist for protocol for systematic literature review examining the evidence for impact of school policies on physical activity. 2020. http://www.doi.org/10.17605/OSF.IO/26QYF

[21] Woods CB, Mutrie N: Putting Physical Activity on the Policy Agenda. Quest. 2012; 64(2): 92–104. Publisher Full Text

[22] World Health Organisation: Global Health Risks: Mortality and burden of disease attributable to selected major risks. Geneva: World Health Organisation. 2009. Reference Source

[23] World Health Organisation Regional Office for Europe: Physical Activity Strategy for the WHO European Region 2016 - 2025. Vilnius: World Health Organisation, 2015. Reference Source

[24] World Health Organization: Global action plan for the prevention and control of noncommunicable diseases 2013-2020. 2013. Reference Source

[25] World Health Organization: Global action plan on physical activity 2018-2030: more active people for a healthier world. World Health Organization. 2019. Reference Source

[26] World Health Organization Regional Office for Europe: Review of physical activity promotion policy development and legislation in European Union Member States. Copenhagen: WHO Regional Office for Europe. 2010. Reference Source

Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity

Abstract

Keywords

Introduction

Protocol

Table 1. Search terms.

Table 2. Search terms based on the seven best investments document.

Eligibility criteria

Table 3. Population related inclusion criteria.

Study selection and data extraction

Risk of bias (quality) assessment

Strategy for data synthesis

Dissemination

Study status

Conclusion

Data availability

Underlying data

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Are you a HRB-funded researcher?

Thank you!

Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity

Abstract

Keywords

Introduction

Protocol

Table 1. Search terms.

Table 2. Search terms based on the seven best investments document.

Eligibility criteria

Table 3. Population related inclusion criteria.

Study selection and data extraction

Risk of bias (quality) assessment

Strategy for data synthesis

Dissemination

Study status

Conclusion

Data availability

Underlying data

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Competing Interests Policy

Stay Updated

Are you a HRB-funded researcher?

Thank you!