2) A B D → X
3) A B E D C → X
Based on the type of data being used, the context of what is being studied, and what is already known about a particular area of interest, a researcher will begin by selecting one of two commonly used analyses, crisp-set (csQCA) or fuzzy-set (fsQCA). In csQCA, conditions and the outcome are dichotomized, meaning that a given case’s membership to a condition or outcome is either fully in or fully out ( 76 ). Alternatively, membership on a fuzzy-set can fall into three, four, six-point, or continuous value set; enabling the researcher to qualitatively assess the degree of membership most appropriate for a case on any given condition.
Procedures for conducting a QCA are illustrated in Figure 2 below. Prior to beginning formal analyses, several steps including determining outcomes and conditions, identifying cases, and calibrating membership scores inform the development of a data matrix. QCA relies heavily on substantive knowledge, and decisions made throughout the analytics process are guided by a theoretical framework rather than inferential statistics ( 76 ). In the first step, researchers assign weights to constructs based on previous knowledge and theory, rather than basing thresholds on means or medians. The number of conditions is carefully selected, as having as many conditions as cases will result in uniqueness and failure to detect configurations ( 55 ). Once conditions have been defined and operationalized, each case can be dichotomized for membership. In crisp-set analysis, cases are classified as having full non-membership (0) or full membership ( 1 ) in the given outcome by using a qualitative approach (indirect calibration) or a quantitative approach with log odds (direct calibration) ( 76 ).
QCA as an approach and as an analytic technique
Source: Kane et al. ( 44 ), with permission.
Once a data matrix has been created, formal csQCA can commence using fs/QCA software, R suite, or other statistical packages for configurational comparative methods (a comprehensive list can be found on www.compass.org/software ). Analyses should begin with determining whether all conditions originally hypothesized to influence the outcome, are, indeed, necessary ( 81 ). Using a Boolean algebra algorithm, a truth table is then designed to provide a reduced number of configurations. A truth table may show contradictions (consistency score = .3-.7), indicating that it is not clear whether this configuration is consistent with the outcome ( 76 ). Several techniques can be used to resolve such contradictions ( 77 ), many of which entail revisiting the operationalization and/or selection of conditions, or reviewing cases for fit.
Once all contradictions are resolved and assigned full non-membership or full membership on the given outcome, sufficiency analyses can be conducted. Initial sufficiency analysis is usually based on the presence of an outcome. The Quine-McCluskey algorithm produces a logical combination or multiple combinations of conditions that are sufficient for the outcome to occur. Three separate solutions are given: parsimonious, intermediate, and complex. Typically, the intermediate solution is selected for the purposes of interpretation ( 76 ). One interest of interpreting findings is to explicitly state that this combination will almost always produce the given outcome. This is measured by consistency. While a perfect score of 1 indicates that this causal pathway will always be consistent with the outcome, a score ≥ .8 is a strong measure of fit ( 76 , 77 ). Complementary to consistency is coverage, or identifying the degree to which all cases were explained by a given causal pathway. While there is often a trade-off between these two measures of fit, without high consistency, it is not meaningful to have high coverage ( 76 ).
QCA has increasingly been used in health services research to evaluate program effectiveness and implementation where outcomes are dependent on interconnected structures and practices ( 15 , 28 , 30 , 46 , 47 , 48 , 90 , 92 ). For instance, Kane and colleagues ( 43 ) used QCA to examine the elements of organizational capacity to support program implementation that result in successful completion of public health program objectives in a public health initiative serving 50 communities. The QCA used case study and quantitative data collected from 22 awardee programs to evaluate the Communities Putting Prevention to Work (CPPW) program. The results revealed two combinations for combining most work plan objectives: 1) having experience implementing public health improvements in combination with having a history of collaboration with partners; and 2) not having experience implementing public health improvements in combination with having leadership support.
A third approach to quantitizing qualitative information used in evaluation research has been the coding and scaling of responses to interviews guided by existing implementation frameworks. These techniques call for assigning a numeric value to qualitative responses to questions pertaining to a set of variables believed to be predictive of successful implementation outcomes and then comparing the quantitative values across implementation domain, different implementation sites, or different stakeholder groups involved in implementation ( 24 , 95 ).
In an illustration of this approach, Damschroder and Lowery ( 22 ) embedded the constructs of the Consolidated Framework for Implementation Research (CFIR) ( 21 ) in semi-structured interviews conducted to describe factors that explained the wide variation in implementation of MOVE!, a weight management program disseminated nationally to Veterans Affairs (VA) medical centers. Interview transcripts were coded and used to develop a case memo for each facility. Numerical ratings were then assigned to each construct to reflect their valence and their magnitude or strength. This process is illustrated in Figure 3 below. The numerical ratings ranged from −2 (construct is mentioned by two or more interviewees a negative influence in the organization, an impeding influence on work processes, and/or an impeding influence in implementation efforts) to +2 (construct is mentioned by two or more interviewees as a positive influence in the organization, an impeding influence on work processes, and/or an impeding influence in implementation efforts). Of the 31 constructs assessed, 10 strongly distinguished between facilities with low versus high MOVE! implementation effectiveness; 2 constructs exhibited a weak pattern in distinguishing between low versus high effectiveness; 16 constructs were mixed across facilities; and 2 had insufficient data to assess.
Team-based work flow for case analysis. Source: Damschroder and Lowery, ( 22 ).
In the absence of quantification of the qualitative data in these three analytical approaches, a thematic content analysis approach ( 43 ) might have been used for analysis of the data obtained from the small group concept mapping brainstorming sessions or the interviews or focus groups that are part of the QCA. A qualitative framework approach ( 33 ) might have been used for analysis of the data obtained from the interviews using the CFIR template. The analysis would be inductive for data collected for the concept mapping exercise, inductive-deductive for data collected for the QCA exercise, and deductive for the data collected for the framework exercise. With the quantification, these data are largely used to describe a framework (concept mapping) that could be used to generate hypotheses (implementation framework) or to test hypotheses (qualitative comparative analysis).
In addition to their use to evaluate intervention effectiveness and implementation, mixed methods have increasingly been employed to develop innovative measurement tools. Three such recent efforts are described below.
The SIC is an 8-stage assessment tool ( 9 ) developed as part of a large-scale randomized implementation trial that contrasted two methods of implementing Treatment Foster Care Oregon (TFCO [formerly Multidimensional Treatment Foster Care] ( 10 ), an EBP for youth with serious behavioral problems in the juvenile justice and child welfare systems. The eight stages range from Engagement (Stage 1) with the developers/ purveyors in the implementation process, to achievement of Competency in program delivery (Stage 8). The SIC was developed to measure a community or organization’s progress and milestones toward successful implementation of the TFCO model regardless of the implementation strategy utilized. Within each of the eight stages, sub activities are operationalized and completion of activities are monitored, along with the length of time taken to complete these activities.
In an effort to examine the utility and validity of the SIC, Palinkas and colleagues ( 64 ) examined influences on the decisions of administrators of youth-serving organizations to initiate and proceed with implementation of three EBPs: Multisystemic Therapy ( 43 ), Multidimensional Family Therapy ( 52 ), and TFCO. Guided by the SIC framework, semi-structured interviews were conducted with 19 agency chief executive officers and program directors of 15 youth-serving organizations. Agency leaders’ self-assessments of implementation feasibility and desirability in the stages that occur prior to (Pre-implementation), during (Implementation), and after (Sustainment) phases were found to be influenced by several characteristics of the intervention, inner setting and outer setting that were unique to a phase in some instances and found to operate in more than one phase in other instances. Findings supported the validity of using the SIC to measure implementation of EBPs other than TFCO in a variety of practice settings, identified opportunities for using agency leader models to develop strategies to facilitate implementation of EBP, and supported using the SIC as standardized framework for guiding agency leader self-assessments of implementation.
The development of the SMS to measure sustainment of prevention programs and initiatives is another illustration of the use of mixed methods to develop evaluation tools. Palinkas and colleagues ( 69 , 70 ) interviewed 45 representatives of 10 grantees and 9 program officers within 4 SAMHSA prevention programs to identify key domains of sustainability indicators (i.e., dependent variables) and requirements or predictors (i.e., independent variables). The conceptualization of “sustainability” was captured using three approaches: semi-structured interviews to identify experiences with implementation and sustainability barriers and facilitators; a free list exercise to identify how participants conceptualized sustainability, program elements they wished to sustain, and requirements to sustain such elements; and a checklist of CFIR constructs assessing how important each item was to sustainment. Interviews were analyzed using a grounded theory approach ( 14 ), while free lists and CFIR items were quantitized; the former consisting of rank-ordered weights applied to frequencies of listed items and the latter using a numeric scale ranging from 0 (not important) to 2 (very important) ( 69 ). Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations and networks; infrastructure and capacity to support sustainability; ongoing evaluation of performance and outcomes; and availability of funding and resources) and five elements were identified by two of three data sets (community need for program, community buy-in and support, supportive leadership, presence of a champion, and evidence of positive outcomes).
Another innovation in the assessment of implementation outcomes is the RE-AIM Qualitative Evaluation for Systematic Translation (RE-AIM QuEST), a mixed methods framework developed by Forman and colleagues ( 31 ). The RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework is often used to monitor the success of intervention effectiveness, dissemination, and implementation in real-life settings ( 35 ), and has been used to guide several mixed method implementation studies ( 6 , 50 , 54 , 79 , 82 , 85 ). The RE-AIM QuEST framework represents an attempt to provide guidelines for the systematic application of quantitative and qualitative data for summative evaluations of each of the five dimensions. These guidelines may also be used in conducting formative evaluations to help guide the process of implementation by identifying and addressing barriers in real time.
Forman and colleagues ( 31 ) applied this framework for both real-time and retrospective evaluation in a pragmatic cluster RCT of the Adherence and Intensification of Medications (AIM) program. Researchers found that the QuEST framework expanded RE-AIM in three fundamental ways: 1) allowing investigators to understand whether Reach, Adoption and Implementation varied across and within sites, 2) expanding retrospective evaluation of effectiveness by examining why the intervention worked or failed to work and explain which components of the intervention or the implementation context may have been barriers; and 3) explicating whether or not and in which ways the intervention was maintained. This information permitted researchers to improve implementation during the intervention and inform the design of future interventions.
Purposeful sampling is widely used in qualitative research for the identification of information-rich cases related to the phenomenon of interest ( 18 , 71 ). While criterion sampling is used most commonly in implementation research ( 68 ), combining sampling strategies may be more appropriate to the aims of implementation research and more consistent with recent developments in quantitative methods ( 8 , 27 ). Palinkas and colleagues ( 68 ) reviewed the principles and practice of purposeful sampling in implementation research, summarized types and categories of purposeful sampling strategies and provided the following recommendations: 1) use of a single stage strategy for purposeful sampling for qualitative portions of a mixed methods implementation study should adhere to the same general principles that govern all forms of sampling, qualitative or quantitative; 2) a multistage strategy for purposeful sampling should begin first with a broader view with an emphasis on variation or dispersion and move to a narrow view with an emphasis on similarity or central tendencies; 3) selection of a single or multistage purposeful sampling strategy should be based, in part, on how it relates to the probability sample, either for the purpose of answering the same question (in which case a strategy emphasizing variation and dispersion is preferred) or for answering related questions (in which case, a strategy emphasizing similarity and central tendencies is preferred); 4) all sampling procedures, whether purposeful or probability, are designed to capture elements of both similarity (i.e., centrality) and differences (i.e., dispersion); and 5) although quantitative data can be generated from a purposeful sampling strategy and qualitative data can be generated from a probability sampling strategy, each set of data is suited to a specific objective and each must adhere to a specific set of assumptions and requirements.
Conducting integrated mixed methods research poses several challenges, from design to analysis and dissemination. Given the many methodological configurations possible, as described above, careful thought about optimal design should occur early in the process in order to have the potential to integrate methods when deemed appropriate to answer the research question(s). Considerations must include resources (e.g., time, funding, expertise; see 58 ), as integrated mixed methods studies tend to be complex and non-linear. After launching an integrated mixed methods study, the team needs to consistently evaluate the extent to which the mixed methods intentions are being realized, as the tendency in this type of study is to work (e.g., collect data) in parallel, even through analysis, only then to find that the sources of data are not reconcilable and the potential of the mixed methods design is not reached. This lack of integration may result in separate publications with quantitative and qualitative results rather than integrated mixed methods papers. Several sets of guidelines and critiques are available to facilitate high-quality integrated mixed methods products (e.g., 17 , 60 , 61 ).
Another consideration of integrating the two sets of methods lies in assessing the advantages and disadvantages of doing so with respect to data collection. Of course, there are tradeoffs involved with each method introduced here. For instance, Rapid Assessment Procedures enable more time efficient data collection but require more coordination of multiple data collectors to insure consistency and reliability. Rapid Assessment Procedure – Informed Clinical Ethnography also enables time efficient field observation and review procedures that constitute ideal “nimble” mixed method approaches for the pragmatic trial, along with minimizing participant burden, allowing for real-time workflow observations, more opportunities to conduct “repeated measures” of qualitative data through multiple site visits, and greater transparency in the integration of investigator and study participant perspectives on the phenomena of interest. However, it discourages use of semi-structured interviews or focus groups that may allow for the collection of data that would provide greater depth of understanding. Concept mapping offers a structured approach to data collection designed to facilitate quantification and visualization of salient themes or constructs at the expense of a semi-structured approach that may provide greater depth of understanding of the phenomenon of interest. Collection of qualitative data on implementation and sustainment processes and outcomes can be used to validate, complement and expand as well as develop quantitative measures such as the SI, SMS and RE-AIM, but can potentially involve additional time and personnel for minimal benefit. The advantages and disadvantages of each method must be weighed when deciding whether or not to use them for evaluation.
Finally, consideration must be given to identifying opportunities for the appropriate use of the innovative methods introduced in this article. Table 2 below outlines the range of mixed method functions, research foci, and study design for each innovative method. For example, Rapid Assessment Procedures could be used to achieve the functions of convergence, complementarity, expansion and development, assess both process and outcomes in effectiveness and implementation studies. However, we anticipate that these methods can and should be applied in ways we have yet to anticipate. Similarly, new innovative methods will inevitably be created to accommodate the functions, foci and design of mixed methods evaluations.
Opportunities for use of innovative methods in mixed methods evaluations based on function, focus and design.
Methods | Mixed method function | Focus | Design |
---|---|---|---|
Collecting QUAL Data | |||
RAP | Convergence Complementarity Expansion Development | Process Outcomes | Effectiveness/ implementation |
RAPICE | Convergence Complementarity Expansion Development | Process Outcomes | Effectiveness/ implementation |
Analyzing (Quantitizing) QUAL Data | |||
Concept Mapping | Development | Predictors | Effectiveness/ implementation |
Qualitative Comparative Analysis | Development | Predictors Outcomes | Effectiveness/ implementation |
Implementation Frameworks | Expansion | Predictors | Implementation |
Measuring Evaluation Outcomes | |||
States of Implementation Completion | Development Convergence Complementarity Expansion | Outcomes Process | Implementation |
Sustainment Measurement System | Development Convergence Complementarity Expansion | Outcomes Process | Implementation |
RE-AIM QuEST | Convergence Complementarity Expansion | Outcomes | Effectiveness/ implementation |
Sampling | Sampling | Predictors Process Outcomes | Effectiveness/ implementation |
As evaluation research evolves as a discipline, the methods used by evaluation researchers must evolve as well. Evaluations are performed to achieve a better understanding of policy, program or practice effectiveness and implementation. They assess not just the outcomes associated with these activities, but the process and the context in which they occur. Mixed methods are central to this evolution ( 16 , 57 , 96 ). As they facilitate innovations in research and advances in the understanding gained from that research, so they must also change, adapt, and evolve. Options for determining suitability of particular designs are becoming increasingly sophisticated and integrated. This review summarizes only a fraction of the innovations currently underway. With each new application of mixed methods in evaluation research, the need for further change, adaptation and evolution becomes apparent. The key to the future of mixed methods research will be to continue building on what has been learned and to replicate designs that produce the most robust outcomes.
We are grateful for support from the National Institute on Drug Abuse (NIDA) (R34DA037516–01A1, L Palinkas, PI and P30DA027828, C. Hendricks Brown, PI) and the Department of Veterans Affairs (QUE 15–272, A. Hamilton, PI).
Disclosure Statement
The authors are not aware of any affiliations, memberships, funding, or financial holdings that might be perceived as affecting the objectivity of this review.
Summer is here, and so is the sale. Get a yearly plan with up to 65% off today! 🌴🌞
HubSpot CRM
Google Sheets
Google Analytics
Microsoft Excel
Try without registration Quick Start
Read engaging stories, how-to guides, learn about forms.app features.
Inspirational ready-to-use templates for getting started fast and powerful.
Spot-on guides on how to use forms.app and make the most out of it.
See the technical measures we take and learn how we keep your data safe and secure.
Defne Çobanoğlu
You have created a program or a product that has been running for some time, and you want to check how efficient it is. You can conduct evaluation research to get the insight you want about the project. And there are more than one method and way to obtain this information.
Afterward, when you collect the appropriate data about the program on its effectiveness, budget-friendliness, and opinions from customers, you can go one step further. The valuable information you collect from the research allows you to have a clear idea of what to do next. You can discard the project, upgrade it, make changes, or replace it. Now, let us go into detail about evaluation research and its methods.
Basically, evaluation research is a research process where you measure the effectiveness and success of a particular program, policy, intervention, or project. This type of research lets you know if the goal of that product was met successfully and shows you any areas that need improvement . The data gathered from the evaluation research gives a good insight into whether or not the time, money, and energy put into that project is worth it.
The findings from evaluation research can be used to form decisions about whether to continue, modify, discontinue, and improve future programs or interventions . Therefore, in other words, it means doing research to evaluate the quality and effectiveness of the overall project.
What is evaluation research?
Conducting evaluation research is an effective way of usability testing and cost-effectiveness of the current project or product. Findings gathered from evaluative research play a key role in assessing what works and what doesn't and identifying areas of improvement for sponsors and administrators. This type of evaluation is a good means for data collection, and it provides a concrete result for decision-making processes.
There are different methods to collect feedback ranging from online surveys to focus groups. Evaluation research is best used when:
When you want to conduct evaluation research, there are different types of evaluation research methods . You can go through possible methods and choose the most suitable one(s) for you according to your target audience, manpower, and budget to go through with the research steps. Let us look at the qualitative and quantitative research methodologies.
These are the type of methods that asks questions to get tangible answers that rely on numerical data and statistical analysis to draw conclusions . These questions can be “ How many people? ”, “ What is the price? ”, “ What is the profit rate? ” etc. Therefore, they provide researchers with quantitative data to draw concrete conclusions. Now, let us look at the quantitative research methods.
Surveys involve collecting data from a large number of people using appropriate evaluation questions to gather accurate feedback . This type of method allows for reaching a wider audience in a short time in a cost-effective manner. You can ask about various topics, from user satisfaction to market research. And, It would be quite helpful to use a free survey maker such as forms.app to help with your next research!
Phone surveys are a type of survey that involves conducting interviews with participants over the phone . They are a form of quantitative research and are commonly used by organizations and researchers to collect data from people in a short time. During a phone survey, a trained interviewer will call the participant and ask them a series of questions.
This type of research method basically aims to explore audience feedback. These methods are used to study phenomena that cannot be easily measured using statistical techniques, such as opinions, attitudes, and behaviors . Techniques such as observation, interviews, and case studies are used to form evaluation for this method.
Case studies involve the analysis of a single case or a small number of cases to be explored further. In a case study, the researcher collects data from a variety of sources, such as interviews, observations, and documents. The data collected from case studies are often analyzed to identify patterns and themes .
Using focus groups means having a small group of people and presenting them with a certain topic. A focus group usually consists of 6-10 people. The focus groups are introduced to a topic, product, or concept, and they present their reviews . Focus groups are a good way to obtain data as the responses are immediate. This method is commonly used by businesses to gain insight into their customers.
Conducting evaluation research has helped many businesses to further themselves in the market because a big part of success comes from listening to your audience. For example, Lego found out that only around %10 of their customers were girls in 2011. They wanted to expand their audience. So, Lego conducted evaluation research to find and launch products that will appeal to girls.
No matter the type of method you decide to go with, there are some essential questions you should include in your research process. If you prepare your questions beforehand and ask the same questions to all participants/customers, you will end up with a uniform set of answers. That will allow you to form a better judgment. Now, here are some good questions to include:
1 - How often do you use the product?
2 - How satisfied are you with the features of the product?
3 - How would you rate the product on a scale of 1-5?
4 - How easy is it to use our product/service?
5 - How was your experience completing tasks using the product?
6 - Will you recommend this product to others?
7 - Are you excited about using the product in the future?
8 - What would you like to change in the product/project?
9 - Did the program produce the intended outcomes?
10 - What were the unintended outcomes?
Generative research is conducted to generate new ideas or hypotheses by understanding your users' motivations, pain points, and behaviors. The goal of generative research is to define the possible research questions and develop new theories and plan the best possible solution for those problems . Generative research is often used at the beginning of a research project or product.
Evaluative research, on the other hand, is conducted to measure the effectiveness of a project or program. The goal of evaluative research is to measure whether the existing project, program, or product has achieved its intended objectives . This method is used to assess the project at hand to ensure it is usable, works as intended, and meets users' demands and expectations. This type of research will play a role in deciding whether to continue, modify, or put an to the project.
You can determine either to use generative or evaluation research by figuring out what you need to find out. However, of course, both methods can be useful throughout the research process in obtaining different types of evidence. Therefore, firstly determine your goal of conducting evaluation research, and then you can decide on the method to go with.
Conducting evaluation research means making sure everything is going as you want them to in your project or finding areas of improvement for your next steps. There are more than one methods to go with. You can do focus groups or case studies on collecting opinions, or you can do online surveys to get tangible answers.
If you choose to do online surveys, you can try forms.app, as it is one of the best survey makers out there. It has more than 1000 ready-to-go templates. If you wish to know more about forms.app, you can check out our article on user experience questions !
Defne is a content writer at forms.app. She is also a translator specializing in literary translation. Defne loves reading, writing, and translating professionally and as a hobby. Her expertise lies in survey research, research methodologies, content writing, and translation.
Işılay Kırbaş
forms.app Team
What is quantitative research, about this guide, introduction, quantitative research methodologies.
The purpose of this guide is to provide a starting point for learning about quantitative research. In this guide, you'll find:
Research involving the collection of data in numerical form for quantitative analysis. The numerical data can be durations, scores, counts of incidents, ratings, or scales. Quantitative data can be collected in either controlled or naturalistic environments, in laboratories or field studies, from special populations or from samples of the general population. The defining factor is that numbers result from the process, whether the initial data collection produced numerical values, or whether non-numerical values were subsequently converted to numbers as part of the analysis process, as in content analysis.
Citation: Garwood, J. (2006). Quantitative research. In V. Jupp (Ed.), The SAGE dictionary of social research methods. (pp. 251-252). London, England: SAGE Publications. doi:10.4135/9780857020116
Watch the following video to learn more about Quantitative Research:
(Video best viewed in Edge and Chrome browsers, or click here to view in the Sage Research Methods Database)
Researchers will compare two sets of numbers to try and identify a relationship (if any) between two things.
Researchers will attempt to quantify a variety of factors at play as they study a particular type of phenomenon or action. For example, researchers might use a descriptive methodology to understand the effects of climate change on the life cycle of a plant or animal.
To understand the effects of a variable, researchers will design an experiment where they can control as many factors as possible. This can involve creating control and experimental groups. The experimental group will be exposed to the variable to study its effects. The control group provides data about what happens when the variable is absent. For example, in a study about online teaching, the control group might receive traditional face-to-face instruction while the experimental group would receive their instruction virtually.
Researchers will attempt to determine what (if any) effect a variable can have. These studies may have multiple independent variables (causes) and multiple dependent variables (effects), but this can complicate researchers' efforts to find out if A can cause B or if X, Y, and Z are also playing a role.
Surveys can be considered a quantitative methodology if the researchers require their respondents to choose from pre-determined responses.
Library services.
Database search tips.
In most databases, the truncation symbol is *
Health Research Policy and Systems volume 22 , Article number: 114 ( 2024 ) Cite this article
62 Accesses
Metrics details
Poverty-related diseases (PRD) remain amongst the leading causes of death in children under-5 years in sub-Saharan Africa (SSA). Clinical practice guidelines (CPGs) based on the best available evidence are key to strengthening health systems and helping to enhance equitable health access for children under five. However, the CPG development process is complex and resource-intensive, with substantial scope for improving the process in SSA, which is the goal of the Global Evidence, Local Adaptation (GELA) project. The impact of research on PRD will be maximized through enhancing researchers and decision makers’ capacity to use global research to develop locally relevant CPGs in the field of newborn and child health. The project will be implemented in three SSA countries, Malawi, South Africa and Nigeria, over a 3-year period. This research protocol is for the monitoring and evaluation work package of the project. The aim of this work package is to monitor the various GELA project activities and evaluate the influence these may have on evidence-informed decision-making and guideline adaptation capacities and processes. The specific project activities we will monitor include (1) our ongoing engagement with local stakeholders, (2) their capacity needs and development, (3) their understanding and use of evidence from reviews of qualitative research and, (4) their overall views and experiences of the project.
We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change. A series of interconnected qualitative and quantitative data collections methods will be used, including knowledge translation tracking sheets and case studies, capacity assessment online surveys, user testing and in-depth interviews, and non-participant observations of project activities. Participants will comprise of project staff, members of the CPG panels and steering committees in Malawi, South Africa and Nigeria, as well as other local stakeholders in these three African countries.
Ongoing monitoring and evaluation will help ensure the relationship between researchers and stakeholders is supported from the project start. This can facilitate achievement of common goals and enable researchers in South Africa, Malawi and Nigeria to make adjustments to project activities to maximize stakeholder engagement and research utilization. Ethical approval has been provided by South African Medical Research Council Human Research Ethics Committee (EC015-7/2022); The College of Medicine Research and Ethics Committee, Malawi (P.07/22/3687); National Health Research Ethics Committee of Nigeria (01/01/2007).
Peer Review reports
Sub-Saharan Africa (SSA) has the highest under-five mortality rate in the world [ 1 ]. Although the global under-five mortality rate declined from 76 to 38 per 1000 live births between 2000 and 2019, more than half of the deaths in children and youth in 2019 were among children under 5 years, approximately 5.2 million deaths [ 1 ]. Poverty-related diseases including pneumonia, diarrhoea and malaria remain amongst the leading causes of death in children under-5 years [ 2 ].Thus, despite progress in the health of young children globally, most countries in SSA fall below the average gains and do not meet maternal and child health targets set by the United Nations Sustainable Development Goal 3 to ‘ensure healthy lives and promote wellbeing’ (1). As of December 2021, under-five mortality rates were reported as 113.8, 38.6 and 32.2 per 1000 live births for Nigeria, Malawi and South Africa, respectively [ 3 ]. Factors accounting for regional disparities in child mortality rates include poverty, socioeconomic inequities, poor health systems and poor nutrition, with coronavirus disease 2019 (COVID-19) adding substantially to the burden [ 4 ].
Addressing healthcare issues such as these requires an evidence-informed approach, where intervention design and implementation are based on the best available evidence, to ensure that scarce resources are used effectively and efficiently, avoid harm, maximize good and improve healthcare delivery and outcomes [ 5 , 6 , 7 ]. Evidence-informed practices have been growing in SSA [ 6 ], and evidence ecosystems are becoming stronger. The evidence ecosystem reflects the formal and informal linkages and interactions between different actors (and their capacities and resources) involved in the production, translation and use of evidence [ 6 , 8 , 9 ]. Guidance that can be developed through this ecosystem includes evidence-based health technology assessments (HTA) and clinical practice guidelines (CPGs). CPGs include recommendations that are actionable statements that are informed by systematic reviews of evidence, and an assessment of the benefits and harms of alternative care options and are intended to optimize patient care [ 10 ]. They can help bridge the gap between research evidence and practice and are recognized as important quality-improvement tools that aim to standardize care, inform funding decisions and improve access to care, among others.
Over the past decade, internationally and in SSA, there has been a rapid growth of CPGs developed for a range of conditions [ 11 ]. In particular, rapid evidence syntheses and guideline development methods has advanced in response to urgent evidence needs, especially during COVID [ 12 , 13 ]. For example, WHO has developed guidelines for all key infectious conditions that cause most deaths. This development has been accompanied by a growing volume of research evidence around CPGs, including the processes for their rapid development, adaptation, contextualization, implementation and evaluation, and further spurred on by COVID. For example, global knowledge leaders, such as the WHO and the GRADE Working Group, have set standards for CPG development, outlining the steps of what is known as ‘de novo’ (from scratch) CPG development [ 14 ]. Another global group, the Guidelines International Network (G-I-N), is a network dedicated to leading, strengthening and supporting collaboration in CPG development, adaptation and implementation. They have published minimum standards and the G-I-N McMaster guideline checklist, which contains a comprehensive list of topics and items outlining the practical steps to consider for developing CPGs [ 15 ].
As CPG standards have evolved, however, so has the complexity of development and adaptation. In the context of poorer settings, such as sub-Saharan Africa (SSA), CPG development is prohibitively human and finance resource intensive. It requires scarce skills, even in the growing evidence-based healthcare (EBHC) community, and financial investments by government where resources are often directed to healthcare services, rather than policymaking processes. Against this backdrop, several studies have found that CPGs in the region often perform poorly on reporting on their rigour of development and editorial independence [ 16 , 17 , 18 ]. Other, more resource-efficient methods for guideline development in SSA are, therefore, essential and urgently needed. Moreover, investment in the overall management of the process is needed, including convening the guideline group and moving stepwise through a rigorous process.
There is also increased international recognition of the value of taking guidelines developed in one country and applying them to other countries. This can avoid duplication of effort and research waste in de novo guideline development, when useful guidelines may exist elsewhere [ 12 , 19 ]. Against this backdrop, several adaptation methods are emerging for contextualization of recommendations to country needs (e.g. ADAPTE, adolopment and SNAP-it, amongst others) [ 19 , 20 , 21 ]. For example, WHO is developing strategies for adapting and implementing their CPGs at country level. One example is the WHO Antenatal Care Recommendations Adaptation Toolkit lead by the Department of Sexual and Reproductive Health and Research [ 22 ]. Their approach is pragmatic and transparent. Another approach is so-called ‘adolopment’, a GRADE method, in which the original guideline evidence is used, either adopted or adapted, considering contextual evidence such as costs and feasibility and local values [ 20 ]. Adolopment involves convening a guideline panel, reviewing available evidence and local contextual evidence and weighing up the panel’s judgements to make recommendations that are fit for purpose [ 20 ].
Despite these advances in CPG adaptation methods, many countries and professional associations in sub-Saharan Africa still use expert opinion-based approaches or proceed to prepare their own systematic reviews and guidelines, ultimately perpetuating resource wastage and duplication of efforts [ 23 ]. Moreover, when countries do adapt and contextualize other countries’ guidelines, there is frequently a lack of transparency and reporting on changes, without clarity on why or by whom. This in turn casts doubts on the recommendation’s credibility. For example, guidelines for child health in sub-Saharan Africa are usually derived from the WHO and UNICEF. However, adaptation of such guidelines and recommendations to national contexts is not well described [ 24 ]. Transparency in guideline adaptation is critical for creating trustworthy, context-sensitive recommendations. What guideline adaptation methods work best and how these can be transparently implemented in the context of lower resource settings, remain key research questions. Therefore, despite the emergence of several guideline adaptation approaches, we need to explore and understand how best to adapt recommendations from one context to another [ 25 ].
Another major advancement within guideline research has been growing recognition of the potential contribution of qualitative research evidence [ 26 , 27 ]. Traditionally, guidelines have been informed by systematic reviews of the effectiveness of specific interventions [ 14 ]. Such reviews provide robust evidence about which interventions ‘work’. However, there is appreciation that evidence regarding the potential effectiveness of an intervention is not sufficient for making recommendations or decisions. Policymakers also need to consider other issues, including how different stakeholders’ value different outcomes, the intervention’s acceptability to those affected by it and the feasibility of implementing the intervention [ 28 , 29 , 30 ]. Evidence from qualitative research is particularly well suited to exploring factors that influence an intervention’s acceptability and feasibility [ 31 , 32 ]. The use of qualitative research to inform recommendations by guidelines has become easier in recent years as systematic reviews of qualitative studies have become more common, and the methods for these reviews are now well developed [ 33 ]. The first WHO guideline to systematically incorporate reviews of qualitative studies was published in 2012 in the field of task-shifting for maternal and child health [ 31 ]. The inclusion of this qualitative evidence helped shape the panel’s recommendations [ 32 ], and this approach is now included in the WHO Handbook for Guideline Development and has been applied in many other WHO CPGs [ 34 , 35 ].
However, a key challenge in using findings from systematic reviews of qualitative evidence is communicating often complex findings to users such as guideline panel members to facilitate effective knowledge translation. While there is now considerable research on communicating findings from reviews of intervention effectiveness [ 36 ], there is limited experience on the usefulness of different options for packaging and presenting findings from systematic reviews of qualitative evidence to CPG panels. To make best use of this evidence, we need presentation formats that are accessible to users who may be unfamiliar with qualitative methods, are concise and simple while retaining sufficient detail to inform decisions and clearly present ‘confidence in the evidence from systematic reviews of qualitative evidence’ (GRADE-CERQual) assessments of how much confidence users should place in each finding [ 37 ]. In addition, we need to understand how qualitative evidence included in global guidelines, such as those produced by WHO, is interpreted and used in country-level guideline adaptation processes.
A final key guideline method advancement has been around the development of multi-layered and digitally structured communication formats for end users [ 38 , 39 ]. Guidelines are not an end in themselves. Recommendations may lack impact if not adequately communicated and disseminated to those who need to implement them, namely healthcare providers, managers and the public. Indeed, in a South African study of primary care guideline national policymakers, subnational health managers and healthcare providers agreed that dissemination is a particular gap [ 40 ]. While guidelines typically are produced as static documents (e.g. PDF formats), information technology is needed to enhance dissemination. The MAGIC authoring and publication Platform (MAGICapp/) was developed for this purpose ( https://magicevidence.org/magicapp/ ). MAGICapp is a web-based tool that enables evidence synthesizers and guideline organizations to create, publish and dynamically update trustworthy and digitally structured evidence summaries, guidelines and decision aids in user-friendly formats on all devices. Such digital multi-layered formats allow different users to rapidly find recommendations, while having the supporting evidence for them one click away [ 41 ]. MAGICapp, used by WHO, NICE and professional societies across the world, holds potential to enhance the impact of evidence-informed guideline recommendations in practice, in an enhanced evidence ecosystem [ 9 ]. However, the usability of the MAGICapp in sub-Saharan Africa, based on local user preferences for different communication formats, are key research questions.
Against this backdrop, the Global Evidence, Local Adaptation (GELA) project will maximize the impact of research on poverty-related diseases through enhancing researchers and decision makers’ capacity to use global research to develop locally relevant guidelines for newborn and child health in Malawi, Nigeria and South Africa. These guidelines will build on and add value to the large-scale programme of child health guideline development from agencies such as the WHO, to support adaptation and implementation led by national ministries in collaboration with WHO Afro regional office.
The overarching aim of GELA is to bridge the gap between current processes and global advances in evidence-informed decision-making and guideline development, adaptation and dissemination by building skills and sharing resources in ways that can be sustained beyond the project period. The project has seven linked and related work packages (WPs) to support delivery of the planned project deliverables. Table 1 provides a brief summary of the activities of each WP. This protocol outlines our approach for the monitoring and overall evaluation of the project activities and impact (WP 6).
The project will be implemented in three SSA countries: Malawi, South Africa and Nigeria over a 3-year period. The project adopts a multi-faceted multidisciplinary research and capacity strengthening programme using primary and secondary research, guideline adaptation methodology and digital platforms to support authoring delivery and dynamic adaptation. These processes will offer bespoke capacity strengthening opportunities for policy makers, researchers and civil society. Throughout the project, we plan for innovations in the tools we use, accompanied by comprehensive evaluation of all aspects of the research, research uptake into policy and capacity strengthening.
Ongoing monitoring and evaluation of project processes and activities will help facilitate ongoing engagement between researchers and stakeholders throughout the research project. This will in turn help ensure that the project is centred on a common goal, with clear understandings of the different research activities and potential impact. This can also promote research uptake and enable researchers to make adjustments to project activities, maximizing stakeholder engagement and research utilization.
The overarching aim of the monitoring and evaluation work package is to monitor and evaluate the various GELA project activities and processes, including whether, how and why activities took place or if goals were met.
The specific monitoring and evaluation objectives are to:
Monitor ongoing engagement with local stakeholders across work packages and explore what worked and didn’t and why;
Assess the capacity development needs of guideline panels and steering group committees and explore their views and experiences of the project’s capacity development activities;
Explore guideline panelists’ experiences with reading and using evidence from reviews of qualitative research, including their preferences regarding how qualitative review findings are summarized and presented;
Evaluate guideline panelists’, steering group committees’ and project team members’ overall views and experiences of the project, including the what works or not, to influence evidence-informed decision-making and guideline adaptation processes
We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change (Table 2 ). The theoretical underpinning for the GELA project across all work packages is related to the three-layered behaviour change wheel comprising opportunity, capability and motivation [ 42 ]. The design, delivery and implementation of multi-stakeholder integrated activities based on identified priority areas and needs is expected to lead to guideline related improved capacity, practice and policy within each country’s health system. Certain objectives also have specific underpinning theoretical frameworks, in addition to the overarching project Theory of Change, which are explained under the respective objectives below. A series of interconnected qualitative and quantitative data collections methods will be used to address each objective.
In what follows, we describe each objective and the methods we will use to achieve it, separately. However, in many cases the qualitative data collection cuts across objectives, with the same interviews and observations being used to explore multiple issues simultaneously (e.g. knowledge translation, capacity, overall views and experiences of the project, etc.). The relationship between the different objectives and associated methods are depicted in Tables 3 and 4 . Table 3 outlines the stakeholder groups included in the monitoring and evaluation work package, including their composition and for which objectives they are targeted. Table 4 provides the timeline for the different data collection methods and how they relate to each across the objectives.
Overall approach for this objective.
This objective will be guided by an integrated knowledge translation (IKT) approach. IKT focuses on the important role of stakeholder engagement in enhancing evidence-informed decision-making [ 43 ]. As part of work package 4 (‘dissemination and communication’), knowledge translation (KT) champions have been identified in each of the three countries and will work together to develop and implement country-level KT strategies. This will include defining KT objectives, identifying and mapping relevant stakeholders, prioritizing those we will actively engage and developing a strategy for engaging each priority stakeholder. We will monitor these engagements through the development and implementation of a tracking sheet, qualitative case studies and semi-structured interviews.
Participants will comprise of knowledge translation (KT) champions and relevant country-level stakeholders. KT champions are GELA project staff who have dedicated time to work on the communication, dissemination and engagement aspects at a country-level. At least one KT champion has been identified for each of Malawi, Nigeria and South Africa.
Relevant country-level stakeholders will be identified as part of the KT strategy development (WP4) and will comprise any health decision-makers, e.g. health practitioners, community groups, health system managers, policy-makers, researchers and media.
A tracking sheet will be used to capture information for each stakeholder related to the purpose, message, medium or forum, messenger, timing and resources for engagement. KT champions in each country will be responsible for tracking these details on a continuous basis, and the tracking sheet will be monitored bi-monthly at a meeting with KT champions from the three country teams. This will help us monitor whether and how engagement activities are taking place, as well as the strategies for implementation. The tracking sheets will consist of different in-country stakeholders (e.g. government officers, health professional associations, researchers, media, etc.), and there may be several goals for engaging each individual stakeholder. The engagement strategy will be reviewed and updated as priority stakeholders change over the research stages and project period. As such, the sample size will be determined iteratively.
We will analyse information with descriptive statistics. For example, we will group and count by categories: number and type of stakeholders, type of engagement activities, type of KT products produced, type of forum or medium used for dissemination, frequency and duration of engagement, follow-ups, intensive engagement period and resources required for engagement.
We will also develop case stories (or impact stories) describing engagement activities and processes between project staff and relevant stakeholders. The case studies will help us monitor successful engagement, disseminate best practice scenarios and draw out lessons for future engagements. We will identify case stories through the tracking sheet and at bi-monthly meetings with the KT co-ordinator, where KT champions will be asked to share success stories or learning moments. KT champions will not know which ‘case’ will be selected for the case study in advance. The information will be collected by the KT co-ordinator, who is not involved in any of the country strategy implementation. The information collected from the KT champions (and messenger, if the messenger is not the KT champion) will be via a standard case story template, including aim of engagement, what the engagement was, experiences from both sides (quotes to be included in stories), success of engagement, lessons learnt and any future engagement plans. The number of cases will be determined iteratively. The intention is to develop one case story from each country annually, showcasing different cases, e.g. type of KT goal, type of stakeholder, type of KT medium/forum, etc.
At project close (month 30), we will conduct semi-structured interviews to explore if, why and how project KT goals were met and what planned stakeholder engagements worked (and did not work) and why. The interviews will be conducted with KT champions, other messengers (e.g. communication officers), country leads and selected stakeholders. At least two people from each county (KT champion and messenger and/or stakeholder) will be interviewed, and so there will be six to eight interviews in total. Participants will be selected purposively for information-rich cases that can help yield insights and in-depth understanding of the nature and success (or not) of our stakeholder engagements [ 44 ].
These interviews will form part of the interviews conducted with project team members more broadly as part of objective 4, the methods of which are therefore described in more detail below.
Overarching theoretical lens.
We will draw on the Kirkpatrick model [ 45 ] as the underpinning theoretical framework for this objective. This model evaluates training effectiveness across four levels: (1) reaction, (2) learning, (3) behaviour and (4) results. The ‘reaction level’ assesses the degree of satisfaction of participants with the training event. The ‘learning level’ examines learning among participants both before and after the training event to determine any change in knowledge [ 46 , 47 ]. The ‘behaviour level’ assesses whether the training event has provided any favourable change in behaviour among participants. The final ‘results level’ assesses the use of knowledge gained through the training event within the workplace [ 46 , 47 ].
To assess the potential difference that project capacity development activities make, the outcomes of interest will be those related to training in evidence-based healthcare (EBHC). An overview of systematic reviews by Young and colleagues identified that EBHC training often aims to ‘improve critical appraisal skills and integration of results into decisions, and improved knowledge, skills, attitudes and behaviour among practising health professionals’ [ 48 , 49 ].
We will employ mixed methods to achieve this objective, including three rounds of online surveys (at baseline, mid-line and at the project close) as well as semi-structured interviews (at project close) and non-participant observations of meetings (various). The first online survey at baseline will assess the capacity needs of the guideline panels and steering group committees in South Africa, Malawi and Nigeria, and the two subsequent online surveys will assess the potential difference project capacity development activities make on these groups across all the four levels of the Kirkpatrick model, i.e. reaction, learning, behaviour and results. The capacity needs and progress of these groups will also be explored qualitatively through semi-structured interviews and observations of meetings.
Details of the project capacity development activities that will be implemented as part of work package 5 (‘capacity strengthening and sharing’) of the GELA project are outlined in Table 1 (above). All members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria will be invited and encouraged to attend all project capacity development activities. ‘On the job’ capacity building will also take place during the various meetings convened with these groups, as they are supported to identify priority topics, to appraise and discuss the evidence used to inform the recommendations and to formulate the final recommendations.
Participants will comprise members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria. Table 3 (above) provides details of the composition of the guideline panels and steering group committees.
Procedures and data collection tools.
At baseline (at approximately 6 months before engagement in any project training activities), at mid-line (month 18) and at the project close (month 30), all members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria will be invited, via email, to participate in a survey. In each of the three countries the guideline development group and steering group committees will include approximately 20 and 10 members, respectively; we will therefore aim to have 90 participants in total complete the survey. The email invitation to all three survey rounds will inform participants about the nature of the study and direct them to an online survey. The landing page of the survey will provide information about the purpose of the research project and what is being requested from the participants, with a consent statement at the end which the participant will be required to agree to before being able to continue with the survey. Data will only be collected from participants who consent to freely participate in the study. The survey will be carried out using a secure online survey platform (such as Microsoft Forms) where all cookies and IP address collectors will be disabled to protect the confidentiality of the participants and to avoid tracking of the participant activities online. Unique identifiers (last six numbers of their ID) will be used to track participants responses over time and link data from baseline to project close.
The baseline survey will be a short (10–25 min) form that will ask participants about their capacity needs and knowledge/skills in evidence-based healthcare (EBHC) and decision-making. The survey will capture demographic variables of participants at baseline, mid-term and at the end of the project. It will assess the training needs of participants at baseline, participants’ satisfaction at the end of each training activity, the knowledge and skills at baseline, mid-term and at the end of the project. Participants’ behaviour will also be assessed using open-ended questions and vignettes. The surveys will focus on all four levels (i.e. reaction, learning, behaviour and results) of the Kirkpatrick model.
All data collected on the secure online survey platform will be coded, cleaned and entered into STATA. Data collected for the baseline survey will be analysed using descriptive statistics to determine the frequency of the various training needs and qualitative data gathered using the open-ended questions will be analysed thematically using manual coding (or if available and dataset is large), and NVivo or a similar tool will be used to identify the recurring themes which emerge in the data collected about the key training needs of participants.
Data collected for the surveys conducted at midpoint and at project close will be analysed using descriptive statistics to determine if there has been a change in the learning, knowledge gained and behaviours over time, as well as the extent of the potential application of evidence-based practice, while the data collected using the open-ended questions will be analysed using thematic analysis outlining how project capacity development activities informed particular outcomes and results in the participant’s workplace. To determine change in skills (and trends over time such as confidence improvement or decay), the descriptive statistics will be supplemented by appropriate inferential statistics for repeated measures (paired data) such as McNemar or paired t -tests, reporting change in percentages as mean differences (such as self-reported confidence) with 95% confidence intervals or/and frequencies. Descriptive trends over time will also be presented graphically using line graphs or other visual aids as appropriate. However, these will be interpreted with caution as the primary analysis is descriptive. Statistical significance will be set at a p value of 0.05.
At project close (month 30), we will conduct semi-structured interviews with a sample of members from the guideline panels and steering group committees in South Africa, Malawi and Nigeria. Sampling will be purposive, with the aim of understanding the broad range of needs, experiences and perspectives and ensuring that the sample reflects a range of socio-demographic characteristics and stakeholder categories. We will begin with a sample size of 10–15 participants in each country; however, sampling will continue if we have not reached saturation of the data through the initial sample size [ 44 ].
Participants will be contacted, either by telephone or via email, and invited to participate in an interview. Interviews will be conducted face-to-face or electronically (e.g. using Microsoft Teams) at a date and time chosen by participants. Face-to-face interviews will take place at a location convenient to participants, which is conducive to a confidential exchange. The interviews will last between 45 and 60 min and will be conducted by researchers trained in qualitative research methodologies and interviewing techniques. The interviews will be guided by a semi-structured topic guide and will include questions informed by the four levels (i.e. reaction, learning, behaviour and results) of the Kirkpatrick model. Specifically, the questions will explore participants’ views and experiences regarding their capacity development needs and expectations of the project; whether and why these expectations were met (or not), the project capacity development activities, what they learned (or not) from these activities and what impact participants believe they have had (or may have) on their practices.
Verbal and written information about the study will be provided to all participants taking part in interviews. Written informed consent will be obtained from all participants before proceeding with the interview. With the permission of participants, all interviews will be digitally recorded.
We will conduct non-participant observations of guideline panel and steering group committee meetings. Observational methods can provide useful data on what people do, how they interact with each other and how they engage with particular artefacts in situ (rather than their accounts of these) [ 50 ]. The steering group committees in each country will meet approximately twice over the project duration (with the option for additional meetings): an initial meeting for project orientation (month 2/3) and again to identify priority topics and guideline gaps (month 6). Guideline panels in each country will meet approximately three times over the project duration (with the option for additional meetings): an initial meeting for project orientation and outcome prioritization (month 6/7), another potential meeting if necessary to finalize outcome prioritization and a final meeting to draft recommendations for the guideline (months 17–20). Meetings for both groups will be held virtually or in person, informed by preferences of the committee.
With the exception of the initial steering group committee (month 2/3), at least one researcher will be present to observe guideline panel and steering group committee meetings. The observer will aim to identify any capacity-related needs, expectations, gaps, strengths, achievements and challenges and the contexts in which these occur. He or she will also pay particular attention to group dynamics and the interactions between members and different stakeholder groups, and the potential impact of these on capacity-related issues. Observations will be informed by Lofland’s [ 51 ] criteria for organizing analytical observations (acts, activities, meanings, participation, relationships and settings). The observer will take detailed observational notes. With consent of the attendees, all meetings will also be digitally recorded. The recordings will be used to identify further issues not identified and to deepen or clarify issues noted, through the real-time observations of verbal engagements.
Interview and meeting recordings will be transcribed verbatim, and all personal identifying information will be removed from transcripts. The anonymized transcripts, together with observational notes, will be downloaded into Nvivo, a software programme that aids with the management and analysis of qualitative data. Analysis of the qualitative data will proceed in several rounds. First, as with all qualitative data analysis, an ongoing process of iterative analysis of the data will be conducted throughout the data collection period. Second, we will use a thematic analysis approach, using the phases described by Braun and Clarke [ 52 ], to identify key themes pertaining to participants’ capacity development needs and expectations and whether, how and why project capacity development activities met (or not) these needs and expectations. Finally, findings from the surveys (as described above) will also be integrated with the findings from the thematic analysis using a ‘narrative synthesis’ approach, a technique recommended by the Cochrane Collaboration as a way of synthesizing diverse forms of qualitative and quantitative evidence in mixed methods studies [ 53 , 54 ]. This approach will allow for both robust triangulation, and a more comprehensive interpretation of the difference project capacity development activities may have made on the guideline panels and steering group committees.
Objective 3 of the monitoring and evaluation stakeholder matrix work package explores how guideline panels view and experience evidence from the review(s) of qualitative research, including how it is summarized and presented. Here, we will employ a user testing approach, drawing on the methods and guidance of the SURE user test package 2022 developed by Cochrane Norway ( https://www.cochrane.no/our-user-test-package ) and which has been used to test various evidence-related products [ 55 , 56 , 57 , 58 ]. User testing involves observing people as they engage with a particular product and listening to them ‘think-aloud’. The goal is to gain an understanding of users’ views and experiences, the problems they face and to obtain suggestions for how a product may be improved [ 55 , 56 , 57 , 58 ].
We will begin by identifying or preparing relevant reviews of qualitative research. We will then develop review summary formats and explore guideline panel members’ views and experiences of these formats. We will revise the formats in multiple iterative cycles.
Identifying or preparing relevant reviews of qualitative research
As part of WP2 of the project (‘evidence synthesis’), we will identify relevant review(s) of qualitative research, including reviews exploring how people affected by the interventions of interest value different outcomes, the acceptability and feasibility of the intervention and potential equity, gender and human rights implications of the intervention. These reviews need to be assessed as sufficiently recent and of a sufficient quality. They also need to have applied GRADE-CERQual assessments to the review findings. Where necessary, we will update existing reviews or prepare reviews ourselves.
Developing the review summaries
In WP3 of the project (‘decision-making’) the evidence from these reviews will be provided to guideline panels as part of the evidence-to-decision (‘EtD’) frameworks that will inform the recommendations they develop (see Table 1 for further details about project work packages 2 and 3). Our next step will therefore be to prepare summaries of the reviews in a format that can easily be included in the EtD frameworks.
Each summary needs to present review findings that are relevant to specific parts of the EtD framework (typically the ‘values’, ‘acceptability’, ‘feasibility’ and ‘equity’ components). It also needs to include information about our confidence in these findings. Finally, the summary needs to indicate where this evidence comes from and to allow guideline panels to move from the summary to more detailed information about the evidence.
Most of this information is found in the review’s Summary of Qualitative Findings tables. However, these tables are usually too large for EtD frameworks and are not tailored to each framework component. We will, therefore, start by creating new summaries, using a format that we have previously used in EtD frameworks [ 59 , 60 , 61 ] but that we have not user tested. As opposed to the Summary of Qualitative Findings tables, where each finding and our confidence in the finding, is presented individually in separate rows, this format involves pulling the findings and confidence assessments together in short, narrative paragraphs.
User testing the summary format
For our first set of user tests, we will observe guideline panels participating in the CPG panel simulation workshops. For our second round of user tests, we will observe how the guideline panels experience and interact with this qualitative evidence during the real guideline processes. Third, we will then test a potentially refined format with a selection of guideline panel members using a semi-structured interview guide. Finally, at the end of the project, we will conduct semi-structured interviews with a selection of guideline panel members to explore their broader views and experiences of interpreting and using evidence from reviews of qualitative studies in their deliberation processes. Figure 1 provides a visual depiction of this iterative process.
Iterative approach for user testing evidence from reviews of qualitative research
We will draw on the adapted version of Peter Morville’s original honeycomb model of user experience [ 62 ] as the underpinning theoretical framework for this objective [ 63 ] (Fig. 1 ). This adapted version extends and revises the meaning of the facets of user experience depicted in the original model. It includes eight facets: accessibility, findability, usefulness, usability, understandability, credibility, desirability and affiliation. Accessibility involves whether there are physical barriers to gaining access; findability is about whether the person can locate the product or the content that they are looking for; usefulness is about whether the product has practical value for the person; usability comprises how easy and satisfying the product is to use; understandability is about whether the person comprehends correctly both what kind of product it is and the content of the product (and includes both user's subjective perception of her own understanding and an objective measure of actual/correct understanding); credibility comprises whether the product/content is experienced as trustworthy; desirability is about whether the product is something the person wants and has a positive emotional response to it; affiliation involves whether the person identifies with the product, on a personal or a social level, or whether it is alienating and experienced as being not designed for ‘someone like me’. The adapted model also adds to the original model a dimension of user experience over time, capturing the chronological and contingent nature of the different facets.
Participants will comprise members of the guideline panels in South Africa, Malawi and Nigeria. Table 3 (above) provides details of the composition of the guideline panels.
We will conduct non-participant observations of the CPG panel simulation workshops and the subsequent guideline panel meetings for developing the recommendations. The CPG panel simulation workshops will run a simulation of a real guideline process and give guideline panels an opportunity to understand how the guideline process works before they participate in real panel meetings. The guideline panels in all three countries will be invited and encouraged to attend these workshops, which will form part of the project capacity development activities of WP5 (Table 1 ).
With the participants’ consent, both the simulation workshops and meetings will be digitally recorded and at least two observers will observe and take notes. The observations will focus on how guideline panel members refer to and interact with the summaries of qualitative evidence. Drawing on a user testing approach ( https://www.cochrane.no/our-user-test-package ), we will also look specifically for both problems and facilitators in the way the qualitative evidence is formatted, including ‘show-stoppers’ (the problem is so serious that it hindered participants from correct understanding or from moving forward), ‘big problems/frustrations’ (participants were confused or found something difficult but managed to figure it out or find a way around the problem eventually), ‘minor issues/cosmetic things’ (small irritations, frustrations and small problems that do not have serious consequences, as well as likes/dislikes), ‘positive/negative feedback’, ‘specific suggestions’, ‘preferences’ and any other ‘notable observations’, e.g. feelings of ‘uncertainty’.
Based on the insights gained from the non-participant observations (above), we may make changes or refinements to our original summary format (Fig. 1 ). Once the guideline panel meetings have concluded (approximately by month 20), we will then conduct structured user testing interviews to test the potentially refined summary format. These interviews will be conducted with a sample of members from the guideline panels in South Africa, Malawi and Nigeria. Sampling will be purposive, with the aim of understanding the broad range of experiences and perspectives and ensuring the sample reflects a range of socio-demographic characteristics and stakeholder categories. As recommended ( https://www.cochrane.no/our-user-test-package ), we will begin with a sample size of six to eight participants in each country; however, sampling will continue until saturation is achieved [ 44 ].
Participants will be contacted, either telephonically or via email, and invited to participate in an interview. Interviews will be conducted face-to-face or electronically (e.g. using Skype or Teams) at a date and time chosen by participants. Face-to-face interviews will take place at a location convenient to participants, which is conducive to a confidential exchange. In line with the SURE user test package 2022 guidance, the interviews will last approximately 60 min ( https://www.cochrane.no/our-user-test-package ). They will be facilitated by a test leader, who will accompanied by at least one observer who will take notes. Both the test leader and observer(s) will be trained in user testing interviewing methodology and techniques. Verbal and written information about the study will be provided to all participants taking part in interviews. Written informed consent will be obtained from all participants before proceeding with the interview. With the permission of participants, all interviews will be video recorded.
For these interviews we will show panel members the latest version of the format, explore immediate first impressions, and then opinions about different elements of the summary. We may also show panel members different formats where we think this may be helpful. We will use a structured interview guide which draws heavily on other interview guides that been developed to user test evidence-related products [ 55 , 56 , 57 , 58 ]. It will include questions related the participant’s background; their immediate first impressions of the summary format(s); in-depth walk-through of the summary format(s), with prompts to think aloud what they are looking at, thinking, doing and feeling; and suggestions for improving the way the summary is formatted and for improving the user testing itself. We may ask follow-up questions to specific issues we observed in the simulation workshops and guideline panel meetings and/or create scenarios that resemble issues we observed in the workshops/meetings. This will be decided upon based on the findings that emerge from these workshops/meetings. The guide will be finalized once the relevant qualitative evidence (from WP2) has been produced and we have gained insights from the workshops and meetings.
As with the non-participant observations of meetings and workshops, throughout the interview, the observers will make notes about the participant’s experience as heard, observed and understood. Drawing on a user testing approach, they will look specifically for both problems and facilitators, specific suggestions, preferences and any other notable observations (as described above under ‘non-participant observations’).
At project close (month 30), we will also conduct semi-structured interviews with a sample of members from the guideline panels in South Africa, Malawi and Nigeria. These will be the same interviews with guideline panel members as described in objective 2. In addition to exploring participants’ capacity development needs, expectations and achievements, the semi-structured topic guide will also explore their views and experiences of (and specific capacity in) interpreting and using evidence from reviews of qualitative studies in guideline processes. More specifically, questions will investigate participants’ familiarity/experience with qualitative evidence; their perceptions of different types of evidence, what constitutes qualitative evidence and the role of qualitative evidence in guideline processes; and their experiences of using the qualitative evidence in their deliberations as part of the project, including what influenced its use and whether they found it useful. Details pertaining to sampling, data collection procedures and collection tools are described in objective 2.
All interview and meeting recordings will be transcribed verbatim, and all personal identifying information will be removed from transcripts. The anonymized transcripts, together with observational notes (from the workshops, meetings and interviews), will be downloaded into a software programme that aids with the management and analysis of qualitative data. Analysis of the data will be guided by the user testing analysis methods described in the SURE user test package 2022 ( https://www.cochrane.no/our-user-test-package ). The analysis will proceed in several, iterative rounds to develop and revise the summary format and to inform the focus of subsequent data collection. After each user test, we will review our notes, first separately and then together. In line with the SURE user test package 2022 guidance, we will look primarily for barriers and facilitators related to correct interpretation of the summary’s contents, ease of use and favourable reception, drawing on the facets of the revised honeycomb model of user experience (Fig. 2 ). We will trace findings back to specific elements or characteristics of the summaries that appeared to facilitate or hinder problems. Before the next set of user tests, we will discuss possible changes that could address any identified barriers and make changes to the summary format.
Adapted version of Peter Morville’s honeycomb model of user experience
This objective explores overall views and experiences of the project, with a focus on guideline panelists, steering group committees and project team members. Specifically, it seeks to gain an understanding of these three stakeholder groups’ more general views and experiences of the project activities they were involved with and whether, why and how these activities may influence (or not) evidence-informed decision-making and guideline adaptation processes. This will be achieved through semi-structured interviews.
Participants will comprise members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria, as well as members of the project team (as described in Table 3 above).
At project close (month 30), we will conduct semi-structured interviews with a sample of members from the guideline panels and steering group committees in South Africa, Malawi and Nigeria. These will be the same interviews and participants as described in objective 2. In addition to exploring issues around capacity development and qualitative evidence, the interviews will also investigate participants’ views and experiences of the various project activities they were involved with, and whether, why and how these activities may influence (or not) evidence-informed decision-making and guideline adaptation processes. Details pertaining to sampling, data collection procedures and collection tools are described in objective 2.
At project close (month 30), we will also conduct semi-structured interviews with members of the project team (see Table 3 for details of project team composition). We will begin by interviewing all project management team members, WP leads and KT champions. Additional participants will be determined iteratively (depending on what emerges from initial interviews) and purposively, with the aim of understanding the broad range of experiences and perspectives and ensuring the sample reflects the various groups which make up the project team. Interviews will be conducted face-to-face or electronically (e.g. using Skype or Teams) at a date and time chosen by the interviewee. The interviews will last between 45 and 60 min and will be guided by a semi-structured topic guide. The questions will explore participants’ views and experiences of the respective work packages in which they were involved, including what the primary goals of the work package were; if, why and how these goals were met; and what worked and what did not work and why.
The same qualitative data analysis procedures and methods will be used as described in objective 2. For this objective, the thematic analysis will identify key themes pertaining to views and experiences of project activities, including what worked (or not) and why, whether, why and how the project may (or not) influence evidence-informed decision-making and guideline development, adaptation and dissemination processes in South Africa, Malawi and Nigeria and potential barriers and facilitators to the sustainability of this influence.
Evidence-based guideline development is a multi-stakeholder, multi-perspective, complex set of tasks. There is limited, if any, research that has followed these steps from the perspectives of policymakers or researchers from start to end. The GELA project protocol sets out to monitor and evaluate various key steps in the process, using in-depth qualitative methods alongside appropriate surveys not only to inform the project as it progresses but also to understand the overall impact of all steps on development of transparent and contextually-rich guideline recommendations. Following WHO’s guideline steps, the tasks range from scoping stakeholder-informed priority topics to conducting relevant data gathering and evidence synthesis, followed by guideline panel meetings to reach consensus decisions and finally to produce recommendations that can be useful to end-users and improve health and care outcomes. The GELA project is undertaking a 3-year project to conduct these tasks in the context of newborn and child health priorities. We are doing this in collaboration with national ministries of health, academics, non-governmental partners and civil society groups in Malawi, Nigeria and South Africa. Overall, we aim build capacity across all collaborators for evidence-informed guideline development, while producing fit for context guideline recommendations, in accessible formats that benefit children, caregivers and health care providers.
As such, this is a practical research project, in that the products should directly impact care decisions at the national level but with the added benefit of being able to learn about what works or does not work for collaborative guideline development in country. We will also be applying emergent guideline adaptation methods to explore reducing duplication of expensive guideline development efforts in our lower resource settings. Our project addresses newborn and child health, keeping this most vulnerable population in our focus, hoping that producing sound evidence-based recommendations has the potential to impact care.
Through some of our formative work, we have completed a landscape analysis identifying and describing all available newborn and child health guidelines in each of the partner countries. In all countries there were similar findings, (1) there is no easy access to guidelines for end-users, thus locating a guideline requires effort and screening through multiple sources; (2) considering national priority conditions in this age group, there were often gaps in available current guidelines for managing children; and (3) when we appraised the guidelines using the global standard, AGREE II tool, we found that the reporting of guideline methods were poor, leaving it uncertain whether the recommendations were credible or whether any influences or interests had determined the direction of a recommendation. Finally, we expected to find many adapted guidelines, based on WHO or UNICEF or similar guidance available globally; however, very few of the identified guidelines stated clearly whether they had been adapted from other sources and, if so, which recommendations were adopted and which adapted.
Given progress globally in methods for guideline development, the continued poor reporting on guideline methods at the country level speak to a breakdown in skills-sharing globally, for example, WHO produces guidelines that are recognized as rigorous and follow good practice and reporting, but the same standards are not supported in country. Overall, GELA aims to address these key gaps in national guideline approaches for adaptation, but we need to recognize that this will be a long term process and that we need to learn from each other about what works and what may not serve us. Therefore, this protocol outlines our approach for monitoring several aspects of the project in our efforts to move closer to trustworthy and credible guidelines that all can use and trust for countries like ours.
Not applicable.
Poverty-related diseases
Sub-Saharan Africa
Evidence-informed decision-making
Evidence-based healthcare
Global Evidence Local Adaptation
Knowledge translation
Levels and trends in child mortality: report 2020. Estimates developed by UNICEF. ISBN: 978-92-806-5147-8. https://www.who.int/publications/m/item/levels-and-trends-in-child-mortality-report-2020 . United Nations Children’s Fund. Accessed Aug 2024.
Liu L, Oza S, Hogan D, et al. Global, regional, and national causes of under-5 mortality in 2000–15: an updated systematic analysis with implications for the sustainable development goals. Lancet. 2016;388(10063):3027–35.
Article PubMed PubMed Central Google Scholar
Under-five mortality. 2021 https://data.unicef.org/topic/child-survival/under-five-mortality/ . United Nations International Children’s Fund. Accessed Aug 2024.
Stewart R, El-Harakeh A, Cherian SA. Evidence synthesis communities in low-income and middle-income countries and the COVID-19 response. Lancet. 2020;396(10262):1539–41.
Article CAS PubMed PubMed Central Google Scholar
Mijumbi RM, Oxman AD, Panisset U, Sewankambo NK. Feasibility of a rapid response mechanism to meet policymakers’ urgent needs for research evidence about health systems in a low income country: a case study. Implementation Sci. 2014;9:114.
Article Google Scholar
Stewart R, Dayal H, Langer L, van Rooyen C. The evidence ecosystem in South Africa: growing resilience and institutionalisation of evidence use. Palgrave Commun. 2019;5(1):90.
Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Promotion of evidence-informed health policymaking in Nigeria: bridging the gap between researchers and policymakers. Glob Public Health. 2012;7(7):750–65.
Article PubMed Google Scholar
Young T, Garner P, Clarke M, Volmink J. Series: clinical epidemiology in South Africa Paper 1: evidence-based health care and policy in Africa.: past, present, and future. J Clin Epidemiol. 2017;83:24–30.
Vandvik PO, Brandt L. Future of evidence ecosystem series: evidence ecosystems and learning health systems: why bother? J Clin Epidemiol. 2020;123:166–70.
Graham R, Mancher M, Wolman D, Greenfield S, Steinberg E. Clinical practice guidelines we can trust. Washington, DC: The National Academy Press; 2011.
Book Google Scholar
Kredo T, Bernhardsson S, Machingaidze S, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8.
McCaul M, Tovey D, Young T, et al. Resources supporting trustworthy, rapid and equitable evidence synthesis and guideline development: results from the COVID-19 evidence network to support decision-making (COVID-END). J Clin Epidemiol. 2022;151:88–95.
Tricco AC, Garritty CM, Boulos L, et al. Rapid review methods more challenging during COVID-19: commentary with a focus on 8 knowledge synthesis steps. J Clin Epidemiol. 2020;126:177–83.
World Health Organization. WHO handbook for guideline development. Geneva: World Health Organization; 2011.
Google Scholar
Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–42.
Kredo T, Gerritsen A, van Heerden J, Conway S, Siegfried N. Clinical practice guidelines within the Southern African development community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases. Health Res Policy Syst. 2012;10:1.
Malherbe P, Smit P, Sharma K, McCaul M. Guidance we can trust? The status and quality of prehospital clinical guidance in sub-Saharan Africa: a scoping review. African J Emerge Med. 2021;11(1):79–86.
McCaul M, Clarke M, Bruijns SR, et al. Global emergency care clinical practice guidelines: a landscape analysis. African J Emerge Med. 2018;8(4):158–63.
Dizon JM, Grimmer K, Louw Q, Kredo T, Young T, Machingaidze S. South African guidelines excellence (SAGE): adopt, adapt, or contextualise? South African Med J. 2016;106(12):1177–8.
Article CAS Google Scholar
Schünemann HJ, Wiercioch W, Brozek J, et al. GRADE evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.
ADAPTE Collaboration. 2009 The ADAPTE process: Resource toolkit for guideline adaptation, version 2. http://www.g-i-n.net/ . Accessed Aug 2024.
Barreix M, Lawrie TA, Kidula N, et al. Development of the WHO antenatal care recommendations adaptation toolkit: a standardised approach for countries. Health Res Policy Syst. 2020;18(1):70.
Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.
Opiyo N, Shepperd S, Musila N, English M, Fretheim A. The, “child health evidence week” and grade grid may aid transparency in the deliberative process of guideline development. J Clin Epidemiol. 2012;65(9):962–9.
Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implemen Sci. 2018;13(1):72.
Langlois EV, Tunçalp Ö, Norris SL, Askew I, Ghaffar A. Qualitative evidence to improve guidelines and health decision-making. Bull World Health Organ. 2018. https://doi.org/10.2471/BLT.17.206540 .
Carmona C, Baxter S, Carroll C. Systematic review of the methodological literature for integrating qualitative evidence syntheses into health guideline development. Res Synth Methods. 2021;12(4):491–505.
Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Res Alloc. 2012;10(1):9.
Verkerk K, Van Veenendaal H, Severens JL, Hendriks EJ, Burgers JS. Considered judgement in evidence-based guideline development. Int J Qual Health Care. 2006;18(5):365–9.
Alonso-Coello P, Schunemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016. https://doi.org/10.1136/bmj.i2016 .
Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process. Implement Sci. 2016;11:98.
Lewin S, Glenton C. Are we entering a new era for qualitative research? using qualitative evidence to support guidance and guideline development by the World Health Organization. Int J Equity Health. 2018;17(1):126.
Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Chapter 21: Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook . Accessed Aug 2024.
Glenton C, Lewin S, Norris S. Using evidence from qualitative research to develop WHO guidelines (Chapter 15) World Health Organization Handbook for Guideline Development. 2nd ed. WHO: Geneva; 2016.
World Health Organisation. WHO recommendations: intrapartum care for a positive childbirth experience. Geneva: WHO; 2018.
Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620–6.
Lewin S, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table. Implement Sci. 2018;13(Suppl 1):10.
Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.
Brandt L, Vandvik PO, Alonso-Coello P, et al. Multilayered and digitally structured presentation formats of trustworthy recommendations: a combined survey and randomised trial. BMJ Open. 2017;7(2): e011569.
Kredo T, Cooper S, Abrams AL, et al. ’Building on shaky ground’-challenges to and solutions for primary care guideline implementation in four provinces in South Africa: a qualitative study. BMJ Open. 2020;10(5): e031468.
Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9.
Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42.
Jessani NS, Rohwer A, Schmidt B-M, Delobelle P. Integrated knowledge translation to advance noncommunicable disease policy and practice in South Africa: application of the exploration, preparation, implementation, and sustainment (EPIS) framework. Health Res Policy Syst. 2021;19(1):82.
Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229–45.
Roos M, Kadmon M, Kirschfink M, et al. Developing medical educators–a mixed method evaluation of a teaching education program. Med Educ Online. 2014;19:23868.
Bijani M, Rostami K, Momennasab M, Yektatalab S. Evaluating the effectiveness of a continuing education program for prevention of occupational exposure to needle stick injuries in nursing staff based on Kirkpatrick’s model. J Natl Med Assoc. 2018;110(5):459–63.
PubMed Google Scholar
Carlfjord S, Roback K, Nilsen P. Five years’ experience of an annual course on implementation science: an evaluation among course participants. Implement Sci. 2017;12(1):101.
Young T, Rohwer A, Volmink J, Clarke M. What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLoS ONE. 2014;9(1): e86706.
Young T, Dizon J, Kredo T, et al. Enhancing capacity for clinical practice guidelines in South Africa. Pan Afr Med J. 2020;36:18.
Green G, Thorogood N. Qualitative methods for health research. London: Sage; 2004.
Lofland J. Analyzing social settings. Belmont, CA: Wadsworth Publishing Company; 1971.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101.
Popay J, Roberts H, Sowden A. Guidance on the conduct of narrative synthesis in systematic reviews a product from the ESRC methods programme. I Health Research: Lancaster; 2006.
Pope C, Mays N, Popay J. How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers? Healthc Manage Forum. 2006;19(1):27–31.
Glenton C, Santesso N, Rosenbaum S, et al. Presenting the results of Cochrane systematic reviews to a consumer audience: a qualitative study. Med Dec Making. 2010;30(5):566–77.
Rosenbaum SE, Glenton C, Cracknell J. User experiences of evidence-based online resources for health professionals: user testing of The Cochrane library. BMC Med Inform Decis Mak. 2008;8:34.
Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607–19.
Rosenbaum SE, Glenton C, Wiysonge CS, et al. Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ. 2011;89(1):54–61.
Downe S, Finlayson KW, Lawrie TA, et al. Qualitative evidence synthesis (QES) for guidelines: Paper 1 - using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):76.
Lewin S, Glenton C, Lawrie TA, et al. Qualitative evidence synthesis (QES) for guidelines: Paper 2 - using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Res Policy Syst. 2019;17(1):75.
Glenton C, Lewin S, Lawrie TA et al. Qualitative Evidence Synthesis (QES) for Guidelines: Paper 3 – Using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Res Policy Sys. 2019;17:74. https://doi.org/10.1186/s12961-019-0450-1
Morville P. 2004 User Experience Design [honeycomb model]. http://www.semanticstudios.com/publications/semantics/000029.php .
Rosenbaum S. 2010 Improving the user experience of evidence: A design approach to evidence-informed health care. PhD thesis. Oslo, Norway: The Oslo School of Architecture and Design.
Download references
We gratefully acknowledge the representatives from the National Ministries of Health in Nigeria, Malawi and South Africa for their support and partnership. We would also like to thank the appointed Steering Committees who have been providing input for the research project and guiding the prioritization of topics. We would also like to thank Joy Oliver and Michelle Galloway for their contribution an support of the project.
The GELA project is funded by EDCTP2 programme supported by the European Union (grant number RIA2020S-3303-GELA). The funding will cover all the activities for this Monitoring and Evaluation work package, including costs for personnel and publication of papers.
Authors and affiliations.
Health Systems Research Unit, South African Medical Research Council, Cape Town, South Africa
Tamara Kredo, Denny Mabetha, Bey-Marrié Schmidt & Simon Lewin
Division of Epidemiology and Biostatistics, Department of Global Health, Stellenbosch University, Cape Town, South Africa
Tamara Kredo, Anke Rohwer, Michael McCaul, Idriss Ibrahim Kallon, Taryn Young & Sara Cooper
Division of Clinical Pharmacology, Department of Medicine, Stellenbosch University, Cape Town, South Africa
Tamara Kredo
School of Public Health and Family Medicine, University of Cape Town, Cape Town, South Africa
Tamara Kredo & Sara Cooper
University of Calabar Teaching Hospital, Calabar, Nigeria
Emmanuel Effa
Kamuzu University of Health Science, Lilongwe, Malawi
Nyanyiwe Mbeye
School of Public Health, University of the Western Cape, Cape Town, South Africa
Bey-Marrié Schmidt
Western Norway University of Applied Sciences, Bergen, Norway
Susan Munabi-Babigumira & Claire Glenton
Norwegian University of Science and Technology, Trondheim, Norway
Simon Lewin
MAGIC Evidence Ecosystem Foundation, Oslo, Norway
Per Olav Vandvik
Department of Health Economics and Health Management, Institute for Health and Society, University of Oslo, Oslo, Norway
Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa
Sara Cooper
You can also search for this author in PubMed Google Scholar
T.K., S.C., T.Y., S.L., C.G. and P.O.V. conceptualized the protocol idea and S.C. drafted the protocol with input from TK, D.M., A.R., B.M., M.M., I.I., C.G., T.Y., S.L. and P.O.V.; all authors approved the final version for submission for publication.
Correspondence to Tamara Kredo .
Ethics approval and consent to participate.
Ethics approval has been obtained in each partner country (South Africa, Malawi and Nigeria) from the respective Health Research Ethics Committees or Institutional Review Boards. Information about the project will be provided to, and consent obtained from, all participants completing the online surveys and interviews and all participants taking part in the meetings. The consent forms will make explicit the voluntary nature of participation, that there will be no negative consequences if they decide not to participate and in the case of the interviews and meetings observations will ask explicitly for permission for the interview or meeting to be recorded. The online surveys will ask participants to provide the last six numbers of their ID as a unique identifier to track their capacity development needs and progress throughout the project. To help protect their confidentiality, the information they provide will be private, deidentified and no names will be used. In addition, all cookies and IP address collectors will be disabled to ensure confidentiality. All interview and meeting recordings on the digital recorders will be destroyed following safe storage and transcription, and any identifying information will be redacted from all transcripts. All study data, including recordings, will be stored electronically using password-controlled software only accessible to key project members and project analysts. Reports of study findings will not identify individual participants. We do not anticipate any specific harms or serious risks to participants. However, there is a risk of breaches of confidentiality for participants who take part in guideline panel and steering group committee project meetings. At the start of all meetings, participants will be introduced to each other. The member names of these groups will not be anonymous as they will play an ongoing role in the GELA project. At the start of each meeting, we will discuss the importance of maintaining confidentiality by everyone. As part of guideline development processes, all guideline members will need to declare conflicts of interests and sign a confidentiality agreement. We will explain, however, that while the researchers undertake to maintain confidentiality, we cannot guarantee that other meeting participants will, and there is, thus, a risk of breaches of confidentiality. We will ensure participants are aware of this risk. Participants may also feel anxiety or distress expressing negative views about project activities. Where there is this potential and where participants identify concerns, we will reassure participants of the steps that will be taken to ensure confidentiality.
Competing interests.
All authors declared no competing interests.
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .
Reprints and permissions
Cite this article.
Kredo, T., Effa, E., Mbeye, N. et al. Evaluating the impact of the global evidence, local adaptation (GELA) project for enhancing evidence-informed guideline recommendations for newborn and young child health in three African countries: a mixed-methods protocol. Health Res Policy Sys 22 , 114 (2024). https://doi.org/10.1186/s12961-024-01189-5
Download citation
Received : 02 June 2023
Accepted : 20 July 2024
Published : 19 August 2024
DOI : https://doi.org/10.1186/s12961-024-01189-5
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1478-4505
VIEW LARGER COVER
Rural highways account for a significant portion of the National Highway System and serve many vital mobility purposes. The Highway Capacity Manual , the standard reference for traffic analysis methodologies, contains analysis methodologies for all of the individual segments or intersections that may constitute a rural highway; however, it does not include a methodology or guidelines for connecting the individual roadway segments into a connected, cohesive, facility-level analysis.
NCHRP Research Report 1102: Reliability and Quality of Service Evaluation Methods for Rural Highways: A Guide , from TRB's National Cooperative Highway Research Program, presents a guide for traffic analysis of rural highways that connects the individual highway segments into a connected, cohesive, facility-level analysis.
Supplemental to the report is NCHRP Web-Only Document 392: Developing a Guide for Rural Highways: Reliability and Quality of Service Evaluation Methods .
National Academies of Sciences, Engineering, and Medicine. 2024. Reliability and Quality of Service Evaluation Methods for Rural Highways: A Guide . Washington, DC: The National Academies Press. https://doi.org/10.17226/27895. Import this citation to: Bibtex EndNote Reference Manager
Chapters | skim | |
---|---|---|
i-x | ||
1-2 | ||
3-7 | ||
8-33 | ||
34-47 | ||
48-66 | ||
67-68 | ||
69-70 | ||
71-80 | ||
81-93 | ||
94-101 | ||
102-106 | ||
107-112 | ||
113-117 | ||
118-121 | ||
122-149 | ||
150-164 | ||
165-166 | ||
167-173 | ||
174-202 | ||
203-227 | ||
228-255 | ||
256-265 | ||
266-272 | ||
273-278 |
The Chapter Skim search tool presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter. You may select key terms to highlight them within pages of each chapter.
An uncorrected copy, or prepublication, is an uncorrected proof of the book. We publish prepublications to facilitate timely access to the committee's findings.
The final version of this book has not been published yet. You can pre-order a copy of the book and we will send it to you when it becomes available. We will not charge you for the book until it ships. Pricing for a pre-ordered book is estimated and subject to change. All backorders will be released at the final established price. As a courtesy, if the price increases by more than $3.00 we will notify you. If the price decreases, we will simply charge the lower price. Applicable discounts will be extended.
What is an ebook.
An ebook is one of two file formats that are intended to be used with e-reader devices and apps such as Amazon Kindle or Apple iBooks.
A PDF is a digital representation of the print book, so while it can be loaded into most e-reader programs, it doesn't allow for resizable text or advanced, interactive functionality. The eBook is optimized for e-reader devices and apps, which means that it offers a much better digital reading experience than a PDF, including resizable text and interactive features (when available).
eBook files are now available for a large number of reports on the NAP.edu website. If an eBook is available, you'll see the option to purchase it on the book page.
View more FAQ's about Ebooks
Internet Explorer is no longer supported by Microsoft. To browse the NIHR site please use a modern, secure browser like Google Chrome, Mozilla Firefox, or Microsoft Edge.
Published: 21 August 2024
The management of the Better Methods for Better Research (BMBR) Programme will be moving from UKRI Medical Research Council (MRC) to NIHR this summer. NIHR will also increase its financial contribution, enhancing opportunities for high quality research.
The BMBR Programme is a collaboration between MRC and NIHR. It aims to ensure that optimal research methods are used to advance biomedical, health and care research, policy and delivery.
BMBR has funded a variety of important methods research. This includes methods for adaptive clinical trials, which allowed multiple treatments for Covid-19 to be rapidly tested in the RECOVERY trial. BMBR has also improved how insights into treatments can be measured using real-world evidence.
The BMBR Programme will remain a close collaboration between MRC and NIHR, and the remit and approach will stay the same. The regular schedule for advertising funding opportunities will continue. Both MRC and NIHR will also continue to fund methods research through other schemes.
Researchers will now need to apply for BMBR funding through NIHR’s application system. While the process will be very similar, there may be some differences. In particular, NIHR has requirements for patient and public involvement and research inclusion . NIHR also remains keen to encourage applications led by early-career researchers.
In addition to taking over management, NIHR’s extra funding is increasing the overall budget of the BMBR Programme. In 2025, there will also be a complete review of the programme to reflect on its achievements to date. This will help to optimise future effectiveness.
Professor Danny McAuley, Scientific Director for NIHR Programmes, said: “This is an exciting next step for the BMBR Programme. It is a clear indication of NIHR’s commitment to funding research that underpins life-changing impact. We look forward to continuing to work with MRC to help researchers improve and enhance the methods they use.”
MRC Executive Chair, Professor Patrick Chinnery, said: “BMBR has supported excellence in methodology research since its inception in 2008 and MRC welcomes the matched increase in NIHR contribution to support the programme’s continuing impact. MRC remains committed to supporting methods research via BMBR and our other funding opportunities, and we are looking forward to working with NIHR in this new phase of the BMBR Programme”.
The next BMBR funding opportunity will open in September 2024. Full application guidance will be available.
If you have any queries before then, you can contact the programme at [email protected] .
NIHR awards £33.2m to inspire students into research
Ketamine-assisted therapy for alcohol disorder trial now recruiting
Half a million people sign up to Be Part of Research
PANORAMIC study learnings are key to pandemic preparedness
New guiding principles for community engagement and involvement in global health research
Run remote usability tests on any digital product to deep dive into your key user flows
Learn how users are behaving on your website in real time and uncover points of frustration
A tool for collaborative analysis of qualitative data and for building your research repository and database.
How-to articles, expert tips, and the latest news in user testing & user experience
Detailed explainers of Trymata’s features & plans, and UX research terms & topics
Conduct user testing, desktop usability video.
You’re on a business trip in Oakland, CA. You've been working late in downtown and now you're looking for a place nearby to grab a late dinner. You decided to check Zomato to try and find somewhere to eat. (Don't begin searching yet).
It was hard to find the bart station. The collections not being able to be sorted was a bit of a bummer
Feedback from the owners would be nice
The flow was good, lots of bright photos
I like that you can sort by what you are looking for and i like the idea of collections
You're going on a vacation to Italy next month, and you want to learn some basic Italian for getting around while there. You decided to try Duolingo.
I felt like there could have been a little more of an instructional component to the lesson.
It would be cool if there were some feature that could allow two learners studying the same language to take lessons together. I imagine that their screens would be synced and they could go through lessons together and chat along the way.
Overall, the app was very intuitive to use and visually appealing. I also liked the option to connect with others.
Overall, the app seemed very helpful and easy to use. I feel like it makes learning a new language fun and almost like a game. It would be nice, however, if it contained more of an instructional portion.
All accounts, tests, and data have been migrated to our new & improved system!
Use the same email and password to log in:
Legacy login: Our legacy system is still available in view-only mode, login here >
What’s the new system about? Read more about our transition & what it-->
A UX (user experience) audit is defined as a comprehensive evaluation process where a website, application, or product is analyzed to understand its usability, accessibility, and overall user experience.
The primary aim is to identify issues that hinder users from achieving their goals efficiently and effectively. A UX audit typically involves reviewing the design, structure, and functionality of the interface, alongside gathering and analyzing user feedback and behavior data. This process helps in pinpointing areas that require improvement and provides actionable insights to enhance user satisfaction and engagement.
During a UX audit, various methods and tools are employed to scrutinize the user experience and interface. These can include usability testing, UX testing, UI testing, user interviews etc.
The benefits of conducting a UX audit are numerous. Firstly, it helps in identifying and rectifying usability issues, leading to a more intuitive and user-friendly product.
Additionally, a UX audit can help in uncovering conversion bottlenecks, thereby improving key performance indicators (KPIs) such as conversion rates, engagement levels, and user retention. By addressing these issues, businesses can optimize their product to better meet user needs and expectations, ultimately driving business growth and success.
The key components of a UX audit encompass several aspects that collectively provide a holistic evaluation of the user experience. These components include heuristic evaluation, usability testing, user research, and analytics review.
The UX audit process involves several key steps that ensure a thorough and systematic evaluation of the user experience.
UX audits can be categorized based on their focus and methodologies. Here are four common types of UX audits, each with examples to illustrate their applications:
Conducting a UX audit effectively in 2024 involves adopting best practices that reflect current trends, technologies, and user expectations. Here are some key best practices:
Interested in learning more about the fields of product, research, and design? Search our articles here for helpful information spanning a wide range of topics!
A complete guide to usability testing methods for better ux, ux mapping methods and how to create effective maps, a guide to the system usability scale (sus) and its scores.
IMAGES
COMMENTS
Evaluation research also requires one to keep in mind the interests of the stakeholders. Evaluation research is a type of applied research, and so it is intended to have some real-world effect. Many methods like surveys and experiments can be used to do evaluation research. The process of evaluation research consisting of data analysis and ...
Evaluating Research refers to the process of assessing the quality, credibility, and relevance of a research study or project. This involves examining the methods, data, and results of the research in order to determine its validity, reliability, and usefulness. Evaluating research can be done by both experts and non-experts in the field, and ...
Evaluation Research Methodology. There are four major evaluation research methods, namely; output measurement, input measurement, impact assessment and service quality. Output measurement is a method employed in evaluative research that shows the results of an activity undertaking by an organization.
Sage Research Methods & Evaluation is at the forefront of research and scholarship, providing classic and cutting-edge books, video collections, reference materials, cases built on real research, key journals, and our online platform for the community Methodspace.. Download special issues, collections, and a selection of most read articles.Browse our journal portfolio.
Evaluation research examines whether interventions to change the world work, and if so, how and why. Qualitative inquiries serve diverse evaluation purposes. Purpose is the controlling force in determining evaluation use. Decisions about design, data collection, analysis, and reporting all flow from evaluation purpose.
Overall, evaluators believed research and evaluation intersect, whereas researchers believed evaluation is a subcomponent of research. Furthermore, evaluators perceived greater differences between evaluation and research than researchers did, particularly in characteristics relevant at the beginning (e.g., purpose, questions, audience) and end ...
This evaluation comprised 4 components: technical accessibility, user experience (UX), quality and learning design; 10 experts were involved in its design and validation. "The combination of qualitative studies through interviews with MOOC providers and learners and the quantitative information provided by the MOOC survey data has provided an ...
evaluation opportunities and challenges, for the sake of brevity our default position will be to refer to 'programme' evaluation unless there is a need to distinguish one from another. Defining evaluation according to method Another group of evaluation definitions identified by Mark et al. (2006) outline evalu - ation in terms of methods.
Evaluation is a set of research methods and associated methodologies with a distinctive purpose. They provide a means to judge actions and activities in terms of values, criteria and standards. At the same time evaluation is also a practice that seeks to enhance effectiveness in the public sphere and policy making. In order to improve as well ...
Evaluation is an essential aspect of research. It is ubiquitous and continuous over time for researchers. Its main goal is to ensure rigor and quality through objective assessment at all levels. It is the fundamental mechanism that regulates the highly critical and competitive research processes.
Evaluation research, also known as program evaluation, is a systematic analysis that evaluates whether a program or strategy is worth the effort, time, money, and resources spent to achieve a goal. Based on the project's objectives, the study may target different audiences such as: Employees. Stakeholders.
It is concluded that evaluation research should be a rigorous, systematic process that involves collecting data about organizations, processes, programs, services, and/or resources that enhance knowledge and decision making and lead to practical applications. Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes, as a specific ...
Approaches (on this site) refer to an integrated package of methods and processes. For example, ' Randomized Controlled Trials ' (RCTs) use a combination of the methods random sampling, control group and standardised indicators and measures. Evaluation approaches have often been developed to address specific evaluation questions or challenges.
Monitoring and Evaluation (M&E) methods encompass the tools, techniques, and processes used to assess the performance of projects, programs, or policies. These methods are essential in determining whether the objectives are being met, understanding the impact of interventions, and guiding decision-making for future improvements.
Building evidence is not one-size-fits all, and different questions require different methods and approaches. The Administration for Children & Families Common Framework for Research and Evaluation describes, in detail, six different types of research and evaluation approaches - foundational descriptive studies, exploratory descriptive studies, design and development studies, efficacy ...
A broadly accepted way of thinking about how evaluation and research are different comes from Michael Scriven, an evaluation expert and professor. He defines evaluation this way in his Evaluation Thesaurus: "Evaluation determines the merit, worth, or value of things.". He goes on to explain that "Social science research, by contrast, does ...
Introduction to Evaluation. Evaluation is a methodological area that is closely related to, but distinguishable from more traditional social research. Evaluation utilizes many of the same methodologies used in traditional social research, but because evaluation takes place within a political and organizational context, it requires group skills ...
Evaluative research is defined as a research method that assesses the design, implementation, and outcomes of programs, policies, or products. ... This means that the evaluation considers the unique characteristics, needs, and conditions of the environment and population being studied. Context-specific evaluations provide more relevant and ...
Evaluation has its roots in the social, behavioral, and statistical sciences, and it relies on their principles and methodologies of research, including experimental design, measurement, statistical tests, and direct observation. What distinguishes evaluation research from other social science is that its subjects are ongoing social action programs that are intended to produce individual or ...
Academic research focuses primarily on testing hypotheses; a key purpose of program evaluation is to improve practice. Research is generally thought of as requiring a controlled environment or control groups. In field settings directed at prevention and control of a public health problem, this is seldom realistic.
The fourth edition of Michael Quinn Patton's Qualitative Research & Evaluation Methods Integrating Theory and Practice, published by Sage Publications, analyses and provides clear guidance and advice for using a range of different qualitative methods for evaluation. Contents. Part 1. Framing Qualitative Inquiry: Theory Informs Practice, Practice Informs Theory
Characteristics of Mixed Methods Designs in Evaluation Research. Several typologies exist in mixed methods designs, including convergent, explanatory, exploratory, embedded, transformative, and multiphase designs ().These, along with other mixed method designs in evaluation research, can be categorized in terms of their structure, function, and process (1, 4, 63, 65, 73).
Basically, evaluation research is a research process where you measure the effectiveness and success of a particular program, policy, intervention, or project. This type of research lets you know if the goal of that product was met successfully and shows you any areas that need improvement. The data gathered from the evaluation research gives a ...
Research involving the collection of data in numerical form for quantitative analysis. The numerical data can be durations, scores, counts of incidents, ratings, or scales. Quantitative data can be collected in either controlled or naturalistic environments, in laboratories or field studies, from special populations or from samples of the ...
Welcome to the EMDD 600 Usability & Evaluation Research Methods research guide. Library Services. University Libraries. The main page for the Ball State University Libraries. Ask a Librarian. Contact a librarian by chat, phone, email, in-person or schedule a reference appointment.
Research Tip: Database Exploration. Start your search by doing keyword searches. Look at the subject headings of relevant records to determine the terminology used in the database for your topic. Use Limits to limit results to Scholarly/Peer Reviewed articles, by publication date, and more. <<
Overall approach. We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change (Table 2).The theoretical underpinning for the GELA project across all work packages is related to the three-layered behaviour change wheel comprising opportunity, capability and motivation [].The design, delivery and implementation of multi-stakeholder integrated ...
NCHRP Research Report 1102: Reliability and Quality of Service Evaluation Methods for Rural Highways: A Guide, from TRB's National Cooperative Highway Research Program, presents a guide for traffic analysis of rural highways that connects the individual highway segments into a connected, cohesive, facility-level analysis.
It aims to ensure that optimal research methods are used to advance biomedical, health and care research, policy and delivery. BMBR has funded a variety of important methods research. This includes methods for adaptive clinical trials, which allowed multiple treatments for Covid-19 to be rapidly tested in the RECOVERY trial. BMBR has also ...
Usability testing provides direct insight into how actual users experience the product, revealing pain points that might not be evident through expert evaluation alone. User Research: User research involves gathering qualitative and quantitative data directly from users through various methods such as surveys, interviews, and focus groups.