Terms of Reference for CHASP Mid-term Evaluation – Mogadishu, Somalia

Terms of Reference for CHASP Mid-term Evaluation – Mogadishu, Somalia

• BACKGROUND

The Community Health and Social Accountability Program (CHASP) Phase II, continued to build on the gains made in Phase I (2017-2020) by providing quality gender-sensitive and disability inclusive reproductive, maternal, newborn and child health (RMNCH) services, which are accessible, acceptable, affordable and equitable underpinned by the national framework of essential package for health services (EPHS). The program is funded by the Swedish International Development Cooperation Agency (Sida) and Swiss Agency for Development and Cooperation (SDC) and commenced on 1st May 2021 and will end on 30th April 2024. This report covers the period from 1st May 2021 to 31st October 2022. The overall objective of the project is to ‘increase utilization of quality Reproductive, Maternal, New-born, Child Health (RMNCH) services which are accessible, acceptable, affordable and equitable through provision of Essential Package of Health Services (EPHS).

The program delivers its objective through its three main outcomes:

• Evidence-base of successful innovative solutions generated adequately scaled-up.
• Essential Package of Health Services (EPHS), including Nutrition, reproductive health, and SGBV case management delivered to women and children.
• Health Governance and Management structures enhanced for sustainability.

The program is implemented in the three federal member states of Puntland, Jubbaland and Galmudug, covering 13 districts (up from 11 in phase one) and 76 health facilities (up from 72 in phase 1). The catchment population for the program is estimated at 644,043 people, out of which 143,252 are women of childbearing age and 128,808 children under the age of five. Learning from Phase I, and the end line evaluation, the current phase of the program has been designed with full recognition of prevailing gender norms and deep-rooted gender inequalities. Norms limit women and girls including persons with disabilities from accessing basic services such as health and nutrition and expose them to a higher risk of Sexual and Gender Based Violence (SGBV). To address these challenges, the program has incorporated a strong gender and child protection component and is working to remove the barriers by collaborating closely with communities, government authorities and other implementing partners in the country working in country such UNICEF, WFP, UNFPA, ICRC, IMC, locational organization such as HRDO(horsed Relieve and development organization), Himilo organization for development and Hirdo. Some of the key activities during year 1 to achieve this change has been the set-up of community structures such as child welfare committees, community sensitization on harmful gender norms and staff capacity building on gender and child protection in all the program regions.

1. Overall Aim of the assignment

The overall goal of this MTR is to assess and document how the program has contributed to strengthen the decentralized health system and its sub-components and to identify areas for improvement in Jubaland, Galmudug and Puntland states of Somalia.

2.1 Specific Objectives

As half of the project life time elapsed, this midterm evaluation will aim:

• To assess how the planned outcomes and impacts of the program have been realized, assess both intended and unintended consequences, and identify success and hindering factors (both internal/external factors) that positively or negatively influenced program implementation (Including effects of COVID19, droughts.)
• To conduct theory of change critical analysis to assess the chain of causality, how the results were produced, and identify if there were synergism effects of the combined interventions and integrated program within the CHASP program itself and other complementary projects in the project target locations.
• To evaluate to what areas the program innovations strengthened existing health systems with specific focus on health systems building blocks
• Assessment of how the project implementation was adapted across the three member states ( Jubaland, Galmudug and Puntland) in terms of project governance and changing context including droughts, conflicts, and other shocks

• To document recommendations, lessons learned, success stories, case studies; formulate policy implications based on the recommendation and results from the review to inform policy making, health systems reform, future program development adaptations, and determine the direction and areas to prioritize by health programmers in Somalia and other similar humanitarian and fragile contexts.
• Examine social inclusion in representation (gender, age, disability, location/migration status, clan minorities) in social accountability mechanisms (CHC, community outreach etc); and assess quality of care
• examine how the project upholds to the principle of do no harm, centrality of protection and conflict sensitivity

1. METHODOLOGY OF THE REVIEW

This review will use mixed methods, both quantitative and qualitative approaches to gather data. Quantitative data will be from health service catchment population level survey, triangulated with project indicator performance, HMIS and nutrition data from the MoH. While for qualitative data, individual and group in-depth interviews will be conducted with relevant stakeholders including Ministry of Health directors at federal level, three-member states, beneficiaries, district health management board member, donors, SCI field staff and other authorities in the project area. Strategies to be used to collect relevant information include:

• Conduct Interviews with a representative range of samples of beneficiaries to capture information related to the relevance and appropriateness of the program innovation strategies utilized. (this includes children, adolescent girls, IDP women and women with disabilities)
• Conduct In-depth interviews MoH directors and SCI field staff and other key stakeholders including key district level authorities and community leaders.
• Document success stories, case Studies, lessons learned, provide recommendation and policy implications/briefs.

An in-depth approach to data analysis is required. The statistical analysis of the quantitative component is expected to identify factors influencing program outcomes, thus establishing relationships between different variables and the outcomes of interest, as well as validating or challenging assumptions about the status of access and children affected by the crisis. It is expected that qualitative data will be analyzed using emerging codes, thus allowing for the identification of unanticipated factors/patterns. Extensive triangulation of qualitative and quantitative data will be required for validation of results and in-depth understanding of the effects observed. Findings should be disaggregated at multiple levels, including but not limited to sex, disability, location, rural, urban, type of school, IDP/non-IDP condition, and where relevant, most prevalent community livelihood (such as pastoralism, agro-pastoralism, agriculture).

1. Output and Midterm Evaluation Criteria

In assessing project interventions, the midterm evaluation must follow The Organization for Economic Co-operation and Development (OECD) six evaluation criteria: relevance, coherence, effectiveness, efficiency, impact and sustainability plus health system resilience .

• Was the project design relevant to identified needs?
• to what extent was the project design adaptive to evolving needs and changing project implementation environment?
• Is there compelling evidence that points to the need for adjustment of the theory of change and the underlying assumptions that informed the project design?
• Are the financial resources, human resources and other inputs being used efficiently to achieve project progress?
• What specific measures can be taken to enhance value for money in delivering project outputs?

Coherence

• To what extent has there been cooperation with other organization/actors working on the same thematic components (in term of complementarity, harmonization and co-ordination with others. Including other SCI projects)
• To what extent are the objectives aligned to the national priorities and strategies?
• To what extent was the project design and implementation aligned to community needs?

Effectiveness:

• To what extent are the planned project outputs delivered, and how are they contributing to the attainment of project objectives within the required timeframe?
• What has worked/not worked well in terms of activity implementation and the processes/systems deployed to facilitate the same?

Impact:

• What impact on refugee-host relations could be attributed to the activities under each objective?
• Have key project results and lessons learnt been incorporated into advocacy and training work?

Sustainability:

• What has the project done to ensure sustainability of the health systems supported/developed in the first half of the project?
• What needs to be done to strengthen the sustainability of the project deliverables in the second half of project implementation

1. Review team

A hired consultant will technically lead this program review, Program Director will oversee overall process of this exercise and provide support throughout the course of the assignment. Furthermore, SCI field and project team will provide necessary logics and support at field level.

1. Timeframe

The projected timeline of this review is about 4 weeks including; field work and drafting and submitting the final report after feedback is incorporated and validation of the report.

1. Data Sources and Data Collection Methods / Tools

All primary data collected during the study must facilitate disaggregation by gender, age, and disability, location or remoteness, vulnerability status. Save the Children will provide guidance on tools and classification schemes for this minimum dataset.

Quantitative methods using structured questionnaires will be used to establish quantitative benchmarks in the Phase 1 log frame. Focus Group Discussions, Individual questioning with a scientific representation of all the stakeholders, will be used to collect detailed qualitative information. It is expected that the consultant will come up with innovative ways of combining the various research tools and methodologies to get the right and accurate information.

During the MTR, the consultant will interview all the stakeholders including targeted beneficiary communities, students, Ministry of health of federal/member state authorities, Community Health Committees, Facility in charge and staff of implementing, men/women groups, religious leaders, opinion leaders, women leaders.

A range of project documentation will be made available to the study team that provides information about the design, implementation and operation of the Program. Documents include: project progress report, Log frame and other previous evaluation reports.

The study team is required to adhere to the Save the Children Child Safeguarding; Protection from Sexual Exploitation and Abuse; Anti-Harassment, Intimidation and Bullying; and Data Protection and Privacy policies throughout all project activities.

Data collection the instruments include: semi-structured interview questionnaires; one- on-one personalized

in-depth interview questionnaires for key informants; focus group discussions (FGDs) with discussion guidelines; institutional fact establishment sheets. The questionnaires will be administered to the respondents, while Institutional fact establishment sheets will be used to collect specific quantitative and qualitative facts at the sampled institutions, including physical and non-physical facilities, enrolment, teacher establishment, etc.

1. Ethical clearance and considerations

Ethical clearance letter will be obtained from the MoH ethical committee, in addition, objectives of the review will be communicated to all participants, and then verbal or signed informed consent will be obtained from participants prior to the start of interviewing. The review team will further be required to follow the SCI ‘Child Safe-guarding and ‘Do No Harm’ policies throughout the period of the review.

1. Work Plan

The detailed work plan for the midterm evaluation will be finalized by January 30th, 2023 in consultation with the program team after a consultant is identified to undertake this evaluation. The roles and responsibilities to execute the process will be determined based on consultation with SC-Sweden/Switzerland and Save the Children Somalia Country Office.

1. Scope of review

The scope of this assignment will be based on assessing outcomes, impacts, innovative interventions, objectives and key deliverables as stated in this Term of Reference.

• Key deliverables

• A detailed proposal outlining the methodology and instruments of the review, including review guiding questions – Inception Report/Proposal.
• Clear plan of the key roles and responsibilities of the proposed team, the plan should contain short background of the person, how his/her experience fits well with proposed activities, time frame for each activity, etc.
• A debriefing presentation for Save the Children International, to talk over the proposed approaches and methods in conducting the review and combine the feedback from the team.
• Shared the final detailed proposal with detailed implementation plan
• Shared the draft review report with SC team for inputs
• Present the results from the review to SC and MoH and other stakeholders
• Submit the final draft of the report
• Draft evidence dissemination plan
• Write up the Manuscript for publication
• Outline of the report

The proposed report format will encompass the following headings and sub headings:

1. Title Page
2. Table of Contents
3. Executive Summary
4. Preface the Report
5. Introduction and Background description
6. Scope and purpose of review
7. Methodology (detailed)
8. Results and Findings (detailed)
9. Discussions
10. Conclusion and recommendations
11. Case studies, success stories, lessons learned, policy implications
12. List of references
13. Annexes

• Obligations of the parties

1. The Consultant

The lead consultant will assume overall responsibility for the deliverables.

1. The Client

• SCI will support all stages of the evaluation process, including providing relevant documentation, assisting in the organization of data collection (giving contact details, ensuring the availability of interviewees and relevant data, providing feedback on drafts of all agreed outputs, including methodology).

The SCI MEAL advisor and research and evaluation manager (child survival) will guide the evaluation, and they will:

• Provide input into the proposed methodology and tools
• Sign off final deliverables (inception report, final report)
• Financial Proposal

Save the Children seeks value for money in its work. This does not necessarily mean “lowest cost”, but quality of the service and reasonableness of the proposed costs. Proposals shall include personnel allocation (role / number of days / daily rates / taxes), as well as any other applicable costs. Schedule of Payment.

The following payments will be made to the consultant using and agreed mode of payment

• Upon approval of inception report and tools: [30%]

• Upon submission of First Draft study Report: [30%]

• Upon approval of final study report: [40%]