Biomedical Research

Abstract

Background: Robust monitoring and evaluation systems can strengthen results-based management, establish a culture of results’ measurement, institutionalize evidence-based decision making and improve accountability of results at all levels of service delivery.

Objective: The objective of this paper is to share the findings of an internal assessment we conducted on the capacity of the twelve components of the monitoring and evaluation system at the department of health in Vihiga county.

Methods: After a three days’ training in a workshop on institutionalizing of monitoring and evaluation, officers at the department reviewed key documents and held discussions amongst themselves using a standardized tool developed in excel. Once consensus was reached, the scored were entered appropriately per question per domain. The tool automatically aggregated the scores under status, quality and autonomy in terms of financial and technical support.

Results: On status, costed annual work plan and supervision and auditing scored the highest (10.00/10.00). on quality routine monitoring scored the highest (9.38/10.00). Performance on technical autonomy was average with the best components scoring (5.00/10.00). Human capacity scored highest on financial autonomy (10.00/10.00).

Conclusion: Overall, majority of the components had relatively higher capacity on status and quality compared to the ratings on technical and financial autonomy. There is need for the county and partners to sustain gains made on the various components as well as strengthen those that are weak.

Keywords

Monitoring, Evaluation, M&E system, Components of M&E system, Vihiga county, Monitoring health services, Capacity of M&E system.

Introduction

Vihiga is a county in the western region of Kenya. It comprises of five sub-counties and twenty-five wards, covering 531KM2 in land area. It is projected that the county will have a population of about 606,160 in the year 2022, with a density of 1,075 per square kilometer [1]. With such a high population density, there is need to prioritize the health of the people. The county’s department of health aims at providing equitable, affordable and quality health care of the highest standard to all its residents as stipulated in the bill of rights in the Constitution of Kenya 2010 [2]. The goal of the department is to achieve a healthy and nationally competitive county and its people. The county’s integrated development plan 2018-2022 espouses the need for focused and well-coordinated monitoring and evaluation strategies for its successful implementation. Monitoring and evaluation (M&E) is critical in fostering sound governance and accountability [3]. In health, continuous monitoring and evaluation helps in determining the performance of service delivery.

Robust monitoring and evaluation systems can strengthen results-based management, establish a culture of results’ measurement, institutionalize evidence-based decision making and improve accountability of results at all levels of service delivery [4]. M&E systems provide data on programs and priorities to inform county and national decisions while guiding the delivery of high-quality services by reliably evaluating the successes or failures. M&E systems are critical tools for looking into the past and the future of programs. They provide the basis for strategic planning, learning, and sound management. Findings from M&E activities are meant to influence decision making, including decisions to improve, reorient, or discontinue the evaluated intervention or policy. It is important for organizations to continuously design, assess, review and redesign M&E systems to ensure they are responsive to the urgent and long-term needs [5].

We discuss the findings of the assessment we conducted on the capacity of the twelve components of the M&E system. This was an internal assessment to understand the status, quality, financial and technical autonomy of the M&E system at the department of health in Vihiga County, Kenya. We conceptualized status as being the existence of a certain element such as the M&E unit; quality as the degree of quality that a specific task or deliverable meets according to established norms or guidelines; technical autonomy as the internal capacity to accomplish technical tasks in the twelve components; and financial autonomy as the capacity to support its undertakings on key tasks in the twelve components financially.

Methods

In July 2021 officers from the department of health were trained in a three-day workshop by experts from the national government M&E unit and Tupime county project. The training focused on institutionalizing M&E in the department of health. The outcome of the training was to assess the M&E system within the department of health to tease out the strengths, weaknesses, and opportunities available for a robust M&E system. The team used a validated Monitoring and Evaluation Capacity Assessment Toolkit (MECAT) based on the 12 components of an M&E system [6-8]. The tool was comprised of structured questions on each component. The questions focused on the following four dimensions: status, quality, technical assistance, and financial system for each of the twelve functional areas. The 12 functional areas of the M&E capacity assessment were scored using a series of statements based on three response scales a) a 3-point scale (Yes mostly, Yes partly, Not at all); b) a 4-point scale (Strongly Agree, Agree, Disagree, Strongly Disagree), (Less than 1 year, 1-2 years, 2-3 years, Greater than 3 years); c) a 5-point scale (Weekly, Monthly, Quarterly, Bi-annually, Annually). The tool was designed on an Excel platform for real time aggregation and color coding. Based on discussions and review of the relevant materials, the appropriate responses were selected from a drop-down menu in the “answer” column and would appear as follows; “Yes-mostly” in green, “Yes Partly” in orange, and “Not at all” in red. The visualization in color helped with summarizing the assessment results and with prioritizing actions. The team referred to key strategic documents such as the county health sector strategic plan, annual workplans, and the M&E plan in order to reach consensus on particular area. A summary dashboard of the results would automatically be generated showing the distribution of the overall scores and for each of the 12 functional areas by dimension (Status, Quality and technical and financial Autonomy). The tool and assessment were based on an aggregated scale of 1.00 to 10.00, where 1 was very weak while 10 was very strong.

Results

In terms of status, organizational capacity and evaluations and research scored the lowest tying at a score of 3.33. The other components scored as follows: county and sub- county databases and surveys and surveillance (5.00); county M&E plan (6.00); data demand and use (6.67); routine monitoring (7.50); partnerships and governance (8.57); human capacity (9.00) while costed annual work plan and supervision and auditing were the best (10.00). The department has an integrated annual work plan incorporating M&E activities. The M&E activities are derived from the 10-year M&E work plan 2018-2028 at the department. Figure 1 presents the performance of the 12 components as regards their status.

Figure 1: Capacity of the M&E components on status.

On quality, evaluations and research scored the least (2.22). The other components scored as follows surveys and surveillance (4.17); County M&E plan (4.33); organizational and advocacy, communication, and cultural behavior (5.00); annual costed work plan (5.56); partnerships and governance (5.94); supervision and auditing (6.25); data demand and use (6.67); county and sub-county databases (7.50); human capacity (8.15); routine monitoring (9.38). Figure 2 presents performance of all the twelve in terms of quality. Routine monitoring is based on data derived from the stable Kenya health information system. In addition, the department through its programmes organizes and conducts regular supervisions to ensure that there is quality of services in the county.

Figure 2: Capacity of M&E components on quality.

On technical autonomy, annual costed M&E work plan, organizational leadership and evaluation and research scored 2.50. County M&E plan scored 3.33 while all the other components tied at 5.00. Figure 3 presents performance of the components in terms of technical autonomy. The department still depends on partners to build the capacity of the M&E system. The establishment of the division of health information, M&E and research is critical in achieving the desired goals in building the technical capacity of the staff.

Figure 3: Capacity of the M&E components on technical autonomy.

Regarding financial autonomy, advocacy, communication and cultural behavior scored the lowest (0.00), partnerships and governance (2.00), followed by evaluations and research (2.50), county M&E work plan (3.33), the other components scored 5.00 apart from human capacity for M&E which scored the highest at 10.00. Figure 4 presents the capacity of the 12 components in terms of financial autonomy. The department has staff, majority of whom are hired on permanent and pensionable basis.

Figure 4: Capacity of M&E components on financial autonomy.

The assessment of the organizational capacity for M&E focused on the mission statement of or stated objectives, values and ethics statements, structure of the M&E unit or division or directorate, effective leadership, resource management, adequate number of skilled M&E staff and defined career path in M&E. The assessment revealed that there were objectives contained in the M&E plan (2018-2028). The activities of the M&E unit are derived from the objectives and strategies of the County Strategic plans and annual working plan. These activities are then monitored and evaluated by the M&E team periodically. The activities are aligned to the mission and objectives of the department of health. However, it was noted that majority of staff are not able to state the mission statement and objectives. In terms of autonomy, the M&E strategic plan was developed with support from external technical experts. At the time of the assessment, the unit did not have direct financial support from any partner. The M&E system scored 58% on organizational capacity.

Assessment of the human resources capacity focused on staff M&E skills and competences, costed Human Capacity Building Plan, costed Human Capacity Building Plan for organizational development, costed Human Capacity Building Plan for data demand and information use, validated M&E training curriculum, and regular M&E unit meetings. The assessment showed that there were only three staff available on full time basis at county level for M&E. Two of the staff were fully focused on health information and records. With over twenty- four sub-programs being supported, the three staff were overwhelmed. The M&E system scored 88% on human capacity. Generally, it was noted that the county had enough staff with relevant skills for M&E. There was need to pool the staff together so as to work as a team on M&E.

The capacity of partnerships and governance was assessed by looking at M&E performance, standard operating procedures that define roles and responsibilities related to M&E functions and activities, Technical Working Groups (TWGs), Commitment from stakeholders in the Division’s M&E activities and performance, an updated inventory of stakeholders for the division M&E, clear mechanisms such as feedback reports, and newsletters to communicate M&E activities and decisions. It was noted that the division had the M&E plan as the main strategic document driving good M&E performance. This was an important document to help correct weak or incorrect M&E performance. The plan had not yet been reviewed. The M&E system scored 77% on partnerships and governance.

To assess the capacity of the M&E plan, the assessment focused on the ability of the division to prepare accurate project work plans, budgets and schedules, clear guidelines with dates specifying when information or reports need to be both received and distributed by the division, availability of a county multi-sectoral M&E plan. The division, had a work plan which had a monitoring and evaluation framework with expected results and activities. The M&E plan had incorporated finance and technical resources. The development process was participatory, with clear linkage to the county strategic plan. It adheres to international and national technical standards. However, it had not been reviewed. The system scored 73% on county M&E plan.

The department develops annual work plans every year. The workplans are costed and contains activities, responsible implementers, timeframe, activity costs and identified funding. There was explicit linkage between the work plan and the county government Medium Term Expenditure Framework (MTEF) budgets, resources (human, physical, financial) committed to implement the M&E work plan. The workplan is developed in a participatory manner and it is usually endorsed by all relevant stakeholders. The M&E system scored 93% on annual costed workplan.

The M&E system scored 100% on advocacy, communication and cultural behavior. There were M&E champions. The department had a communication officer in place. In addition, each program within the department had specific communication strategies. M&E issues, strategies and products were being included in the county policies and Integrated Strategic Plans (ISP). M&E materials were being shared with relevant stakeholders.

In determining the capacity of routine monitoring, the assessment looked at availability of essential tools and equipment for data management (which includes collection, transfer, storage, analysis), guidelines to document procedures for collecting, recording, collating, and reporting program monitoring data from health information system. Overall, the M&E system scored 81% on routine monitoring. All reporting health facilities were using standardized data collection forms. The tools capture essential indicators for routine performance monitoring. The data is entered into the Kenya health information system. The unit through its officers (health records and information officers) at county, subcounty and facility levels identify gaps in the existing tools that need to be updated and report to the national M&E team. The changes are then implemented on the Kenya health information system. The data is abstracted and used to inform decisions in various programs within the department.

The system scored 75% on surveys and surveillance. Questions based on survey and surveillance inventory, protocols for surveys and surveillance and a functioning surveillance system were used. Protocols for all surveys and surveillance were found to be in line with national and international standards. Specified schedules for data collection were linked to stakeholders’ needs, including identification of resources for implementation.

On county and subnational databases, the M&E system scored 60%. Questions based on databases for electronically capturing and storing data generated for and by the national M&E systems and subcounty level databases linked to the county system were used. The main database being used in the county, the Kenya health information system, was designed to respond to the decision-making and reporting needs of different stakeholders and programs within the county. There were no linkages between other small systems providing relevant program-based data.

The system scored 65% on supervision and auditing. The questions that were used explored guidelines and tools for supportive supervision (M&E) and data quality audit. There were no guidelines for integrated supportive supervision in the county. Currently, supportive supervision was mainly facility based with little effort on community-based supervision. The department relied mostly on program based routine supervision visits, including data assessments and feedback from and to local staff. There were periodic data quality audits per program. Supervision and audit reports should be disseminated, and action points taken into consideration.

The system scored 80% on evaluation and research. Mainly, evaluation was assessed in consideration of the availability of data on the Kenya health information system. Further, every year the department releases an annual performance report and the medium-term expenditure framework report. These reports provide data on performance against set targets. There are usually county fora where the findings on the performance as well as research findings are shared. The fora are usually attended by most of the key stakeholders. Normally the deliberations from the fora identify clear action plans for the department both in terms of policy and practice. The fora are funded by the county government and partners.

The system scored 67% on data demand and use. Questions were coined around availability of a county data use plan, dissemination of information products and data analysis and presentation guidelines. There was a county data use plan embedded in the county M&E plan (2018-2028). The data use plan was conforming to best practices on collecting, recording, collating and reporting.

Discussion

Overall organizational capacity scored relatively low. At the time of assessment, the division only had few staff skewed towards health information without substantive officers aligned to M&E and research development. However, the county had commenced the process of improving the organizational capacity by creating the office of head of division, M&E and research development. Having a clear vision of the division would provide the strategic direct aimed at ensuring that all stakeholders work towards the same goals [9]. Whereas human resources domain scored relatively higher, it was important that individuals with the prerequisite skills be consolidated by creation of the M&E unit or a rigorous technical working group [10]. Some of the critical skills required for establishment of a robust M&E unit include qualifications and experience in health informatics, epidemiology, economics and social sciences. The department had a robust partnerships office. However, there was need to have a standard operating procedure that define partner roles and responsibilities related to M&E functions and activities within the department of health. Recent studies show that efficient stakeholder engagement in M&E results in improved performance of the programs [11].

Development of county level and subcounty level databases is also needed. The department fully depends on the Kenya Health Information System. There are however some programs that have their own systems and databases for management of data. There is need for a clear roadmap on use of health information systems and databases. This would ensure that there is consistency and that all data feeds into one county system and one national level system. The department needs to streamline its support supervision and auditing activities. Studies show that if well structured, supportive supervision improves service delivery and program outcomes. Effective approaches during supportive supervision include building the capacity among the cadres at facility and community levels through mentorship and using technology for real time data analysis and decision making [12,13]. The department needs to embed research into programs to turn the tide towards evidence-based decision making. Research produces data and information that improves knowledge and informs policy and programs.

Conclusion

The department of health, M&E system in Vihiga county has gained tremendous growth considering that devolution of health only happened in 2013 in Kenya. Overall, majority of the components had relatively higher capacity on status and quality compared to the ratings on technical and financial autonomy. There is need for the county and partners to sustain gains made on the various components as well as strengthen those that are weak.

Recommendations

1. Firstly, there is need to urgently re-vamp the M&E unit at the department. This would require an assessment of the skills and professional competencies among key staff. These would then be followed by training needs assessment, identification of opportunities for M&E capacity development.

2. Secondly, implementation and mid-term review of the already available 10-year (2018-2018) M&E plan is critical. The current funding mechanisms of M&E activities where individual programs conduct M&E activities though suffices, there is for change for long term strategy such as creating a pool of funds where then the M&E can be managed centrally and in an integrated and holistic manner.

3. Thirdly, it is important to digitize data collection process. This will go a long way improving data management and reducing the costs of printing hard copy tools.

4. Fourthly, there is need to build capacity for research, evaluations, and surveillance. This will help in mobilization of funds to fund M&E activities in a sustainable manner.

Acknowledgement

We acknowledge the county government of Vihiga through the Governor, the CECM and chief officer for the support provided during the Tupime county project implementation phase and drafting of this paper. We are indebted to the United States Agency for International for funding the project and the Palladium group for implementation of the project.

Competing interests

All the authors declare no competing interests in writing and publishing of this article.

Author contributions

All authors met the criteria for authorship as outlined in the authorship policy and author contribution statement policies as follows:

Anyiendah Mary Susan and Collins Mukanya Mudogo participated in the assessment of the M&E system, conceptualized the paper, wrote the paper and therefore are equal first authors. Polycarp Opiyo, Wafula Cheng’oli and Eric Ong’ute participated in the assessment of the M&E system. Professor Mwanje Justus and Dr. Wilber Otichillo mentored and guided the authors in the entire process of writing the paper. All the authors reviewed the paper and approved it for publication.

Funding

Tupime county project with funding from USAID facilitated the training of officers on using the MECAT.

Data availability

All data and information on the assessment is available on request with permission from the chief officer department of health services.

Disclaimer

The views and opinions presented in this article are the authors’ views and not the funding institution’s views.

Ethical statement

This article is not based on primary data collected from individuals. There was therefore no need for consent. All the officers who participated in the assessment participated in writing the article and are listed as authors.

References

1. County Government of Vihiga. Annual development plan. 2021.

2. Comparative Constitutions Project. Kenya?s Constitution of 2010.  2019; 1-172.

3. County Government of Vihiga. County Government of Vihiga Second County Integrated Development (2018-2022). 2017.

4. Nash D. Strategies for more effective monitoring and evaluation systems in hiv programmatic scale-up in resource-limited settings: Implications for health systems strengthening. J Acquir Immune Defic Syndr 2009; 52: 58-62.

   [Crossref] [Google Scholar] [PubMed]

5. Boerma T. Monitoring and Evaluation Systems Strengthening Tool. Evaluation 2007; 1-9.

   [Google Scholar]

6. Measure Evaluation. Monitoring and Evaluation Capacity Assessment Toolkit User Guide. Univ North Carolina 2017; 81.

7. USAID, PEPFAR, MEASURE Evaluation. Technical Brief?: Measurement of M & E System Strengthening. 2014.

8. Palladium LLC, Tupime Kaunti. County measurements, learning and accountability program: tupime kaunti. 2019.

9. Ministry of Health Republic of Bangladesh. Monitoring & Evaluation Strategy and Action Plan. 2014; 1-4.

10. MacQueen K, Harlan SV, Slevin KW, Hannah S, Bass E, Moffett J. Design a Monitoring and Evaluation System. Stakehold Engagem Toolkit HIV Prev Trials 2012; 25-140.

11. Ahuja R, Anil S, Arif M. Prioritizing BIM Capabilities of an Organization: An Interpretive Structural Modeling Analysis. Procedia Eng 2017; 196: 2-10.

    [Crossref] [Google Scholar]

12. Tegegne S. The role of supportive supervision using mobile technology in monitoring and guiding program performance: A case study in Nigeria, 2015-2016. BMC Public Health 2018; 4.

    [Crossref] [Google Scholar]

13. Djibuti M, Gotsadze G, Zoide A, Mataradze G. The role of supportive supervision on immunization program outcome - A randomized field trial from Georgia. BMC Int Health Hum Rights 2009; 1-12.

    [Crossref] [Google Scholar] [PubMed]