A monitoring and evaluation

platform for nonprofits:

DHIS2 Quick Start

Gillian Kerr, Ph.D., C.Psych.

Martha McGuire, M.S.W., CE
October 11, 2015 www.logicaloutcomes.net
http://dx.doi.org/10.5281/zenodo.31995
Contents
The problems with monitoring and evaluation

Comparison of monitoring and evaluation software

About DHIS2

DHIS2 Quick Start

Phase 1. Define requirements

Phase 2. Define indicators and design reports

Phase 3. Set up a functioning system with DHIS2

DHIS2 use cases

About LogicalOutcomes

| 2
The problems with
monitoring and
evaluation
Not all evaluation is
Monitoring and evaluation systems often fail: They tend to go wildly over budget, or
over schedule, or don't deliver what they promised, or all three. Even when they are
implemented correctly, there is little evidence that they improve program
effective effectiveness.* Yet funders expect nonprofits - even small ones - to evaluate their
programs as though it's a simple task. Why are M&E systems so difficult to
implement? And how can we make them less expensive and more useful?

There are so many problems with the usual approaches …

In our experience, drawn from 25 years of working with funders and agencies:

 Agencies create logic models that are uninformed by research because they don’t
have the resources to review the research literature.

 Services are based on untested assumptions, imitations of other unevaluated

programs, or ‘the way things have always been done’.

 Funders require agencies to design evaluation plans but don’t have the expertise to
assess them for feasibility or usefulness.

 Evaluators tend to select indicators that are technically weak, and in any case,
agencies don’t have the capacity to collect the data.

 Even when agencies collect service data, they do not have the capacity to test its
quality, aggregate it and report it to users in a way that supports decision-making.

 Narrow funder-defined goals can lead to unethical behaviour.**

| 4
*Powers, L.C. (2009). A framework for evaluating the effectiveness of performance measurement systems. RealWorld Systems Research Series 2009:1.
Available at SSRN: http://ssrn.com/abstract=1371158
**E.g., Ethical breakdowns (2011) Bazerman and Tenbrunsel, Harvard Business Review, https://hbr.org/2011/04/ethical-breakdowns
Difficulties in
Most monitoring and evaluation Nonprofits move too quickly to
(M&E) implementations go over software implementation and get
budget, over schedule, or don’t paralyzed
implementing M&E deliver what they promised
Nonprofits tend to select a software
systems
M&E systems are difficult to
program with the mistaken idea that
implement because of the lack of
it will solve their evaluation needs. In
standards around outcomes and
fact, most of the work involves the
indicators. Every funder and
definition of indicators, reports, user
nonprofit uses different definitions,
permissions and other elements that
LogicalOutcomes interviewed over and each must define its indicators
don’t depend on any particular
40 staff and consultants at from scratch.
software. Software developers don’t
nonprofits in Canada, the U.S., have the expertise to define
Organization-wide data
Europe, Asia and Africa and indicators, and the project gets stuck.
aggregation requires sophisticated
several software vendors*. meta-data management & data
Neither nonprofits nor vendors are
models
We also reviewed the research satisfied with the implementation
literature on M&E software It is relatively simple to collect data of M&E systems
implementations for nonprofits for a single project and a single
Because nonprofits have an
using Google Scholar, and funder.
unrealistic concept of the complexity
reviewed web forums and news
As soon as a nonprofit needs to of M&E systems, implementers feel
groups devoted to monitoring and frustrated and unappreciated.
report to multiple funders or
evaluation.
combine data across different
Project managers report that their
There was broad consensus that programs, it is a completely different
own managers don’t appreciate their
challenge. And most nonprofits do
large scale M&E implementations efforts, vendors report that
not have the expertise to manage the
are very difficult to manage, nonprofits expect unreasonable
added complexity.
regardless of the software. deliverables for the budget, and
nonprofits report that they sink vast
amounts of money with
| 5
* Much of this research was done for SNV - Kerr, G. 2015. PME software and functions after 2015. Unpublished report, SNV. unsatisfactory returns.
Summary of
Difficulty of M&E implementation Selecting software
Organization-wide M&E systems are  M&E requirements are so
comparable to Enterprise Resource complex that no single software
comments from Planning (ERP) implementations. In
some ways they are more difficult
program can meet all of them.
 Every software program will

nonprofits on M&E because of the lack of common

vocabulary in M&E. Major points:
require workarounds and
compromises unless you are
willing to invest large amounts of
implementations  For a full organizational M&E
implementation, count on a
money on custom development.
 If you want customization, ensure
minimum of 18 months and many your software has a well-defined
frustrations. roadmap and the ability to
 Full implementations should be negotiate with the developer or
championed by a member of hire your own developers.
Organization-wide M&E implementations
executive/leadership team.  You cannot have both flexibility
are extremely difficult and the time and
costs are underestimated  Much of the development time and ease of use in an enterprise
would be essentially the same for data management tool. Small
It takes a minimum of 18 months for any software tool. Defining well- differences in wording create
organization-wide implementation designed indicators, aggregation massive headaches at an
categories, data validation rules, enterprise level if you are trying
No software is ideal, all of them are buggy, to aggregate data.
data entry forms and reports are
and all of them require compromises
essential and time-consuming  Mobile data collection tools are
There is a direct trade-off between flexibility tasks. essential for improving data
(ability to customize) and ease of use  Nonprofits are struggling with quality but you can combine two
trade-offs between flexibility, the software programs for that.
Mobile data collection is essential for
ability to aggregate data across  Look for the ability to aggregate
adequate data quality
projects, and ease of use. data in different ways to meet
Implementations require M&E skills as well  There is a growing interest in needs of funders & global office.
as skills in rolling out technical processes sharing templates and indicators  To reduce complexity, consider
among nonprofits to decrease the adjusting your processes around
costs of M&E implementations. the software’s capabilities rather
 M&E implementations require than customizing the software.
staff or consultants with technical See if you can accept off-the-
| 6
skills in designing good indicators. shelf functionality.
Comparison of
monitoring and
evaluation software
Selecting software Dozens of software programs claim to provide monitoring and
evaluation. LogicalOutcomes carried out a comparison of over 35 of

for monitoring and them, including platforms like SalesForce, SharePoint and Microsoft
CRM as well as specialized programs like DevResults, ActivityInfo

evaluation and DHIS2.

We based the analysis on a list of needs that we identified by

interviewing 40 staff in international and Canadian nonprofits.

The needs covered:

 Design of an evaluation framework

 Data collection

 Reporting

 Implementation and roll-out

 Flexibility and resilience

 Building local capacity

 Cost

See the summary of the requirements in the next three pages. Only
one software program satisfied all of them: District Health
Information Software (DHIS).

| 8
Capture theories of change and indicators for each program
Software  Can power-users create logic models and evaluation frameworks during

requirements.1 proposal development and then revise them at project setup?

 Can power-users create or select indicators for programs, allowing
aggregation in different combinations (e.g., age groups and gender) to
meet the differing needs of funders?
Collect data
 Are basic users provided help to collect data with an adequate level of
quality, including data collection tools and automatic validation rules?

Monitoring software is complex, so we  Can basic users easily enter and process data on a mobile device
(smartphone) or web form?
assume three levels of expertise at the
 Can basic users collect information about individual service users and/or
agency:
events, or qualitative information, or rating scales?
 Can power-users design data entry forms with indicators disaggregated by
Power-users are agency staff who are different categories (e.g., age, location, program, etc.) based on funder
familiar with the software. They requirements?
don’t need to be software Report information
programmers.  Can power-users build automated monthly reports that meet agency
Project managers are agency staff needs?
who are given 3 to 4 hours of  Can project managers quickly design customized reports for individual
funders to meet their changing reporting requirements?
training, mostly to create reports.
 Can project managers generate and tailor attractive reports, defining
Basic users just enter data or view various combinations of indicators and time frames, aggregating on many
dashboards. variables, and exporting in PDF or spreadsheet formats?
 Can project managers easily get information out of the system in flexible
formats once it is put into the system, aggregating by program, client type
and/or sector?

| 9
Implement and roll-out
Software  Does the software system provide good updated documentation and
requirements.2 training materials (e.g. video tutorials)?

 Can the software run on popular web browsers on all major operating
systems?

 Can power-users make most changes without a software developer's

support, including designing the framework, creating reports, revising the
data collection instruments, etc.?

 Can basic users view, enter or download data even when internet
connection is not available?
Manage and protect data

 Does the software protect data integrity from corruption, e.g., when
internet connectivity is disrupted?

 Does the software employ security protocols when transferring data and
when data is at rest? Does it follow good practices for protecting
confidential information?

 Is the software updated frequently (a few times a year) using good

development practices including a clear and transparent roadmap?

| 10
Software Build community capacity and knowledge

 Can the software measure key elements (e.g., core values, success factors)
requirements.3 that are important to the agency and its communities?

 Can agencies use and adapt the software freely without limitation? Does
the software use open standards for importing, exporting and
communicating data to support the work of partners?

 Does the software empower local communities and service providers by

giving them more control over their own information and the ability to get
insights from it?

 Does the software provide additional value to agency's contribution by

sharing tools and strengthening the capacity of partners? (e.g., promoting
local ownership of data)
Cost

 What is the cost of design, configuration and implementation per project?

 What is the annual cost per basic user and per project manager, including
the expected level of technical support and hosting?

 How long will it take to train for each role (basic user, project manager,
power-user)?

 How long does it take to create new templates, indicators and elaborate
data entry forms?

| 11
12 key software
In summary, nonprofits seem to want software that is infinitely flexible,
inexpensive to configure and implement, and extremely easy to use.

requirements This is not an unusual set of requests for enterprise software, but it is difficult
to achieve. It requires a complex, flexible software platform that supports a
variety of user roles and the capacity to develop and share templates. That, in
turn, led to a strong preference for open source software that would not be
locked down by a vendor.

When we combined the findings from nonprofits and vendors, we identified

the following requirements for monitoring and evaluation software programs:

Ability to create complex indicators Open source and ability to share templates

Used successfully by large international

Ability to collect data on mobile devices
nonprofits
Ability to aggregate data in different Ability to be configured without software
combinations developers

Ability to store, import, export data Large community of developers (to prevent
vendor lock-in)
Ability to create on-demand attractive Frequent revisions of the software (to
and flexible reports prevent obsolescence)
Specifically designed for M&E; does not
require extensive customization Posted development roadmap

| 12
We identified about 35 software programs through searches on the web, discussion forums and

Software recommendations from nonprofits and narrowed them down to 24 after an initial review.
Where possible we requested information from their respective vendors; not all of our

comparison questions were answered so there are many gaps in the table.

Only 2 programs satisfied all criteria, and only one of them (DHIS2) was well-tested and mature.

| 13
This table is a simplified summary of our analysis. We used a set of clearly defined tasks to test each software platform. In some cases the platform passed some elements of
the test but not others (and was scored with an x and y). We rejected software once we established that they failed in at least one of the 12 key requirements.
It’s not entirely about
M&E software can be divided into four categories:

Multifunctional enterprise software platforms like Salesforce, Microsoft

the software CRM, SAP ByDesign, and other CRMS and ERPs. These solutions require
extensive customization for M&E implementations and are typically very costly
to develop.

Applications built on enterprise software, such as TaroWorks for SalesForce.

The applications take advantage of the power of the underlying platform and
simplify implementation, but add costs over the base licensing fees and
require a significant amount of customization.

Full-featured M&E software programs like DevResults, ActivityInfo, NewDea

All software programs have serious and District Health Information Software, which try to cover all the major M&E
trade-offs. No nonprofit and no functions. They vary in their usability and flexibility.
vendor claimed that M&E software
Limited-function M&E software programs that can integrate with others to
implementation was simple. build a full system, such as mobile data collection tools like Akvo FLOW,
KoboToolbox and CommCare HQ. They tend to be more user-friendly for the
data collection phases at the expense of formal data management.

All of them have been used successfully in some organizations, and have failed
in others (as defined by being over budget, over schedule, or not providing
the expected functionality).

Typically there is a trade-off between flexibility and ease of use. Software that
is quick and easy to configure has less capability in terms of monitoring and
evaluation functions.

Even the most expensive software requires a large staff investment from
organizations to define outcomes, indicators and data models. As one
informant stated, “90% of our work would have been exactly the same if we
| 14
had chosen another software program”.
About DHIS2
What DHIS2 does

Individual data records Data management and Reporting

analytics
DHIS2 enables you to collect, DHIS2 lets you explore and
manage and analyse transactional, understand your data through great
DHIS2 lets you manage aggregate
case-based data records. It lets you visualization features. Get the
data with a flexible data model which
store information about individuals complete overview through the pivot
has been field-tested for more than
and track these persons over time table feature, spot trends in your data
15 years. Everything can be
using a flexible set of identifiers. As with charting and visualize your
configured through the user
an example, you can use DHIS2 to geographical data aspects using the
interface: You can set up data
collect and share essential clinical GIS functionality.
elements data entry forms, validation
health data records across multiple
rules, indicators and reports in order
health facilities. Individuals can be
to create a fully-fledged system for
enrolled for longitudinal programs
data management. DHIS2 has
with several stages. You can
advanced features for data
configure SMS reminders, track
visualization, like GIS, charts, pivot
missed appointments, generate visit
tables and dashboards which lets you
schedules and much more.
explore and bring meaning to your
Excerpted from www.dhis2.org
data. | 16
About the software
Based on community empowerment principles

DHIS2 is an open source program that has been in development for over 20 years.
It emerged in post-apartheid South Africa in 1994 as a collaboration between local
public health activists and Scandinavian action researchers. Its mission: To build the
capacity of local communities while contributing to an effective national health
system. It is now used in many applications beyond health.

Stable and well-supported

DHIS2 releases new versions every three months. It is supported by the University of
Oslo, plus an international network of experts and consultants. It is funded by
NORAD, PEPFAR, the University of Oslo, the Global Fund, CDC, Gates Foundation
etc. and is accompanied by detailed documentation, video tutorials and training
materials. A free online Academy is being launched in early 2016.

Resilient
DHIS2 is designed to handle intermittent internet connections and low cost data
collection. Agencies can collect data offline with free phone apps or light-weight
feature-phone browsers and upload it when the internet is up. They can download
their own data and work with it, syncing when they wish.

Flexible
DHIS2 is designed to aggregate data that is gathered in multiple formats and
locations. It can import and export data through csv files or a web API. It also
provides built-in data collection apps for individual client tracking.

Decentralized
DHIS2 is designed to be independent of any one organization. Expert nodes have
been set up in India, Vietnam, Malawi, Namibia, South Africa and several other
countries to ensure that local expertise can develop. The University of Oslo has
supported dozens of graduate students from developing nations to carry out| 17
research on health systems using DHIS2.
Implemented in 47
National standard for Health Management Information Systems in 20
countries. The uses go well beyond health, though:

countries Health programs - HIV, TB, Malaria and more

Logistics - Uganda, UNCOLSC

Facility registry - OpenHIE

Facility surveys - SA Core Standards/WHO SARA

Disease surveillance - CDC Global Health Security, Ebola outbreak

Education - Dean’s Dashboard

DHIS2 is massively scalable. One
Food security - Bangladesh
instance can support:
Case-based registers - Ghana, Kenya
 8 million women and children
Water and sanitation - Zambia, DRC
registered in Bangladesh
Tracking of pregnant women and children - Uganda, India

 75,000 reporting Tracking of TB patients – Rwanda

organizations and field Social franchising - PSI, MSI
offices in PEPFAR
Licensing of medical doctors - Vietnam

 45,000 field offices and org Forestry - Rwanda

units in PSI Finance - Rwanda Health Finance

Results based finance - Bluesquare / Benin | 18

DHIS 2 Adoption

| 19
Partners Pilot/early phase Scaling up Nation-wide rollout
Implemented by
Service providers Global initiatives
World Health Organization - Program
Population Services International
dozens of NGOs (PSI)
dashboards, Malaria, DQA tool

UNICEF - HQ / Data use in countries

Médecins Sans Frontières (MSF)
Economic Community Of West
International Rescue Committee (IRC)
African States (ECOWAS)
Futures Group
East African Community (EAC)
IMA World Health (IMA)
PEPFAR - DATIM
Management Sciences for Health
Global Fund - Support in 10+
(MSH)
countries
EngenderHealth
USAID - Nigeria PEPFAR reporting
Marie Stopes International (MSI)
CDC - Global Health Security
Family Health International (FHI 360)
UNAIDS - National HIV programs
ICAP

Africare

American Leprosy Missions

| 20

From University of Oslo, Expert Academy, 2015

DHIS2 is an open
platform

| 21
From http://www.slideshare.net/dhis2/global-citizen2
Committed to In January 2016, University of Oslo HISP is launching
an online DHIS2 Academy

training and support  It will use Open edX, a MOOC platform developed by
MIT.

 It will probably be no cost for access to the courses,

and may charge a fee for certification.

 The first course, Fundamentals of DHIS2, is being

tested in September.

Oslo will continue to promote a network of DHIS2

experts

 See Expert Community at

https://www.dhis2.org/expert-community

 Two LinkedIn groups (for users and experts)

 Two Launchpad newsgroups (for users and

developers)

Documentation and sharing will continue to improve

 Launch of Trainingland, a demo of DHIS2 designed for

training that will replace the Sierrra Leone demo

 A sharing group for apps, spreadsheets etc. that the

community is setting up

 A growing interest in templates

| 22
Online Academy - session
New versions
every 3
months

| 24
From https://launchpad.net/dhis2/+milestone/2.21
A simple Client The University of Oslo is investing heavily in the
ability to track the progress of individual clients. The
Management System ‘tracker’ can be used to create a basic client
management system.

By permission from Community Empowering Enterprises Toronto | 25

Android apps for off- Android applications are integrated with DHIS2 and
used for client tracking and facility assessments.
line assessments They can be used off-line (with no Internet access) for
weeks or months.

| 26

From Google Play store screenshots for DHIS2 apps

DHIS2 Quick
Start
An effective
approach to International experience has led to a consensus on the elements

monitoring and
of effective measurement systems:

evaluation  Defining clear outcomes that can be communicated through indicators.

 Defining effective programs that are informed by evidence and meet local
needs and priorities.

 Selecting valid and useful indicators that will actually change the behaviour of
managers and funders.

 Defining indicators and measures clearly enough that they can be shared and
aggregated across jurisdictions, using standard formats.

 Collecting data securely using tools that do not incur an unreasonable cost
burden on front line workers and agencies.

 Validating key information with objective external audits.

 Combining, cleaning and aggregating data from many sources to meet the
needs of multiple users.

 Reporting information in various formats to multiple users – funders, donors,

managers, communities and partners.

This approach requires many different skill sets all working

together. It’s really hard to do from scratch.

We have a 3 stage process that will rely heavily on templates. | 28

Workplan

Define
Define
Indicators and Set up DHIS2
Requirements
Reports
1. Launch project 3. Define program indicators 5. Set up DHIS2 system
Hold kick-off with team, client Review and clarify existing program Configure DHIS2 instance from
working group. indicators. worksheets.
Finalize project charter and Define data elements, Test data collection and reporting
workplan. disaggregations, option sets with internal users.
Deliverable: Project Charter Validate evaluation framework with Pilot test DHIS2 system with selected
2. Define DHIS2 requirements users. users.
Identify decision-makers and user Deliverable: List of Program Collect user feedback & incorporate
groups. Indicators and Data Elements changes.
Interview users and staff regarding 4. Design reports Deliverable: Beta version of DHIS2
M&E needs. Design reports using sample data system
Assess business processes and Define format of import/export 6. Transition to maintenance phase
existing IT system. tables and APIs. Deliver training to client staff.
Define organizational units, user Validate report designs with users Transfer project to ongoing hosting
roles, datasets, reports. Deliverable: DHIS2 Report and maintenance plan.
Deliverable: DHIS2 Requirements Worksheet Deliverable: Working DHIS2 system
Worksheet

| 29
Schedule This schedule would be customized for each project.
WEEKS
ACTIVITY 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
1. Launch project
Hold kick-off with team, client working group
Finalize project charter and workplan
Deliverable: Project Charter
2. Define DHIS2 requirements
Identify decision-makers and user groups
Interview users and staff regarding M&E needs
Assess business processes and existing IT system
Define organizational units, user roles, datasets, reports
Deliverable: DHIS2 Requirements Worksheet
3. Define program indicators
Review and clarify existing program indicators
Identify relevant indicators from external sources if applicable
Define data elements, dissagregations, option sets
Validate evaluation framework with users
Deliverable: List of Program Indicators and Data Elements
4. Design reports
Design reports using sample data
Define format of import/export tables and APIs
Validate report designs with users
Deliverable: DHIS2 Report Worksheet
5. Set up DHIS2 system
Configure DHIS2 instance from worksheets
Test data collection and reporting with internal users
Pilot test DHIS2 system with selected users
Collect user feedback & incorporate changes
Deliverable: Beta version of DHIS2 system
6. Transition to maintenance phase
Deliver training to client staff
Transfer project to ongoing hosting and maintenance plan
Deliverable: Working DHIS2 system
7. Close project
Finalize project tasks & debrief
Archive or delete files
Document lessons learned

| 30
M&E requires a system, not just a software program
Indicators Data warehouse
Indicators are defined using international Service data are stored in encrypted databases
metadata standards. on secure servers with data integrity safeguards.

Report templates and dashboards Data can be imported from other systems,
combined, and exported in various formats.
Standard data visualizations, reports and
dashboards can be selected from templates. Standard disaggregations

Survey question bank The system provides standard options for

categories like age groups, gender and other
Survey questions and data collection forms are demographic characteristics. (For example,
available in a survey bank. Agencies are able ‘youth’ may be defined as ages 14 to 29 or 16
to select individual questions by filtering for to 35.)
activities, populations and desired outcomes.
Technical support
Coaching and setup
Agencies have access to levels of technical
Design and setup of the system is done in support (basic to advanced) at a predictable
collaboration with agency staff. By the end of and reasonable cost.
an implementation process, staff are able to
maintain the system without the vendor. | 31
Estimated pricing

Rock bottom DHIS2 Quick Start Complex multi-program

implementations
If you want to play with DHIS2 with a We recommend a ‘quick start’ DHIS2
minimum investment, we suggest implementation that sets you up with
These projects are like ERP initiatives
getting a basic hosting package from a customized instance for your
and tend to be massive. At this point,
Knowarth and some coaching from program that you can change or
we suggest beginning with a ‘quick
us. grow once you get a feel for the
start’ on one or two programs and
system.
Estimated cost: About $10,000 for then deciding whether to expand
setup, including access to our DHIS2 across the organization.
Estimated cost: $30,000 for design,
templates and basic assistance. Any setup, training and coaching, testing
We can refer you to other experts if
additional help would be charged by and launch.
you want to get a price for a full-
the hour (generally between $60-
scale implementation. Prepare for
$80/hour). Hosting on Swiss servers at
$200-$300,000.
$250/month including 3 levels of
Hosting on Amazon servers at tech support with a set budget for
$52/month. support tickets.
| 32
Phase 1. Define
requirements
What do you need?
 Number of indicators, reports and  Event trackers, to monitor the delivery and efficacy of
dashboards individual events

 Existing data that must be imported and  Client trackers, to track the registration and progress of
combined with new data individual clients (DHIS2 can provide a simple Client
Management System)
 Number of users and user roles
 Facility checklists to track program fidelity and quality
 Graphic and UX design of data collection
forms  Registry of all service locations, including services and
catchment areas. Facility registries can support flexible
 Validation rules and skip logic for data
reporting by service type, location and so on
import and collection
 Geographic mapping of indicators and services
 A web portal that dynamically reports on
selected indicators for a public audience  Program fidelity checklists and rules engines to track the
quality of program delivery based on defined milestones
 Enhanced reports that combine multiple
and attributes
data sources and indicators to
communicate trends in service delivery  Training and development instances of DHIS2

 Expansions to other projects and regions  Technical support and training

 Survey bank of questions to assess staff,  Online training material customized to your agency
partner and participant engagement
 Ongoing coaching through the implementation phase
| 34
Use spreadsheets to
define system
requirements
A ‘Quick Start’ approach uses a configuration spreadsheet with
individual worksheets including:
 Indicators
 Data elements
 Categories
 Option sets
 Indicator groups
 Data element groups
 Indicator group sets
 Organizational units
 Datasets
 Report types
 Organizational roles
 Users

The configuration can then be finalized manually to capture the complex

many-to-many relationships that can be mapped within DHIS2.

| 35
Define feasible
Invite (don’t force) teams to participate in pilots of monitoring and evaluation
tools, and select projects that can tolerate ambiguity and the frustrations that
are part of early adoption. Pilots should be championed by critical and
objectives knowledgeable project managers.

Focus on user needs. For example, who is actually using the information?
When do they need it, and how do they want to report it? Include corporate
users (like business development) as well as the project managers.

What are the minimum reports necessary to achieve user objectives? You
don’t need to solve everything at once. Aim for quick wins and build
excitement across the organization by delivering products that work.

Decide how important it is to aggregate high quality information across the

Some of these tips are from the organization. If it’s important, be aware that the complexity and cost of the
implementation is far greater than if you tolerate variations at the local level.
research literature (see selected
references below) and other are from Long term fixed-price contracts are not good at handling complexities and
experienced consultants and project unforeseen situations – a phased approach ensures the nonprofit has a
managers of M&E implementations. concrete deliverable and gives a basis to estimate the cost of the next phase.

Be willing to work with less-than-perfect datasets. The research literature on

DHIS2 implementations* suggest that it is unwise to clean up all of the existing
information sources in a system before launching DHIS2. Live with uneven
data quality for a while, and clean it gradually. After a few years the old,
inaccurate data will be archived.

*https://scholar.google.ca/scholar?hl=en&as_sdt=0,5&q=DHIS2
and
http://www.mn.uio.no/ifi/english/research/networks/hisp/Researc
h%20Library/Recent%20Publications and | 36
http://www.mn.uio.no/ifi/english/research/networks/hisp/Researc
h%20Library/phd-thesis-list.html
Phase 2. Define
indicators and
design reports
Defining outcomes
International experience has shown that shared measurement systems
should get to the level of indicators as quickly as possible. They should be

by using indicators brief, clear, achievable and measurable. Two examples:

| 38
Millennium Development Goals http://www.undp.org/
Canadian Index of Wellbeing https://uwaterloo.ca/canadian-index-wellbeing
Characteristics of a
good indicator -
examples

| 39
From Harmonized Reproductive Health Registries (hRHR) Working Group at www.fhi.no/dokumenter/1d23cd1b4e.pdf
Characteristics of a 1. Action Focused
“It is clear what needs to be done to improve outcomes associated with this
good indicator – indicator (e.g., immunise to reduce neonatal tetanus)”

summary* 2. Important
“The indicator and the data generated will make a relevant and significant
contribution to determining how to effectively respond to the problem”

3. Operational
"The indicator is quantifiable; definitions are precise and reference standards
are developed and tested or it is feasible to do so”

4. Feasible
“It is feasible to collect data required for indicator in the relevant setting”

5. Simple and valued

“The people involved in the service can understand and value the indicator”

We have added a sixth requirement for the DHIS2 Quick Start:

6. Open access
“The indicator is available at no cost and can be shared freely.”

Adopted from Harmonized Reproductive Health Registries (hRHR) Working Group at www.fhi.no/dokumenter/1d23cd1b4e.pdf | 40
Example indicator –
GAVI Alliance

DHIS2 provides the ability to

use international metadata
standards to define indicators.

This allows for standardized

measurement systems.

From Gavi at http://www.gavi.org/results/goal-level-indicators/ | 41

Example indicator -
PEPFAR

| 42
From http://www.pepfar.gov/documents/organization/240108.pdf
Example
indicator –
social services

| 43
Draft Indicator Reference Sheet from working document, Prosper Canada and Canadian Bankers Association, 2015.
Indicators will be The challenge

shared through Common measurement systems require shared definitions of indicators,

measures and data collection tools. Organizations tend to be reluctant to
open access tools share this kind of intellectual property, and most funders explicitly forbid
open access in their contracts with agencies and consultants. (If you’re in
doubt about this, check out the legalese in your contracts regarding property
rights and ownership.)

As a nonprofit human service sector, we need a way to share freely while

recognizing the contribution of authors and sponsors.

How DHIS2 can help

We will assign a DOI (Digital Object Identifier) to each complete and

Other metadata standards that validated Indicator Reference Sheet and/or to a collection of approved
can be supported by DHIS2: indicators through Zenodo, an international research repository.

The indicator itself will have its own license and authorship (e.g., Statistics
IATI (http://iatistandard.org/)
Canada uses the Open Government Licence – Canada).
HXL (Humanitarian Exchange Language,
http://hxlstandard.org/) Zenodo, as the publisher, will maintain accessibility of the indicator(s) even if
the original dataset is taken down.
ADX (Aggregate Data Exchange,
www.dhis2.org/doc/snapshot/en/develo
per/html/ch01s12.html)

DOI - http://blog.apastyle.org/apastyle/2014/07/how-to-use-the-new-doi-format-in-apa-style.html
ORCID - https://orcid.org/organizations/funders and http://orcid.org/content/initiative
FUNDREF - http://www.crossref.org/fundref/index.html | 44
ZENODO - https://zenodo.org/features
Open Government License – Canada - http://open.canada.ca/en/open-government-licence-canada?_ga=1.156660539.1898951134.1438269552
Generic logic model We are using a standard 8-step logic model to provide consistency for
coding indicators into the evaluation system. At the top level of the data
dictionary are Indicator Group Sets divided into four outcome groups and
four output groups.

OUTCOMES

1. Impact – covering all timeframes from immediate to long term, and that
refer to the impact on the intended beneficiary groups. Examples:
employment, income, housing status, etc.

2. Organizational practices – The desired changes in organizational

policies, procedures and practices that are necessary to lead to the
desired impact

3. Behaviours – individual behaviours among the participants, target group

members and/or service providers

4. KASA – Knowledge, attitudes, skills, aspirations – for participants and

service providers

OUTPUTS

5. Experience – satisfaction or engagement from participants and

stakeholders

6. Reach – the extent to which the program reaches the targeted number
and type of participants or audience

7. Activities – the program activities

8. Management – the extent to which the program is well managed,

| 45
including financial and human resources
Designing reports
that lead to action

Evaluation and data nerds

tend to love information.

Senior managers and

decision-makers tend to
prefer simplicity. If you give
them detailed reports, they
may not spend the time to
understand them.

We need to design reports

that are relevant, accurate
and meaningful to decision-
makers.

| 46
This is a real chart with real data. From http://junkcharts.typepad.com/junk_charts/2010/08/
Designing reports In this example, vaccinations have fallen in the Bird
District. The organization needs to find out why
that lead to action vaccinations have been declining, especially in late
2014, and how to increase the rate again.

| 47
From Trainingland – DHIS2 training site - to be launched in early 2016 on www.dhis2.org
GAVI Vaccine Alliance

From Gavi Vaccine Alliance at http://www.gavi.org/results/goal-level-indicators/. Gavi has begun to use DHIS2 to track and report on indicators. | 48
Use PowerPoint to Use PowerPoint or Excel to create prototypes of desired reports
using dummy data. Then consult with key stakeholders and
decision-makers. Is this what they want?
create mockups of We will have a few basic reports that are part of the template,
desired reports including participant satisfaction and number of people served.

| 49
Phase 3. Set up a
functioning system
with DHIS2
DHIS2 reports
information in Dashboards can be created for individual users and funders. They can be posted on
the integrated web portal or shared privately.
various formats

| 51
Use DHIS2 to test
mockups of
dashboards

From www.dhis2.org | 52
Defining logic model Each group set (see below) is linked to multiple indicator groups.

through group sets

| 53
Assigning indicators Group Sets (e.g., REACH) are linked to Indicator Groups.

Each Indicator Group can have unlimited numbers of Indicators.

to groups

| 54
DHIS2 use
cases
Managing malaria in
To improve malaria reporting in Kenya, the Ministry of Health in 2010
approved the use of DHIS2 to report on malaria commodities at the sub-
national level.*
Kenya With support from USAID, Kenya’s Malaria Control Unit transitioned its
reporting system to DHIS2 in October 2012. Use of DHIS2 improved
reporting rates from about 45 percent to 70 percent in the months after
its implementation (see figure to the left).

Kenya is now working with 13 county governments to promote reporting

through DHIS2 for family planning, HIV, nutrition, and laboratory
commodities.

The Health Information Systems unit of the Ministry of Health and staff from
the HIV, TB, malaria, reproductive health and family programs participated in a
conference on the impact of DHIS2, facilitated by USAID and Ministry staff.

Participants heard how an effective health information solution contributed to

Kenya's standing with The Global Fund to Fight AIDS, Tuberculosis and
Malaria going from a C to an A2 rating: Using DHIS2 made it possible and easy
for the country to track the reporting and non-reporting health facilities.
Through better tracking of commodities with DHIS2, donor confidence in the
Reporting rate for malaria commodities across malaria program was restored and additional funding was secured.
all health facilities in Kenya, June 2012-January
2013. Effective health information systems leads to better health systems

Effective health information solutions like DHIS2 can help improve

accountability across the health system. Scale up of DHIS2 can further
strengthen the management and use of health commodities and improve the
use of data for decision making at all levels of the health system.

* This page is excerpted and paraphrased from | 56

https://www.msh.org/news-events/stories/managing-data-with-dhis2-
improving-health-commodities-reporting-and-decision
Use of technology in
Ebola response in
West Africa
DHIS2 is one of the key
technology platforms used
to manage Ebola and
HIV/AIDS.

It has been nationally

implemented in at least 13
African countries, and in the
process of adoption in 50
countries worldwide.

Excerpted from https://www.msh.org/resources/use-of-

technology-in-the-ebola-response-in-west-africa | 57
unicef

Many of Unicef’s projects rely

on DHIS2.

From http://www.unicef.org/search/search.php?q_en=dhis&go.x=0&go.y=0 | 58
Afghanistan
Algeria
Bangladesh
Benin

National
Bhutan
DHIS2 is being used or in the process of Burkina Faso
Burundi
adoption by over 50 countries so far.
implementations of
Cameroon
Colombia
Here is a somewhat out-of-date list. Congo Brazzaville

DHIS
Cote d'Ivoire
DRC
Key: Ghana
Guinea Bissau
India (Bihar, Orissa, Maharashtra,
Kerala, Punjab, Haryana, H Pradesh)
Kenya
Laos
Liberia
Malawi
Mexico
Mozambique
The University of Oslo’s DHIS2 program trains Myanmar

doctoral students in health management. A search Namibia

Nepal
of Google Scholar of DHIS2 shows over 400
Niger
research articles on implementation and health Nigeria

system management. North Korea

Rwanda
Samoa
Senegal
Sierra Leone
Solomon Islands
South Africa
South Sudan
Sri Lanka
Sudan
Tajikistan
Tanzania
The Gambia
Timor Leste
Togo
Uganda
Vanuatu
Vietnam
Zambia
From https://www.dhis2.org/deployments. | 59
Also see https://scholar.google.ca/scholar?q=dhis2&btnG=&hl=en&as_sdt=0%2C5 Zanzibar
Zimbabwe
The Global Fund

The Global Fund (www.theglobalfund.org) is an international partnership that

provides funds to accelerate the end of AIDS, tuberculosis and malaria. It
raises and invests nearly $4 billion US/year to support local programs.

It is a heavy user of data standards, and promotes the use of DHIS2 to track
health status. In fact, it funds DHIS2 implementations as part of its ‘Health
Systems Strengthening’ initiative, and most of its national partners use DHIS2
to collect and report on health data. In November 2014, Global Fund reported
that:

“Strengthened country data systems are crucial to making robust plans and
measuring and evaluating impact. Data needed for results reporting and
impact assessments require country-based data systems and structures … Of
the high impact countries, 17 out of 23 are using DHIS2 2 as a reporting
platform, with funding from grants going to support rollout and training.”*

The entire web site provides a model for good funding practices and
resources. They use indicators that have been defined within DHIS2, including
PEPFAR’s, and show examples of how to build in workplan deliverables and
milestones.

From http://www.theglobalfund.org/documents/fundingmodel/progressupdate/FundingModel_2014-12-Progress_Update_en/ | 60
Data linked to
national data
systems.

Dynamic and
verified results
instead of static
results.

Click and zoom on

public interfaces of
national data
systems

| 61
About
LogicalOutcomes
About us LogicalOutcomes is a nonprofit organization based in Toronto, Canada.

We provide evaluation and consulting to support collective impact

through shared measurement systems and stakeholder engagement.

We work in virtual teams, with a roster of dozens of analysts and specialists

from around the world. And we’re led by three Board members, described
below (in alphabetical order).

Gillian Kerr, Ph.D., C.Psych. Martha McGuire, M.S.W., CE Neil Price, M.A.

 Organizational psychologist with  Founder of Cathexis Consulting,  Leads stakeholder

background in qualitative methods one of Canada’s leading consultation projects
and information systems evaluation firms
 Led multiple youth programs
 Formerly VP United Way of Greater  Former President of Canadian related to post-secondary
Toronto, on UW America Taskforce Evaluation Society, provincial education
on Measuring Program Outcomes, and national bodies
1996  CivicAction DiverseCity Fellow
 Has carried out over 130 and community advocate
 Background in working with people evaluations
with disabilities, ethnocultural  Teaches community
services, immigration policy  Teaches evaluation at Ryerson development at George
University Brown College

| 63
The DHIS2 Team
LogicalOutcomes has an international The DHIS2 Network
network of analysts and contractors.

We work with Canadian and international The University of Oslo, the NonProfit
analysts, software developers, writers Organizations Knowledge Initiative
and evaluators. (NPOKI), Metrics for Management,
Population Services International
For DHIS2 implementations, we work (PSI), The Global Fund and many
with HISP India and HISP Uganda, two of other organizations are building a
the international hubs for DHIS2 community of practice to create
development, as well as several shared measurement systems for
independent experts. nonprofits across the world.

We have relationships with two DHIS2 LogicalOutcomes will help agencies

hosting services. Entuura Ventures, our tap into the network for advice and
main partner, is led by Steven support as well as engaging in the
Uggowitzer, an international leader in community ourselves.
DHIS2 implementations, and provides a
full DHIS2 service including level 2 and 3
technical support. Knowarth, a reliable
and low-cost Indian hosting provider,
uses US-based Amazon servers. Both
providers include SSL encryption,
monthly patching and testing, backups
and 24 hour emergency support.

In addition, we have a long-term

relationship with SolutionAnalysts, a
technology firm that develops web
applications, mobile apps and complex
enterprise systems. They have designed,
| 64
developed, built and maintained more
than 350 solutions.
Project management Project Management Processes Fixed Price or Cost Plus Budgets

and costing Project budgets are based on an

estimate of effort for each phase.
Clients can choose whether they
prefer a fixed price contract or a
We bill only $10/hour over what we ‘cost plus’ contract:
actually pay our consultants, with a
Our project tools
$45/hour minimum. For fixed price Fixed price: we make our best
contracts we price our projects to estimate for the hours that will be
We use free software tools to run our required for the project, and will not
cover our costs with little left over
projects, customized to make us more
for unexpected expenses. charge for additional time if we go
efficient. They include:
over budget.
With overheads so low, we need to
OneNote Notebooks, designed as self-
control project scope carefully so Cost plus: we will charge for the
contained ‘projects in a box’. OneNote is
extraordinarily effective if it is set up that we don’t go over budget. We hours we work. Some clients like the
properly. use formal project management flexibility to assign us to new or
processes, working closely with our changing tasks as the project
Office365 groups to encourage collaboration clients to keep on track. We use progresses, or train internal staff to
and reduce reliance on emails. agile methodologies to deliver replace our team members (things
products in short modules that cannot be budgeted ahead of
Zotero for literature reviews. time.)
(generally at 6 to 8 week intervals)
DHIS2 to track deliverables, to produce to ensure that we are creating
useful tools that meet the project’s In either case we work closely with
monthly status reports, and to prototype
goals. the client to ensure that objectives
evaluation systems.
are met within the resources that
nCrypted Cloud to protect confidential Projects have clearly defined roles are available. In both cases the
personal information on laptops and to share and responsibilities for quality, cost, clients may terminate the contract
it securely with team members. schedule and stakeholder with two weeks’ notice for any
relationships. reason.
| 65