Talking Metrics, Part III: Going Beyond Metrics - Monitoring & Evaluation

Metrics can answer whether your organization is operating as it should be.  Metrics don’t answer why you’re seeing the results you’re seeing, whether your organization is tackling the right problem, or if there could be a better way. If your organization is looking to answer those questions, you’ll want to engage in an evaluation.  An evaluation can help you to go deeper into results – getting behind the metrics – to troubleshoot challenges and learn.  

People often use the terms monitoring and evaluation interchangeably, and there are some overlaps. But I find it helpful to think of them as distinct concepts that complement one another. 

Monitoring vs. Evaluation

Monitoring

Monitoring is using data to drive incremental and continuous improvement, provide rapid feedback, and demonstrate accountability.  This often looks like reviewing metrics on a regular basis. It also helps to answer the questions: “Are we doing what we expected to do?” and “Are we getting the results we expected to get?”  This work is typically performed by organization staff and is done frequently (e.g., monthly, weekly, or quarterly), with lower levels of ongoing effort than evaluation, but a constant state of effort.   Monitoring needs to be rigorous enough to help move forward continuous organizational improvement, but no more - you don’t need a controlled trial to determine if your organization is operating as expected. 

Evaluation

In contrast to monitoring, evaluation is more periodic, in-depth research, which is used to justify major decisions and demonstrate program effectiveness in a statistically valid way.  Evaluation helps your organization understand causation - showing why change is happening. Evaluation typically takes the form of episodic research projects that help to answer strategic questions such as “Are we engaging in the right activities?” or  “What difference is our work making?”   Evaluation explores something longer-term, so it doesn't make sense to review data on a regular basis.  Conducting evaluations and reviewing results on a yearly or bi-yearly basis is often sufficient.  The level of effort put into an evaluation project is higher than monitoring, but it occurs for a shorter time period. In other words, evaluation takes the form of a project, while monitoring typically takes the form of a habit.  Evaluation can be conducted by internal staff or external consultants; using external firms is often helpful to demonstrate that the results aren’t biased. How rigorous an evaluation needs to be varies widely and depends on the questions you are asking and for what audiences.   

Why does this distinction matter? Some kinds of questions that your organization needs to answer are better addressed by a one-time evaluation project than by the habit of regularly monitoring performance metrics. Often both approaches can complement one another, but each has its place. 

Does this leave you with more questions than answers? Feel free to contact me and I’m happy to share more!