Two years ago I sat in on a board meeting where an organization proudly presented its brand-new evaluation framework: a full logic model, a color-coded outcomes matrix, quarterly review cycles, the works. Six months later, nobody on staff could tell me what question the framework was actually supposed to answer.
The framework existed. The evaluation didn't.
Program evaluation frameworks are everywhere right now, and I think that's mostly a good thing. Funders expect them. Boards ask for them. There's a real appetite in this field to move past gut instinct and toward something more disciplined. But somewhere along the way, adopting a framework started to feel like the finish line instead of the starting point.
A framework is a container. It only holds something if you've decided, in advance, what you're actually trying to find out.
The Framework Isn't the Question
This is the stance I've come to hold pretty firmly: a program evaluation framework is only as good as the question it was built to answer, and many organizations skip that step entirely. They start with the framework's structure, outputs, outcomes, indicators, rather than starting with the decision the evaluation is supposed to inform.
Is this program working well enough to expand? Should we cut a component that isn't pulling its weight? Are we serving the population we think we're serving? Those are different questions, and they require different data, different timelines, and different tolerance for ambiguity. A generic logic model answers none of them particularly well, because it was never built to answer a specific one.
The CDC's Framework for Program Evaluation in Public Health, published in 1999 and still widely used well outside public health, gets this right in a way many adapted versions miss: it treats evaluation as a cycle that starts with engaging stakeholders and clarifying purpose, not with picking indicators.
Rigor and Usefulness Are Not the Same Thing
The most rigorous framework is not automatically the most useful one. A strand of evaluation theory built around utilization has argued for decades that an evaluation's value is determined by whether the people who receive it actually use it to make a decision, not by how methodologically sophisticated it is. A perfectly designed evaluation that sits in a PDF nobody reads has failed, regardless of its rigor. A simpler evaluation that changes how a program runs next quarter has succeeded.
I've watched organizations chase rigor for its own sake–adding control groups, expanding data collection, layering on statistical methods–because rigor signals seriousness to funders. Sometimes that's warranted. Other times it produces an evaluation that answers a more precise question than anyone actually needed answered, at a cost the organization didn't need to pay. The frameworks that hold up over time are the ones built around what a specific team, at a specific moment, actually needs to decide.
Evaluation Has to Be Built to Be Used, Not Just Built to Be Complete
A framework needs a home inside the organization's actual decision-making rhythm, or it becomes ceremonial. Evaluation researchers have documented this exact pattern for decades, and the more recent work shows it has not gone away. Alkin and King's 2016 Historical Development of Evaluation Use traces how the field's thinking shifted from asking whether a report changed a decision to asking how evidence moves through an organization at all, a shift that came from repeatedly finding that findings arrive where no one is positioned to act on them. Michael Quinn Patton’s reflections on that series, published in the American Journal of Evaluation, describe where the field has landed.
Evaluation is not a single event that follows a report. It unfolds over time through relationships, context, and many interacting influences, so evaluation has to be analyzed as a complex, dynamic system rather than as a straight handoff from finding to decision. Work on nonprofits points the same way: Carman and Fredericks’ Evaluation Capacity and Nonprofit Organizations found that many organizations collect evaluation data mainly to satisfy funders, which means it is built for a reporting calendar rather than for the decisions leaders actually face. Findings get produced and then quietly ignored, not because they were wrong, but because nobody built a pathway connecting the finding to a decision that was still open. If the funding decision or the program redesign already happened by the time the evaluation report lands, the framework did its job on paper and nothing else.
This is why I help clients build evaluation timing around real decision points, board votes, renewal cycles, program redesign windows, instead of an arbitrary annual calendar. Our Advisory engagements are guided by the principle that strategy lives in how an organization operates, not in a deck. For clients like Painting Pandas, whose work centers on community building, that means measuring what the community is actually experiencing as the program grows, so leaders can respond while it still matters instead of waiting for a year-end report. Through Compass, we hold impact data to the same standard, grounding it in transparent evidence so decisions rest on what the work is actually producing.
A Different School of Thought Entirely
Starting in the late 1990s, a strand of evaluation practice began borrowing directly from complexity science and systems thinking, and it offers something closer to an actual answer, though not a clean one. A key proponent of this shift was Ben Ramalingam, whose book Aid on the Edge of Chaos argues that we have spent too long treating economies and societies as if they were machines. In his view, social systems behave much more like living ecosystems, constantly changing, made up of many different parts, and impossible to predict with confidence.
The core departure is treating a program not as a fixed mechanism with a predictable chain from input to outcome, but as a complex adaptive system: something shaped by feedback loops, shifting conditions, and actors who change their behavior in response to the evaluation itself. Under that view, a static logic model is an input error, applying a linear tool to something that was never linear in the first place.
This complexity-informed tradition produces evaluation that runs alongside a program rather than after it, feeding observations back into the work while conditions are still shifting, instead of waiting for a fixed endpoint to render judgment. It closes the timing gap structurally, because the evaluation is embedded in the decision-making process itself rather than appended to a reporting cycle.
But the nuance matters here, and I don't want to oversell it. This approach asks a great deal of an organization's tolerance for ambiguity, and of a funder's willingness to accept findings that resist a clean before-and-after story. It's genuinely harder to standardize across a grant portfolio, harder to summarize in a one-page dashboard, and it can be misused as a shield, a way for a program to avoid hard questions by claiming its context is too complex to measure cleanly. That failure mode is real, and I've seen it used in bad faith.
This school of thought earns its keep with programs still finding their shape. A ten-year-old program with a proven model doesn't need to be treated as an unfolding complex system. It needs clear, fixed indicators, because the uncertainty complexity-informed evaluation is built to hold has mostly already been resolved. The judgment call is knowing which situation you're actually in, and most organizations default to whichever framework is fashionable rather than asking that question honestly.
The Standard I'd Hold Any Framework To
I don't think complexity is the enemy here, and I don't think it's a universal fix either. Some programs need the discipline of fixed metrics. Others need room to keep changing shape while they're being measured. I think that too many organizations are running frameworks sized for questions they aren't asking, and the fix isn't a better template in either direction. It's starting from the decision that needs to get made, being honest about how settled or unsettled the program actually is, and choosing a structure that matches that reality instead of whichever one arrived first.
