Archive for the ‘evaluation’ Category

Smart(er) Measures

July 1, 2015


Which of these challenges is most important to solve: teen pregnancy, hunger, diabetes, or drunk driving?

Where should we focus time and money to improve lives and our communities?

Just as sports and commerce have been recast by data, the health field is leading the way for different mental models that help us calculate value in the social sector. In the US last year, more than 600,000 people died from heart disease while about 75,000 died from diabetes. Does that make heart disease eight times worse than diabetes?

We’re well aware death is inevitable. Still, many conditions aren’t terminal but very costly for individuals and society. In fact, the non-fatal conditions account for the vast majority of health costs. With this rationale, health economists have created new measures. One is a unit identified as “disability-adjusted life years” or DALYs. DALYs calculate the years of healthy life. A DALY is generated by identifying the years of life lost when a person dies (compared with a projected average without the condition). Then, the total years lived with a disability are tallied. A DALY is generated based on estimates of how nonfatal conditions detract from perfect health.

Using this new measurement unit, one can rank health problems. For example, in 2012, scientists reported 200,000 more deaths by lung cancer than traffic accidents worldwide. But, when calculated in DALYs, road injuries are far worse. Most lung cancer deaths are among older folks. Those who tend to die in car accidents with frequency are young – in their 20s and 30s. Importantly, road injuries cause about 40 times more disabilities when people survive them. Should we invest in anti-smoking or road safety campaigns?

Mexico has moved to this kind of analysis for treatments. Now, childhood cancer treatments and emergency care for car accidents are high priorities. Australia has also used DALYs to focus on childhood obesity and other issues. The application of DALYs in the US identifies low back pain, depression and anxiety as enormous health concerns. They generate substantial costs because of prevalence along with significant pain and suffering.

New statistics offer different and valuable viewpoints. They can threaten the status quo. Capable leaders use them to support change and progress. DALYs are a great example of how, with new measures, we can enable rational decisions. Isn’t that smarter? In fact, it can affect how we live and die.

Lisa Wyatt, Ed.D. is chief strategy officer and managing partner in Phillips Wyatt Knowlton, Inc. PWK is a performance management resource for systems and social change with clients worldwide. Lisa has cross-sector and international experience. She is an author and W.K. Kellogg Leadership Fellow. See:

A Blinding Blizzard

March 23, 2015


Blinded by the data blizzard?

In 2014, this volume of data was produced, each minute:

  • 204 million email messages were sent
  • Google received more than 4 million search queries
  • 2.46 million pieces of content were shared on Facebook
  • 277,000 tweets were sent, along with 48,000 apps downloaded
  • 26,380 reviews were posted on Yelp! and 216,000 photos posted on Instagram
  • 3,472 images were pinned to Pinterest, and
  • 72 hours of new video were uploaded to YouTube

In Digital Destiny, Shawn DuBravac, PhD, reminds us there’s no need to remember these figures. They are obsolete. The quantities are far greater today. However, these facts show something very important: the huge scale and speed of data production.

Data is everywhere in your organization, community, home and life. Managing effectively depends on measuring accurately. The careful use of data sets strategy, creates programs, provides feedback, shows potential for improvement and displays  outcomes.

With increasing frequency, we see metrics, indicators and findings mis-used. To support a conclusion or point of view, some people consciously (and unconsciously) will generate or select data to suit their purpose. It’s a strong way to market any message.

There’s no public or private “regulator” that practically sorts this for you. The volume and quality of data used across many contexts presents tremendous challenges for those with little measurement experience or awareness.

Professionals who handle data routinely know and practice ethical standards for data use. What can you do? Here’s a start: listen to skeptics; trust your intuition; ask hard questions to challenge assumptions, methods and sources; read more about metrics; understand limitations in findings; secure an independent review by an ethical evaluator.

Data can be very powerful in the right hands, heads and hearts. Because of this, every manager-leader needs data literacy.  Sorting out the signals from the noise is a vital skill in demonstrating value, for learning and creating change.

Lisa Wyatt, Ed.D. is chief strategy officer and partner in Phillips Wyatt Knowlton, Inc. PWK is a performance management resource for systems and social change with clients worldwide. Lisa has cross-sector and international experience. She is an author and W.K. Kellogg Leadership Fellow. See:

Smart Measurement

August 4, 2014

If data told you where to improve and whether you’re successful or off-course, would you want it? Would you use it? When talk turns to metrics –people often shut down. Mostly, it’s because relevance and utility are absent. Sometimes it’s because people prefer to avoid accountability.

The sticky and too-common problem: people collect and report a vast range of items that are easy to measure instead of deeply understanding and choosing a vital few metrics.

Key performance indicators (KPIs) should be vital navigation gauges to understand where your work stands compared to planned results. The right data is used to improve both processes and impact. In your selection, make sure data is both  relevant and actionable.

One way to organize indicators is to see your organization in several big parts. A simple four-piece dashboard common to the private sector includes: customers (or participants), financial performance (outcomes), internal processes, and talent. Those noted in parentheses are a social sector comparable.

What do you need to know about each to make decisions?

1.Customers or Participants It makes sense to have information about your customers or participants…without them, the enterprise quickly fails. For both exempt and for-profit organizations, you might want to know what is your participant retention rate? You also might want to know about “reach,” which is closely associated with the effectiveness of your marketing strategy. How well do you translate inquiries, invitations, presentations, web views into new participants? This is a “conversion rate.” To determine growth, the size of your slice of the pie, compared to others in the same market is important to know, too. This is relative market share.

2.Financial Performance or Outcomes The private sector identifies a revenue growth rate or net profit margin. In this context, it’s also relatively simple to calculate return on investment. Although more complex, the social sector can describe outcomes, too. Results often come in the form of reductions or gains, e.g., fewer teens in the criminal justice system or better science scores by 9th grade. It’s important to be able to describe short-, intermediate and long-term outcomes. Specifying those outcome chains in a time sequence  informs smart plans, adaptation and progress. It’s essential to know where you are …to get where you’re going. A social service agency or foundation cites the aggregate of their effort across multiple programs or grantees. For example, Pew Trusts delivers public value from environmental partnerships in several selected priorities.

3.Internal Processes The cost and quality of internal processes are ripe areas for review because these can be directly influenced by capable managers. We know the overall enterprise performance is deeply connected with routine functions. The maturation or relative sophistication in processes is an important measure. So, for example, in planning: Does rigor, discipline and evidence occur in a standard process and cycle? In addition, capacity as well as productivity are areas to assess against standards, as is the value on-going projects contribute to operations.

4.Talent High performing organizations rely on great talent. Organizational leadership as well as human resource practices vary in quality. Both significantly influence workforce and the workplace. “Best in class” human resource practices focus on talent identification, selection, development and retention. The talent domain raises these and many more questions: What competencies and attributes matter most in our staff? Do we have the right people? Will staff recommend their employer as a great place to work? Do staff attitudes and behaviors contribute to the enterprise goals? How do staff rate each other, themselves, their supervisors? What’s the depth of the management bench? Is the prevailing culture healthy or toxic?

Einstein said: “Not everything that counts can be counted, and not everything that can be counted counts.” Steering your ship requires the right information at the right time to inform choices. It is why skillful management measures carefully.

-Lisa Wyatt, Ed.D. is chief strategy officer and partner in Phillips Wyatt Knowlton, Inc. PWK is a performance management resource for systems and social change with clients worldwide. Lisa has cross-sector and international experience. She is an author and W.K. Kellogg Leadership Fellow. See:

Seeking Social Benefit

March 17, 2013


How do we lead new, more effective ways to deliver social benefit? For many decades, private sector management has applied this lever to their parallel performance challenges. Bill Gates recently extolled its value. A reasonable answer to the question: measurement.

The Gates Foundation 2013 annual letter spells out accomplishments and an ambitious agenda. But, a key message is measuring for managing. “I have been struck again and again by how important measurement is to improving the human condition,” wrote Gates. He cites the intent to eradicate polio. And, describes timely, local, accurate measurement is a prequel to “figure out what is wrong, and fix it.”

Performance Management
Measurement is intimately connected to performance management. Performance management relies on data collection, analysis and course correction. Increasingly, the public, stakeholders, funders and others ask:
• What value are we getting?
• What more can our work deliver?
• How are we changing lives and systems?
These are all fair questions that paid (and volunteer) manager-leaders are eager to answer, too.

In Michigan, measurement has had serious application in early childhood development efforts. First Steps, in partnership with Grand Rapids Public Schools, has focused on what works under what conditions in high-risk neighborhoods with vulnerable children. For example, in less than a week, a pre-kindergarten “camp” positively affected the socio-emotional status of children along with adoption of routines. In addition, Play & Learn groups showed changes in children’s language and literacy skills. Because of measurement, it’s possible to demonstrate progress, thoughtfully adapt programs and identify the value interventions contribute to children, their families and the education system.

Funds for Results
In most contexts, funds are given for the promise of a desired change or intended result in the nonprofit sector. However, as resources are more scarce, connecting funding with proven success may become more common. New financing instruments called social impact bonds require explicit results to continue funding. The planned results and associated cost savings are built into the economic model. And, even the US government is exploring the idea of results-based resources in their programs that offer financial support for social challenges (called Pay for Success).

The discipline of measurement is underutilized, perhaps because of the distinct skills it requires. It also carries some risk, because it points out poor program design, plans and/or implementation. Measurement can certainly identify waste. Notably, not everyone has the same vibrant passion for efficiency and effectiveness.

Social Progress
The old adage goes: You can’t manage what you don’t measure. While “social engineering” has plenty of detractors and critical issues in it – much of our tax exempt or civil sector have huge caches of social, political and economic capital aimed at social change that can deliver more, better value.

Literacy and competencies in measurement are essential to both leapfrog and routine progress. These days, a noble cause is only a great start. Authentic claims about impact or serious change must be grounded in precise measurement. Ask Bill Gates.

-Lisa Wyatt, Ed.D. is chief strategy officer and partner in Phillips Wyatt Knowlton, Inc. PWK is a performance management resource for systems and social change with clients worldwide. Lisa has cross-sector and international experience. She is an author and W.K. Kellogg Leadership Fellow. See:

In Pursuit of Ignorance

September 11, 2012

Charles Darwin, 1881

“Knowledge is a big subject. Ignorance is bigger…and it is more interesting,” says Columbia University neuroscientist Stuart Firestein. He claims exploring the unknown is the true engine of science. Ignorance, he says, helps scientists concentrate their research.

Firestein’s book, “Ignorance: How It Drives Science” argues that what we don’t know is more valuable than building on what we do know. He believes ignorance follows knowledge. Knowledge enables scientists to propose and pursue interesting questions. Rather than big tangles like the “How was the Universe formed?” Firestein favors the more specific. In the social and private sectors, this perspective has enormous merit for both routines and innovation.

Great Questions

Inquiry can catalyze learning and support change. In a recent proposal to an influential funder, we posed work with colleagues as applied research. The primary question: What early childhood learning investment works best with which kids? Why? This raises others:

  • When is the best time to provide intervention and enrichment?
  • How many opportunities are available in each intervention and enrichment opportunity?
  • What gains are made at what cost?
  • What proportion of 4-year olds are most at risk?
  • How are children distributed along a continuum of need? 

Thoughtful Ignorance

According to Firestein, “Thoughtful ignorance looks at gaps in a community’s understanding and seeks to resolve them.” A historic example underscores this opinion. Deeply religious Victorian society in the late 1800s was shocked by Darwin’s suggestion that humans and animals shared common ancestry. His “non-religious biology” asked some vital questions about the origin of the species and revealed  new, big ideas. Apparently, Charles Darwin was a prescient forecaster for Firestein. Long ago Darwin said: “Ignorance more frequently begets confidence than does knowledge.”

Identify what you don’t know. Be willing to ask great questions; vigorously pursue discovery. These attitudes and practices yield improvements and change.  

Lisa Wyatt, Ed. D. is chief strategy officer and partner in Phillip Wyatt Knowlton, Inc. PWK is a performance management resource for systems change with clients worldwide. Lisa has cross-sector and international experience. She is an author and W.K. Kellogg Leadership Fellow. See :

New: The Logic  Model Guidebook (2013) just published by SAGE.

Break Throughs Take Will and Capacity

June 17, 2012

Yiannis Boutaris, 70, a successful Greek winemaker has been mayor of Thessaloniki for 18 months. Thessaloniki, Greece, is a sprawling city of 800,000 people on the Aegean Sea. It is second in size only to Athens.

Bankrupt & Corrupt

Boutaris inherited a city on the brink of bankruptcy (nearly $130 million in debt), with outdated laws and regulations, corruption, manufacturing decline and few tourists. His predecessor and 17 colleagues have been indicted – accused of stealing about $38 million.

Under these conditions, pundits gave the new mayor little prospect for success. Bloated municipal employment, inadequate basic services, discord with Turkey, and tangled regulations were all accepted as “normal.” However, this wiry septuagenarian who sports a pierced earring and frequently punctuates his point of view with profanity, knows two fundamental factors vital for change: will and capacity.

Will & Capacity

Will is the practical and political determination to persist. Will endures both obstacles and critics. It prevails. Boutaris advises: “When you propose the slightest change, people say no. If you do it all at once, it is a different thing. Something has to break through.” He adds: “You cannot step back, if you step back you lose.”

Capacity is about the strategic management acumen to make smart, hard choices that enable performance. Capacity reflects knowledge, skills, training and experience. Boutaris has made unpopular but effective decisions about budgets, employees, public policy and external relations. He is changing practices with a focus on different and better.

Whether a community, an organization or individual, will and capacity are requisites for change to occur.

Boutaris is undeterred in his reforms. (For more, see NYT Saturday Profile.) To date he has begun recycling programs, resumed relations with Turkey, grown tourism, and instituted unheard of practices at City Hall: job descriptions, goals and evaluations. He has cut city costs by 30%. One man insists on making his hometown a place of progress and growth.

I bet it happens.

Lisa Wyatt, Ed. D. is chief strategy officer and partner in Phillip Wyatt Knowlton, Inc. PWK is a performance management resource for systems change with clients worldwide. Lisa has cross-sector and international experience. She is an author and W.K. Kellogg Leadership Fellow. See :

Staking A Claim

March 15, 2011

We need to “move the needle. “ It’s one of the most common clichés used to describe the need to generate change. It might be the best one but there are others, like: “bring home the bacon”, “beat the street”, “go gang-busters.” These translate to “help me show results.” They are about performance. As frequently said but not-so-original or interesting phrases they are often delivered with a tone of slight desperation as people try to validate their work.

Precious Feedback
The private sector learned a long time ago to incorporate metrics in their day-to-day work. Metrics and their use are also a key part of formal training that prepares you for the “world of business.” Sales, price–to-earnings, indexes, customer satisfaction scores, debt ratios, profit margins, projections, and efficiencies are all aspects central to performance in the private sector. They offer both guidance on progress as well as terminal performance ratings. They help us describe current status, aspirations and results. Metrics are essential feedback in operations, acquisitions, mergers and value. This is because they are integral factors in managing.

Many of the debates about the use, skills with and request for metrics in the nonprofit and philanthropic sector are puzzling. It’s vital, for sure, to recognize that social change is not a “controlled” space like a drug trial or cookie factory. In fact, I think the complexities of social change suggest an even greater value for the critical feedback metrics provide in managing and leading. Both evaluative thinking and evaluation are essential to all sectors.

Claims and Cause
While those who declare “move the needle” may not know it… the needle will move. Regardless of their efforts or lack of. Change is constant. There are external and internal dynamics that absolutely will influence the gauge reading or needle status. Frequently, when this phrase is used to inform/define an organization or program’s work, the construct in play is cause. It’s an intention to declare value or stake a claim. Can I declare contribution or attribution, neither or both?

The difference in contribution and attribution are relative proportion of cause. I hit the car when I ran a red light is an example of direct cause. The event is attributed to me. I gave $500 to the $4.8M capital campaign goal deems me a contributor. I am a small part of a big result. When seeking attribution it means particular protocols must be used to determine the portion of effect. It is a high standard.

For example, in determining the influence of a pre-literacy intervention for preschool children we compared children and teachers in Head Start with a Head Start control group. Those with the special intervention and preparation exceeded learning and performance measures many times greater than the control group. This allows the intervention program, with significant confidence, to claim attribution (or direct cause).

Determining contribution is not as difficult and it is more common. The role or sequence of factors in an intervention can be essential to understanding contribution. The strength of a plausible connection allows us to claim contribution. It is a part of cause. Specification of contribution should rely on multiple techniques. Models, plans and other items are useful tools in this determination.

Metrics Application
It is vital that independent and quality assessment occurs to assure credibility. There are big challenges in understanding the use and mis-use of evaluation. Evaluation is most certainly affected by politics. If not high quality, the metrics represented by evaluation can easily be “cover” or marketing. Capable professionals know this is why the field has quality standards. They also can spot shoddy quality, marketing and promotional or “lite” use of evaluation. People have anxiety about metrics because they are useful with accountability.

Individually and in our organizations, we all want to “make a difference.” We want our lives and our organizations to be relevant, have meaning and credibility. Whether the sole cause or a contributor, it’s vital to use metrics to set targets, review progress and determine influence on results. This makes evaluation an essential literacy for anyone managing and leading. The dangers and costs of mishandling are extraordinary. When done well, so is the value.

-Lisa Wyatt Knowlton, Ed. D. is a strategy architect and partner in Phillip Wyatt Knowlton, Inc. PWK is a performance management resource for systems and social change with clients worldwide. She is also a W.K. Kellogg Leadership Fellow. Contact her via:

Look for Hay in your Haystack

October 6, 2010

Every day you and your colleagues experience an “infolanche.” Quite simply an avalanche of information pours in to your desk through the phone and computer hour after hour each day.  Skillful navigation of this overload is critical to progress. Digging out is a big job. This deluge creates yet another challenge to managing and leading.

What’s the right information for your attention?

Focus, Focus, Focus

Our resident psychometrician often suggests that clients “look for hay in your haystack.” Before  launching a hurried search begin with careful thought. In a recent preschool literacy evaluation we conducted the standard for performance was a score of 62. One of the exciting programs we assessed scored 58. Did it fail? By the explicit federal standard it did. By any other measure, it was a huge success. How could that be?

Before the literacy intervention, children entering the program were tested by several valid and reliable tools. After a school year of experience with the program, children were tested again. They showed substantial gains, in fact, statistically significant changes in pre-literacy awareness, knowledge and skills. The program was effective but risked discard because a pre-determined value was not secured. Moreover, by analyzing student gains by teacher, it became obvious which teachers had delivered a high quality “dose” with fidelity to program design. This provided the clues on which teacher’s could improve and how they needed to adapt their instruction practice.

Great Questions Matter

Asking great questions is the first most important step sorting your haystack. Great questions guide data navigation towards high value information. Specification of your information needs can focus your data collection, analysis and interpretation.  How you frame your inquiry matters lots. In the preceding example, the right question was: Did the program positively affect literacy skills? It would be an error to “quit early” and simple ask: Did we make the standard score?

Data-driven decisions are the new daily bread. There’s plenty of information to use and confuse  us. The next time you review data, recognize what vital clues it provides for the challenges your organization faces . If you are clear about what questions must be explored, it will help you sort your pile.

Lisa Wyatt Knowlton, Ed. D. is a strategy architect and partner in Phillip Wyatt Knowlton, Inc. PWK is a performance management resource for systems and social change with clients worldwide. She is also a W.K. Kellogg Leadership Fellow. For more information, see :

%d bloggers like this: