Grant Evaluation

Building Community Capacity for Participation in Evaluation

council-webteam — Tue, 19 Nov 2013 20:08:12 +0000

Building Community Capacity for Participation in Evaluation council-webteam Tue, 11/19/2013 - 15:08

Building Community Capacity for Participation in Evaluation

From Grantmakers for Effective Organizations (GEO), what does it mean to truly and authentically engage a community in evaluation? Grantmakers working on place-based grantmaking already have the incentive and drive to support in-depth evaluation, to understand the impact of the initiative, and to identify opportunities to improve their work. Effective evaluation of these complex and multi-faceted efforts is grounded in the perspectives of community stakeholders.

Go to External Resource

Building Community Capacity for Participation in Evaluation

Questions?

Connect with Council Staff

Related Resources

Sample Post-Grant Self-Evaluation

council-webteam — Tue, 19 Nov 2013 18:16:52 +0000

Sample Post-Grant Self-Evaluation council-webteam Tue, 11/19/2013 - 13:16

Sample Post-Grant Self-Evaluation

This sample post-grant evaluation can be customized to your foundation and provided to your grantees to submit at the conclusion of their grant.

Download Document

Sample Post-Grant Self-Evaluation

Questions?

Connect with Council Staff

Related Resources

Related Events

Knowing How and When to Monitor, Assess, and Evaluate Grants

council-webteam — Tue, 19 Nov 2013 17:52:37 +0000

Knowing How and When to Monitor, Assess, and Evaluate Grants council-webteam Tue, 11/19/2013 - 12:52

Knowing How and When to Monitor, Assess, and Evaluate Grants

A family foundation's legal responsibilities for monitoring, assessing, or evaluating the grants it makes to organizations with 501(c)(3) tax exempt status are minimal. The IRS requires little in the way of detailed reporting on the outcome of specific grants—except for grants to organizations that lack 501(c)(3) status. Even without required reporting requirements, however, most foundations ask their grantees for a final report that gives a financial accounting of how funds were used, a narrative description of any accomplishments of the project or program, and copies of any media coverage of the project. For small grants, this type of self-evaluation is generally considered sufficient.

Alfred L. Castle of the Samuel N. and Mary Castle Foundation in Honolulu, Hawaii (assets of $40 million) offers the following advice on how small foundations can get grantees to prepare self-evaluations:

"Trustees of small foundations who would like to have an inexpensive self-evaluation completed by a grantee should…let grant applicants know at the time they apply that this is a condition of receiving the grant. It also helps to send them a copy of the questions at the onset of the grant. Where possible, the grantee should play a role in developing the questions. This process works best when approached as a collaborative effort, since all foundations are partners of nonprofits in achieving mutual goals."

Beyond self-evaluation, many choices and styles of grants evaluation are available. Some family foundations opt to use their in-house personnel, while others rely on hired consultants to conduct the analysis. Some choose to conduct their analysis on the basis of interviews and written responses to questions, others conduct site visits while the grant is in progress or at its conclusion, and still others employ a combination of these methods. In the case of larger grants, foundations sometimes decide to pay for a more objective analysis, which may mean an investigation using all the rigorous tools of modern social science and statistics. Others take a more relaxed approach, using a consultant to interview program participants and staff and to draw conclusions with the detachment of an outside observer.

William J. O'Neill, Jr. and the William J. and Dorothy K. O'Neill Foundation of Cleveland, Ohio (assets of $5.2 million), require different levels of evaluation, depending on the size of the grant and its importance to the foundation's strategy. For "small" grants of less than $5,000, no evaluation is required. For mid-size grants (up to $15,000), self-evaluation by the grantee is required. For larger grants, a site visit by a foundation representative is conducted to verify the self-evaluation. Finally, its current "focus area project"—a project to which the foundation has awarded more than $200,000 for two years—the foundation collaborated with another area family foundation to include the services of an independent evaluation specialist in the project. Of course, the detailed evaluation of the focus area project is intended not only as verification but also as part of the overall strategy to learn from a model project and to disseminate important findings.

Often, outside assistance for evaluations can be obtained through collaboration with other grantmakers. Such a collaboration might establish one evaluation process for a project, even though the project is receiving grants from many different foundations. In addition, it may be possible to borrow someone with evaluation skills from a community foundation or larger private foundation at less cost than tapping the open consultant market.

Approaches to Monitoring and Evaluating Grants

Most family foundations with assets of less than $10 million are managed by trustees and family members without the support of a large administrative budget or professional staff. Trustees of these foundations often assume that monitoring and evaluating a grant is difficult and expensive. As a result, they choose to fund only those organizations with which they are familiar. This assumption is incorrect: monitoring and evaluating grants can be accomplished relatively easily and cost-effectively.

Defining Monitoring and Evaluation. Grants monitoring and evaluation attempt to determine the worth or effectiveness of grants based on the grantmaker's criteria. Family foundations of every size can benefit from these activities; they are essential to an ongoing process of self-development and "course correction."

Monitoring is the process of ensuring that a grantee fulfills certain commitments during the course of a grant—files reports on schedule, allocates grant money to the purposes designated in the grant agreement, and carries out promised activities. Orderly monitoring can be very important in detecting management or fiscal accountability problems before they get out of hand. Monitoring is no substitute, however, for the actual assessment of results, which is evaluation.

IRS regulations do not require grantmakers to monitor grants or other activities. Grantmakers can ask for appropriate IRS documentation from the grantee and then monitor the grantee to determine whether the funds were spent according to the grant agreement and possibly whether a project or program has been completed. If time permits, a brief site visit by a board member can also provide useful information.

Evaluation has been defined as "the systematic collection of information about the activities, characteristics, and outcomes of programs for use by specific people to reduce uncertainties, improve effectiveness, and make decisions with regard to what those programs are doing and affecting."

Evaluation goes beyond monitoring in that it tells trustees how well their grants have been implemented and to what extent the foundation's mission is being fulfilled. Some smaller family foundations find it useful to use evaluation to determine community needs or the impact of grants. The key point is that evaluation can be a useful tool for those who wish to use it, and it is a tool that can be employed in greater or lesser degrees as the foundation governors and managers see fit.

There are, technically, six types of evaluation:

Needs Assessment: Describes the extent of a social or other problem, identifies targets of a program and helps design a new program, or justifies program continuation
Monitoring: Provides periodic information to the foundation on whether a grantee is implementing a grant as planned, identifies problems, and facilitates timely a resolution
Formative: Provides information to the foundation and the grantee organization that can improve the project or program as it is being implemented
Process: Is intended to help the grantor and grantee understand how the program made or failed to make an impact and how it might be replicated
Impact or Outcome: Measures program effectiveness in producing desired change
Summative: Assesses the overall equality and effectiveness of programs with respect to accountability and decision making; typically conducted after the program is terminated

Information from evaluations can help a family foundation board to:

Ensure that the foundation is accountable as a public trust
Improve the foundation's grantmaking abilities
Increase the effectiveness of funded projects
Plan and implement new programs

Techniques for Conducting Evaluations. Small foundations can rely on grantee self-evaluations or make use of an experienced consultant or a graduate student at a local university to conduct evaluations. They can also perform site visits. Each of these approaches is discussed briefly below.

Grantee Self-Evaluation.The most common form of grant evaluation employed by smaller foundations is the grantee self-evaluation. One family-managed foundation with assets of $2.5 million, for example, requires that each grantee sign a contract stating that it will provide a narrative evaluation of its project at the end of twelve months.

Grantees are sometimes reluctant to admit to their project's weaknesses, or may be unfamiliar with the self-evaluation process and thus unsure of what to report. Nevertheless, when a relationship of trust between the foundation and grantee has developed, grantees almost always appreciate participation in the evaluation process, and are more likely to accept and implement findings.

To help grantees with the self-evaluation process, foundations might ask that they answer the following five questions:

What were the lasting benefits of the project?
Were goals set in the grant application fulfilled?
What problems arose during the project?
What are your specific plans, if any, for continuing the work started by this project?
Do you have any comments, suggestions, or criticisms about working with the foundation staff?

Other helpful information to request includes copies of press releases, media stories, or other materials published regarding the funded program. Such information is very useful in preparing an annual report or other communications.

Outside Assistance. Consultants can bring objectivity, autonomy, and credibility to grant evaluations. Family foundations sometimes locate consultants by checking with staff of a larger private foundation or a community foundation or through peer networks.

Foundations with limited funds for part-time consultants might consider these means for getting an external assessment at the lowest cost:

Undertake joint evaluations and share costs with other small grantmakers funding in the same field
Ask a larger, staffed foundation to lend expertise
Build the costs of evaluation into the grant and have the evaluator work for the grantee. This is the least troublesome form of external evaluation for a grantee, although it may raise questions for trustees about objectivity

Trustee Site Visits. Site visits, conducted after a grant has begun, are relatively inexpensive and allow funders to see grantees and their grant dollars at work. During these site visits, foundation trustees might ask grantee personnel such questions as:

What differences exist between the way the organization planned and actually implemented the project? What do the differences mean to the staff, project, population served, and organization?
What were the project's results, for the people served and for the organization? Did the project accomplish its goals? What significant internal and external factors affected outcomes?
What unanticipated benefits or problems arose?
How could the foundation have been more helpful?
How will the project results be disseminated?

In addition, trustees might want to consider asking the intended beneficiaries of the foundation grant:

What difference did the foundation's support of the program make in their lives?
What were some of the benefits and problems of implementing the project?
How would they, the grantee, improve the project?

As William S. White, president and CEO of the Charles Stewart Mott Foundation in Flint, Michigan, notes, the bottom line for any evaluation is to help a foundation ensure that its objectives are on target and that its programs are meeting those objectives:

"Evaluation is a continuous process, one that should provide useful information to decision makers both inside and outside the organization. The single most important feature of any evaluative system is its ability to prompt learning. And it is in the process of reflection and dialogue about what is learned that a foundation can apply lessons in order to improve its service."

Tips for Making Evaluations

Drawing on several sources in the field, here are some tips from experienced grantmakers:

Review grantmaking guidelines to ensure that grantseekers understand the importance of evaluation, build reporting procedures into their proposal and project administration, and provide the information that the grantmaker wants
Encourage grantees to monitor and evaluate their work
Select for evaluation only those projects that have long-term implications for the foundation's grantmaking. Start early, set the wheels for evaluation in motion, bring grantees into the planning process, and budget sufficient funds for a proper job of evaluation
Join with other funders in the state or region to exchange information about what works and what doesn't
Track the ways in which the evaluation is used

What Foundations Are Really Doing: Two Surveys

In the past few years, the Council on Foundations on its own, and the University of Pittsburgh, in collaboration with the Lilly Endowment, have sought to determine what foundations are doing to evaluate the performance of their grantees. Results of their two surveys provide insights into the realities of grantmaker evaluation practices.

Foundation Management Survey. The 1994 Foundation Management Survey was the first survey in which the Council on Foundations explored grantmakers' evaluation practices in some depth. The questions were intended to determine the variety of program and self-evaluation practices among the different types and sizes of grantmaking organizations: what kinds of grants are evaluated, how evaluations are conducted, and how the findings are used. Selected excerpts of the analysis:

Types of Evaluation Performed or Funded. Most respondents (93 percent or 498) indicated that they evaluate the projects or programs they fund; about one-half said that they conduct self-evaluations and fewer than one-in-five support evaluations of organizations or programs funded by others.

National Survey on Role of Evaluation. The second study is called "A National Survey of Small and Mid-Size Foundations: The Role of Evaluation," and was published in July 1996. The Lilly Endowment supported the survey as part of the Endowment's partnership effort with the University of Pittsburgh to enhance its capacity to evaluate its grantmaking in education. An additional goal has been to contribute to what is known about evaluation use in foundations.

The survey queried 238 members of the Council on Foundations, all of which had assets below $250 million and 25 percent of which had assets below $10 million. Family foundations were not identified as a distinct category. The authors developed twelve findings from their data:

The authors found that evaluation "clearly centers on grantees," which, after foundation personnel themselves, are the most likely evaluators. The authors concluded that:

This emphasis on self-evaluation has mixed results from the perspectives of foundation personnel. There are concerns about the grantees technical adequacy and the natural tendency to be less than candid. However, the grantees involvement in evaluation reflects the notion of the 'reflective practitioner" that is considered a best practice in the literature on effective organizations.

Kinds of Grants Evaluated. Three-quarters of respondents indicated that they evaluate grants made to a new organization or an organization that they (the grantmaker) are funding for the first time. About 60 percent evaluate grants above a certain amount, and an equal proportion indicated that they evaluate grants made outside of typical program areas (such as special initiatives).
How Evaluations Are Conducted. For 80 percent of respondents (424 of 527), evaluation is handled by staff or board members; 179 of the 424 use this approach exclusively. The second most frequent approach was to include evaluation as part of grant activities; that is, funded through the grant and carried out by or under the direction of the grantee. Grantmakers were less likely to have evaluations implemented by outside consultants under contract to the grantmaker (22 percent of the 527), and even fewer reported having an evaluation supported through separate grants (11 percent).
How Evaluation Findings Are Used. Most respondents (90 percent or 496 of 522) use evaluation findings to assist them in making future funding decisions or to measure program effectiveness (82 percent). About half of the respondents used evaluation findings to capture program results or to demonstrate the strategic value of a program. Few (12 percent) indicated that they release evaluation findings to media and/or general public.
- Most small and mid-size foundations do evaluate as part of their standard operating practices. Much of this evaluation takes the form of grant monitoring by foundation staff and grantee self-reports.
- Small and mid-size foundations fund evaluation through administrative/operating budgets and as a component of grant activities with greater frequency than through grants to external evaluators.
- Thirty-one percent of responding small and mid-size foundations conducted at least one external evaluation in the past three years. Five foundations conducted fifteen or more external evaluations in the past three years.
- Most small and mid-size foundations do not trace dollars expended on evaluation.
- There is no meaningful trend for recent years in the number of evaluations being funded by the responding small and mid-size foundations.
- Small and mid-size foundations make the decision to conduct external evaluations usually for the purpose of gaining new knowledge. The controversial nature of a grant or program context does not appear to increase the likelihood that foundations will use an external evaluator.
- The evaluation process tends to be the responsibility of the program officer of record in the foundations surveyed. There is no widespread use of "evaluation directors" and/or staff dedicated to evaluation efforts in these foundations.
- Evaluation results are typically shared more often with audiences within the foundation.
- Evaluation is most often used for the purposes of holding grantees accountable, assessing program impact, and informing strategic thinking in foundations.
- Small and mid-size foundations do not often use evaluation to hold themselves accountable to the public.
- Small and mid-size foundations find evaluations unhelpful when specific qualities in the evaluation are useless (as in the collection of data simply for the sake of research) and limitations inherent when the grantee is the source of evaluation information (such as that consisting wholly of anecdotal evidence).
- The most commonly cited barrier to conducting evaluations among respondents was a lack of resources—notably, staff to oversee or conduct evaluation, staff time to commit to evaluation, and dollars to support evaluation activity.

Questions?

Connect with Council Staff

Grant Evaluation

Related Events

Related Resources

Getting Started with Evaluation

council-webteam — Sat, 16 Nov 2013 21:32:25 +0000

Getting Started with Evaluation council-webteam Sat, 11/16/2013 - 16:32

Getting Started with Evaluation

As different as foundations can be from one another, they all share the need to know what works and, especially, what works well. The more foundations can show how their grants are making a difference, the more value they can bring to their communities.

To know what works, foundations must evaluate their grants. Evaluation has many benefits. It helps the foundation assess the quality or impact of funded programs, plan and implement new programs, make future grant decisions, and demonstrate accountability to the public trust.

But how do foundations evaluate what works? There are many possible approaches, ranging from informal, anecdotal methods to full-fledged program studies. With so many options, the choice may seem overwhelming. Yet, evaluation doesn’t have to be as complicated as it sounds. As one expert put it, “Evaluation is the process of asking good questions, gathering information to answer them, and making decisions based on the answers.”

Does this sound simple? It can be. Beyond the fancy terminology and debates over whether qualitative or quantitative assessments are better, evaluation boils down to things you already do every day: Ask questions and get answers.

Evaluation Types—What You Want to Know

There are, of course, different ways to ask questions and different reasons for asking. Let’s take a look at some common types of evaluation and what purpose they serve. Some evaluations are categorized according to when they are conducted. For example:

Formative evaluations provide information about how well a project is designed, planned, and implemented, with the goal of improving and refining a developing program. Grantmakers typically look at process (what happened), impact (what changed), or both, giving feedback to newer programs.

Participatory evaluations involve an evaluator working with a grantee's staff and participants. The purpose is to empower grantees by involving them in generating data for the decision-making process. Instead of waiting until the grant ends, the evaluator collects the data throughout the grant period, giving the grantee the opportunity to adjust the project along the way.

Summative evaluations provide summary information about the outcomes of a project. They involve judgments of cause and effect, in particular the effectiveness and worth of a project’s activities in solving specific problems. Summative evaluations are usually conducted for accountability and decision-making purposes.

Other evaluations are determined by the kind of data being collected:

Process evaluations determine if a program is reaching the desired target group and if it is being executed in the way it was planned. It’s common to use an evaluation form to collect this data.

Outcome evaluations (sometimes called impact evaluations) assess how a program produced change. Rather than focus on the process (what did we do), they focus on the outcomes (what changed as a result). Usually, the evaluation focuses on harder to quantify questions, such as how the program made a difference in people’s lives. Choose this type of evaluation when you want to demonstrate how well grantees met their objectives, highlight an innovative model or program, or evaluate your overall grantmaking program.

Evaluation Methods—How to Collect the Data

Now that you know a little more about different types of evaluation, let’s take a look at the actual activities that evaluations involve. Some primary methods to choose from include:

Grantee self-evaluation is a do-it-yourself model that works well and requires limited resources, although some see it as little more than grant monitoring. At the end or midway point of a grant, foundations ask grantees to fill out a self-evaluation form. Sometimes the forms are followed by an interview with a foundation staff member. This method can be a good way to build evaluation skills within grantee organizations.

Case studies are used to collect in-depth information about a program through a single participant or site. For example, it might be the story of one person’s experience with a program.

Expert peer review relies solely on judgment-based information. The assigned evaluator may be an individual; an advisory committee of experts; or community members who read documents, conduct site visits, and interview grantees. This method can usually be completed quickly and inexpensively. However, the evaluation depends on the knowledge, experience, and opinions of those chosen, so this method can be biased.

Focus groups encourage people to share their feelings and perceptions. Although led by a facilitator, these conversations often flow naturally and explore unexpected or unintended areas. Evaluators can qualitatively analyze areas in which participants’ responses converge, as well as the tone or feeling that accompanies those responses.

The logic model method follows a linear, logical format that breaks down a project into various components and tracks it from beginning to end. The logic method uses a specific set of words to describe what the project intends to achieve; the resources and processes used to conduct the work; and the immediate, mid-term, and long-term results. This form of evaluation can be complex and requires training.

Descriptive analysis uses statistics to characterize a program; its participants; and the relevant social, political, or economic environment to understand how and why a program works.

Comparison-group design is used to measure effects and attribute them to a project or program. For example, a group of people who receive an innovative treatment or participate in a new program are compared to a group of people who don’t. The study measures the difference between the two groups and attributes those differences to the treatment or program.

Frequently Asked Questions

What is the difference between grant monitoring and grant evaluation?

Grant monitoring requires grantees to set goals and periodically report back to grantmakers on their success or failure. It also requires grantees to perform financial reporting. Grantees submit progress and/or final reports to the foundation, which helps keep them accountable to the project goals as well as the budget. While all grants should be monitored, not every grant is worth evaluating. For projects that receive small amounts of funding, monitoring is the most reasonable expectation. Here’s an example: A foundation asks a grantee to fill out a self-evaluation form describing his progress relative to the goals determined in the proposal and the grant agreement and how the money is being spent compared to budget.

Grant evaluation takes a more in-depth look at the program’s accomplishments, helping the foundation learn what happened because of the grant—how it made a difference. An example of evaluation: A foundation invites grantees and community members to a focus group where it collects feedback on how the grant or program made a change in the community. From the information collected, the foundation writes a report, has it reviewed by an expert evaluator for key findings, and uses the information to guide future grant decisions.

What should we think about when considering an evaluation?

There usually is more than one acceptable way to evaluate a given grant, project, or program; however, it is difficult to decide which direction to choose. It’s important to think through what you want out of the evaluation up front. The form it takes—and the results it yields—will depend on the choices you make regarding the:

Purpose of the evaluation
Target audience for the evaluation
Questions to be answered
Evaluation methods to be used
Qualifications and experience of the person or group conducting the evaluation
Cost of the evaluation relative to the grant’s size

What is the most common and simple way to evaluate grantees?

Most foundations ask grantees to fill out an evaluation form, either midway through a grant period or at the end, as a form of grant monitoring. This is a simple, cost-effective method for gathering data—plus it introduces grantees to the concept of accountability if they aren’t already familiar with it. One difference between grant monitoring and grant evaluation is what you as a foundation decide to do with the data you collect. You are monitoring the grant if you use the form to check how the money was spent. If you use the data to assess how the grant actually made a difference, you are moving toward evaluation.

One potential downside to self-evaluation forms is that grantees may be reluctant to admit their project’s weaknesses or may be unsure what to report. Nevertheless, when there is a relationship of trust between the grantee and foundation, grantees almost always appreciate participating in the evaluation process and are likely to accept and implement findings.

Keep in mind that not all of your grants will require the same level of evaluation, and your criteria will vary from grant to grant. Some foundations don’t ask for evaluation forms for grants below a certain dollar level.

What should a grantee evaluation form look like?

What the actual form looks like doesn’t matter. You can choose from dozens of samples or design your own. What does matter is that you are asking the right questions and tracking the answers. The key is to define—or have the grantee define—what determines “success” for a particular grant or initiative and follow up from there.

Ask grantees up front: What is the change that you want to see happen as a result of the project? How will you know if it happens?

Here’s some advice from your foundation colleagues:

“On the evaluation form, we copy what they wrote in their grant proposal, telling them: ‘This is what you said you would do, this is what you said would take place, and this is how you said you would measure it.’ Then we ask them to write a narrative comparing the project against what they said would happen.”
“Be sure to ask grantees about the pitfalls they encountered as well as the successes. Remind them that the ‘negative’ information is just as valuable as the positive.”
“If foundations have specific expectations of grantees for evaluation or monitoring, they should be cognizant that there may be a cost associated with the activity and be prepared to pay for it as part of the grant.”

A good rule for evaluation forms is to keep it simple. The fewer—but more pertinent—questions you ask grantees, the more effective they will be in giving you answers.

When should we conduct a formal evaluation?

Evaluations can be simple or elaborate, depending on the size and scope of the grant as well as on your foundation’s goals and capacity. In simple evaluations, foundations ask grantees for a final report and perhaps interview the grantee or their clients. If a foundation has the resources, it might conduct a formal evaluation— particularly if the grant is part of a large-scale initiative.

Formal evaluations can be costly and time consuming, however. According to one individual, the only time a small or place-based foundation should do a formal evaluation is for a large, multiyear strategic grant—such as a major grant to a nonprofit trying to bring about social change.

Why should you bother with formal evaluations at other times? “Evaluation is hard work, and the results can be arbitrary,” said a foundation program director, “You have to be committed at least one, two, or even three years before you are going to see real outcomes from a grant.”

When considering when to conduct a formal evaluation (or if it’s even appropriate for your foundation), it helps to set an evaluation policy to guide you. For example, some smaller foundations only evaluate major initiatives or grantees that received more than a certain dollar amount.

Who conducts an evaluation?

The board establishes policy on how the foundation will conduct its evaluations and who will be responsible for the task, such as the staff, the board, community volunteers, or an outside evaluator. For simple evaluations, the staff will typically be in charge. If you have the resources, you might work with an outside evaluator who can help guide the process. For larger initiatives, you might contract an outside expert to manage the whole project.

Outside evaluators can be objective and may have technical skills and resources that you don’t have in-house. However, they can be costly. Fees will vary based on the type and scope of evaluation, but expect to pay between $10,000 and $100,000. Check with your local university research department or ask your foundation colleagues for names of consultants, nonprofit research institutes, or experts in your field of funding.

What data can we collect in an evaluation?

The kind of data you collect depends on the type of project and the purpose of your evaluation. For some grants, you might look at quantitative data on how many people were served and how that made a difference. With others, you might examine the qualitative results of the grant, answering questions such as, “How do clients feel about their circumstances because of the grant?” In any case, criteria should relate to the grant or program purpose and should be defined at the grant’s onset. (For grant monitoring purposes, you will look at the grant amount given and how closely the grantee adhered to the proposed budget.)

Technology makes it much easier today for foundations to evaluate grants not only locally but regionally and even nationally. Foundations can now use online surveys and data tabulation tools, among other resources, making evaluation more accessible and immediate and requiring less labor and costs than in the past.

What do we do with the data once we get it?

Again, this will depend on the type and purpose of the evaluation, the audience, and your own mission and goals. With any evaluation, though, you will use the data to determine if the community foundation’s investment (for example, funding) was worth it, and if so, how. Here are some examples of how you can use the data:

Create a grant summary in which you report highlights from the grant: what was awarded, what it accomplished, what was successful about it, and what could be improved.
Report the evaluation results to the board to help inform future grant decisions.
Include the grant summary in newsletters to donors and the community.
Include the grant summary on your website, in your e-newsletters, and in your annual report.
Create and distribute a news release announcing the grant or initiative results.
Give the data to an outside expert to draw conclusions.
Commission a research paper based on the evaluation results.
Share the evaluation process and results (warts and all) with the community foundation field; sometimes, admitting where you went wrong is the most valuable information.
Assess how cost effective it was for your foundation to conduct the evaluation.

How can we evaluate the impact of our entire grantmaking program on our community?

It used to be that community foundations evaluated only grants—if they evaluated their work at all. Now, with changing roles and increased accountability, many want to show how all of their activities make an impact—their programs, leadership, services, and organizational effectiveness. But how should they go about doing it?

Even large private foundations with greater resources struggle with this question. The short answer is: It simply can’t be done—or rather, it can’t be done simply.

According to one community foundation colleague, “A better approach for community foundations may be to redefine ‘impact.’ Rather than focusing on grantmaking effectiveness, it’s more reasonable and useful to focus on mission effectiveness. In other words, instead of asking what difference you are making… the question becomes how effective[ly] you are meeting the intent of your own mission and purpose.

“Shifting to a mission focus will force you and your board to recognize that you simply can’t meet every need in your community to the degree you would hope. Ultimately, it’s about knowing your community foundation’s place and recognizing your limitations as well as your strengths.”

Other small foundations emphasized the importance of knowing what you can (and cannot) handle. “There are so many levels of evaluation—it all depends on the capacity of your foundation as far as what you can take on. You simply can’t evaluate everything at once; it’s important to prioritize. Start by evaluating individual grants and work your way up from there.”

Ask Yourself

When developing an evaluation process, consider these questions:

What do you want to know one, two, three, or five years from now that you don’t know today?
How do you get the board’s commitment to the evaluation process?
How do you know you have the in-house capacity to conduct (or oversee) an evaluation?
What is your budget for such an endeavor?
Which projects are the most important to measure?
What are the three most important things you want to learn from a particular grant/initiative?
With whom will you share your evaluation findings?
How should you receive evaluation results?
How will you use the information from the evaluation?
How can you involve grantees in the evaluation process?

Resources

Evaluation Techniques: A Series of Brief Guides. Each guide explains the basics of one evaluation technique and how some grantmakers are applying it. Visit www.grantcraft.org.

Foundations and Evaluation: Contexts and Practices for Effective Philanthropy. JosseyBass, 2004. A good read for both newcomers to evaluation and those with more experience. Visit www.josseybass.com.

Grantmakers for Effective Organizations. GEO’s mission is to maximize philanthropy’s impact by advancing the effectiveness of grantmakers and their grantees. Visit www.geofunders.org.

Learning Together: Collaborative Inquiry among Grant Makers and Grantees. This guide explores an increasingly popular method called “collaborative inquiry.” Grantmakers define the practice, consider potential benefits, and grapple with common challenges. Visit www.grantcraft.org.

W.K. Kellogg Foundation Evaluation Handbook. This book provides a framework for thinking about evaluation as a relevant and useful program tool. Visit www.wkkf.org.

Questions?

Connect with Council Staff

Grant Evaluation

Related Events

Related Resources

Grant Evaluation Approaches and Methods

council-webteam — Sat, 16 Nov 2013 20:49:05 +0000

Grant Evaluation Approaches and Methods council-webteam Sat, 11/16/2013 - 15:49

Grant Evaluation Approaches and Methods

One of the greatest challenges encountered in thinking about evaluation is that there usually is more than one acceptable way to evaluate a given grant, project, or program.

The form that an evaluation takes and the products that it yields will depend on choices made about the following issues:

The purpose of the evaluation
The target audience for the evaluation
The evaluation questions to be answered
The evaluation methods to be used
The qualifications and experience of the person or group conducting the evaluation
How much to spend

Purposes and Priorities

A foundation must identify its greatest needs and the ways it, or others, will use evaluation results, and then use this information to set evaluation priorities. Most foundations would probably assign a high priority to accountability and to improving their abilities as grantmakers. Beyond this, the reasons for supporting evaluation will depend on the foundation's own goals and programs.

The Audience

Possible audiences for evaluation findings include foundation trustees and staff, grantees, other program funders, policymakers, the field at large, and the general public. When deciding whether and how an evaluation ought to be undertaken, a foundation should also decide upon its desired audience for the evaluation findings. The information needs of various groups can be very different, and the costs of obtaining certain kinds of information—notably, the kinds of information obtained by scientifically rigorous evaluation—can be high.

Evaluation Questions

Evaluations are distinguished by the nature of the questions they attempt to answer. Many evaluation projects will involve more than one of these characteristics.

Needs Assessments. These evaluations verify and map the extent of a problem. They answer questions about the number and characteristics of the targeted institutions or individuals. Needs assessments can help design a new program or justify continuation of an existing program.

Monitoring. These activities produce regular, ongoing information that answers questions about whether a program or project is being implemented as planned. They also identify problems and facilitate their resolution in a timely way.

Formative Evaluations. These evaluations answer questions about how to improve and refine a developing or ongoing program. Formative evaluation usually is undertaken during the initial, or design, phase of a project. However, it also can be helpful for assessing the ongoing activities of an established program. Formative evaluation may include process and impact studies. Typically, the findings from formative evaluations are provided as feedback to the programs evaluated.

Process Evaluations. Studies of this kind are directed toward understanding and documenting program implementation. They answer questions about the types and quantities of services delivered, the beneficiaries of those services, the resources used to deliver the services, the practical problems encountered, and the ways such problems were resolved. Information from process evaluations is useful for understanding how program impact and outcome were achieved and for program replication. Process evaluations are usually undertaken for innovative service delivery model projects, where the technology and the feasibility of implementation are not well known in advance.

Impact or Outcome Evaluations. These evaluations assess the effectiveness of a program in producing change. They focus on the difficult questions of what happened to program participants and how much of a difference the program made. Impact or outcome evaluations are undertaken when it is important to know how well a program’s objectives are being met, or when a program is an innovative model whose effectiveness has not yet been demonstrated.

Summative Evaluations. Summative evaluations answer questions about program quality and impact for the purposes of accountability and decision making. They are conducted at the conclusion of a project or program and usually include a synthesis of process and impact or outcome evaluation components.

Evaluation Methods

Some primary evaluation methods presented in the context of evaluation choices:

Expert peer reviews associated with judgment-based information. The peer may be an individual or a committee. The review may consist of reading documents; conducting site visits and interviews with project staff, participants, or other individuals; or both. The benefits of this approach are that it can be done quickly and at low cost. The hazard of the approach is that it depends totally on the knowledge, experience, and viewpoints of the experts chosen, and so runs the risk of being biased.

Data-based evaluation uses other methods. A descriptive analysis, for example, uses descriptive statistics to characterize a program, its participants, and attributes of the relevant social, political, or economic environment in order to understand how and why a program works. A case study makes extensive use of descriptive analytic methods.

A comparison-group design is called for when it is important to measure effects and to attribute those effects to a project or program. In such designs, a group of people or institutions who receive an innovative treatment or participate in a new program are compared to a similar group who do not receive the treatment or participate in the program. Differences in prespecified measures of impact or outcome between the two groups are attributed to the intervention.

Who Conducts the Evaluation?

Evaluation activities may be conducted by the program itself, foundation staff, one or more outside experts, an independent grantee or contractor, or a combination of these.

How Much to Spend

The cost of an evaluation project can range from a few hundred dollars for an expert-judgment assessment of a completed research grant to a few million dollars for a randomized controlled experiment of an innovative service program at multiple sites. Between these extremes, many evaluations can be conducted for less than $10,000, such as a visiting-committee expert review involving travel to several program sites; a descriptive study of program clients' characteristics, use of services, and satisfaction; a telephone survey of grantees; or a study of the number of publications yielded by a research program. A descriptive evaluation of a program that entails primary data collection at one site might range from $10,000 to $50,000. Process and impact evaluations of programs in more than one community requiring data collection from individuals at two or more points in time might cost between $100,000 and $300,000.

Here are some examples of factors that can increase costs:

A desire to attribute causal impact to the program, which means using a comparison-group design (and hence more data collection)
Programs that target whole communities rather than specific groups of individuals
Multi-site rather than single-site programs
Programs that try to make relatively small reductions in problems, so that evidence of impact is hard to discern
A need to collect primary data, when suitable records or published statistics are not available
Designs that require data collection in person
Designs that require collecting data at multiple points in time
A need for data that must be collected through highly technical procedures