Knowing How and When to Monitor, Assess, and Evaluate Grants

A family foundation's legal responsibilities for monitoring, assessing, or evaluating the grants it makes to organizations with 501(c)(3) tax exempt status are minimal. The IRS requires little in the way of detailed reporting on the outcome of specific grants—except for grants to organizations that lack 501(c)(3) status. Even without required reporting requirements, however, most foundations ask their grantees for a final report that gives a financial accounting of how funds were used, a narrative description of any accomplishments of the project or program, and copies of any media coverage of the project. For small grants, this type of self-evaluation is generally considered sufficient.

Alfred L. Castle of the Samuel N. and Mary Castle Foundation in Honolulu, Hawaii (assets of $40 million) offers the following advice on how small foundations can get grantees to prepare self-evaluations:

"Trustees of small foundations who would like to have an inexpensive self-evaluation completed by a grantee should…let grant applicants know at the time they apply that this is a condition of receiving the grant. It also helps to send them a copy of the questions at the onset of the grant. Where possible, the grantee should play a role in developing the questions. This process works best when approached as a collaborative effort, since all foundations are partners of nonprofits in achieving mutual goals."

Beyond self-evaluation, many choices and styles of grants evaluation are available. Some family foundations opt to use their in-house personnel, while others rely on hired consultants to conduct the analysis. Some choose to conduct their analysis on the basis of interviews and written responses to questions, others conduct site visits while the grant is in progress or at its conclusion, and still others employ a combination of these methods. In the case of larger grants, foundations sometimes decide to pay for a more objective analysis, which may mean an investigation using all the rigorous tools of modern social science and statistics. Others take a more relaxed approach, using a consultant to interview program participants and staff and to draw conclusions with the detachment of an outside observer.

William J. O'Neill, Jr. and the William J. and Dorothy K. O'Neill Foundation of Cleveland, Ohio (assets of $5.2 million), require different levels of evaluation, depending on the size of the grant and its importance to the foundation's strategy. For "small" grants of less than $5,000, no evaluation is required. For mid-size grants (up to $15,000), self-evaluation by the grantee is required. For larger grants, a site visit by a foundation representative is conducted to verify the self-evaluation. Finally, its current "focus area project"—a project to which the foundation has awarded more than $200,000 for two years—the foundation collaborated with another area family foundation to include the services of an independent evaluation specialist in the project. Of course, the detailed evaluation of the focus area project is intended not only as verification but also as part of the overall strategy to learn from a model project and to disseminate important findings.

Often, outside assistance for evaluations can be obtained through collaboration with other grantmakers. Such a collaboration might establish one evaluation process for a project, even though the project is receiving grants from many different foundations. In addition, it may be possible to borrow someone with evaluation skills from a community foundation or larger private foundation at less cost than tapping the open consultant market.

Approaches to Monitoring and Evaluating Grants

Most family foundations with assets of less than $10 million are managed by trustees and family members without the support of a large administrative budget or professional staff. Trustees of these foundations often assume that monitoring and evaluating a grant is difficult and expensive. As a result, they choose to fund only those organizations with which they are familiar. This assumption is incorrect: monitoring and evaluating grants can be accomplished relatively easily and cost-effectively.

Defining Monitoring and Evaluation. Grants monitoring and evaluation attempt to determine the worth or effectiveness of grants based on the grantmaker's criteria. Family foundations of every size can benefit from these activities; they are essential to an ongoing process of self-development and "course correction."

Monitoring is the process of ensuring that a grantee fulfills certain commitments during the course of a grant—files reports on schedule, allocates grant money to the purposes designated in the grant agreement, and carries out promised activities. Orderly monitoring can be very important in detecting management or fiscal accountability problems before they get out of hand. Monitoring is no substitute, however, for the actual assessment of results, which is evaluation.

IRS regulations do not require grantmakers to monitor grants or other activities. Grantmakers can ask for appropriate IRS documentation from the grantee and then monitor the grantee to determine whether the funds were spent according to the grant agreement and possibly whether a project or program has been completed. If time permits, a brief site visit by a board member can also provide useful information.

Evaluation has been defined as "the systematic collection of information about the activities, characteristics, and outcomes of programs for use by specific people to reduce uncertainties, improve effectiveness, and make decisions with regard to what those programs are doing and affecting."

Evaluation goes beyond monitoring in that it tells trustees how well their grants have been implemented and to what extent the foundation's mission is being fulfilled. Some smaller family foundations find it useful to use evaluation to determine community needs or the impact of grants. The key point is that evaluation can be a useful tool for those who wish to use it, and it is a tool that can be employed in greater or lesser degrees as the foundation governors and managers see fit.

There are, technically, six types of evaluation:

Needs Assessment: Describes the extent of a social or other problem, identifies targets of a program and helps design a new program, or justifies program continuation
Monitoring: Provides periodic information to the foundation on whether a grantee is implementing a grant as planned, identifies problems, and facilitates timely a resolution
Formative: Provides information to the foundation and the grantee organization that can improve the project or program as it is being implemented
Process: Is intended to help the grantor and grantee understand how the program made or failed to make an impact and how it might be replicated
Impact or Outcome: Measures program effectiveness in producing desired change
Summative: Assesses the overall equality and effectiveness of programs with respect to accountability and decision making; typically conducted after the program is terminated

Information from evaluations can help a family foundation board to:

Ensure that the foundation is accountable as a public trust
Improve the foundation's grantmaking abilities
Increase the effectiveness of funded projects
Plan and implement new programs

Techniques for Conducting Evaluations. Small foundations can rely on grantee self-evaluations or make use of an experienced consultant or a graduate student at a local university to conduct evaluations. They can also perform site visits. Each of these approaches is discussed briefly below.

Grantee Self-Evaluation.The most common form of grant evaluation employed by smaller foundations is the grantee self-evaluation. One family-managed foundation with assets of $2.5 million, for example, requires that each grantee sign a contract stating that it will provide a narrative evaluation of its project at the end of twelve months.

Grantees are sometimes reluctant to admit to their project's weaknesses, or may be unfamiliar with the self-evaluation process and thus unsure of what to report. Nevertheless, when a relationship of trust between the foundation and grantee has developed, grantees almost always appreciate participation in the evaluation process, and are more likely to accept and implement findings.

To help grantees with the self-evaluation process, foundations might ask that they answer the following five questions:

What were the lasting benefits of the project?
Were goals set in the grant application fulfilled?
What problems arose during the project?
What are your specific plans, if any, for continuing the work started by this project?
Do you have any comments, suggestions, or criticisms about working with the foundation staff?

Other helpful information to request includes copies of press releases, media stories, or other materials published regarding the funded program. Such information is very useful in preparing an annual report or other communications.

Outside Assistance. Consultants can bring objectivity, autonomy, and credibility to grant evaluations. Family foundations sometimes locate consultants by checking with staff of a larger private foundation or a community foundation or through peer networks.

Foundations with limited funds for part-time consultants might consider these means for getting an external assessment at the lowest cost:

Undertake joint evaluations and share costs with other small grantmakers funding in the same field
Ask a larger, staffed foundation to lend expertise
Build the costs of evaluation into the grant and have the evaluator work for the grantee. This is the least troublesome form of external evaluation for a grantee, although it may raise questions for trustees about objectivity

Trustee Site Visits. Site visits, conducted after a grant has begun, are relatively inexpensive and allow funders to see grantees and their grant dollars at work. During these site visits, foundation trustees might ask grantee personnel such questions as:

What differences exist between the way the organization planned and actually implemented the project? What do the differences mean to the staff, project, population served, and organization?
What were the project's results, for the people served and for the organization? Did the project accomplish its goals? What significant internal and external factors affected outcomes?
What unanticipated benefits or problems arose?
How could the foundation have been more helpful?
How will the project results be disseminated?

In addition, trustees might want to consider asking the intended beneficiaries of the foundation grant:

What difference did the foundation's support of the program make in their lives?
What were some of the benefits and problems of implementing the project?
How would they, the grantee, improve the project?

As William S. White, president and CEO of the Charles Stewart Mott Foundation in Flint, Michigan, notes, the bottom line for any evaluation is to help a foundation ensure that its objectives are on target and that its programs are meeting those objectives:

"Evaluation is a continuous process, one that should provide useful information to decision makers both inside and outside the organization. The single most important feature of any evaluative system is its ability to prompt learning. And it is in the process of reflection and dialogue about what is learned that a foundation can apply lessons in order to improve its service."

Tips for Making Evaluations

Drawing on several sources in the field, here are some tips from experienced grantmakers:

Review grantmaking guidelines to ensure that grantseekers understand the importance of evaluation, build reporting procedures into their proposal and project administration, and provide the information that the grantmaker wants
Encourage grantees to monitor and evaluate their work
Select for evaluation only those projects that have long-term implications for the foundation's grantmaking. Start early, set the wheels for evaluation in motion, bring grantees into the planning process, and budget sufficient funds for a proper job of evaluation
Join with other funders in the state or region to exchange information about what works and what doesn't
Track the ways in which the evaluation is used

What Foundations Are Really Doing: Two Surveys

In the past few years, the Council on Foundations on its own, and the University of Pittsburgh, in collaboration with the Lilly Endowment, have sought to determine what foundations are doing to evaluate the performance of their grantees. Results of their two surveys provide insights into the realities of grantmaker evaluation practices.

Foundation Management Survey. The 1994 Foundation Management Survey was the first survey in which the Council on Foundations explored grantmakers' evaluation practices in some depth. The questions were intended to determine the variety of program and self-evaluation practices among the different types and sizes of grantmaking organizations: what kinds of grants are evaluated, how evaluations are conducted, and how the findings are used. Selected excerpts of the analysis:

Types of Evaluation Performed or Funded. Most respondents (93 percent or 498) indicated that they evaluate the projects or programs they fund; about one-half said that they conduct self-evaluations and fewer than one-in-five support evaluations of organizations or programs funded by others.

National Survey on Role of Evaluation. The second study is called "A National Survey of Small and Mid-Size Foundations: The Role of Evaluation," and was published in July 1996. The Lilly Endowment supported the survey as part of the Endowment's partnership effort with the University of Pittsburgh to enhance its capacity to evaluate its grantmaking in education. An additional goal has been to contribute to what is known about evaluation use in foundations.

The survey queried 238 members of the Council on Foundations, all of which had assets below $250 million and 25 percent of which had assets below $10 million. Family foundations were not identified as a distinct category. The authors developed twelve findings from their data:

The authors found that evaluation "clearly centers on grantees," which, after foundation personnel themselves, are the most likely evaluators. The authors concluded that:

This emphasis on self-evaluation has mixed results from the perspectives of foundation personnel. There are concerns about the grantees technical adequacy and the natural tendency to be less than candid. However, the grantees involvement in evaluation reflects the notion of the 'reflective practitioner" that is considered a best practice in the literature on effective organizations.

Kinds of Grants Evaluated. Three-quarters of respondents indicated that they evaluate grants made to a new organization or an organization that they (the grantmaker) are funding for the first time. About 60 percent evaluate grants above a certain amount, and an equal proportion indicated that they evaluate grants made outside of typical program areas (such as special initiatives).
How Evaluations Are Conducted. For 80 percent of respondents (424 of 527), evaluation is handled by staff or board members; 179 of the 424 use this approach exclusively. The second most frequent approach was to include evaluation as part of grant activities; that is, funded through the grant and carried out by or under the direction of the grantee. Grantmakers were less likely to have evaluations implemented by outside consultants under contract to the grantmaker (22 percent of the 527), and even fewer reported having an evaluation supported through separate grants (11 percent).
How Evaluation Findings Are Used. Most respondents (90 percent or 496 of 522) use evaluation findings to assist them in making future funding decisions or to measure program effectiveness (82 percent). About half of the respondents used evaluation findings to capture program results or to demonstrate the strategic value of a program. Few (12 percent) indicated that they release evaluation findings to media and/or general public.
- Most small and mid-size foundations do evaluate as part of their standard operating practices. Much of this evaluation takes the form of grant monitoring by foundation staff and grantee self-reports.
- Small and mid-size foundations fund evaluation through administrative/operating budgets and as a component of grant activities with greater frequency than through grants to external evaluators.
- Thirty-one percent of responding small and mid-size foundations conducted at least one external evaluation in the past three years. Five foundations conducted fifteen or more external evaluations in the past three years.
- Most small and mid-size foundations do not trace dollars expended on evaluation.
- There is no meaningful trend for recent years in the number of evaluations being funded by the responding small and mid-size foundations.
- Small and mid-size foundations make the decision to conduct external evaluations usually for the purpose of gaining new knowledge. The controversial nature of a grant or program context does not appear to increase the likelihood that foundations will use an external evaluator.
- The evaluation process tends to be the responsibility of the program officer of record in the foundations surveyed. There is no widespread use of "evaluation directors" and/or staff dedicated to evaluation efforts in these foundations.
- Evaluation results are typically shared more often with audiences within the foundation.
- Evaluation is most often used for the purposes of holding grantees accountable, assessing program impact, and informing strategic thinking in foundations.
- Small and mid-size foundations do not often use evaluation to hold themselves accountable to the public.
- Small and mid-size foundations find evaluations unhelpful when specific qualities in the evaluation are useless (as in the collection of data simply for the sake of research) and limitations inherent when the grantee is the source of evaluation information (such as that consisting wholly of anecdotal evidence).
- The most commonly cited barrier to conducting evaluations among respondents was a lack of resources—notably, staff to oversee or conduct evaluation, staff time to commit to evaluation, and dollars to support evaluation activity.