Earlier this year IIE’s Center for Academic Mobility Research (CAMR) hosted a roundtable on program evaluation and impact assessment. We brought together practitioners and researchers from various fields of study to discuss ways to measure impact of international fellowship programs. I was pleased to greet participants from the Ford Foundation, Mastercard Foundation, Social Science Research Council, German Academic Exchange Service, and others at the event held in IIE’s New York office.
The roundtable came at an opportune time. We are currently preparing the methodology for a ten-year tracking study of the Ford Foundation International Fellowships Program (IFP). This event provided an opportunity to discuss some of the considerations and challenges in developing the study’s research design, which I encounter every day.
While most evaluations happen as a program is in process or as it is ending, less research has been done on program impact of international fellowships over time, sometimes years after program implementation. Yet this is precisely the impact practitioners would like to see, the evidence that their program has promoted long-term change in the lives of their fellows and their communities. Program implementors also share the desire to document impacts that go beyond the individual experience, impacts at the organizational, community or even national and international levels.
We plan to use an innovative approach in our tracking study design, collecting data on IFP alumni over a period of ten years. A tracking study intends to “systematically analyze the lasting or significant changes – positive or negative, significant or not – in people’s lives brought about by a given action or series of actions.”[1] Our study focuses on the participants of the program and their personal trajectories, and the extent to which the program intervention may have contributed to their life choices and opportunities.
The roundtable discussion covered practical techniques for designing studies to measure alumni impact:
- Tracking alumni over time
- Promoting alumni engagement
- Measuring impact through continuous assessment
We discussed the importance of ensuring that alumni have a sense of purpose or value in the study, creating a feedback mechanism for them to respond and contribute to the study’s implementation. This is very difficult for our project, as the evaluation will span ten years. Opportunities for further alumni engagement, whether through networking events or alumni awards, are effective ways to maintain interest among alumni beyond the fellowship.
According to participants at the roundtable, impact measures of the fellowship should not only tie to the initial program goals, but should also take into consideration the changes in values and perceptions of the fellowship among its recipients. Studies of impact should consider how the fellow views the program over time within the context of other achievements in his or her life. Being able to identify and group these various achievements will provide a unique view into the trajectories of alumni and the way in which they frame their fellowship experience.
Keep checking back for a continued discussion on impact assessment as I examine these topics and more at IIE’s Opening Minds blog!