The ISE Online Project Monitoring System (OPMS) collects data from ISE projects at the beginning and end of their grants, and each year in between. This month’s article about the ISE OPMS tackles a topic that is perhaps the most important, yet challenging component of the online surveys—impacts and indicators. Projects use the baseline survey to provide a wealth of data about their anticipated deliverables, including how they expect to affect the audiences who experience their projects; projects revisit this information in their closeout report to document the extent to which their impacts were achieved. These impacts target the heart of the ISE program—how projects intend to make their public and professional audiences more skilled, knowledgeable, active, engaged, and/or enthusiastic about STEM. This article explains why providing these impacts is important and how to maximize their utility. It also provides some information about impact data that have been reported by current and previous projects.
Importance: Why NSF needs information about impacts and indicators
Impacts describe the changes a project hopes to achieve as a result of its efforts to engage public and professional audiences, while indicators outline how projects will demonstrate that those achievements have occurred. These data help NSF:
- Document the wide array of project-related accomplishments;
- Determine what content or skill areas are being targeted by projects;
- Determine what areas are not being addressed; and
- Identify how future projects can operationalize and measure similar achievements.
Equally important, these impacts provide NSF with clear, concise descriptions of the accomplishments that can be attributed to the ISE program—descriptions that can be shared with Congress and others stakeholders.
Impacts: Describing success with a broad brush
An impact is a broad goal that an ISE project intends to achieve with its public and/or professional audiences. The value of a project impact entered into the ISE OPMS depends on the level of detail that is provided about what will be accomplished. Similarly, impacts are most useful when they are specific and measurable. To be as useful as possible, impacts in the ISE OPMS should be written so that someone who knows little about the project will be able to understand the anticipated goals for participants. A few key tips can help maximize the value of impacts for NSF and the informal science education field:
- Identify the target audience(s) (e.g., will this impact target all project participants, or only high school-aged students?)
- Identify how the audience(s) will change (e.g., increasing knowledge, developing a more positive attitude)
- Identify the STEM content area (e.g., cicadas, the interaction between insects and the ecosystem)
Indicators: Defining success
An indicator gives more specific information about an impact and explains how the project will know if an impact has been achieved. The indicator might provide:
- A breakdown of goals by audiences (e.g., one indicator might focus on elementary school students, another on high school students).
- A breakdown of content (e.g., if the impact is that visitors will increase their knowledge of cicadas, one indicator might be that visitors will know the anatomy of cicadas, and another that visitors will be able to identify the lifecycle stages of cicadas).
- An explanation of how the impact will be measured (e.g., visitors will be able to list all phases of the cicada lifecycle). This explanation should not simply list data collection methods, but rather should say what information those measures will collect.
Examples: Impact statements and their corresponding indicators
Impact—Film viewers will learn more about the cicada lifecycle. [Identifies audience (film viewers), category (knowledge), and content (cicada lifecycle).]
Indicator 1—Film viewers will know about the length of a cicada lifecycle. [Identifies audience (film viewers), category (knowledge), and content (cicada lifecycle).]
Indicator 2—Children will be able to describe at least two stages of the cicada's lifecycle. [Delineates a more specific goal for target audience and how it will be measured.]
Consider the following example of a professional audience impact and its corresponding indicators. In this example, each indicator defines a more specific example of what the professional audience will know about the curriculum.
Impact—After-school professionals will understand the curriculum after training. [Identifies audience (after-school professionals) and category (knowledge/awareness/understanding.]
Indicator 1—After-school professionals will know several interesting facts about cicadas to incorporate into their activities for their program participants.
Indicator 2 – After-school professionals will be able to correctly answer questions about any part of the program’s content.
Additional Impact Data: Keeping everything consistent
Projects use the ISE OPMS to enter one or more impacts that describe the changes anticipated in the public or professional audience as a result of their exposure to a given deliverable. For each impact statement, projects enter an appropriate impact category, the study design, data collection methods that will be used to evaluate the degree to which the impact was achieved, and up to five indicators (see Figure 1).
All of the information entered for an impact should “align,” or match. For example, if an impact describes a new skill that people will develop, then the selected impact category should be “skills,” and the entered indicators should also describe how the project will know that people have actually acquired those skills (e.g., a breakdown of the skill components that people will be able to perform, or examples of ways that people will perform them). All of the information entered for a given impact should relate to the same topic and category.
Similarly, the evaluation methods should match the impact and indicators. For example, an indicator that people will read more books about science after visiting an exhibit will presumably include a data collection component that occurs after people leave the exhibit (unless the indicator will not be measured).
Keeping all of this information aligned ensures that NSF can easily search for projects that seek to increase “engagement” (or any other impact category)—and identify all impacts, indicators, and data collection methods related to engagement.
Findings from the ISE OPMS
Data collected in the pilot version of the ISE OPMS reveal trends about anticipated project impacts for the full-scale implementation projects funded between 2006 and 2009. Projects could provide information about an unlimited number of impacts (the average project identified four impacts). As shown in Figure 2, almost all projects (91 percent) described at least one impact pertaining to awareness, knowledge, or understanding, while over half described an impact pertaining to engagement or interest (69 percent) and/or behavior (50 percent). Less than half sought to impact participants’ skills or attitudes (46 and 39 percent, respectively). These data are shown broken out by audience type in Figure 3. Projects also have the option of entering “other” and specifying their own impact categories.
For each impact, ISE OPMS respondents were asked to specify which data collection methods would be used to assess whether the impact would be achieved. The most commonly used methods were surveys collected at informal science venues or programs (76 percent) or via the web (71 percent). Almost two-thirds of projects were planning to use direct observations of visitors’/participants’/educators’ conversations and/or behavior at informal science venues or programs (62 percent), semi-structured or informal interviews conducted via phone (62 percent), and semi-structured or informal interviews conducted at informal science education venues or programs (62 percent).
Conclusion
ISE-funded projects will have the opportunity to enter impacts and indicators into the system in their baseline reports, or when reviewing their annual reports this spring. We hope the information in this article helps you develop impacts that are specific, measurable, and useful. The ISE OPMS contains several tools to help projects craft measurable impacts and indicators—including a PowerPoint presentation about impacts and indicators on the ISE OPMS help screen. The CAISE website has useful resources, including a webinar that walks respondents through a sample OPMS baseline survey. In addition, the Framework for Evaluating Impacts of Informal Science Education Projects provides detailed information about developing and evaluating project impacts. We also strongly encourage working with your evaluator early in your project’s timeline to construct your impacts, indicators, and evaluation plan. In addition, as your project enters data into the ISE OPMS, you are encouraged to contact Westat personnel at ISEhelp@westat.com with any questions you might have about impacts and indicators (or anything else pertaining to your ISE OPMS submission).
Next month’s article theme has not yet been determined. If you have a topic about the ISE OPMS that you would like addressed in an upcoming CAISE newsletter, please email HannahPutman@Westat.com with your question or idea.