caise

Within the next week, all PIs of ISE-funded projects will receive an email announcing the posting of the NSF ISE Online Project Monitoring System for measurable impacts and indicators. Developed by Westat, the system was created to enable NSF to track its funded projects and aggregate data about products, audiences served, evaluation methods, intended impacts (re: knowledge, awareness, attitudes, behaviors, skills, and other outcomes of experiential informal learning: see Friedman, ed. 2008).

As of next week, all PIs of ISE-funded projects will be required (and, hopeful developers of 2010 grant applications to ISE are well advised) to use the matrix on the Monitoring System to articulate products, audiences and intended outcomes on this new reporting system.

Right off the bat, the first thing you need to know is that though the Monitoring System has been approx. 18 months in development, Westat is fully aware that its System is still in a prototype stage. They are quite welcoming and willing to confer with PIs on the phone and help them fill out the Worksheets, and—this is a most important point—they want and are eager to know where you are encountering difficulties in fitting the ideals, scope, and values of your innovative project into their pre-designated categories. Their job is not an easy one, and they are open to your input. At the workshops presented on the new ISE Online Project Monitoring System, some participants judiciously noted that many of the audience categories that are served by ISE projects, such as families, ESL learners, entire communities, have not yet been included by Westat on the Online System. You are well advised to speak with Westat and help them broaden their categories so that your project can be accurately defined and measured for its success as well as its challenges (yes! These are included because it is in the challenges where ISE professionals often find their most valuable lessons). It’s a new system, it’s required, and it’s in progress. In the end, it will be an immensely valuable tool to allow ISE at NSF to aggregate data and report on the value that we as informal educators bring to the educational infrastructure of the nation. Right now, as pioneers, of course there’ll be rocks on the road, but it’s very good to know that Westat designers are eager listeners and offer all a helping hand.

Personally, I would want the field as a whole to agree—and vociferously agree—that though the Online Monitoring System is a valuable tool for NSF purposes, nothing in the field replaces the richness and complexity of well-written evaluation reports posted on www.informalscience.org. Often times, the story of how a project evolves, its unexpected alliances, its unavoidable difficulties and serendipitous (or more often, arduously unraveled) solutions is the narrative data that I and many others find most useful to inform our thinking about innovation and transformation in the field. Think for a moment about the wealth of data embedded in these reports just on the vicissitudes of collaboration between organizations!

The Online Monitoring System is a welcomed quantitative tool and complements well the narratives posted by PIs and their evaluators on Informalscience.org. Both are equally valuable to maintaining a vibrant community and building an infrastructure of shared knowledge.