Educators at the January 2015 Hive NYC meet-up brainstorm indicators for Connected Learning

This past winter, Hive Research Lab had the opportunity to collaborate with folks from the Connected Learning Research Network (CLRN) Longitudinal Study of Connected Learning team at the University of Colorado-Boulder and SRI International on a process they were going through to improve a survey instrument around Connected Learning. The team, led by Bill Penuel at CU-Boulder, had been working to figure out how the survey could do two things. First, it aimed to measure which Connected Learning design principles a young person experienced in a program or activity. And second, it sought to understand whether that young person was experiencing valued outcomes in four areas: deepening civic engagement, advancing career goals and goal discovery, increasing academic success and bonding to school, and generally a deepening in satisfaction, fulfillment and joy.

But measurement instruments can only be effective if there’s clarity around what’s being measured –  similar to how curricula can only be judged as effective once there’s a actually good sense of what you’re trying to teach. At the time they came to us, the survey team’s understanding of outcomes of related to Connected Learning was evolving along with the larger network of scholars that were part of the CLRN. As a result, parts of the instrument weren’t working perfectly, particularly some of the questions related to measuring the valued outcomes mentioned above. So the team was iterating on various parts of the survey, figuring out what was and wasn’t working from a measurement standpoint, what wasn’t matching with qualitative findings and grappling how to address those issues.

The challenge was determining what a young person could say or do that would be a good indicator that certain outcomes were happening for them, while also not creating a situation where every young person taking the survey reported that those outcomes are occurring when it’s not possible for that to be the case (a statistical phenomenon called ‘ceiling effects’).

The approach the CLRN team wanted to take to solving this issue and to improving the survey was to bring in the views of both young people and on the ground educators that are closest to these issues. Rather than going back to the literature, or talking more to statisticians and psychometricians, they took a community-oriented and participatory approach – talk directly to educators and young people about what they’ve heard (or said themselves) that have been indicators that things are going well, or not so well, around a particular area for a young person like being civically involved or having a sense of future career goals. The purpose was both to help clarify the outcomes to be measured and to help design new survey items around those outcomes.

Hive educators share out things they’re heard from youth that could be indicators around career outcomes

So in a series of workshops that Hive Research Lab helped to design and facilitate in early 2015, youth and adults at the Colorado Alliance for Creative Youth Development and educators from Hive NYC shared perspectives and accounts from their experience. From there, the survey development team, and the broader Connected Learning Research Network, built on what was shared in order to generate new, more valid survey items. The team just finished putting touches on a report that goes into more detail about the process of taking a participatory, community-based approach like this. It also includes the outcomes of the workshops: new survey items that educators can use to measure Connected Learning at their sites. You can check it out here.