[report] Leveraging Hive’s Collective Intelligence To Better Measure Connected Learning

Educators at the January 2015 Hive NYC meet-up brainstorm indicators for Connected Learning

Educators at the January 2015 Hive NYC meet-up brainstorm indicators for Connected Learning

This past winter, Hive Research Lab had the opportunity to collaborate with folks from the Connected Learning Research Network (CLRN) Longitudinal Study of Connected Learning team at the University of Colorado-Boulder and SRI International on a process they were going through to improve a survey instrument around Connected Learning. The team, led by Bill Penuel at CU-Boulder, had been working to figure out how the survey could do two things. First, it aimed to measure which Connected Learning design principles a young person experienced in a program or activity. And second, it sought to understand whether that young person was experiencing valued outcomes in four areas: deepening civic engagement, advancing career goals and goal discovery, increasing academic success and bonding to school, and generally a deepening in satisfaction, fulfillment and joy.Screen Shot 2015-05-18 at 3.30.58 PM

But measurement instruments can only be effective if there’s clarity around what’s being measured –  similar to how curricula can only be judged as effective once there’s a actually good sense of what you’re trying to teach. At the time they came to us, the survey team’s understanding of outcomes of related to Connected Learning was evolving along with the larger network of scholars that were part of the CLRN. As a result, parts of the instrument weren’t working perfectly, particularly some of the questions related to measuring the valued outcomes mentioned above. So the team was iterating on various parts of the survey, figuring out what was and wasn’t working from a measurement standpoint, what wasn’t matching with qualitative findings and grappling how to address those issues.

The challenge was determining what a young person could say or do that would be a good indicator that certain outcomes were happening for them, while also not creating a situation where every young person taking the survey reported that those outcomes are occurring when it’s not possible for that to be the case (a statistical phenomenon called ‘ceiling effects’).

The approach the CLRN team wanted to take to solving this issue and to improving the survey was to bring in the views of both young people and on the ground educators that are closest to these issues. Rather than going back to the literature, or talking more to statisticians and psychometricians, they took a community-oriented and participatory approach – talk directly to educators and young people about what they’ve heard (or said themselves) that have been indicators that things are going well, or not so well, around a particular area for a young person like being civically involved or having a sense of future career goals. The purpose was both to help clarify the outcomes to be measured and to help design new survey items around those outcomes.

Hive educators share out things they're heard from youth that could be indicators around career outcomes

Hive educators share out things they’re heard from youth that could be indicators around career outcomes

So in a series of workshops that Hive Research Lab helped to design and facilitate in early 2015, youth and adults at the Colorado Alliance for Creative Youth Development and educators from Hive NYC shared perspectives and accounts from their experience. From there, the survey development team, and the broader Connected Learning Research Network, built on what was shared in order to generate new, more valid survey items. The team just finished putting touches on a report that goes into more detail about the process of taking a participatory, community-based approach like this. It also includes the outcomes of the workshops: new survey items that educators can use to measure Connected Learning at their sites. You can check it out here.

Hive Research Lab at the AERA 2015 Annual Meeting

AERA 2015 imageHive Research Lab will be heading to Chicago this week for the American Educational Research Association’s 2015  Annual Meeting.

The theme of the meeting this year is “Toward Justice: Culture, Language and Heritage in Education Research and Praxis”. Check out the official meeting page.

Here’s a run-down of where you can find our work at the conference:

Friday, April 17th

4:05 to 5:35pm, Sheraton, Second Level, Superior A

Symposium: Learning as Transformation: Examining How Youth Author New Learning Pathways/Ecologies in Science, Engineering, and Technology

Chair: Daniel Birmingham, Loyola University Chicago

Presentation: “From Half-Pipe to Full-fillment: Leveraging Interest-Driven Identities as a Strategy for Technology Learning”, Dixie Ching (NYU, presenting author), Rafi Santo (IU), Tal Bar-Zemer (Citylore), Jessica Forsyth (Harold Hunter Foundation), Chris Hoadley (NYU).

 

Saturday, April 18th

Sheraton, Ballroom Level, Sheraton V

Structured Poster Session: New Tools, New Voices: Innovations in Understanding and Analyzing Life-Wide Ecologies for Youth Interest-Driven Learning

Chairs: Kylie Peppler (IU), Chris Hoadley (NYU)

Poster: Affordances of Social Learning Ecology Maps for Examining the Importance of Social Support in the Pursuit of Digital Media Making Activities Dixie Ching, NYU (presenting author); Rafi Santo, IU; Kylie Peppler, IU; Christopher Hoadley, NYU

[white paper] On-ramps, Lane Changes, Detours and Destinations – New Community-Developed White Paper on Supporting Pathways Through Brokering.

In the summer of 2014, Hive Research Lab facilitated a design charrette focused on supporting youth interest-driven learning trajectories in the Hive. As part of the preparation for the meeting, mRoad signs groupedembers were asked to provide illustrative examples of what successful pathways or trajectories looked like on the ground. From the extraordinarily rich stories that members provided emerged our latest publication, a community-developed white paper entitled: On-ramps, Lane Changes, Detours and Destinations: Building Connected Learning Pathways in Hive NYC through Brokering Future Learning Opportunities.

This paper [pdf] makes a strong case for the importance of brokering as a key strategy for supporting youths’ interests in sustained and robust ways. Brokering (1) connects youth to meaningful future learning opportunities including events, programs, internships, individuals, and institutions that will support their ongoing interest-driven learning; and (2) enriches youth social networks with adults and peers that are connected to or have knowledge of future learning opportunities. The paper emphasizes that the Hive NYC community—with its robust social network of educators and professionals—represents an impressive cache of human and social capital that could be leveraged more fully through brokering.

Screen Shot 2015-04-13 at 2.17.17 PMChallenges to brokering are discussed as well, including how a young person’s network orientation or help-seeking orientation may affect her ability to take up and navigate the opportunities brokered by high resource individuals. Several recommendations targeted towards individual organizations and the network are offered as generative starting points for thinking about ways to extend the impact of brokering.

We wish to thank the 70+ individuals (named in the Appendix) who engaged in the collaborative sensemaking discussions, reflective conversations, and feedback-giving that resulted in this paper. We hope this collective effort felt of value to all who participated and that HRL, as facilitators of the process, was able to capture adequately the thinking and expertise of the community.

And of course, we look forward to more conversations and collective knowledge building around this core youth development practice!

Using a Collaborative Design Research Approach in Hive NYC: Hive Youth Meet-ups Case Study

RafiJuanHive Research Lab seeks to provide value to Hive members through the sharing of formative reports, acting as embedded research partners, and engaging in collaborative design research efforts with members. This post reports on our recent design and evaluation work with the Hive Youth Meet-up team—Juan Rubio from Global Kids, Chris Amos from Carnegie Hall, Brian Cohen from Beam Center, Erica Kermani from Eyebeam, and Zac Rudge from Parks and Recreation Computer Resource Centers. While Juan and Zac have provided wonderful perspectives on various aspects of the project (also see Global Kids’ Chanell Hastings’ recap of the first two meet-up events), here we focus on the design research process itself. Since this was one of the most substantive projects where we’ve applied this process, we wanted to document and reflect on how things played out and how these methods might be applied in other contexts within Hive. Continue reading

What me, mentor? Introducing the Connected Mentor Framework for out-of-school environments

CM_logoMentoring is often described in terms of its intensity: the long arc of interactions, the deep bonds that form, the great investments made on both sides. However, when reflecting on the range of experiences that youth have in out-of-school contexts like those found in Hive Learning Networks, one can point to many instances when adults mentor youth in ways that don’t quite fit the traditional model, but are still valuable in their own right. For example: the video game designer hired to run a 7-week program, who not only provided instruction but also created a temporary conversation space to ‘geek out’ about the latest video games and the game design industry itself. Or the fashion designer invited to speak at a youth event about his journey to owning a small business. Couldn’t such “mentoring moments” be just as valuable or consequential to a youth’s learning trajectory as more sustained mentoring?

This is the question that the Chicago-based Mentoring Working Group—Tené Gray, Director of Professional Development at the Digital Youth Network (DYN), Elsa Rodriguez, program manager at Hive Chicago, and Bernadette Sánchez, associate professor at DePaul University—set out to address. Their Connected Mentor site, which launches tomorrow, represents a synthesis of rich research and practical knowledge around mentoring, including a mentoring framework that was developed specifically for the out-of-school context.

As our own on-the-ground observations here in NYC indicate that many ‘fleeting’ interactions between youth and helpful individuals do provide key forms of support contributing to the ongoing pursuit of interest-driven activities, we find this framework provides a valuable foundation to begin teasing apart and further leveraging these relationships. Furthermore, this issue of common language is something we’ve seen as important for any type of coordinated action and so we applaud this effort to develop a powerful and shared language around the practice of mentoring.

Tene and Elsa were kind enough to chat with us about their work and give us some insight into their Connected Mentor Framework. Continue reading

HRL@Mozfest2014: A Network of Networks, a Hive of Hives

Global Hive Meetup @ MozFest 2014
Photo by Mozilla Hive NYC Learning Network

As we’ve studied Hive NYC, one of the big achievements we’ve seen is what can happen when previously unconnected educational organizations come together from across a given city. Organizations just miles apart geographically may have previously seen themselves as being in different sub-fields, but in Hive created a shared community platform where they can exchange ideas, find the places they have common values and organize around collective goals. One of the biggest value propositions of Hive, as contrasted with, say, broader field-level structures like conferences or online communities, is its locality. From an organizational perspective, it’s an ongoing field-level community located right in the backyard of your city. This core feature, and value, of locality is what sets Hive apart in many ways in an increasingly ‘flat’ virtual world.

And so the idea of having a ‘global’ Hive movement can seem antithetical to this local commitment. The whole point of a Hive is rooted in the idea of educators and organizations being able to physically come together with regularity, easily collaborate and design solutions to local problems – wouldn’t some sort of ‘Hive global’ be missing the point that the model made in the first place?

What I saw at Mozfest made me think about this issue in a new way. It made me realize just how powerful it is to have interaction among the growing number of cities that have Hives, be they full fledged Hive Learning Networks (in NYC, Chicago, Pittsburgh and Toronto), emerging Hive Learning Communities (in Kansas City, Chattanooga, San Francisco-Bay Area, India) or bunches of others in places that are exploring like Denver, Seattle, Vancouver, Manchester, Berlin and elsewhere. What I saw at Mozfest was not some sort of unitary ‘Hive Global’, some singular über-community that subsumes the commitment to local urban community. Rather, the Hive track represented a ‘Hive of Hives’, with Hive members and leaders sharing, discussing and creating solutions to shared problems together, using common language and based in shared values.

Kevin Miklasz, a Hive NYC member, reflected beautifully on the power of Hives coming together in his post about the festival:

It’s like that feeling you have when you meet up with an old friend and pick up right where you left off, as if no time had gone by. Except we had never met before, but we were able to just pick things up and start going anyway. We talked about the same things, using the same language and ideas, and were comfortable using the same kinds of open-ended design-thinking techniques to work through problems.

The rich interactions amongst ‘Hivers’ at Mozfest was what has convinced me of the value of having a ‘network of networks’. People were not only able to talk about issues that were relevant to actors in every city (e.g., what are effective strategies for making learning relevant for youth?), but also for sharing ideas about how a Hive might work, and how it might organize itself (e.g., what are the ways that your Hive designs opportunities for members to share ideas?). The former example, sharing thoughts on common issues like pedagogical approaches, is certainly powerful. But it is also the kind of thing that happens at many conferences and workshops. But that latter issue – on how Hives might facilitate and organize themselves – this is the place where a ‘network of networks’ can act as a context to innovate on the Hive model, to share thoughts on what works and doesn’t when it comes to building a Hive in a city. A Hive of Hives acts as an infrastructure for innovation on the idea of Hive itself.

As more cities explore organizing Hives in their backyards, it will be critical that there is a common space where the ‘theory of Hive’ and the ideas that Hives care about can be discussed, debated, learned from and about, and most importantly, advanced. Just as a local Hive is a great opportunity for educational organizations and actors that are committed to exploring and inventing around ideas of Connected Learning and web literacy, a Hive of Hives presents a critical opportunity for those committed to such city-based models to share knowledge and collectively level up what Hives around the world can be.

[white paper] What does it mean to Work Open in Hive NYC? New Hive community whitepaper

Working Open Whitepaper front page imageIn the summer of 2014, a group of Hive NYC members and stakeholders came together to think, talk and design around the idea of ‘working open’ in the Hive. The context was the Network that Learns design charrette. Led by Hive Research Lab, we created this two day sprint as a space to address key knowledge management issues that were brought up State of the Hive meeting in March 2014. Members had voiced needs around making it easier to locate expertise in the network, continually capture best practices from ongoing projects, and figure out how the network could be a context for accumulating collective wisdom. In our ongoing work studying innovation in a ‘networked’ context, we’ve see how just such issues can be critical in supporting organizations to effectively leverage Hive as they dive into new areas of work and strengthen existing ones, and so the charrette acted as a space where we could collectively think them through.

We used the idea of ‘working open’ during the charrette as somewhat of a grounding theme that cut across these knowledge management issues. As a way of working that values collaboration, failing early and often, ongoing storytelling, community building and an experimental and flexible spirit, ‘working open’ felt like it might have something to offer these issues that were raised at the State of the Hive. If understood and supported in a way that speaks to the Hive community’s distinct context, working open could be considered a mode of engagement that allows Hive members to progress both individually and collectively.

The white paper we’re sharing now [pdf] is the result of those discussions, and aims to synthesize many views about what constitutes ‘working open’, what it looks like in practice, what tensions are involved and what this all might mean for the Hive NYC community. Drawing on the voices of participating Hive members, as well as our field research in Hive NYC, the white paper offers somewhat of a vision for collective organizational learning within the network. We’d love to hear what you think, and how you could see these ideas applying to the way you do your everyday work.