Using a Collaborative Design Research Approach in Hive NYC: Hive Youth Meet-ups Case Study

Hive Research Lab seeks to provide value to Hive members through the sharing of formative reports, acting as embedded research partners, and engaging in collaborative design research efforts with members. This post reports on our recent design and evaluation work with the Hive Youth Meet-up team—Juan Rubio from Global Kids, Chris Amos from Carnegie Hall, Brian Cohen from Beam Center, Erica Kermani from Eyebeam, and Zac Rudge from Parks and Recreation Computer Resource Centers. While Juan and Zac have provided wonderful perspectives on various aspects of the project (also see Global Kids’ Chanell Hastings’ recap of the first two meet-up events), here we focus on the design research process itself. Since this was one of the most substantive projects where we’ve applied this process, we wanted to document and reflect on how things played out and how these methods might be applied in other contexts within Hive.

Using a collaborative design research approach in Hive NYC
In developing the Hive Research Lab model, we knew that basic research would be central to helping the network understand key issues, but also felt that much could be gained by also working with Hive members to develop new initiatives based on our learnings. Hive NYC deeply embodies the spirit of educational experimentation, producing a range of programs and tools around Connected Learning and digital literacy and we saw an opportunity to bring principled inquiry approaches from the design research world into the mix. Such approaches blend design thinking with formative data collection in order to rigorously inform whether and how progress towards certain valued goals have been made. This process also helps to accumulate knowledge that could benefit others.

Our design research process is rooted in three disciplines: improvement science, participatory design, and learning sciences. From improvement science, we utilize the Plan-Do-Study-Act (PDSA) tool, which involves rapid cycles of testing and measurement and constantly asking how we know when a change is actually an improvement. From participatory design we maintain a core value of building with, as opposed to building for—we assume collaborative work with Hive stakeholders in developing projects. And finally, we maintain a learning sciences commitment that the development of learning theory best happens in the context of designing initiatives within real world learning environments, as opposed to lab settings.

The spark for Hive Youth Meet-ups
We saw from our basic research on youth pathways that there are a number of persistent challenges around supporting learning pathways, including a phenomenon we call “post-program slump,” referring to the decline in support that some youth perceive after a Hive program they were enrolled in ends. So in the summer of 2014, we facilitated the Youth Pathways Design Charrette, where groups of Hive educators engaged in design sprints around initiatives that sought to address stubborn barriers to supporting youth pathways. Juan Rubio of Global Kids presented his idea to develop youth meet-ups within the Hive and quickly gained allies at the meeting who were very excited to think about what this might look like. After the charrette, the Hive Youth Meet-up team applied for and was awarded funds by Mozilla to pilot three events. HRL joined the planning meetings, which began in fall of 2014.

Developing a baseline model for Hive Youth Meet-ups
Engaging in a process of iteration-towards-improvement of something requires first defining the object of examination. Models can be a good tool for this as they represent a distillation of all the important aspects of this ‘thing’—a tool, program, event, etc. Math education researcher Alan Schoenfeld has described modeling as “a rigorous way to test theoretical ideas…Models are falsifiable, which means that one’s explanations are put to the test” (2014, p. 405). Furthermore, as hard as it might be to model a complex social event (like a youth meet-up), by making earnest and careful efforts to do so, there’s an opportunity to make stronger contributions towards real world solutions.

Developing a model for Hive Youth Meet-ups meant both specifying the important components—its goals, design features, intended audience, roles within the event, etc.—that might contribute to our intended outcomes, and getting clearer on what exactly those outcomes should be. We began this process by collaboratively developing a ‘design document’ that would help the project team rally around a common mission, identify barriers to that mission that a youth meet-up might be able to address, and then sum up this line of work by developing a list of goals to design towards. Through this process, the team also started to create common language around and understanding of our goals—which turned out to be extremely important for our reflection periods and for remaining on the same page throughout the process. The design document also helped HRL develop evaluation tools for the event itself. Below are the steps we went through to produce our “final” design document and working model for HYMs.

Steps toward developing the HYM model:

1. Articulate guiding mission. The project team began by articulating a guiding mission/challenge around supporting youth pathways and trajectories: “Helping Hive youth develop, sustain, deepen, or expand upon their engagement in their interest-driven learning, both during and after a program or event.”
2. Identify barriers. We then generated a list of common barriers to achieving the mission. In particular, we discussed barriers that we thought a youth-oriented event might help address. Example barrier: “Youth do not develop a sense of affiliation with a Hive organization—this may affect their ability to take advantage of support from that organization.”
3. Identify design goals. Next, we developed a set of key design goals that the team wanted to focus on through the design and implementation of Hive Youth Meet-ups. The goals flowed from our discussion about barriers but also reflected what the HYM team generally expressed was important to them. Example goal: “Increase ways for Hive youth to develop a stronger affiliation with a Hive organization.”
4. Identify design features. Based on discussions at the YT design charrette and during subsequent planning meetings, HRL translated the event format into a set of “design features.” Example design feature: “Youth presentations”—members envisioned three youth presentations at each event and that presentations should be about a Hive program project or a personal project.
5. Design logic mapping. Finally, HRL mapped the design features to the key design goals the team wanted to reach (see figure above). Reviewing this as a group resulted in adjustments to design features and goals as well as more specificity around how a certain feature might achieve a certain goal. For example, we realized that youth presentations had the potential to help youth presenters develop a stronger affiliation with the organization they were representing and discussed how to help ensure that outcome by, for example, advising youth presenters to briefly give an overview of the organization and program they came from before sharing out about their project.

Testing the Hive Youth Meet-ups model
With a general baseline model consisting of key goals and design features in place, HRL was able to develop a formative evaluation plan that could help project members understand to what extent they were meeting their intended design goals at each event. HRL lead the process of developing the plan as well as collecting formative data. We utilized the aforementioned Plan-Do-Study-Act (PDSA) cycle as our general sequence of steps, repeating the PDSA cycle for each event, as follows:

1. (Plan) HRL develops a plan to test aspects of the HYM model.
2. (Do) While the HYM team runs the meet-up, HRL executes the evaluation plan, which includes observing the event, collecting youth impressions, conducting a post-event debrief, collecting post-event reflection notes from the planning team, and administering surveys to youth and non-planning-team adults.
3. (Study) HRL analyzes the data and generates a formative evaluation report that includes a summary of the data as well as recommendations.
4. (Act) At the next HYM meeting, HRL presents the formative evaluation report. The planning team and HRL discuss possible modifications to the model based on the results of the formative research as well as further personal reflections.

Learning from each event
Our post-event reflection meetings represent an opportunity for the team to further refine and define the model, as well as create local theories that might apply to both research and practice. For example, after the first HYM event, we further specified the youth audience that we hoped the event would attract, and clarified the adult and youth roles for each meet-up. In terms of changes to the enactment of the event itself, we decided on strategies that would lead to more impactful brokering. These included: Building a mailing list and creating a moment in agenda where the “brokering coordinator”—a role/design feature we created to meet certain HYM goals—could talk about upcoming programs and encourage adults and youth to visit the “Hive Opportunity Table.”

Reflections on using a Collaborative Design Research model in Hive NYC
Just as we were hoping to help HYM members refine their model, we also wanted to reflect on our own process as well.

• We need to develop a clearer story of why and what a collaborative design research approach entails. We are still figuring out the best ways to articulate and define our approach for our HYM partners. Given that our ultimate hope is that practitioners will adopt some aspect of this process in their own planning and development activities, we know we need to develop a succinct case for why working this way is worthwhile.
• Engaging in collaborative design research is time intensive. The process we just described is complicated and requires a lot of time and mental bandwidth to execute successfully. Given the intense time pressures and limited funding resources that HYM members were under, we recognized that it was a challenge to even get the meet-ups off the ground, and that engaging in this design research process represented a substantial layer of work over that.
• The idea of design goals seemed of value to the planning team. Based on what HYM members talked about during our planning and reflection meetings and when presenting the project to others, it does seem that members valued the process of initially defining goals and referring back to them throughout the project.
• Practitioners can play more of a role in data collection and analysis. Practitioners being part of the data collection and analysis process is a place of potential experimentation for us. We did this to some degree by giving project members observation tasks during the event, and some made reflections based on the raw data that was provided in appendices of reports. We hope to more intentionally develop routines down the line that can help guide how practitioners might engage in observation and analysis activities.
• The form of the initiative will affect the design research process. The HYM project was a great way to engage in principled inquiry because each HYM was considered an independent event and the project team proposed three events, which allowed for two cycles of measurement, reflection and iteration. Additionally, there was plenty of time for data analysis and reflection in between the events. Other initiatives, be they technology tools, more ongoing weekly programming, or large scale events would require an adaptation of the approach.

This was our first attempt to articulate explicitly and implicitly this process of collaborative design research and we will continue to refine it. As a final note, we wish to thank the HYM team for their continued participation, support, and patience throughout this collaboration!

References
Schoenfeld, A. H. (2014). What Makes for Powerful Classrooms, and How Can We Support Teachers in Creating Them? A Story of Research and Practice, Productively Intertwined. Educational Researcher, 43(8), 404-412.