Lessons from Sesame Workshop: Leveraging Research and Learner Testing for Powerful Edtech

With a global value of over $100 billion, the edtech industry is bursting with new players who want a slice of the pie. But because rigorous efficacy studies require the time and resources many companies don’t have before products go to market, it’s challenging for educators, district leaders, and families to understand which tools are likely to have a meaningful impact on learning.

At Digital Promise and Sesame Workshop, we’ve spent a combined six decades thinking about the best ways to design education tools to support learning. Based on our research and the best practices we’ve identified, our teams recommend that edtech vendors incorporate two key phases into their research and development product lifecycle. As a result of focusing on these two phases, those looking to purchase and use the product will better understand the tool’s intentionality and connection to learning.

Phase 1: Design based on learning sciences research

As Digital Promise recently discussed, the use of learning sciences research in product development is critical for crafting tools that have a high likelihood of supporting learner needs. Without diving into the research, designers may be stuck reproducing outdated or baseless pedagogical ideas that, at best, offer no learning opportunity and, at worst, may harm outcomes, particularly for learners from historically and systematically excluded populations. Edtech that successfully incorporates learning sciences research does so by designing based on the field’s best understanding of practices that support learning, as well as focusing on concrete learning outcomes that research has led experts to expect to see.

Phase 2: Iterative improvements based on user testing with intended users

Based on the hypotheses drawn from engagement with learning sciences research, products can be iteratively evaluated with those who will ultimately use the tool to further ensure the quality and accessibility of the digital learning tool prior to release. This type of summative evaluation offers product designers and researchers with the ability to effectively understand how the product is used, providing mechanisms for both product improvement and accountability.

Research and Iterative Testing in the Real World: The Sesame Model

Sesame Workshop is exemplary in utilizing formative and summative research in their evaluation methodologies to inform their product decision-making for both their television and digital learning resources. Edtech product developers and researchers can follow a similar approach by establishing the learning outcomes intended from the onset, conducting formative research to inform product development, and then evaluating learner engagement and outcomes in iterations and cumulatively through user feedback.

Sesame Workshop’s model focuses on conducting learner-testing with children and adults to inform and improve production decisions as well as secondary research on children’s learning and development. The biggest benefit of child-centered research is in the direct changes and improvements that can be implemented after testing a product, app, game, or tool with the consumers themselves: kids! This invaluable feedback reduces the risk of project failures, design incongruencies, and unforeseen challenges and helps us deliver the best educational products for kids and families.

Here are several examples of how user-testing benefits the final product:

The Art of Surprise

• Game developers wanted to try having a narrator give playful “anti-instructions” for The Monster at the End of this Game, such as “Do not go on to the next level!” and “You must not complete the game!” There was concern that, with children still developing abstract thinking, they would not understand the humor and would, instead, stop playing the game. Summative testing with 3–5-year-olds revealed that they loved the anti-instructions, responding positively with reactions like “Oh yes, I will keep playing!” and “Can’t stop me!” Without kid-testing, the developer’s assumptions may have led to the unnecessary removal of the enjoyable anti-instructions.

Playful Learning Experiences

• Children learn through play by actively engaging in activities like exploration, problem solving, and communication. Developers wanted to do formative research to utilize the years of research on guided play and its benefits to design a digital experience that leveraged these findings with augmented reality technology. The result was a game called Grover’s Block Party. In the game, kids play, build, and experiment with words using a physical wooden letter set that Grover can “see” and respond to. This game stimulates a guided play experience for children, allowing them to learn and explore. The second phase of research involved playtesting and revealed that giving the kids a lot of control in the game and immediate feedback on tasks was critical to its success. Kids also loved that they were able to see themselves on-screen while playing, which led to high engagement and learning.

Usability in Action

• An educational game, Bert & Ernie Reuse and Reinvent, was developed to promote recycling and upcycling items. Players were able to take everyday items such as a shoe box and turn it into a guitar. In the summative study design, there was a game point that involved subtabs to add different things such as stickers to the child’s creation. The sub tabs were hard for children to notice, even with in-game highlighting. Children often needed a parent to help or moved on without interacting with the tabs at all completely. To combat these challenges, developers made the subtabs pulse, had characters voice over reminders, and added two different buttons for kids to click so they could navigate the subtabs more efficiently. These changes ultimately benefited kids by increasing their agency and enhancing their learning of STEM concepts including problem-solving and critical thinking.

These examples highlight the essential learnings young children can provide during each phase of the research process. When users can interact with products more easily, they are more likely to engage with them, and thus the products are better able to have a meaningful impact on their learning and growth.

For more tips on how to best incorporate learning sciences research in product development and strategies for user testing with kids, check out the new toolkit created by the Joan Ganz Cooney Center at Sesame Workshop. To see which products have earned Digital Promise’s Research-Based Design Product Certification, dive into our list of certified products.