Though many edtech companies claim their products are
research-based and effective, some educators say they have trouble believing these claims when targeted students and school contexts are ambiguous. Educators can better ensure the products they use are grounded in valuable and relevant research by asking three questions, which we dive into below.
What qualifies as “research”?
Each of the six edtech entrepreneurs we interviewed agreed that research is important in product development. Several edtech entrepreneurs explained that they use learning sciences research to inform the development of their product. Most commonly, those interviewed cited pilot studies as a key source of evidence for iterative improvements to their products.
Julia Dexter, co-founder and CMO at Squiggle Park, said her product is “completely data-driven so every decision we make has to be founded in some kind of data. […] Making data-based decisions is one of our grounding principles.”
Dan Tu-Hoa, senior vice president at Mathspace, and Mitch Slater, co-founder and CEO at Levered Learning, agreed that the best edtech companies use published learning sciences research to inform the pedagogical approaches and instructional designs of their products. Additionally, Dexter at Squiggle Park hired a team member with a doctoral degree in language development to develop the product in alignment with research-based instructional strategies.
Beyond using existing research, product developers also work to gather their own evidence through rapid-cycle pilots. For example, Matt Nupen, CEO and co-founder of Insert Learning, explained that he relies on his own extensive classroom teaching experience in addition to teacher feedback from pilots to develop and refine his product.
While all six edtech entrepreneurs have used rapid-cycle pilots to improve their products, interest in conducting and participating in pilots varies by the development stage of the product. Commonly, developers collect interview and focus group data, student and teacher survey data, assessment data, and data collected by the product itself in pilots to evaluate the product’s efficacy.
Steve Tardrew, vice president of assessment and research at Achieve3000, and Kevin Miklasz, vice president of data and prototyping at BrainPOP, explained that their companies predominantly rely on internal research teams to continuously assess products, often in the form of edtech pilots in collaboration with school districts. Entrepreneurs from well-established companies noted that piloting has to meet a company goal, such as marketing, building a relationship, or beta testing. In contrast, the early-stage companies we spoke with were eager for pilots to learn more about their product’s effectiveness.
What motivates developers to use research?
Most of the entrepreneurs we interviewed explained that rapid-cycle pilots serve two main functions: one, to potentially receive exciting data to demonstrate product efficacy; and two, to initiate and develop a relationship with a school or district.
Tardrew of Achieve3000 explained, “Piloting is always helpful because we know there is interest on the part of the customer or potential customer on learning about our offerings and gives us an opportunity to demonstrate the effectiveness of the product. You also learn a lot along the way and it’s a chance to really build a relationship with the customer.”
Additionally, many entrepreneurs found piloting to be useful in identifying specific student populations that are best served by the product.
What specific decisions are informed or influenced by research?
The research gained from running pilots, specifically feedback from students and teachers, often leads to changes to a product’s interface and implementation supports. For example, some of the entrepreneurs commented that the companies have better customized teacher dashboards based on teacher feedback data collected during pilots.
Information gathered from pilots can also provide companies with an evidence-based approach to onboarding current and future customers so educators can feel better supported in properly implementing a product, likely leading to increased impact and effectiveness.
How does this help me select the best products and programs for my students?
- Demand Research Use. Educators should demand edtech companies to incorporate research into their products. These interviews made it clear that reputable edtech companies are incorporating research into product design and using evidence to inform iterative improvements. When working with a credible edtech company, the general question regarding whether research is used should be about which research is used and how research is incorporated at each stage of product development.
- Ask Questions. Unsurprisingly, research use varies substantially in this landscape. There is no cut and dry explanation for the types of research developers use and how that research is incorporated in the product. Thus, it’s important as consumers to push edtech companies to be transparent and open about their use of research.
Educators need to press edtech companies to discuss the ways in which research was used to design a product and how rapid-cycle piloting may be used to improve a product. This is an excellent opportunity to begin an open dialogue with the edtech vendor you’re working with to discuss specific product features that may be most important to you, like data interoperability, and to understand how your input and feedback will contribute to product improvements.
Many of the edtech entrepreneurs interviewed noted that the availability of high-quality, open-source free tools and resources to support them in conducting pilots has improved their piloting process and yielded valuable results to help improve their products. Edtech developers and educators interested in piloting can find useful tools and resources on the Edtech Pilot Framework.
