Pittsburgh Schools and Developers Share Tips for Improved Ed-tech Pilots

Everyone agrees more information is helpful when it comes to choosing the right ed-tech products to support student learning. But is it worth the extra time and effort for schools and product developers to go beyond an informal product pilot and adopt a more rigorous approach to collecting data?

For ed-tech pilot participants in the Pittsburgh area, the answer is a resounding “yes.”

With guidance from Digital Promise and Carnegie Mellon University (CMU) researchers over the past year, staff members from three districts, as well as ed-tech developers discovered the value of participating in a structured pilot.

Below are two key insights from this project’s full report. These insights focus on the importance of:
1) finding a research partner, and
2) providing opportunities for feedback and learning for all pilot participants.

Key Insight #1: Research Partners Improve the Pilot Process

Because CMU and Digital Promise were “external third parties,” participants trusted the process, and felt safe sharing honest feedback. According to Todd Keruskin, Assistant Superintendent at Elizabeth Forward School District, “It was a relief on my end to have a third party gather feedback. CMU really took their time to talk with our teachers and get feedback from students.” Scott Miller, principal at Avonworth Primary Center, added, “That relationship supported authentic feedback in the process.”

CMU and Digital Promise also provided research and logistics expertise. They managed IRB (Institutional Review Board) processes, created data sharing agreements, established realistic timelines and goals, and designed methods to collect and analyze data. As a result, schools and developers could focus on implementing products and responding to feedback.

The research partners also offered suggestions to help developers continuously improve their products. For example, Karen Wang of Amplify said, “The CMU team was a great collaborator with us. There were some new things we wanted to try out, like an engagement activity we called the Game Squad. They were great with thinking through how to track the impact.”

Key Insight #2: Opportunities for Feedback and Learning Improve the Pilot Process

In the Pittsburgh pilots, there were abundant opportunities for partners to share feedback and learn from one another; this led to improved processes and products. School leaders found it beneficial to review ongoing feedback from teachers and students throughout the project, and this deep well of information increased their confidence when making purchasing decisions.

Also, in contrast to traditional evaluations where companies only receive a final report, developers could capitalize on real-time feedback to make product improvements. For example, in South Fayette Township School District, INVENTORcloud changed its program rollout in response to student requests. In several pilots, developers who responded quickly to feedback also received a secondary benefit: this generated trust with district leaders, and translated into continued contracts with schools — despite some hiccups with products during the pilot.

Finally, the high level of communication between partners helped districts and developers to gain new skills and knowledge in running pilots. For example, Miller said, “If I was doing it over again, I would just use one product. …It would give the researchers more opportunities to have more observations with one tool, and provide a deeper analysis.” Wang, of Amplify, said, “CMU…had a whole script and program for what focus groups would look like. This was not something we had not done in the past with our customers. It was some of the best information we have gotten from customers in the last school year.”

Getting Started: Find Partners for Your Pilot

As the number of ed-tech products grows, the need for trusted information to help connect schools with quality products is increasing. Local pilots with tightly-linked, engaged partners (like those in Pittsburgh) are a promising path forward. Although forming partnerships requires effort, Keruskin said the process aligns with his vision for his school’s culture: one where everyone is “learning together” about what works, and having open discussions about new technologies.

To get started, districts and developers can review this project’s findings, and Digital Promise’s resources for conducting pilots. Additionally, Marti Louw, Director of the Learning Design Media Center at CMU, recommends taking time to find the right research partner, because “trusting the process and trusting the partners is essential.” With the right partners in place, schools and developers will have guidance for how to structure the pilot so they can collect quality feedback.

Developers seeking research partners can check out our Using Research in Ed Tech guide, and Columbia Teachers’ College and Johns Hopkins University services for ed-tech companies. Schools can partner with local universities, or research organizations such as Harvard Research Schools, the University of Chicago Consortium on Chicago School Research, and Regional Educational Laboratories (RELs). The William T. Grant Foundation also offers a wealth of resources on how to bridge research and practice.

With close partnerships, high levels of communication, and open minds, schools and developers have a better chance of gathering useful information that can improve decision-making, the quality of products, and ultimately student learning.

To learn more about the Pittsburgh area ed-tech pilots, read the full report.