Rapid Cycle Pilots: Improving Ed Tech Products through Feedback

This report presents case studies from three Pittsburgh area school districts, all of which are members of the Digital Promise League of Innovative Schools: Avonworth, Elizabeth Forward, and South Fayette. Digital Promise and Carnegie Mellon University researchers worked with each district to pilot one to two digital courseware products as a way to gather evidence of product effectiveness to inform purchasing decisions. The study, conducted in the 2015-2016 school year, included a total of over 700 students and 30 teachers.

Key findings relate to both the pilot process and outcomes. First, districts that conducted a thorough needs analysis and chose a product well-suited to their goals and objectives had a more meaningful pilot and improved results. It was also important to have alignment between a district’s instructional or curricular needs, and the product’s intended use. Further, small to medium-sized pilots that involved only one grade level or subject area were more successful in identifying suitable products, compared to pilots that tested products over multiple grade levels and subject areas. Finally, technological issues, delays in product rollouts, and device limitations posed barriers to pilot success.