Would you buy a car without taking a test drive? Despite the proliferation of trusted expert reviews, consumer feedback, and official ratings, 84 percent of Americans will only buy a car after taking it for a spin first.
Now think about the educational technology market, where there is a shortage of trusted information about available products. It makes sense that school districts rely heavily on testing out products, or “pilots,” before deciding what learning technology to purchase for teachers and students.
Our recent study, Improving Ed-Tech Purchasing, gathered perspectives on educational technology procurement among school district decision-makers and technology providers. We found that school districts don’t always trust the product information that tech companies provide them and don’t regularly use third-party sources of information, either. Instead, we found schools turn to pilots as a proxy for market information to make purchasing choices.
But are these “pilots” leading to the best possible information for making ed-tech purchasing decisions? We’re not so sure.
In interviews for our study, school districts often report pilots as informal processes for trying learning software – more akin to kicking the tires than taking a test drive followed by a 50-point inspection.
For example, according to one technology director,
“[O]ftentimes pilots become a license for [teachers] to use [the product], so they forget that it was a pilot and there’s no real analysis done… It’s a real challenge.” Technology Director
These pilots take on informal structures because the layers of bureaucracy in school districts, especially larger ones, mean limited communication between teachers and district-level administrators, our study found. So either teachers try new products without the guidance of the business office or instructional leadership, or those same administrators bring new products into the district without assessing whether teachers actually need them.
As one superintendent said,
“There is a temptation to just to buy a whole lot of things that then end up sitting on the shelf, end up not being used and not being used in a proper way.” Superintendent
On the other side, technology providers report mixed satisfaction with the chance to have their products piloted, especially in a crowded market. Just 23 percent of providers report satisfaction with their ability to gain visibility in a district. District stakeholders seem to agree.
“If there’s a good vendor out there doing wonderful things, it’s hard to find that vendor.” Superintendent
Our study found even less satisfaction from providers with opportunities to turn pilots into broader implementation within a district. No one would expect every pilot to turn into a purchase, but there should be shared understanding of what success looks like and the process for continuing or scaling up use of a product.
Technology providers we surveyed say that’s not always the case. Only 11 percent report satisfaction with information from districts on buying cycles and purchasing policies.
So what can we do about all of this?
Giving students opportunities for an excellent education is more important than deciding which car to buy. It’s critical that learning software pilots are well-designed from the start, implemented and evaluated with clear methodology throughout, and ultimately lead to purchasing decisions based on results and learning.
This is an area where school leaders and technology providers agree. Both express interest in developing guidelines for better pilots.
Creating better pilots is a key focus area for our Marketplace initiative going forward. By starting with clear learning goals and instructional challenges, getting teacher input from the outset, and establishing guidelines for evaluating success, pilots can be a key tool for schools in putting the right products in the hands of teachers and students.
Do you have great examples of schools and companies piloting learning software programs with rigor to inform data-driven purchasing decisions? Please share them in the comments below, or contact marketplace@digitalpromise.org.
