Designing AI Infrastructure for Longevity and Interoperability: Additional Takeaways from the K-12 AI Infrastructure Grantee Convening

Read part 1 of this blog series.

Building on the convening’s opening discussion of privacy, rigor, and open science, an expert panel focused on how the public goods grantees are building can stay useful over time. As foundation models are rapidly evolving, district policies and vendor terms shift, and classroom needs differ across states and communities, the panel considered what it takes for datasets, benchmarks, and models to remain reliable, interoperable, and trusted long after their initial release.

Moderated by Dr. Pati Ruiz, panelists included: 

• Dr. Yang Jiang, Research Scientist, Educational Testing Service
• Dr. Maciej Pankiewicz, Senior Research Investigator PennGSE | Associate Director, Penn Center for Learning Analytics
• Dr. Bodong Chen, Associate Professor PennGSE | Director, Knowledge Building Innovation Network
• Emma Braaten, Director of Digital Learning, Friday Institute

Bringing together perspectives from measurement science, learning analytics, learning sciences, and district leadership, the discussion initiated by defining artificial intelligence (AI) infrastructure. Dr. Chen described infrastructure at three levels:

• Technical: benchmarks, pipelines, and models that people take for granted until they break.
• Relational: the different ways people depend on, ignore, or are obstructed by those objects.
• Organizational: the curriculum, reforms, and everyday routines that keep schools running.

Citing sociologist Susan Leigh Star’s concept of the “installed base,” Dr. Chen urged developers to study what already exists in schools before building on top of it.

The panelists’ insights pointed to several design considerations for building durable public goods:

Separate Fast and Slow Layers

Knowledge building as a pedagogy has existed for more than 40 years. In contrast, foundational AI models change rapidly, often getting updated by providers every few months or even weeks. To manage this difference, Dr. Chen’s team distinguishes pedagogical designs from the AI model itself. This way, when a model changes, they only have to adjust small components—like prompt structure and output processing—, keeping maintenance minimal. Dr. Chen emphasized that infrastructure is a long-term commitment, and teams must plan and budget for repair and maintenance from the start.

Dr. Pankiewicz reflected on lessons learned in building data pipelines that outlast models, describing his lab’s process for de-identifying transcripts of coach-teacher discussions. His lab developed an LLM to pre-process data to remove personally identifiable information, which can save up to 80% of the time human de-identification would take. Human reviewers then compared original and redacted versions and produced the final, de-identified versions. He highlighted a broader challenge in maintaining AI-based pipelines over time: proprietary models may be discontinued or become unavailable, while new open-weights models are regularly released, creating opportunities to improve performance.  He reflected that making a pipeline reusable for the next model, or for another team, requires a meaningful share of additional effort, and noted it’s rarely accounted for in project planning and budgeting. He also raised a related gap: the field lacks public benchmarks that show whether switching the underlying model helps or harms performance. He argued that evaluation should be built into products or available as public-facing benchmarks, not left as overhead for districts or researchers to absorb.

Pair Data with Context

Drawing on her work with process data, including National Assessment of Education Progress data, Dr. Jiang cautioned that clicks, pauses, and keystrokes are not evidence on their own. The same behavior can mean different things. Spending a long time on an item may signal productive struggle, confusion, or disengagement. Using a calculator before writing an answer may reflect building a solution, while using it afterward may reflect checking work. Dr. Jiang recommended starting from theory about which constructs are being measured. She also recommended validating indicators against multiple sources, such as outcomes, self-reports, human coding, and expert think-alouds. For infrastructure, this means sharing raw process data together with documentation of how indicators were defined, derived, and validated.

Design the Full Lifecycle

Dr. Braaten, who became a district digital learning director on the day schools closed in March 2020, described the friction districts face. Staff time is limited. Vendor terms of use change without notice. Every vendor brings its own customized workflow. She identified offboarding as an often-overlooked reason districts decline tools. “We give a lot of energy and effort into the first 80%, but that last part matters tremendously to schools and districts,” she said.

During the audience discussion, attendees raised the tension between public benefit and district reluctance to share data. One noted that some partners worried their data could reach frontier AI labs. Another suggested starting with minimal data collection and offering opt-in features over time. Dr. Braaten added that a trust page alone is insufficient, calling for clear feedback mechanisms and accountability when problems occur.

She also introduced the idea of stewardship. With frequent staff turnover, districts need systems that anyone can understand and manage. That requires clear vocabulary, visible change logs, and sustained updates. 

Looking Ahead

Durable AI infrastructure requires more than strong initial releases. It depends on planned maintenance, transparent documentation, and partnerships that treat districts as co-designers. Through continued collaboration, K-12 AI Infrastructure Program grantees can build public goods that remain useful as the technology evolves.

Want to stay up-to-date with the K-12 AI Infrastructure Program? Subscribe to our newsletter and learn more about our current request for proposals.