Building Public Goods with Rigor: Key Takeaways from the K-12 AI Infrastructure Grantee Convening

Amid increased “techlash” over whether artificial intelligence (AI) in education supports authentic learning while protecting student privacy, the K-12 AI Infrastructure Program is funding a multi-year effort to develop privacy-focused, open-source digital public assets grounded in learning sciences research. Supporting 12 newly awarded grantee projects, the program focuses on developing high-quality datasets, benchmarks, and models for K-12 formative assessment. To align these public goods with real-world needs, grantees, researchers, educators, and technical partners gathered for the K-12 AI Infrastructure grantee convening. Together, attendees addressed current industry challenges, aligned on research standards, and explored key design choices necessary for broad edtech adoption.

Navigating Barriers with Evidence, Privacy, and Open Science

Opening the convening, Dr. John Whitmer, senior researcher and founder of Learning Data Insights, and Dr. Amy O’Hara, director of Massive Data Institute at Georgetown University, set the stage by addressing the public skepticism surrounding AI in education, particularly when AI-enabled edtech products are developed with inflated expectations, lack of direct educator involvement, and unaddressed data privacy concerns. Dr. O’Hara and Dr. Whitmer provided guidance on avoiding such missteps and underscored the importance of developing products with adequate rigor and baseline evidence.

The discussion emphasized key shifts necessary to lower privacy-related risks in the development of public goods:

• From Administrative Barriers to Proactive Guidance: Developers and researchers face complex hurdles, from shifting administrative orders prohibiting certain privacy methods (e.g., noise infusion) to strict consent frameworks. Our program partners at Georgetown’s Massive Data Institute (MDI) and Learning Data Insights (LDI) address this by providing grantees early proactive hands-on guidance on data governance, licensing, and privacy-preserving techniques to ensure public assets are defensible.
• From Low Replicability to Standardized, Transparent Public Goods: Amid concerns about low replication rates in AI evaluations, our program works directly with grantees to declare methodologies early, adopt pre-registrations, utilize standardized data ontologies (such as evaluation cards), and publish clear code documentation so assets function as reusable public goods.
• From Siloed Research to Knowledge Sharing: Moving beyond isolated research labs and one-off datasets, our goal is to support a community of practice in adopting open science frameworks, exploring automated annotation tooling, and leveraging tiered data-access models informed by exemplars like Cornell’s National Tutoring Observatory (NTO) Million Tutor Moves dataset and AI2’s adaptive tutoring benchmarks.

EdTech Panel: What Developers Need from Digital Public Goods

A highlight of the convening was a developer-focused panel featuring Peter Gault, founder and executive director of Quill.org, and Kedaar C. Sridhar, co-founder and chief product and technology officer of M7E AI. The discussion explored what is needed for the adoption of public goods and centered on what technical features and design decisions will maximize usefulness and uptake by edtech developers.

Core Developer Insights

1. Backward Engineering from the Use Case
   • Panelists stressed that datasets must be designed with the end application in mind. Gault noted that while large repositories, like the Persuade corpus, provided immensely valuable raw student writing, datasets lacking a clear, repeatable pattern are difficult to translate into reliable algorithms.
   • Sridhar shared how M7E AI had to synthesize research across more than 300 papers to create linguistic criteria for math comprehension because existing public datasets lacked not only state-specific nuances (e.g., Texas TEKS vs. Florida BEST), but also sufficient coverage of linguistic accessibility, cultural context, and student-interaction data.
2. Decoupling Annotations from Raw Work
   • Panelists advised researchers to recognize that developers may disagree with a dataset’s underlying pedagogical assumptions, such as the style of Socratic coaching or scaffolding.
   • Making raw student work easily accessible enables external teams to re-annotate datasets for different pedagogical frameworks without having to recreate the foundational data.
3. Designing for Evaluators and Benchmarking
   • Effective dataset development requires rapid testing loops. Gault recommended splitting datasets early; reserving a portion for testing; and using simplified, deterministic scales (such as binary 0/1 ratings instead of complex multi-point rubrics) to reliably evaluate model performance.
4. Iterative Collaboration Over Version 1 Perfection
   • Building AI infrastructure is a long-term, multi-year process. Closing the gap between academic researchers and commercial developers requires participatory design and continuous feedback loops to refine public goods over time.

Looking Ahead

Creating transformative digital public goods requires interdisciplinary coordination. Through systematic check-ins, technical communities of practice, and direct support from project partners, the K-12 AI Infrastructure Program grantees are positioned to set a new standard for responsible, open, and impactful edtech development.

Want to stay up-to-date with the K-12 AI Infrastructure Program?

• Learn more about our new Request for Proposals.
• Register for our RFP Information Session: Integrating Public Goods to Advance AI-Enabled Formative Assessment on Monday, Oct. 26, at 4 p.m. ET.