Analysis of the National Assessment of Educational Progress (NAEP) Mathematics Process

Overview

We are examining the National Assessment of Education Progress (NAEP) process, outcome, and survey data to understand the test-taking behavior and mathematics performance of learners with and without disabilities. Conducted in partnership with Dr. Susu Zhang from the University of Illinois Urbana-Champaign, this project uses innovative statistical and machine-learning techniques.

What Is the NAEP Process Data?

The NAEP process records the series of clicks, entries, and timestamps during test takers’ interactions with the test items. By analyzing this data, our research team can examine the key components of test-taking behaviors of learners with disabilities and uncover patterns that would not be apparent if using outcome or survey data alone. 

We have used innovative machine-learning approaches coupled with expert review to:

• Extract meaningful patterns from process data to reflect key components of students’ mathematics test-taking behavior (i.e., math cognitive process, time on task, level of engagement, and accommodation usage).
• Examine how test-taking behavior differs based on students’ disability status.
• Identify patterns of test-taking behavior that lead to successful performance on the NAEP mathematics eighth grade assessment.
• Investigate the association between test-taking behaviors and test performance for students with disabilities and students without disabilities.

In addition, the research team uses structural equation modeling (SEM) and latent profile analysis (LPA) to understand the degree to which the relationships between math instruction, performance, and key components of testing behavior (i.e., math cognitive process, time on task, level of engagement, and accommodation usage) differ by students’ disability status.

Project Findings

The findings of this study have provided important insights about the test-taking behavior of students with disabilities and their cognitive processes, time on task, level of engagement, and accommodation usage during their interactions with a digital assessment. Results have revealed helpful strategies that students with disabilities can use to solve math problems, as well as features of digital assessments that increase engagement and accessibility and provide more equitable and accurate measures of the performance of students with different abilities.

Grant Award Page

Published Peer-Reviewed Papers

• Kwon, S., Zhang, S. (2025). Explaining Performance Gaps with Problem-Solving Process Data via Latent Class Mediation Analysis. Psychometrika. https://doi.org/10.1017/psy.2025.10038
• Wei, X. (2024). Evaluating chatGPT-4 and chatGPT-4o: performance insights from NAEP mathematics problem solving. Frontiers in Education. https://doi.org/10.3389/feduc.2024.1452570
• Wei, X. (2025). Examining Associations Between Accommodation Types and Student Outcomes: Extended Time and Breaks Individually Versus Bundled. Remedial and Special Education, 0(0). https://doi.org/10.1177/07419325251341178
• Wei, X. (2024). Text-to-Speech Technology and Math Performance: A Comparative Study of Students With Disabilities, English Language Learners, and Their General Education Peers. Educational Researcher, 53(5), 285-295. https://doi.org/10.3102/0013189X241232995
• Wei, X. (2024). Universal Design Element Utilization and Mathematics Performance: Implications for Diverse Student Populations. Journal of Special Education Technology, 0(0). https://doi.org/10.1177/01626434241289951
• Wei, X., & Zhang, S. (2024). Extended Time Accommodation and the Academic, Behavioral, and Psychological Outcomes of Students With Learning Disabilities. Journal of Learning Disabilities, 57(4), 242-254. https://doi.org/10.1177/00222194231195624
• Wei, X., Zhang, S., Zhang, J., & Yu., J. (2023). Mathematics performance, response time, perseverance, and enjoyment of eighth-grade autistic students and their general education peers. Autism. https://doi.org/10.1177/13623613231168241
• Wei, X., Zhang, S., & Zhang, J. (2025). Digital Pencil Usage and Mathematics Performance Among Students with Learning Disabilities and Their General Education Peers. Journal of Special Education Technology, 40(4), 443-455. https://doi.org/10.1177/01626434251314041
• Wei, X., Zhang, S., & Zhang, J. (2024). Identifying student profiles in a digital mental rotation task: insights from the 2017 NAEP math assessment. Frontiers in Education.  https://doi.org/10.3389/feduc.2024.1423602
• Wei, X. (2026). Exploring Time-Use Profiles in Digital Mathematics Assessments for Students With Learning Disabilities. Journal of Learning Disabilities, 59(3), 172-184. https://doi.org/10.1177/00222194251347965

Conference Papers

Our research team has presented on bundled accommodations, text-to-speech technology impacts, universal design in education, and various analyses using NAEP data across several respected conferences, including the Society for Research on Educational Effectiveness (SREE), American Educational Research Association (AERA), National Council on Measurement in Education (NCME), and virtual Principal Investigator meetings for the Institute of Education Sciences. Highlights include the effectiveness of digital tools, accommodations, and profiling cognitive strategies in spatial reasoning tasks:

• Kwon, S. & Zhang, S. (2025). Explaining performance gaps with problem-solving process data via latent class mediation analysis. Electronic board presentation at the Annual Conference of the National Council on Measurement in Education, Denver, CO.
• Kwon, S., & Zhang, S. (2023, November). Explaining performance gaps with problem-solving process data via latent class mediation analysis. Presentation at the 2023 Ideas in Testing Research Seminar, Chicago, IL.
• Kwon, S. & Zhang, S. (2024, July). Explaining performance gaps with problem-solving process data via latent class mediation analysis. Paper presentation at the 2024 International Meeting of the Psychometric Society, Prague, Czech Republic.
• Wei, X. (2024, September). Are Bundled Accommodations More Beneficial Than Individual Accommodations? Effectiveness of Breaks and Extended Time Accommodations. Paper was presented at the SREE conference, Baltimore, Maryland.
• Wei, X. (2024, April). Examining the impact of text-to-speech technology usage on math performance: Insights from the 2017 NAEP assessment. Paper was presented at the AERA conference, Philadelphia, Pennsylvania.
• Wei, X. (2025, April). Evaluating chatGPT-4 and chatGPT-4o: Performance insights from NAEP mathematics problem solving. Paper presented at the AERA conference, Denver, Colorado.
• Wei, X. (2024, April). Universal design element utilization and math performance: Implications for diverse student populations. Paper was presented at the AERA conference, Philadelphia, Pennsylvania.
• Wei, X. (2025, April). When More Isn’t Better—Extended Time and Separate Sessions vs. Individual Accommodations’ Impact on Performance. Paper presented at the NCME conference, Denver, Colorado.
• Wei, X., Zhang, S., & Zhang, J. (2023, April). Use NAEP process and performance data to examine the impact of extended time accommodation for students with learning disabilities. Paper presented at the AERA conference, Chicago, IL.
• Wei, X., Zhang, S., & Zhang, J. (2023, April). Use NAEP process data to profile cognitive strategies for solving spatial problems. Paper presented at the NCME conference, Chicago, IL.
• Wei, X., Zhang, J., & Zhang, S. (2022, April). A tale of two disabilities: Finding from NAEP response and process data. Paper presented at NCME conference, San Diego, CA.
• Wei, X., Zhang, S., & Zhang, J. (2022, May). Use NAEP process and performance data to examine the impact of extended time accommodation. Paper presented at the Virtual IES PI meeting.
• Wei, X., Zhang, S., & Zhang, J. (2022, September). The effect of digital drawing tools on solving math problems in a large-scale assessment. Poster presented at SREE conference, Washington, DC.
• Zhang, S., Wei, X., & Zhang, J. (2022, May). Examining the differences in item revisit patterns across disability groups. Paper presented at the Virtual IES PI meeting.
• Zhang, S., & Wei, X. (2023, April). Math test-taking behavior of students with learning disabilities: A natural language processing approach. Paper presented at the Virtual IES PI Meeting, Virtual.
• Zhang, S., & Wei, X. (2023, April). Math test-taking behavior of students with learning disabilities: A natural language processing approach. The paper presented at the AERA conference, Chicago, IL.
• Zhang, S., & Wei, X. (2023, April). Explaining performance gaps with problem-solving process data via exploratory mediation analysis. Paper presented at the NCME conference, Chicago, IL.
• Zhang, S. (2024, April). Revision and Review Behavior in Large-Scale Computer-Based Assessments: An Analysis of NAEP Mathematics Process Data. Invited poster presentation at the 2024 American Educational Research Association (Excellence in Education Research: Early Career Scholars and Their Work), Philadelphia, PA.

Follow Our Updates

Follow our updates through Digital Promise’s LinkedIn.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324P230002 to Digital Promise. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.