October 28, 2021 | By Nidhi Hebbar and Madison Jacobs
And while we’ve seen some districts investing in culturally responsive teaching, educators checking their assumptions at the door to better understand their students’ needs and perspectives, and administrators chartering new teams to ensure equitable instruction for students—we have also seen horror stories of technology product failures in other industries: racially-biased pre-trial risk assessment in the criminal justice system, discriminatory job recruitment and healthcare technologies, racist surveillance predictions, consumer-lending discrimination, and unfair algorithmic loan decisions.
Edtech tools and digital solutions are leveraging artificial intelligence (AI) in education at a rapid speed to address the promise of efficiency and personalization in schools. We must get ahead of the threats racial bias poses to AI in the education and education technology industry. As we’ve developed the first Product Certification to hold edtech tools accountable for racial equity —Prioritizing Racial Equity in AI Design—we’ve heard questions from our thought partners about race and racism in product design. Below, we share insights into the way we’ve structured this certification and considered these important questions.
It’s not really that race on its own has to do with artificial intelligence (AI), but rather, racial bias. Products that leverage artificial intelligence use data, or information, to learn how to interact with users.
Edtech companies are responsible for designing for Black and Brown learners, which requires edtech companies to understand them and champion their needs. Without rigorous and intentional design along with ongoing oversight and accountability, edtech products that use AI and machine learning can amplify the existing racial biases already present in the data within our school systems and introduce new biases through assumptions in algorithmic design.
Though it may sound counter-intuitive, AI-driven products that do not explicitly collect learners’ race data can still create racially-biased tools. Why? Because racial bias impacts many aspects of students’ educational experiences; this impact is reflected in commonly used, seemingly race-agnostic data in ways you might not expect. This is, after all, the definition of a blindspot.
As AI becomes commonplace in education solutions, the design decisions and assumptions that affect students’ learning experiences become increasingly opaque to students, teachers, and families. As schools begin to adopt more AI-driven technologies, it’s important that the edtech market develop the language to bring schools and families into the conversation about how products are designed to ensure fair and transparent products for Black and Brown students.
Our tools provide tangible practices edtech teams should implement at each stage of the design and development process to uncover and address racial bias. It’s time edtech companies take a deeper look into the data they collect and how their products impact Black and Brown students.
In education, as well as many other societal institutions, there is already a long history of systemic racism embedded into many of the practices we consider standard in schools today. While all aspects of bias in AI are concerning, AI can exacerbate racial bias by amplifying existing bias and by encoding new biases into edtech products used in schools. This further impacts students and communities that are most often underserved and underestimated in schools. This makes it paramount to start with race so that both edtech companies and schools are equipped to identify and mitigate for racial bias as AI becomes commonplace in schools.
Black and Brown students are consistently the most underserved and underestimated in U.S. schools across behavioral and academic support. This further intensifies the challenges that Black and Brown students face when race intersects with students’ other statuses such as language, income, and disability status.
This is not the end of the conversation, but rather, the beginning. The principles encouraged by the certification can and should be used to consider issues of bias pertaining to income, opportunity, disability status, and other under-supported communities of students and families.
First and foremost, this product certification provides language and vocabulary for companies to identify and communicate the processes through which they mitigate racial bias in their products’ design. We’ve seen a lot of AI principles and frameworks designed to guide responsible development, but they felt too abstract to really hold the edtech market accountable. By defining a foundation of tactical basic practices that companies should adopt to prioritize learners of color, we hope this shines a light on the path for companies that need to address gaps in their design and development processes—which should lead to products that are designed for learners of color from the start. The certification also aims to highlight companies who have already built products with learners of color in mind.
The certification is a first step in our vision to require companies to design for and provide evidence of practices that prioritize racial equity in their products. By certifying companies that do so, we hope to make procurement and product selection easier for educators to identify products that were designed with their learners in mind.
We also believe that schools must champion the needs of their Black and Brown students and hold edtech companies accountable for their impact on those students—before purchase. In addition to working with edtech developers, we have resources that empower schools by helping them ask the right questions during the procurement process. Schools can become powerful partners in advocating for racially equitable edtech products.
To learn more, check out Digital Promise’s Product Certifications. Keep an eye out for the launch of our Prioritizing Racial Equity in AI Design certification pilot, which launches November 4, 2021.
