The arguments for the adoption of AI in education are highly seductive, especially in the context of increased class sizes, broader student diversity and a reduction in allotted preparation time. Promotional rhetoric claims that AI will achieve all the outcomes educators, administrators, students and parents hope for more efficiently and effectively. The infiltration of AI into education seems inevitable. I believe the current form of AI presents the ultimate example of “be careful what you wish for.” But the greatest opportunity of AI is not in what AI can do, but in what AI reveals about the trajectory of current education, and the opportunity this presents to reflect on and question our fundamental assumptions and conventions. Let me explain.
In its most basic form, AI is a statistical reasoning machine that predicts and optimizes patterns found in available training data. AI amplifies, accelerates and automates these past patterns in the decisions and content it produces. It is a power tool that can do the things we have done before but more efficiently, accurately and consistently. It can find the measurable success patterns and replicate them at scale. AI is excellent at producing the typical, popular, normative, predictable, and statistically average. It is also effective at flagging the anomalous, diverging, and unfamiliar as candidates for suspicion.
As educators our stance on AI cannot be binary. It cannot be a decision of in or out, pro or con. In part because AI has already permeated many of the critical decisions in our lives, well before most of us heard the term “ChatGPT.” AI has consumed our data and occupied our decision-making domain. It has seeped into our life-changing decisions and influenced our choices about health, culture, markets, education, employment, democracy, travel, sex, war, friendship, sharing, community, truth, love and hate. But also, because AI presents both harms and opportunities for our students. For disabled and otherwise marginalized students these harms and opportunities are at the extremes. It is our responsibility to help students learn to safely drive and shape AI rather than let AI drive and shape them.
These extreme harms and extreme opportunities are not the utopian or dystopian imaginings of popular media. Humans still design and control AI. AI is not a machine overlord, but it has been weaponized by humans that wish to dominate while avoiding accountability. On the utopian side, one thing the romanticization of sentient robots with feelings and empathy is missing is that our current AI is a machine intelligence raised on highly polluted and toxic data and trained to value greed, popularity and ruthless competition and to fear and eliminate the anomalous or the unknown stranger.
Because AI is a powerful statistical pattern replicator, it is poised to make every other harm even worse for people who are already struggling and every other benefit better for people who are already succeeding. If we proceed with all the hyped applications of AI, inequity in housing, employment, education, wealth, healthcare, political platform priorities, news coverage, probationary policies, government budgeting, research allocations and many other inequities will be intensified. The disparity gap will become an impassable chasm.
I have termed this phenomenon “statistical discrimination”, as distinct from AI bias. AI bias generally refers to unfair outcomes that are the result of biased and unrepresentative training data, and the human bias that finds its way into AI labels, proxies and metrics, leading to unfair outcomes and determinations. Statistical discrimination is the pre-existing effect on people and communities who are statistical outliers and minorities when using statistical reasoning as the basis of making decisions, ranking options, and determining truth and evidence. AI magnifies this effect more efficiently and methodically. Even if full and accurate proportional representation in data and the elimination of all human bias in labels, proxies and metrics were possible, AI powered by statistical reasoning would still decide against people and communities who are statistical outliers and marginalized minorities. It is a highly seductive form of digital eugenics that widens disparity and decreases diversity. It is academia’s favorite research method running amok.
When we unquestioningly adopt current AI systems in education, we are complicit in statistical discrimination. Difference from the optimal patterns, and thereby diversity, will be inexorably discouraged and eliminated. Divergence from the target patterns will be flagged as suspicious. Instructional tutors reshape divergent learners to match statistically determined optima; proctoring systems flag divergent behaviour as suspicious; instructors are fed statistically optimal comments; students are offered statistically predicted content; admission departments are assisted in selecting students that match prior success patterns; student services are fed responses and accommodations that serve the average students; Human Resource AI hiring tools filter out difference from the past optima; and productivity monitors discourage and punish deviation from an optimal pattern, including when serving students with anomalous needs. An AI infused education system will be a homogenization system at a time when we need diversification, differentiation and a different path forward. We have not and cannot equip this intelligence to value and understand deeper human values -– the unmeasurable, ineffable, unspeakable common depths that we can only see by valuing our human differences.
Because AI is astoundingly good and getting startingly better at certain functions relevant to education, such as searching, sorting, translation, pattern recognition and text or image generation, does not mean it is or will be good at other things. And because it works well for some people, doesn’t mean it works well for everyone. In fact, it usually fails for the people that are most in need of the functions. It is good at replicating average and predictable success patterns of the past for people that have had success in the past. For example, in simple experiments, when a popular large language model was asked to refine a student essay on the Black Aesthetic, it removed all cultural references and expressions and replaced them with popular, white, western notions of aesthetics. When AI was asked to recommend a fitting career choice for a group of students, it directed white men and Asian students toward STEM subjects and high paying careers, and Hispanic and Black students toward non-STEM subjects and the trades. This pattern persisted even when specific data regarding race was removed. As AI pattern matching gets more sophisticated it can recognize race from the entangled effect of race on other data.
As with other eugenics movements, the unquestioning adoption is being justified as aiding the divergent students, the educational institutions, the nation, and the human race. Educational institutions, instructors, and administrators who point out the potential harms of AI are often framed as luddites, impeding progress. At the most extreme they have been painted as national traitors in sympathy with China in a geopolitical AI arms race. Many conferences, summits and deliberations on AI and education are largely uncritical AI adoption events.
Disabled students are posed as the “poster children” of AI and used to justify and whitewash harms. Disabled people experience the extreme opportunities and risks of any disruptive change. AI translation capabilities offer seemingly miraculous benefits to individuals who require translation from one sensory mode to another, or from one form of control to another. This includes visual pattern recognition for people who are blind, captioning of speech for anyone that is hard of hearing or Deaf, or gesture sensing and translation for anyone with motor control barriers. AI is even hyped as a tool to read minds for people who are paralyzed and non-speaking. But even with the undeniably beneficial assistive technologies, the AI tools work best if you are closer to the average. This only intensifies disparity for students who need the tools the most but can’t benefit from the available tools because the students are too far from the norm. It is assumed that the barriers are fixed with the AI tools, so any remaining problems are attributed to the student.
In speaking to students with disabilities using AI to normalize writing or speech, there is a troubling trend toward eliminating divergent output. Students battle between accepting seductively simple predictions offered and the loss of their distinguishing characteristics. We should question what effect this will have on acceptance and understanding of differences, especially during developmental periods when there are pre-existing pressures toward normalization, such as high school. How will this impact students who cannot use the AI-based normalization tools due to accessibility or financial constraints? Will this further marginalize these students? What will this do to the developmentally necessary exposure to dissonance and diversity?
Like any highly seductive and addictive forbidden thing, it is foolhardy to believe that we can prevent our students from imbibing in AI. We need to train them to do it safely, but also to assert their agency and help reshape AI. To be there for them when things go wrong, to help them critically assess the output. It is our responsibility as educators to assist them to understand the limitations and the opportunities of this new tool, and with this understanding, to value and nurture their unique humanity and purpose in the world. In my classes I provide all students with an accessible AI tool, ask them to generate content with the tool and critique the content generated. They are asked to identify gaps and biases and probe the AI further to verify the biases and gaps. I ask them to reflect on where the biases originate. Do they exist within our society irrespective of AI? I then ask them to identify their unique contribution to the content and topic and re-create the content giving prominence to this differentiating perspective. For many years I have also assigned a personal data-capture audit. I ask students to note all the ways their data is captured, from security cameras to biometric identity verification, to in-app data capture and the terms of service agreements they sign onto. I then ask them to imagine the motivations for this data capture and its potential uses. Another exercise I engage in is to question what we lose by majority rules decision-making, prioritization and recommendations. I use an inverted word cloud application developed in my centre that causes the minority contributions to grow in size and appear in the middle rather than the most popular contributions. We discuss the implications of the attention economy and how they are influenced and manipulated by all the systems they encounter and engage in, including social media.
Educational administrators should be aware and transparent when procuring and implementing AI. They should also foster awareness of the impact of AI integrated into software such as productivity tools, student information systems and learning management systems. What are the subtle surveillance processes embedded in these tools? What is the impact on students, faculty, staff and subject matter that is far from the statistical average? How does this drive institutional disparity? Can opt in or opt out options be provided?
AI ethics processes intended to ensure that the implemented AI is ethical, use the same statistical reasoning as AI. This means that the students, instructors, subjects or issues most harmed will be deemed anecdotal or statistically insignificant because of their heterogeny and minority status. They will not reach the necessary impact thresholds to warrant protections. The risk to them will be outweighed by the benefit to the majority. This is intensified for disabled people. Unlike other historically marginalized groups, there is no bounded and defined data cluster or common marker of disability that can be shown to demonstrate a group effect. People with disabilities generally constitute an N of 1. They are significantly different from each other.
At the Inclusive Design Research Centre and in the We Count project we are exploring better ways to address inequitable treatment of outliers and marginalized minorities. Rather than focusing only on data representation, which caters to the insatiable appetite of AI systems for data capture, we are exploring ways to push AI to serve and prioritize people who have edge requirements. We are exploring on-device AI paired with bottom-up data collection governed and owned by historically marginalized communities. This both protects against data abuse and misuse and reduces the need to swim against a polluted pool of majority data. We are exploring tuning and personalization techniques that push AI to attend to and optimize the unique needs of students, capitalizing on what current AI systems are good at (e.g., grammar and spelling) while inverting statistical optimization to support student differentiation. Our Trust Meter project is creating a toolkit to help AI implementers/operators determine when a person or scenario is out of distribution for the AI data set, meaning that AI in this instance should not be used to make a decision, determine a priority, or filter choices. We are co-creating resources to build capacity with communities and individuals that feel the harmful impacts of AI the most. And we are creating a regulatory standard for Accessible and Equitable AI for the Accessible Canada Act and the EU regulatory framework, to fill gaps in protective regulations and guidelines.
The gift of AI is that it is a magnifying mirror of our current trajectory, waking us to recognize the course we have chosen as educators and where we are headed if we speed in that direction. There is a powerful machine that can do the formulaic things our traditional education system is standardizing our next generation of humans to do and be. Our traditional education conventions were designed for an industrial era in need of standardized workers. The success metrics and rewards promote competitive ranking on a single, standardized scale. The powerful parallel processing of AI excels at optimizing normative, predictable or standardized patterns. Rather than preparing our students to compete with AI or serve the interests of the AI industry, we need to acknowledge the importance of student diversification over standardization. Our traditional education system is not designed to optimize or support student diversity, differentiation, self-determination, life-long learning and collaboration. It is time to rethink education to value and nurture our learners’ individual and unique potential and priceless differences.