Schools are using AI surveillance to protect students. It also leads to false alarms — and arrests

Industry,

Lesley Mathis knows what her daughter said was wrong. But she never expected the 13-year-old girl would get arrested for it.

The teenage girl made an offensive joke while chatting online with her classmates, triggering the school’s surveillance software.

Before the morning was even over, the Tennessee eighth grader was under arrest. She was interrogated, strip-searched and spent the night in a jail cell, her mother says.

Earlier in the day, her friends had teased the teen about her tanned complexion and called her “Mexican,” even though she’s not. When a friend asked what she was planning for Thursday, she wrote: “on Thursday we kill all the Mexico’s.”

Mathis said the comments were “wrong” and “stupid,” but context showed they were not a threat.

“It made me feel like, is this the America we live in?” Mathis said of her daughter's arrest. “And it was this stupid, stupid technology that is just going through picking up random words and not looking at context.”

Surveillance systems in American schools increasingly monitor everything students write on school accounts and devices. Thousands of school districts across the country use software like Gaggle and Lightspeed Alert to track kids’ online activities, looking for signs they might hurt themselves or others. With the help of artificial intelligence, technology can dip into online conversations and immediately notify both school officials and law enforcement.

Educators say the technology has saved lives. But critics warn it can criminalize children for careless words.

"It has routinized law enforcement access and presence in students’ lives, including in their home,” said Elizabeth Laird, a director at the Center for Democracy and Technology.

Schools ratchet up vigilance for threats

In a country weary of school shootings, several states have taken a harder line on threats to schools. Among them is Tennessee, which passed a 2023 zero-tolerance law requiring any threat of mass violence against a school to be reported immediately to law enforcement.

The 13-year-old girl arrested in August 2023 had been texting with friends on a chat function tied to her school email at Fairview Middle School, which uses Gaggle to monitor students' accounts. (The Associated Press is withholding the girl’s name to protect her privacy. The school district did not respond to a request for comment.)