The first recorded use of the term “racism” was in 1902. At the Lake Mohonk Conference held by members of the “Friends of the Indian” movement, Richard Henry Pratt said, “Segregating any class or race of people apart from the rest of the people kills the progress of the segregated people or makes their growth very slow. Association of races and classes is necessary to destroy racism and classism.”

He was referring to the enforced separation of American Indians from much of 1900s American society, adding that though the “negroes” were “not altogether citizen and equal yet,” they had abandoned their old ways and “become a valuable part of our industrial forces.” Pratt’s solution was forcible cultural assimilation — the complete abandonment of Native American culture, religion, homes, and lives. He founded boarding schools for Native American children in which their native languages were quite literally beaten out of them. The American government, in Pratt’s view, had to “kill the Indian ... [to] save the man.” Racism as a concept was first articulated by a person who believed wholeheartedly in the cultural inferiority of an entire group of people — in short, a racist.

But that should come as no surprise. Racism — what it is and why it’s dangerous — has long been at least in part articulated by people who believe wholeheartedly that another group is more criminal, more dangerous, more concerning to the general populace — that is, by racists. To them, if the concept of racism is real at all, it is a cudgel against any and all opposition. To them, Obama is our most racist president, more so than Woodrow Wilson, who encouraged the passage of a bill to ban interracial marriage in the District of Columbia and hosted a special screening of the Klan-centric epic The Birth of a Nation. To them, racism is what happens when you talk about race, when you discuss the disparate impacts of racism, when you open the Pandora’s box of race that they believe we hammered closed in 1968 with the Fair Housing Act.

And no one is more horrified by racism than a racist. Or more accurately, no one is more horrified by being called a racist than a racist. When a West Virginia town official called Michelle Obama an “ape in heels,” the official resigned and then wrote in a statement, “I am truly sorry for any hard feeling this may have caused! Those who know me know that I’m not of any way racist!” She wasn’t sorry for what she said, but she’s very sorry you felt bad about it — because, of course, she’s not a racist.

When Maine governor Paul LePage — who refused to attend a celebration of Martin Luther King Jr. Day because the NAACP was hosting it and believes that men named D-Money, Smoothie, and Shifty are impregnating innocent white teens — was called a racist by a constituent, he responded that being called racist is “like calling a black man the n-word or a woman the c-word. It just absolutely knocked me off my feet.” It isn’t. Telling someone that they’re racist isn’t at all like calling someone a nigger or a cunt. But if you’re in power, the threat of being called racist is apparently a bridge too far — much more so than the penalty for actual racism — and meanwhile, the bar for true, unfettered racism to be called out on the carpet keeps rising higher and higher. If you’ve been watching the news cycle, you’ll see that even white nationalists aren’t racist; they’re “racial realists.” If nothing is racist, then no one is racist.

And yet, despite the blinkered, desperate hopes of governors and politicians and West Virginia town officials, racism remains in America. It wasn’t solved by Martin Luther King Jr. or John Lewis. We didn’t hash it out in a group viewing of Ghosts of Mississippi. Your favorite NFL player is black and yet you may still remain racist. And voting for Barack Obama in 2008 or 2012 did not give anyone a free pass or a “get out of racist jail free” card. Racism — the belief that a group of people is somehow lesser because of an innate characteristic, a collective flaw — exists.

And the problem with this rhetoric, of course, is that the true burden of racism isn’t on the racist — it’s on the person or people subjected to racism. Being called a racist is bad. But being undervalued and underestimated, or considered to be a “credit to the race” because you’re not like “those other ones,” or asked to comment on the actions of someone else who happens to look like you, or told that the racist thing being said around you was only said because “they’d never met one like you before,” or being told that you can’t do something, or, more often, that you should do something because otherwise you’re “voting against your own interests,” is worse. Being told that there will never be another black president because of the president’s perceived “failures” — because all black people, no matter what we do, are the same, tethered together to the same rock thrown into the same lake — is worse. Being perpetually suspect is worse. Being robbed of any and all complexity is worse. Being perceived as immune to literal physical pain is worse.

Racism isn’t bad because it’s insulting. Racism is bad because it’s immoral, and because at its heart, it’s blatantly stupid. Racism requires believing that some people are better than others because of bone structure or melanin or hair quality or other characteristics even more meaningless than those. That is a concept that could only be favored by people who needed it to be true.

And the real damage of racism isn’t done to racists; it’s done to racism’s victims. The people who needed federal legislation to live freely, a Voting Rights Act and a Fair Housing Act to be able to elect officials and rent apartments. The people who continue to be perceived as lesser, whose presence has instigated the rise of a new white nationalism because nothing is more terrifying to the racist who assumes his own superiority than oncoming equality. The people most affected by the whims of racists like Paul LePage or Richard Henry Pratt. Men like LePage and Pratt have the luxury to be horrified by accusations of racism. Their victims got nothing.