When Kenneth Glasgow had his right to vote restored in Alabama more than a decade ago, after serving 14 years in prison for drug charges, he wore his registration card around his neck for months “like a badge of honor.”

“My own mother said, ‘Why are you wearing that? Are you crazy?’” Glasgow, now a pastor and criminal-justice organizer, told MTV News. He told his mother, “No, it proves I'm a citizen.”

Like Glasgow, Michael Hiser has held on to his voting swag; he cast a ballot for the first time in 2016, after receiving a full pardon for 53 drug and theft convictions in 2015 in Kentucky, and still keeps his “I Voted” button next to his bed so he can look at it before he prays.

Barbara Harris in Virginia got the paperwork in the mail saying her rights were restored on August 25, 2016, a date she can recite like a grandchild’s birthday. Seeing that envelope in the mail, she says, “felt like I had gotten married and had a baby at the same day.” June 13 will mark the first time she’s been able to vote in more than a decade, due to her drug conviction and the two years she spent in prison. She carries the document in a briefcase, along with all the other materials she needs to apply for a pardon.

Before they received that treasured proof of their rights, Glasgow, Hiser, and Harris were just three of about 6.1 million Americans stripped of their right to vote by felon disenfranchisement laws. As Harry Enten pointed out at the Guardian a few years ago, keeping ex-felons from voting has had a far bigger impact on elections than voter ID laws, though the latter issue gets much more national attention. These laws disproportionately affect minorities; 1 in 13 black Americans can't vote because of felon disenfranchisement. This number is large enough to potentially change the outcome of elections, the makeup of districts, and the amount of attention elected officials give to all of their constituents.

“Say you have 100 people in one town, 100 people in another” in a state with particularly extreme felon disenfranchisement measures, Julie Ebenstein, an attorney with the ACLU, told MTV News. If one town has a population with no felony convictions while the other has 50, “to a politician, that town is only worth 50 votes. You can assume [these towns are] going to get a lot less attention from political campaigns, and it's less likely their concerns will be heard.” And, as many organizers noted, ex-offenders are paying taxes, despite their inability to vote. “Our country,” says Hiser, “is famous for going to war over that.”

Laws chipping away at the voting rights of the incarcerated first appeared after Reconstruction, joining the poll tax, the grandfather clause, and other tactics intended to deprive minorities of the ballot. After the Civil War, law enforcement in the South was quick to arrest black citizens who were enfranchised in order to create a new form of nearly free labor, leaving behind upticks in incarceration and a repeated loss of voting rights. Many of these measures are still kicking; as Michelle Alexander, author of The New Jim Crow, has pointed out, more black Americans were barred from voting in 2004 because of felon disenfranchisement measures than in 1870. The number of disenfranchised minorities in and out of prisons ballooned across the country once the drug war hit in the ’70s and ’80s. Given the Trump administration's repeated threats to bring back a punitive battle against drugs — despite the noted lack of success of the last offensive — organizers are especially driven to try to tackle these measures.

Right now, the people most affected by these law-and-order changes don't always get the opportunity to fight back at the ballot box. Most states ban felons from voting until their probation or parole is completed, while Iowa, Florida, Kentucky, and Virginia make the formerly incarcerated apply to get their rights back, a process so complicated — and prone to rejection — that few go through it. Only Maine and Vermont, the two whitest states in the country, allow felons to vote while in prison. Though many countries bar prisoners from voting, most automatically restore this right after a sentence is completed.

But change seems to be coming, especially in the South. More than 40,000 released felons were allowed to vote in Maryland last year, after the state legislature overrode a Republican gubernatorial veto. After his blanket pardon was struck down by the courts, Governor Terry McAuliffe individually restored the rights of more than 156,000 people in Virginia over the past year. Organizers in Florida, the state that has more disenfranchised felons than anywhere else in the country, are trying to get a ballot initiative ready for 2018 that would re-enfranchise voters before 2020.

And in Alabama, where Glasgow has been organizing for years, the formerly incarcerated have recently celebrated their own win. The state has a law stating that “no person convicted of a felony of moral turpitude” can vote. Inserted in the state constitution in 1901, it was explicitly designed to “establish white supremacy in this state,” as one politician in charge of drafting the document put it. Figuring out what “moral turpitude” meant, and which ex-felons could have their rights restored, was historically a subjective and confusing affair, differing from official to official. But in May, the Republican governor finally signed a bipartisan bill that defines what a crime of “moral turpitude” actually is — murder, sexual abuse, terrorism, etc. — clearing the way for many of the voters silenced by the drug war back into the fold. Some groups estimate that it could allow thousands more residents to vote. Glasgow even learned that he shouldn't have lost his rights to begin with.

Other states are still resistant to change. Governors in Iowa and Kentucky signed executive orders expanding the franchise for felons that are soon to be reversed by their successors. Organizers in Louisiana are currently trying to open up voting to those on parole or probation, the same change that happened in Maryland in 2016. Earlier this year, a state judge upheld the prohibition, although he added, “I don't like this ruling. I don't like it. It's not fair.”

There is still clear opposition to expanding felon voting rights, despite the bipartisan push for criminal justice reform, partly because voter suppression is tied up in ideology. (Felon disenfranchisement policies disproportionately affect minorities, which means they disproportionately affect Democrats.) Any attempts to expand suffrage, then, are written off as a partisan maneuver, despite the fact that why these laws exist in the first place is just as steeped in cynical strategizing. Shortly before becoming president, Donald Trump himself derided the push to expand voting rights in Virginia. “They are letting people vote in your Virginia election that should not be allowed to vote,” he told a rally on the campaign trail. “Sad. So sad.”

In 2002, the Senate considered a measure that would have expanded suffrage to all ex-felons, at least for federal elections. Some familiar faces opposed the bill: Senator Mitch McConnell, now the majority leader, said, “We are talking about rapists, murderers, and even terrorists or spies,” not mentioning that the majority of people who would be affected by expanding felon voting rights have not committed these types of crimes. Senator Jeff Sessions, now the attorney general, said that “each state has different standards based on their moral evaluation, their legal evaluation, their public interest,” refusing to acknowledge the history of measures to take away the vote. As Hiser notes, deciding that it's necessary to only let citizens who are “good” or “moral” enough take part in democracy is a slippery slope, and an immense power liable to abuse.

But fighting back against felon disenfranchisement laws is only one facet of this voting rights battle. “Eligibility to register is really the first step toward someone with a prior conviction getting to the ballot,” says Ebenstein. “There are also then all the other barriers that everyone else faces.”

State governments are bad at making sure residents know everything they need in order to understand current election law before voting. For example, Hiser thought that all ex-offenders got their voting rights back after five years. “When you're in jail,” he says, “you aren't thinking about voting rights.” Many Americans might believe the opposite. “I was just watching a TV show,” says Nicole Porter at The Sentencing Project, “and it assumed that anyone with a felony conviction can't vote. That is a part of the common narrative. People have been working for years to counter that.” Election officials aren't often up to speed on law changes either, further confusing voters. In Baltimore County last year, some released felons received letters saying that they weren't eligible to vote, weeks after their rights were officially restored.

In Alabama, the newly enfranchised will have to deal with getting voter IDs. Glasgow already has a plan to deal with that hurdle before the 2018 midterms.

“I consider myself Moses," he says. “I want to campaign now, ‘Let my people vote.’” And voter IDs? “[The formerly incarcerated] are identified better than regular citizens. Why? Because you fingerprinted us to lock us up. So we can use our jail and prison IDs. And if the government says we can't use that, and it's a federal or state or local municipality-issued ID, I'm going to have a ball suing everybody involved. And you can quote that.”

All the organizers recognize this is a long-term fight. “I'm a 47-year-old Caucasian male," Hiser says, “and I've had enough of old, fat, white guys. The table is crowded with people not affected by their decisions. We should be there, we have insight. I have 25 years of research as a criminal.” And, he adds, “I'll keep fighting until change comes. And if it doesn't come in my lifetime, my kids will fight for it. And I hope everyone else does too.”