CHARLOTTESVILLE, Va. — In early July, crews showed up downtown for some long-delayed evictions. After years of protest, litigation and even violence, the statues of two Confederate generals, Lee and Jackson, were finally carted out of city parks, expelled by the city’s drive to right its past wrongs.

Now the really hard work awaits.

It has been four years since white supremacists descended on Charlottesville, wreaking bloody havoc in the streets and killing a young woman. The horror of that August weekend sent the city into a deep study of its own racial past and a debate over what to do about its legacy. The catalog of lingering artifacts of that bigoted history is daunting, beginning with statues but quickly getting to the basics of civic life like schools and neighborhoods.

In a city that prides itself on its progressivism, the push for justice has, in general terms, enjoyed broad support. That this push may entail changes to people’s neighborhoods — streets of one- and two-story brick homes, lovely dogwoods and abundant Black Lives Matter signs — is another matter.

Charlottesville’s planning commission is considering a proposal to roll back some of the city’s zoning restrictions in an effort to encourage construction of more affordable housing, a plan that has drawn reaction ranging from fervent opposition to disappointment that it does not go further.

But there has been a particular disquiet, said Lyle Solla-Yates, a member of the planning commission, among a certain part of the population: “smart, educated” white residents who are neither poor nor very wealthy, and who live in charming neighborhoods with a history of discriminating against Black people that they had known nothing about. Now they imagine multi-story apartment buildings going up on their streets.

“There’s fear and anger at being targeted,” he said. “They don’t feel centered in this process. And they are correct.”

For months, residents and city officials have been considering a draft land use map that sketches out what kind of buildings would be allowed where in the years to come.

Propelled by research showing that single-family zoning restrictions have roots in discrimination and consequences in soaring housing prices and more segregated neighborhoods, Charlottesville is joining communities across the country in debating whether to ease these restrictions. Several Democratic presidential candidates in 2020 pledged to encourage the loosening of zoning rules, and President Biden’s infrastructure bill includes grants for cities that do so.

On the right, figures from Donald J. Trump to Tucker Carlson to Mark and Patricia McCloskey, the St. Louis lawyers who were given a speaking slot at the 2020 Republican National Convention after waving their guns at protesters, have accused Democrats of wanting to “abolish the suburbs” by curtailing single-family zoning. The results, Ms. McCloskey said, would be “crime, lawlessness and low-quality apartments.”

This kind of fire-breathing partisanship is relatively rare in Charlottesville, a liberal college town. But the colors on the land use map — especially the gold, which shows up all over the city and particularly in comfortable neighborhoods like Lewis Mountain and Barracks Rugby, indicating that residences of up to 12 units would be allowed in places where single-family homes now sit — were, to many, alarming.

A “huge social experiment on our city,” said a law professor at one of the hours-long virtual planning commission meetings this summer. “I just don’t understand what is driving this,” another commenter said.

In a sign of just how much the political ground has shifted in recent years, the chief argument of the plan’s opponents is that it would actually be bad for the poor, a giveaway to greedy developers. Some have compared the plan to the razing of Black neighborhoods in decades past, and comment threads on the Nextdoor app have crackled with debates about whether the proposal would simply yield a city full of high-end apartments and whether genuinely “horrible injustices” from the past would really be rectified by “destroying neighborhoods in the present.”

Charlottesville stands apart neither for the controversy nor for the foul chapters of its planning history, a record of enforced segregation that it shares with cities nationwide. What is different here is the recent past.

Before the white supremacists descended in August 2017, a shortage of housing for the poor had been a concern but not an emergency, at least not to the people who tended to show up for city planning meetings. After that August, everything changed.

Protesters shut down city meetings, academics and activists publicly scrutinized the city’s racial past, and people who had long sought to draw the city’s attention to its poorest residents suddenly found a broader audience.

Two long-overlooked realities moved to the center of the civic conversation. For one, there was a growing recognition that many of the city’s neighborhoods were once preserved along openly racist lines, with home ownership policed by deeds that did not allow sales to people of color. When such deeds were ruled illegal, zoning restrictions, which reduced the supply of housing and made what was there more expensive, had a similarly exclusionary result.

It also became clear that contemporary Charlottesville, population 47,000 and growing, was a place where many poor and working-class people — Black people to a disproportionate degree — could no longer afford to live. While most of the city is reserved for detached single-family homes, a majority of the residents are renters, with many paying more than half their monthly income in rent. This goes a long way toward explaining why the city’s Black population, now around 18 percent, has been steadily shrinking.

“Black people are being displaced,” said Valerie Washington, 28, who grew up in town but now lives out in the surrounding Albemarle County. As young white professionals and house-flippers have snapped up properties, few of the Black neighbors she knew as a child are left in her old neighborhood. “I’m there all the time,” she said. “But I can’t afford to live there.”

In March, the city endorsed a plan that includes $10 million for housing assistance each year as well as protections for renters, along with a rewrite of the zoning ordinance to allow much more multifamily housing to be built, with some portion of new developments required to include affordable units. The zoning rewrite, officials argued, would release pressure from the pricey and competitive housing market while also breaking up the legacy of the city’s exclusionary past.

Roughly half of the hundreds of people who emailed the city about the latest draft of the map expressed support for the plan, and virtually no one is publicly questioning its ultimate goals.

“If we have to ruin half of our block for racial justice, yes, we’ll go for that,” said Leeyanne Moore, a creative writing instructor who lives on a street of small stucco bungalows. But she contends that the proposal would result only in a lot of expensive apartments for University of Virginia students. “Rezoning would not solve the problem,” she said.

Her neighbor, Diane Miller, also has reservations. She has not joined in the public debates, which tend to be dominated by the opinions, pro and con, of white professionals and academics. “My opinions don’t mean nothing,” said Ms. Miller, who is Black.

But she remembers, as a young girl, hearing her parents talk about a developer who was buying out all the neighbors, most if not all of them Black. She did not know whether their property was taken by eminent domain; all she remembers is that everyone left reluctantly, including her family, which left behind a house that had belonged to her grandmother.

Ms. Miller distrusts any top-down plans to address racial inequities; after all, those inequities came from the top in the first place.

“They took everything that Black people own, everything,” said Ms. Miller, now 65. “Ain’t no trust there.”

Carmelita Wood knows a lot about that history herself. She was raised in Vinegar Hill, a bustling neighborhood of homes and Black-owned businesses that was razed to the ground in the 1960s in the name of “urban renewal.” The idea that any policy could make amends for that does not pass muster with her. “Most of those people are dead and gone,” she said. “And their kids have moved away.”

But while history runs deep and its tragedies are irreversible, Ms. Wood suggested that it was not too late to start doing the right thing. She is now the president of the neighborhood association in Fifeville, a part of town that is majority Black, but by a steadily dwindling margin. In letters and op-eds, she has made the case that the vision in the proposed land use map, of neighborhoods around the city opening up to all kinds of different people, was a good first step.

“I think it will work,” Ms. Wood said. “I think it’ll work because folks will finally see that if we speak up, then maybe they will listen to us.”