It was supposed to be the home of the United States National Slavery museum.
Now, the group that failed once wants to try again.
"We wouldn't be opposed to a small museum, if it's done right and done in a timely manner,” said Judson Honaker, President Commercial division, Silver Companies. “But there just hasn't been any effort put forth to make this a reality.”
“I think this garden behind us is a perfect example,” added Honaker. Inside the Memorial Garden, overgrown vegetation monopolizes the space, and the area in disrepair.
L. Douglas Wilder told a federal judge he wants to scale back the project, sell off a huge portion of the waterfront property to pay off his debtors and use the rest of the land to eventually open the museum.
Wilder owes the City of Fredericksburg $300,000 in back taxes.
"If they build a museum out there that would be great,” said Jim Haney, Treasurer of the City of Fredericksburg. “If they don't build a lot of interest keeps growing.”
But the Silver Companies, which donated the land to Wilder, isn't so optimistic.
"If we had proper assurance that a museum is going to get built," said Haney, "I think we would strongly consider it."
Honaker said his company put restrictions in place to build a museum on this land.
What would it take for them to allow Wilder to move forward? "Money raised...activity planned…some sort of guarantee,” said Honaker. “We really haven't thought that through.”
But if it was definitely going to happen, I think we would strongly consider moving the restriction," said Honaker.
If not, Honaker said, "We would like to get on and find another attraction to generate traffic within the project."
And while everyone is waiting on a decision from the judge, people visiting Fredericksburg have their own opinions.
"It's a great location for it,” said Aspen Pash. “It's sure to bring a lot of people.”
"If you're going to scale it back to a point where it's not going to be a place where people can actually learn, it's going to be more like a monument, I think that's a waste of money," said Ashley Hanson.
A Federal Bankruptcy judge is expected to decide next month on whether or not to accept Wilder's plan.
CBS 6 tried to reach the former Governor and his attorney Sandra Robinson, but no one has returned our phone calls. CBS 6 is following this developing story.