Transportation, architect, and infrastructure experts say the Penn Station revamp won’t happen unless Madison Square Garden relocates. But getting the Madison Square Garden Company to agree to moving to a nearby site could cost a hefty price.

It was one of the hot topics at “The Future of Penn Station”, a panel discussion in the Great Hall at Cooper Union. Moderated by NY1’s transit reporter Jose Martinez, several of the speakers agreed that in order for Governor Andrew Cuomo’s vision of upgraded train stations, retail, and office space, which he estimates would cost nearly $1.6 billion and would be completed by December 2020, to come to fruition, Madison Square Garden would have to go. Governor Cuomo himself has dismissed the idea.

“They have about seven years left in their permit to conduct business,” said Tom Wright, president of the Regional Plan Association, to a packed auditorium. He was referring to a 2013 City Council vote that would extend the Garden’s permit for only ten years, in hopes they would find a new location before their lease is up.

“The window of opportunity is going to close in terms of other sites in the neighborhood they can move to.” he added.

But moving MSG would be no easy feat. The stadium is built on top of Penn Station and rests on over a thousand steel columns that extend all the way into the basement. According to Gina Pollara, the president of the Municipal Art Society of New York, 75 percent of those steel columns would have to be demolished to create more space for tracks.

“In order to free tracks, Madison Square Garden really has to go.” she said.

The price tag for a Madison Square Garden removal would be expensive. According to a report published by NYU’s Rudin Center for Transportation Policy and Management titled “Time to Get Moving”, relocating the Garden could cost $5 billion dollars. It could cost $65 million alone just to demolish the building, and another $1.6 billion to build a new one. That’s not including the estimated $800 million it could cost to acquire the United States Postal Services Morgan Annex between 28th and 30th streets at Ninth and Tenth Avenues, a favorite spot of advocates for a possible relocation, among several other expenses.

Despite the price and overall unwillingness of the Madison Square Garden Company to move, some said the idea is still not far-fetched.

“We need to remember that traditionally Madison Square Garden has moved several times,” said Michelle Young, the founder of Untapped Cities and an adjunct professor at the Columbia University Graduate School of Architecture, Planning and Preservation, after the panel. And she’s right—the Garden was first built near Madison Square–and has been in three different locations. It’s been in its current location at 34th Street since 1968.

“It’s quite a feasible option, considering the history of moving MSG.” Young said. “The idea behind it is to not only allow Penn Station to grow underground, but above ground to make use of that space. Right now, we’re just constrained by a massive sporting stadium.”