NEW YORK (AP) — The debate over a planned Islamic community center and mosque near ground zero became a court fight Wednesday, as a conservative advocacy group sued to try to stop a project that has become a fulcrum for balancing religious freedom and the legacy of the Sept. 11 attacks.

The American Center for Law and Justice, founded by the Rev. Pat Robertson, filed suit Wednesday to challenge a city panel’s decision to let developers tear down a building to make way for the mosque two blocks from ground zero.

The city Landmarks Preservation Commission moved too fast in making a decision, underappreciated the building’s historic value and ‘‘allowed the intended use of the building and political considerations to taint the deliberative process,’’ lawyer Brett Joshpe wrote in papers filed in a Manhattan state court. The Washington, D.C.-based group represents a firefighter who responded to and survived the terrorist attack at the World Trade Center.

City attorneys are confident the landmarks group adhered to legal standards and procedures, Law Department spokeswoman Kate O’Brien Ahlers said. A spokesman for the planned Islamic center, Oz Sultan, declined to comment on the lawsuit but said organizers were continuing to work toward choosing an architect.

The mosque has become a national political flashpoint, pitting several influential Republicans and the nation’s most prominent Jewish civil rights group against New York Mayor Michael Bloomberg and others. In one of the latest signs of the issue’s political reach beyond Manhattan, Massachusetts Gov. Deval Patrick expressed support Wednesday for the proposed mosque.

The group behind the $100 million project, the Cordoba Initiative, describes it as a Muslim-themed community center. Early plans call not only for prayer space but for a swimming pool, culinary school, art studios and other features. Developers envision it as a hub for interfaith interaction, as well as a place for Muslims to bridge some of their faith’s own schisms.

‘‘We want to create a model that shows the world that you can develop moderate Muslim communities,’’ Sultan said Wednesday. ‘‘We would admonish people to, at least, give us a fair shake.’’

Opponents, including some Sept. 11 victims’ relatives, see the prospect of a mosque so near the destroyed trade center as an insult to the memory of the nearly 3,000 people killed by Islamic terrorists in the 2001 attacks. Shouts of ‘‘shame on you!’’ erupted from the audience after the city panel voted Tuesday to deny landmark protection to the existing building, saying the 152-year-old structure wasn’t distinctive enough.

Big-name Republicans including former Alaska Gov. Sarah Palin and former House Speaker Newt Gingrich have criticized the plan — as has the Anti-Defamation League, a Jewish civil rights group known for advocating religious freedom.

Former Rep. Rick Lazio, a Republican running for governor of New York, has raised questions about the Cordoba Initiative’s imam, Feisal Abdul Rauf. In a ‘‘60 Minutes’’ interview televised shortly after the Sept. 11 attacks, Rauf said that ‘‘United States policies were an accessory to the crime that happened.’’

But supporters of the planned Islamic center see it as a monument to tolerance and religious liberty.

‘‘The World Trade Center site will forever hold a special place in our city, in our hearts,’’ Bloomberg, a Republican-turned-independent, said Tuesday. ‘‘But we would be untrue to the best part of ourselves, and who we are as New Yorkers and Americans, if we said no to a mosque in lower Manhattan.’’

For now, the court case centers on the legalities of the landmarks commission’s vote, which the lawsuit seeks to overturn.

The existing, Italianate building was built for shipping magnates and later occupied by the pharmaceuticals giant Merck & Co., among other businesses.

The law center argues it deserves landmark status for its architectural features — and for its newer historical significance as a structure that withstood being hit by debris from one of the hijacked jetliners used in the terrorist attacks.

‘‘The building is the only building of its kind that links the growth of American free enterprise to the present-day events and the aftermath of Sept. 11, 2001, events which stand as a testament to economic, social and political freedom in the face of violence,’’ Joshpe wrote.