ALBANY - The state panel that rules on judges' conduct is trying to remove its chairman for writing a best selling humor book with comic Jackie Mason that says “allegedly” means a charge is true and that affirmative action is “insidious.”

Attorney Raoul Felder, chairman of the state Commission on Judicial Conduct, co-wrote the book, “Schmucks! Our Favorite Fakes, Frauds, Lowlifes, Liars, the Armed and Dangerous, and Good Guys Gone Bad.”

The book is on The New York Times Best Seller list for hardcover nonfiction.

“Loosen up guys, this is humor,” Felder said Friday. “This is America. This can't happen here.”

Felder said he will fight back in court.

“I would feel better as an American if they could point to something, some official act I have done, where I haven't acted appropriately or somebody feels their rights were infringed by anything I've done. And they can't do that.”

“This case will end up in the courts and the appellate courts and that's where it should be,” the Manhattan lawyer said. “Because somebody has to define what right bodies have to curtail free speech.”

The panel said the book is “crude, biased, vulgar and otherwise demeaning” and “repeatedly invokes racial, ethnic and religious invective.”

It provided excerpts including: “Nothing in our country is more insidious than affirmative action.”

The panel said that “raises a reasonable perception that the speaker could not pass fair judgment on anyone he perceived to have benefited from affirmative action.”

Another excerpt, also confirmed by Felder, states: “Anytime you hear the word ‘allegedly,' you can bet it's true.”

The commission stated Friday that it voted unanimously to “express a loss of confidence in the judgment and leadership” of Felder. The panel is considering firing Felder, according to the statement.

The panel stated the book “undermines the appearance of impartiality, and the dignity and probity that is required of the commission and its chair.”

As for the passage about “allegedly,” the panel said that viewpoint is “untenable from a commission member whose role is to evaluate allegations of judicial misconduct and identify those that have merit.”

The commission rules on complaints against judges including conflicts of interest and inappropriate behavior.

Felder's position as chairman is unpaid.