New York City's Chatbot Told Businesses to Break the Law

A government chatbot is supposed to lower the friction of dealing with bureaucracy. New York City's MyCity business assistant, launched in October 2023 and powered by Microsoft's Azure AI, did the opposite in a specific and dangerous way: it answered official-sounding questions with advice that would, if followed, break the law.

In March 2024, the non-profit newsroom The Markup — reporter Colin Lecher, in work co-published with local outlets — tested the bot against real New York City rules and published what it found. The answers were not merely vague. They were confidently, specifically wrong on points where being wrong exposes a business to serious legal liability.

The advice it gave

Asked whether a landlord had to accept tenants on housing vouchers, the bot suggested they did not — source-of-income discrimination that is illegal in New York City. Asked about workers' tips, it indicated an employer could take a cut. It suggested businesses could go cash-free, which city law prohibits, and gave answers that contradicted rules on firing workers who complain about harassment. In at least one exchange it undercut its own disclaimer, telling a user they could rely on it for professional guidance.

Grounding was the missing control

The technical root cause is familiar. A general-purpose language model, asked a legal question, will produce fluent, plausible text whether or not it has the correct rule in front of it. Without tight grounding in the city's actual, current regulations — and a hard instruction to defer rather than improvise when the answer is not in that corpus — the model fills the gap with a confident guess. The interface makes that guess look like a ruling.

This is the same failure mode that produced the Air Canada tribunal decision, but with the state as author. The lesson is not that governments should never deploy assistants; it is that a public-sector bot answering regulatory questions needs a far narrower remit and much stronger sourcing than a general chatbot.

Consider the asymmetry the interface creates. A search engine returns a list of sources and leaves interpretation to the reader; a chatbot returns a single, fluent, first-person answer that reads as a decision. For a small-business owner without a lawyer on retainer, that answer may be the only guidance they get — and acting on it can mean a discrimination complaint, a wage claim, or a fine. The confidence of the presentation is itself a hazard when the substance is unreliable, because it strips away the natural scepticism a page of links would invite.

Notably, the city kept the tool online after the reporting, framing it as an improving pilot and adding stronger disclaimers. That choice underlines a governance question every organisation faces: what is your threshold for pulling an AI system that is demonstrably giving harmful answers?

What defenders should take away

• Ground high-stakes answers in an authoritative, current corpus. For regulatory or legal topics, retrieve from the actual rulebook and cite it; forbid the model from answering when the source is absent.
• Constrain scope to what you can stand behind. A bot that answers everything will be wrong about something that matters. Narrow it to the questions you have grounded and tested.
• Do not let the UI over-promise. A disclaimer the model itself contradicts is worse than none. Test that the assistant honours its own limits.
• Define a kill switch and a threshold to pull it. Decide in advance what level of demonstrated harm takes the system offline, and who owns that decision.

MyCity is a reminder that the risk of a public-facing model is not only data theft — it is authoritative-sounding error at scale, delivered under a trusted name. Before you can govern that risk, you have to know which teams have stood up citizen- or customer-facing assistants in the first place. You cannot defend what you have not discovered.