From Prompt Injection to Code Execution: The MathGPT Case

Most prompt-injection stories end with the model saying something it shouldn't. This one ends with a server executing attacker-controlled code and coughing up its secrets. In 2023, a researcher probing MathGPT — a public app that used GPT-3 to turn natural-language math questions into runnable Python — showed that when a language model's output becomes executable, prompt injection stops being a content problem and becomes remote code execution. The case is catalogued by MITRE ATLAS.

MathGPT's design was the vulnerability. It took whatever a user typed, asked GPT-3 to write Python that answered it, and then ran that Python on the server to compute the result. That is a powerful pattern — and a dangerous one, because the model's output is not just displayed, it is executed. Anything an attacker can steer the model into writing, the server will obligingly run.

Turning a math box into a shell

Rather than asking a math question, the researcher supplied injection-style prompts — the familiar “ignore the above and instead …” framing — to make the model emit Python that did something other than arithmetic. Through several attempts, they produced code that read the host's environment variables, which exposed the application's GPT-3 API key, and demonstrated they could run system-level operations, including a denial-of-service condition on the host.

The pattern is spreading, not shrinking

MathGPT was a niche app, but its architecture is now everywhere. “Agentic” assistants that generate and run code, tools that let a model query a database or call an API, plugins that translate a request into an action — all share the same core risk. The moment a model's output crosses from text into an executed instruction, every downstream system it can reach inherits the injection exposure. The convenience of “just let the model do it” is precisely what an attacker exploits.

Containing executable AI output

• Sandbox everything the model runs. Generated code must execute in an isolated, ephemeral environment with no access to host secrets, the network, or the wider system. Assume the code is hostile.
• Keep secrets out of reach. The API key leaked because it lived in the execution environment. Credentials should never be accessible to code the model can influence.
• Apply least privilege to tools. If a model can call functions, each should do the narrowest possible thing, with permissions scoped to that task and nothing more.
• Validate and constrain outputs. Where feasible, restrict generated actions to a vetted set of operations rather than executing arbitrary code.

The researcher disclosed the issues, and the MathGPT and platform teams responded by filtering certain prompts and rotating the exposed key. But prompt filtering is a fragile patch for a structural problem: if untrusted input can shape executed code, you are one novel phrasing away from the next bypass. The reliable fix is architectural — isolate execution and starve it of anything worth stealing. As models are increasingly given hands as well as a voice, you cannot secure the actions of an agent whose reach you have not deliberately mapped and contained.