Vanna.AI: When a Text-to-SQL Assistant Becomes Remote Code Execution
In June 2024, JFrog's security research team disclosed CVE-2024-5565, a flaw in the open-source Vanna.AI library that turns a friendly feature — ask a question in English, get back a chart — into a remote-code-execution primitive. It is one of the clearest worked examples of prompt injection escaping the text box and reaching the host.
Vanna helps developers query databases in natural language. A user asks a question; an LLM writes the SQL; Vanna runs it and, by default, visualises the result. The visualisation step is where the trouble lives. To draw the chart, Vanna asks the model to generate Python code for the Plotly graphing library, and then executes that generated code with Python's exec().
The gap the attacker jumps
User input flows into the prompt that drives that code generation. Because the model can be steered by what the user writes, a crafted question can cause the generated "plotting" code to contain arbitrary Python — which exec() then runs on the server. The natural-language front door leads, through the model, to code execution on the back end. JFrog assigned it a CVSS score of 8.1, in the high range.
exec() makes the LLM part of your control flow. Since the model bends to user input, any user who can prompt it can, in effect, run code on your server.Fixing it, and the general lesson
The immediate mitigation is straightforward: do not pass externally influenced input to the visualisation path — for instance, set the visualise option to off when the question comes from an untrusted user — and the project added guidance to that effect. But the durable lesson is broader than one library. Any system that executes model-generated code, or feeds model output into an interpreter, shell, or eval, has built a bridge from prompt to execution.
This pattern is spreading as "agentic" tools proliferate: the model writes code or commands and something runs them. Vanna is a small, legible instance of a design that is quietly appearing in many larger systems, and the defensive principle transfers directly.
The pattern deserves naming because it is proliferating faster than awareness of it. "Give the model a tool" has become the default way to make LLM applications useful — let it query a database, call an API, render a chart. Each tool is a capability the model can be steered into misusing, and code generation is the most dangerous because it collapses the distance between the model's output and arbitrary execution entirely. The same shape hides inside any agent that writes and runs code, any notebook assistant, any "let the AI script it for you" feature. The posture that scales is to assume the model will eventually be adversarial and place the boundary at the interpreter — sandbox execution, drop privileges, constrain reach — so that even a fully compromised prompt cannot do more than the sandbox allows. Trusting the model to only write benign code is not a control; it is a hope.
What defenders should take away
- Never
exec()oreval()model output on untrusted input. If code generation is unavoidable, run it in a locked-down sandbox with no filesystem, network, or credentials. - Separate generation from execution. Constrain what generated code may do to a tight allow-list, and validate it before it runs rather than trusting the model to behave.
- Default to the safe configuration. Features that execute code should be off by default and enabled only for trusted contexts.
- Inventory where models feed interpreters. Every prompt-to-code bridge in your stack is a candidate for this exact flaw.
CVE-2024-5565 is a compact reminder that a language model wired to an interpreter is an execution engine, not a chatbot. Finding the places in your architecture where model output becomes running code is the first step — because you cannot defend what you have not discovered.