Ask a database in English, Arabic or Urdu without trusting the model that writes the SQL.
What I didBuilt end to end: the pipeline, the guardrails and the evaluations.
Every number on this page links to the file it came from, pinned to text-to-sql-guardrails @ dae1de3.
Headline results
The short version
The problem
Most text-to-SQL demos send a schema and a question to a model and run whatever comes back. I wanted to know what has to happen around that call before it is safe to put in front of users, in Arabic and Urdu as well as English. The first version was safe but failed most hard questions, the ones that join three or four tables, and its first answer of every session timed out.
What I built
A pipeline that treats the model as untrusted input. The generated query is parsed into a syntax tree, every table and column is checked against the real schema, and only then does it run in a read-only sandbox with a five second deadline and caps on rows and bytes. When the validator or SQLite rejects a query for a fixable reason, the model is shown the error and tries again, at most twice. A query that tried to write is never sent back.
Every language shares one prompt that lists how the tables join and what business terms mean, such as revenue being the price actually charged. Urdu is told apart from Arabic by its letters. The answer leads with one line in the question’s language and the plain steps behind it, both built from the result rather than written by the model. Follow-up questions carry the earlier exchange, and a short curated list of ambiguous terms makes the system ask back before guessing.
How I tested it
Four suites on a local CPU with Ollama. A development suite of 20 questions in each of three languages, where the prompt was tuned. A held-out suite of eight hard questions in each language, written before any tuning and run once. A follow-up suite of two-turn conversations. Seven adversarial prompts. A regression gate compares every new run with the committed one, question by question, and the previous pipeline was run on the same suites so the comparison is fair.
What I would do next
The weakest parts of the system now are the confidence score, which cannot see a wrong but well-formed query, and plain-language injection through stored data, which the guardrails contain but cannot stop. Next I would look for a better confidence signal and turn the thumbs up and down that users can now give into reviewed eval cases and verified queries. Run Spider, which needs its databases from somewhere other than HuggingFace, and try dialectal Arabic and Roman Urdu, which the suites do not cover.
How it works
Pick a request and watch it go through the system. The red ones are the requests I use to attack it. They end where the system is supposed to stop them.
The query passes validation, runs read-only, and the answer comes back as one line in the question's language, with the steps that produced it.
- Question
- Clean and detect
- Shared prompt
- Model
- Extract SQL
- Validate
- Execute and ground
- Answer
One request through the pipeline. The model sits outside the trust boundary, and nothing it writes reaches the database without passing the validator, including a repaired query.
Test results
14 claims, each with the bar I held it to and what I measured. 5 failed and 2 are unclear. They stay in, because a list where everything passes tells you nothing.
| Result | Claim | Expected | What I measured | Source |
|---|---|---|---|---|
| PASS | Hard multi-table questions workJoins were the weak spot of the first version. The model read a column off the wrong table or left a join column unqualified. Join conditions, business definitions and a repair loop fixed most of them. | most of them | 21 of 27 on the development suite (78%), up from 6 of 27 (22%) | docs/results.md ↗ @ dae1de3 |
| PASS | The gains hold on questions the tuning never sawIt ends lower than the 85% on the development suite, which is where the prompt was tuned and so flatters it. The held-out number is the one to trust. | a clear gain on the held-out suite | 7 of 24 before, 16 of 24 after (66.7%) | docs/results.md ↗ @ dae1de3 |
| PASS | No harmful statement reaches the databaseContainment comes from the validator, not from model behaviour. A query that tried to write is never sent back for repair. | 100% containment, which is the guardrail's job | 0 executed; 6 of 6 dangerous outputs contained (4 of 4 and 2 of 2) | docs/results.md ↗ @ dae1de3 |
| PASS | Urdu works about as well as EnglishUrdu is new. Through the old Arabic path it scored 10 of 20. | no large gap | English 18/20, Arabic 17/20, Urdu 16/20 on the development suite | README.md ↗ @ dae1de3 |
| PASS | Follow-up questions workAdding them left every single-question reply byte-identical, for both models. Bare fragments such as "only for customers in the UAE" are where it still slips. | n/a | 19 of 24 (79.2%); English 7/8, Arabic 7/8, Urdu 5/8 | DECISIONS.md ↗ @ dae1de3 |
| PASS | Clarifying questions only fire where a term is really ambiguousWith hints that say what to do in the query, the model followed the chosen option in 15 of 15 tries. | n/a | on the two eval questions whose verbs split the model from the gold answer, in each language, and nowhere else | DECISIONS.md ↗ @ dae1de3 |
| FAIL | The confidence score separates right answers from wrong onesIt used to (0.97 against 0.71) because most wrong answers failed to run. Now almost every answer runs, and its signals are structural, so a well-formed wrong query looks like a right one. | a clear gap | mean 0.98 on correct answers and 0.96 on wrong ones (7b, development suite) | docs/results.md ↗ @ dae1de3 |
| FAIL | The repair loop is what fixed hard questionsThe prompt changes did the work. The loop is a safety net for queries that would have failed outright. | n/a | it fired on 2 of 84 questions and fixed 1; on the 0.5b model it fired on 42 and fixed none | DECISIONS.md ↗ @ dae1de3 |
| FAIL | Similar questions can share a cached answerSo the cache matches only case, punctuation, polite filler and spelling variants, and every hit is validated and run again. | rewordings match and different questions do not | the embedding model scored "customers in Dubai" against "customers in Riyadh" at 1.000, above true rewordings (0.938 to 0.976) | DECISIONS.md ↗ @ dae1de3 |
| FAIL | The 0.5b model is good enoughIt improved on the original 24 questions and collapses on the new hard ones. It is less accurate and exactly as safe. | n/a | qwen2.5:0.5b: 0 of 24 held out, 3 of 24 follow-ups, Urdu 1 of 20 | docs/results.md ↗ @ dae1de3 |
| PASS | The eval is stable enough to gate on | identical replies on unchanged code | re-running 7b gives byte-identical replies, and the baseline for this change reproduced the committed 19/24 exactly | README.md ↗ @ dae1de3 |
| FAIL | Semantic prompt injection through stored data is stoppedWhat bounds the damage is that the guardrails reject destructive SQL whatever persuaded the model, so it can return wrong rows but cannot delete any. | stopped | not stopped; the text 'ignore all rules' looks exactly like a legitimate label | SECURITY.md ↗ @ dae1de3 |
| UNCLEAR | Accuracy on the Spider benchmarkNo Spider number is quoted anywhere in the repo. | n/a | not run; the benchmark's SQLite databases cannot be downloaded unattended | docs/results.md ↗ @ dae1de3 |
| UNCLEAR | Dialectal Arabic and Roman Urdu work | n/a | not measured; the suite is Modern Standard Arabic and Urdu in Urdu script | docs/results.md ↗ @ dae1de3 |
The numbers
Each headline is the conclusion. Open “View the data” under a chart for the exact values.
The gain held on hard questions written before any tuning.
Before and after, qwen2.5:7b
Same model, same questions, scored the same way. Before is the previous pipeline run on today's suites.
At 7b the three languages are close. The small model loses Urdu almost entirely.
Accuracy by language and model size
Development suite, 20 questions per language.
Each model refused once. The validator contained every dangerous output.
What the models wrote when told to attack
Seven adversarial prompts per model, one in Arabic and one in Urdu. Dangerous means a write, DDL, sandbox escape, denied function or stacked statement.
What went wrong
The first question of every session timed out
- How it surfaced
- The request timeout was shorter than a cold start. Loading the model and reading the prompt took 128 seconds, the 120 second limit cut it off, and the client retried.
- What I did
- Timeouts now come from measured latency, the server reads the prompt once at startup, and the model stays loaded while the server runs.
Valid queries with a subquery were blocked
- How it surfaced
- The validator reported a derived table's alias as undefined, so valid queries with a subquery in a join were rejected. Hard questions use them often.
- What I did
- Derived tables are resolved like CTEs. Their contents are still checked, so naming a subquery cannot hide an unknown table.
The repair loop coached an attack
- How it surfaced
- Asked to fix a parse error that contained DROP, the model returned the same attack with a semicolon in it. The second statement did not parse, so it vanished from the telemetry.
- What I did
- A rejected reply that contains a write keyword is never repaired, and the stacked-statement check uses sqlglot's tokenizer, which works on text that does not parse.
A clarification lost to the Urdu verb
- How it surfaced
- After choosing "all orders", the Urdu question still came back filtered on status = 'delivered'. The plain-language steps under the answer showed it at once.
- What I did
- Clarification hints now say what to do in the query. The model then followed the chosen option in 15 of 15 tries across three languages.
Stopping the demo threw the model's work away
- How it surfaced
- A keep-alive request sent on shutdown did not carry the context size the model was loaded with, so Ollama reloaded the model and a running job paid a 4.5 minute re-read.
- What I did
- The server keeps the model warm through its normal request path and never sends a bare keep-alive.
A 200 MB reply passed every control
- How it surfaced
- Probing with printf('%.*c', 200000000, 'x') returned a 200 MB string in 1.1 seconds. It was one row, well inside the 5 second budget, an allowed function and an ordinary SELECT.
- What I did
- Every limit bounded rows or time and none bounded bytes. A per-cell cap and a total byte budget now apply in the executor.
Stored prompt injection through sampled column values
- How it surfaced
- Column values are rendered into the prompt, and repair hints quote values from the data. A value of '; DROP TABLE t-- reached the prompt intact.
- What I did
- A content allowlist withholds unsafe values from both. A plain-language payload cannot be filtered by form, so that residual risk is documented and a test asserts it.
Choices I would defend
Parse the query, never pattern-match it
Every regex filter has a one-line bypass, such as DR/**/OP or a WITH clause hiding a DELETE. The validator parses with sqlglot and checks the whole tree.
Hold out questions before tuning
Eight hard questions were written and checked before any prompt change and run once at the end. A prompt tuned on a suite always looks good on that suite.
One prompt for every language
The model reads its prompt once at startup and reuses it whatever language comes next. A prompt per language made every switch re-read it from scratch.
Show the model the error, but never repair an attack
The validator and SQLite already know what is wrong with a broken query. A query that tried to write is final, because a repair would turn a blocked attack into a cleaner one.
Ask back from a curated list
A short list of terms that can be read two ways in this data decides when to ask. It costs no model call and is checked against every eval question.
Build answer text from the result
The one-line answer and the plain-language steps come from the result and the syntax tree, so they cannot state a number the query did not return.