Where AI helps an analyst and where it hurts

A confident tone and a correct answer are independent things

A language model predicts a plausible continuation of text. Plausible SQL is syntactically correct, runs without errors and returns a result. Whether that result matches reality is a separate question, and generation alone does not answer it. Hence the rule that runs through this course: verification is built into the process, not bolted on at the end.

Where the gain is real

• A draft query against a schema you provide. You give the table DDL and describe the task in words, and you get a 40-line skeleton instead of half an hour of typing.
• Reading someone else's code. The model walks through a 200-line query with six CTEs step by step and names what each block does.
• Translating between dialects and tools. Window functions, date interval syntax, regular expressions.
• Generating checks. "What are three independent ways to re-verify this number" is a strong prompt; the model suggests cross-checks you had not thought of.
• A draft of the report text written over numbers you have already computed and verified.
• Getting oriented in an unfamiliar domain. What a field with this name usually means, which traps show up in billing data.

Where the harm is systematic

• Arithmetic over pasted data. "Sum these 300 rows" gives a number that looks right and is almost certainly wrong.
• Guesses about the schema. The model will add a condition on an is_deleted field you do not have, or drop it because you never showed the DDL.
• Business definitions. What counts as an active user in your company is known only to your company.
• Causation. Ask it to explain a drop and it produces a coherent story over any data at all, including pure noise.
• Agreeing under pressure. Say "I think there is a bug here" and the model will find a bug even in a correct query.

Weak:   "Compute revenue by region"
Strong: schema: orders(id, user_id, created_at timestamptz,
        status, amount_minor int, currency)
        refunds(order_id, refunded_at, amount_minor)
        task: net revenue = paid orders minus refunds,
        a refund belongs to the month of the original order,
        status paid, currency converted to RUB using fx_rates
        on the order date, period 2025-01..2025-06, split by region
        return the SQL and, separately, three ways to check the result

The division of labour rule

The model phrases things and writes code; the database computes. Any model answer that contains a number must either be turned into a query or discarded. That sounds strict, but it removes a whole class of errors: a number from a chat is not reproducible, you cannot recompute it tomorrow and you cannot put it in a report with a link to its source.

Testing for confirmation bias

The model adapts to how you phrase the prompt. Ask it to show that the new price lifted conversion and you get a set of slices where it lifted. Ask for the opposite and you get an equally convincing set. Both answers were generated from the same data. So here is a working technique: run every hypothesis twice, once in the affirmative and once in the disproving form, and see whether the effect survives. If both sides come back with strong arguments, you are looking at noise rather than an effect, and what needs examining is not the conclusions but the sample size.

Cheat sheet

• Plausible SQL runs and returns the wrong thing.
• Code and wording to the model, numbers to the database.
• Show the schema explicitly, set the definitions yourself.
• Ask for the list of assumptions and for ways to check.