dbt run-operation in BigQuery: The Lineage Blind Spot (2026)

The Short Answer

When you run a transformation through dbt run-operation, a BigQuery stored procedure, or EXECUTE IMMEDIATE instead of a proper dbt model, that transformation is invisible to dbt's DAG. No lineage. No tests. No source freshness. Any downstream model that depends on the output is flying blind — and when it breaks, you will spend hours looking for a cause that does not exist inside dbt at all.

This is one of the most common silent correctness problems we see in dbt projects built under time pressure. The team needed to do something dbt could not do cleanly in a model — a dynamic pivot, a multi-statement procedure, a one-off backfill — and the workaround shipped to production. Nobody cleaned it up. Six months later, nobody knows what it does or when it runs, and it is still writing to a table that three mart models depend on.

This post is about finding those blind spots, understanding what they break, and fixing them properly.

---

What Actually Happens When You Use run-operation for Transformations

The dbt run-operation command is designed to invoke macros for operational tasks — things like granting permissions, creating audit log entries, or dropping development schemas. It is explicitly not intended to be part of your transformation pipeline, and for good reason.

When a transformation lives inside a run-operation call:

• dbt has no record of it in the DAG. The output table exists in BigQuery, but dbt does not know how it was built, what it depends on, or what depends on it.
• dbt ls and dbt docs generate will not show it. Any downstream consumer of that table appears to have a root-level dependency on nothing — when in reality it depends on logic that runs outside the orchestrated pipeline.
• Tests cannot be applied automatically. You can write a generic test against the output table, but you cannot use dbt build to guarantee that the procedure ran before the test executed.
• Source freshness checks do not apply. If the procedure reads from a raw source that goes stale, dbt will not flag it.
• Column-level lineage is completely absent. As dbt's static analysis tooling has matured — including the Fusion engine — it parses model SQL to build column-level lineage. A stored procedure or EXECUTE IMMEDIATE call is a black box: the parser cannot see inside it, and you lose that entire lineage branch.

The same problem applies to BigQuery stored procedures invoked via dbt run-operation or via post-hooks that call CALL my_dataset.my_procedure(). The result is a table in your warehouse with no provenance inside dbt.

A pattern we see repeatedly in early-stage analytics teams: a data engineer writes a dynamic pivot using EXECUTE IMMEDIATE — because BigQuery's native PIVOT requires static column lists — wraps it in a macro, and calls it from run-operation in the orchestrator. That pivot output feeds a dashboard that finance checks every morning. Nobody has documented the dependency, there are no tests on the pivot output, and the first time the upstream source schema changes, the pivot silently produces NULLs. Finance uses the broken numbers for two weeks before anyone notices.

---

---

📺 Watch: Easily Start Your dbt Cloud Job After a BigQuery Upload

---

How to Find run-operation Transformations Hiding in Your Project

The fastest way to surface this blind spot is to cross-reference what BigQuery actually executed against what dbt knows about. You need two things: the tables dbt manages, and the tables that were written to by statements dbt did not orchestrate.

Step 1: Get all tables dbt knows about.

Run this from your dbt project root:

dbt ls --output json | python3 -c "

import sys, json for line in sys.stdin: try: node = json.loads(line.strip()) if node.get('resource_type') in ('model', 'seed', 'snapshot'): print(node['relation_name']) except: pass " > dbt_managed_tables.txt

This gives you a flat list of fully qualified table names that dbt owns.

Step 2: Query BigQuery's INFORMATION_SCHEMA.JOBS to find tables written to outside of dbt runs.

Run this in the BigQuery console or via bq cli. Replace your_project and your_dataset with your values:

SELECT
  destination_table.project_id,
  destination_table.dataset_id,
  destination_table.table_id,
  COUNT(*) AS write_job_count,
  MAX(creation_time) AS last_written_at,
  COUNTIF(JSON_VALUE(labels, '$.dbt_invocation_id') IS NULL) AS writes_without_dbt_label,
  COUNTIF(JSON_VALUE(labels, '$.dbt_invocation_id') IS NOT NULL) AS writes_with_dbt_label
FROM
  `your_project`.`region-eu`.INFORMATION_SCHEMA.JOBS
WHERE
  creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND statement_type IN ('INSERT', 'CREATE_TABLE_AS_SELECT', 'MERGE', 'UPDATE', 'SCRIPT')
  AND destination_table.dataset_id = 'your_dataset'
  AND error_result IS NULL
GROUP BY 1, 2, 3
HAVING writes_without_dbt_label > 0
ORDER BY writes_without_dbt_label DESC;

What this does: dbt tags every BigQuery job it submits with a dbt_invocation_id label. Any write job to your dataset that lacks that label was executed outside of dbt — either by a run-operation macro (which does not attach the standard model labels), a stored procedure call, a scheduled query, or a direct SQL session. Tables that appear in this result but do not appear in your dbt_managed_tables.txt file are your blind spots.

Step 3: Inspect SCRIPT statement type jobs specifically.

BigQuery classifies EXECUTE IMMEDIATE and multi-statement stored procedures as SCRIPT jobs. These are worth isolating because they are the most opaque — the inner SQL is only visible in the query field of the job:

SELECT
  job_id,
  user_email,
  query,
  destination_table.table_id AS destination,
  creation_time
FROM
  `your_project`.`region-eu`.INFORMATION_SCHEMA.JOBS
WHERE
  creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND statement_type = 'SCRIPT'
  AND error_result IS NULL
ORDER BY creation_time DESC
LIMIT 100;

If you see EXECUTE IMMEDIATE in the query column writing to a table that downstream dbt models reference, you have found the exact problem.

Running that query for one dataset is manageable. Checking across every dataset in a large project is not. fintel-scan is a free, open-source CLI (MIT licence) that runs this check and fourteen others locally against your dbt project metadata without a live warehouse connection: uvx fintel-scan.

---

Why This Breaks Downstream Models Silently

The insidious part of this problem is not the immediate failure — it is the silent corruption. When a stored procedure writes malformed data, a downstream dbt model does not error. It runs successfully against bad inputs and produces a result that looks plausible.

Consider this dependency chain:

raw_transactions (source)
    |
[stored procedure: dynamic pivot — outside dbt]
    |
int_transactions_pivoted (dbt model — references the pivot output as a source)
    |
fct_revenue_by_channel (dbt model — used in executive dashboard)

From dbt's perspective, int_transactions_pivoted appears to depend directly on raw_transactions — or on nothing at all, if it references the pivot output table as a hardcoded string rather than a {{ source() }} or {{ ref() }} call. The stored procedure step is invisible. If the stored procedure stops running — because an orchestrator job fails silently, because a schema change broke the dynamic SQL, because someone changed the procedure without updating the caller — int_transactions_pivoted keeps reading from the last successfully written version of the pivot table. Yesterday's data. Last week's data. Stale indefinitely, with no alert.

This is precisely the kind of scenario where a reconciliation failure goes undetected for weeks. We have seen a version of this in a global fintech client: a transformation running outside the dbt DAG was producing a subtly wrong aggregation, and because there were no tests on its output and no lineage flagging it as a dependency, the error propagated into executive reporting for over a month before a manual spot-check caught it.

For more on how late-arriving or stale data propagates silently through dbt models, see our post on late-arriving data in dbt BigQuery — the mechanisms overlap.

---

How to Fix It: Bring the Logic Into the DAG

The fix depends on why the transformation was outside dbt in the first place.

Case 1: Dynamic SQL that dbt models cannot express natively.

The most common reason for reaching for EXECUTE IMMEDIATE is a dynamic pivot — BigQuery requires static column names in PIVOT, so engineers generate the column list at runtime. The fix is to pre-materialise the distinct pivot values into a seed or a staging model, then use a Jinja loop in your dbt model to generate static CASE WHEN expressions that replicate the pivot. It is more verbose, but it is transparent, testable, and visible in lineage. If the set of pivot columns changes rarely, a seed refresh is a small operational cost for a large correctness gain.

Case 2: Multi-statement procedures needed for sequenced writes.

If a stored procedure does sequential inserts or updates that a single SELECT cannot express, refactor into multiple dbt models with the correct materialisation strategy. An incremental model with a MERGE strategy can replace most write-heavy procedures. Where a true multi-statement flow is unavoidable, use dbt's pre-hook and post-hook to anchor it to a specific model — and document the dependency explicitly in that model's meta block so it appears in documentation even if it cannot appear in the DAG graph.

Case 3: The transformation is a genuine one-off backfill.

If it was truly a backfill and is not recurring, remove it from the orchestration pipeline and document what it did. Add a test on the output table's key invariants — unique key, not-null on critical fields, expected row count range — so that any future modification to the backfilled data is caught. Add a {{ source() }} declaration for the output table so dbt at least knows it exists.

In all cases: add a test on the output table immediately. Even if you cannot bring the transformation inside dbt this sprint, a generic not_null and unique test on the output's primary key buys you an early warning system. Use dbt test --select source:your_dataset.your_table to run it in isolation.

For hardcoded table references that bypass dbt's lineage graph, our post on hardcoded table names in dbt BigQuery covers the detection and remediation pattern in detail.

---

How to Stop This Recurring

Detection and fix are table stakes. Prevention is the real goal.

1. Lint your macro directory for write-producing SQL. Add a CI check — even a simple grep — that fails if any macro invoked by run-operation contains INSERT, MERGE, CREATE TABLE AS SELECT, or CALL statements. Macros in your macros/ directory should not be writing to tables.

2. Enforce dbt label presence in your monitoring query. Schedule the INFORMATION_SCHEMA query from Step 2 above as a daily BigQuery scheduled query. Alert on any new table that receives a write without a dbt_invocation_id label. This makes the rogue write visible within 24 hours of it first occurring, not six months later.

3. Review run-operation calls in your orchestrator. If you are using Airflow, Prefect, or Cloud Composer, grep your DAG definitions for dbt run-operation. Every occurrence is a candidate for review. Ask: is this call producing data that downstream models consume? If yes, it belongs in the dbt graph, not outside it.

4. Document the exception explicitly if you cannot fix it immediately. If a run-operation transformation genuinely cannot be refactored right now — resource constraints, complexity, time — add a sources.yml entry for its output table with a description block that says exactly what writes to it and when. That costs ten minutes and means the next engineer who looks at the lineage graph has a fighting chance of understanding what they are looking at.

---

Frequently Asked Questions

Q: Can I use dbt run-operation to insert data into BigQuery tables?

A: You can — BigQuery will execute it. But you should not, for production pipelines. Transformations run via run-operation are invisible to dbt's DAG, carry no lineage, and cannot be reliably tested as part of a dbt build run. Use a proper dbt model instead.

Q: How does dbt tag the BigQuery jobs it submits?

A: dbt attaches labels to every BigQuery job it submits, including dbt_invocation_id, dbt_job_name, and dbt_node_id. You can query these via INFORMATION_SCHEMA.JOBS using JSON_VALUE(labels, '$.dbt_invocation_id'). Jobs submitted by run-operation may not carry full model-level labels, and stored procedures called externally will carry none.

Q: Does using a dbt post-hook to call a stored procedure restore lineage?

A: Partially. Anchoring the procedure call to a specific model via a post-hook at least ties its execution timing to that model's run. But the stored procedure's internal logic remains opaque to dbt's static analysis — column-level lineage is still broken, and tests must be written manually against the output. It is better than a completely standalone call, but it is not a substitute for a proper dbt model.

Q: What if my stored procedure uses EXECUTE IMMEDIATE — can dbt parse that?

A: No. EXECUTE IMMEDIATE constructs and executes SQL at runtime, which means the target tables and column transformations are unknowable at parse time. dbt's static analysis — and BigQuery's own Dataplex lineage tracking — cannot trace dependencies through dynamic SQL. The only reliable solution is to refactor the dynamic logic into static dbt models with Jinja templating.

Q: How do I find all tables in BigQuery that were written to by stored procedures in the last 30 days?

A: Query INFORMATION_SCHEMA.JOBS in your region filtering on statement_type = 'SCRIPT' and error_result IS NULL for the target time window. Join the destination_table fields against your list of dbt-managed models to isolate tables written by procedures that dbt does not know about. The full query is in the detection section above.

---

If your dbt project has grown organically under delivery pressure — and most do — there is a real chance that transformations are running outside your DAG right now, writing to tables your mart models depend on, with no tests, no lineage, and no alert when they stop working. At Fintel Analytics, we have diagnosed and remediated exactly this class of problem for analytics teams across fintech, payments, and e-commerce — often as part of a broader data stack audit that surfaces several interconnected issues at once. If your lineage graph does not tell the full story of how your data moves, that is a fixable problem, and fixing it is the foundation everything else depends on.