← All posts
Data Engineering9 min read

dbt Test Severity Warn: Find Tests That Never Fail Builds

A dbt test set to warn reports a problem but never stops the build. Here is how to find every one in your project and decide which should fail.

A dbt test with severity set to warn reports failing rows but never stops dbt build, so downstream models still run and dashboards still refresh on data you already know is wrong. To find every one, parse your project, then query target/manifest.json for tests whose config.severity is warn. That takes under five minutes, and the queries are below.

Most teams set a test to warn for a good reason. The test was noisy, a source was misbehaving, and someone needed the pipeline green before a board pack went out. Then nobody went back. A few months later the project has dozens of tests that can only produce log lines, and the team believes it has test coverage that does not actually protect anything.

This post covers the detection first, then how to triage what you find, then how to stop warn tests creeping back in.

Why Does dbt Test Severity Warn Hide Bad Data?

Because a warning is not a gate. When a test runs with severity warn, dbt records the status as warn in run_results.json and carries on. In dbt build, models downstream of the test still materialise. Nothing alerts anyone unless you have wired up log parsing, and most small teams have not.

The mechanics matter here. With severity error, dbt checks error_if first, then warn_if. With severity warn, it skips error_if entirely and only evaluates warn_if. So a test set to warn cannot fail a run at any threshold. Raising error_if to something like > 100 has a similar effect. The test only fails once the problem is large, which is usually after someone has already noticed it in a dashboard.

This is a trust problem more than a tooling one. dbt Labs' 2025 State of Analytics Engineering report found poor data quality was the challenge most often reported by data teams, cited by over 56% of respondents. Its 2026 edition reports that 71% are concerned about hallucinated or incorrect data reaching stakeholders. A warn test on a primary key is exactly how incorrect data reaches stakeholders while the project looks tested.

A pattern we see repeatedly in early-stage fintech and payments teams: a uniqueness test on a transactions model gets downgraded to warn during a noisy week from a duplicated webhook feed. Duplicates then inflate volume in a finance dashboard for weeks, because the model is built and the dashboard is refreshed anyway. Nobody was negligent. The warning simply never had an owner.


📺 Watch: dbt Incident Management Tutorial: Cut Alert Noise with Orchestra

dbt Incident Management Tutorial: Cut Alert Noise with Orchestra


How Do You Find Every dbt Test Set to Warn?

You need the manifest. Run dbt parse (no warehouse queries, so it costs nothing on BigQuery), then query the JSON with jq.

List every test currently configured as warn, with the model it is attached to:

dbt parse
jq -r '.nodes | to_entries[]
  | select(.value.resource_type=="test")
  | select((.value.config.severity|ascii_downcase)=="warn")
  | [.value.name, (.value.attached_node // "n/a")] | @tsv' target/manifest.json

Then find the sneakier ones: tests that are still severity error but have a custom threshold, so they only fail when the damage is large.

jq -r '.nodes | to_entries[]
  | select(.value.resource_type=="test")
  | select(.value.config.error_if != "!= 0" or .value.config.warn_if != "!= 0")
  | [.value.name, .value.config.severity, .value.config.warn_if, .value.config.error_if] | @tsv' target/manifest.json

Check for blanket downgrades too. Search dbt_project.yml for a project-wide default:

grep -nE "severity|warn_if|error_if" dbt_project.yml models/**/*.yml

If you see +severity: warn under data_tests: (or tests: on older dbt versions), every test in that folder is advisory. We have seen this applied to an entire staging layer to get a first deployment green, then left in place for a year.

One caveat: the key is data_tests from dbt 1.8 onwards and tests before that, so check which your version uses.

Which Warn Tests Are Failing Right Now?

A warn test that passes is harmless. A warn test that fails every day is a known defect with no owner. Separate the two using the last run's artifacts:

jq -r '.results[]
  | select(.status=="warn")
  | [.unique_id, .failures] | @tsv' target/run_results.json

That shows the most recent run only. For a history, turn on store_failures for your warn tests. dbt then writes failing rows to a table in a dbt_test__audit schema, overwritten on each run. On BigQuery you can see which of those tables currently hold rows:

SELECT
  table_id,
  row_count,
  TIMESTAMP_MILLIS(last_modified_time) AS last_refreshed
FROM `your-project.analytics_dbt_test__audit.__TABLES__`
WHERE row_count > 0
ORDER BY row_count DESC;

Replace analytics_dbt_test__audit with your target schema plus _dbt_test__audit. Every row in that result is a test that fired and was ignored. Sort by row_count, then open the biggest table and look at the actual rows. That is usually where the conversation with the model owner starts.

Running that query for one project is fine. Checking every model across a whole project is not. fintel-scan is a free, open-source (MIT) CLI that runs fourteen checks across a dbt project locally, with no warehouse connection: uvx fintel-scan. If you want a second pair of eyes on what it finds, you can also explore our services.

Should a dbt Test Severity Warn Become an Error?

Not always. Promoting every warn to error will break your pipeline on day one and teach the team to switch tests off. Triage each test into one of four buckets instead:

  1. Guards a key or a financial figure (unique, not_null on IDs, amounts, currency codes): promote to error now. If it fails, the data is wrong and downstream consumers should not see it.
  2. Fails because of a real upstream defect: fix the source or add a cleaning step in staging, then promote to error. If the fix needs a week, keep the warn but give it an owner and an expiry date.
  3. Fails for a known, bounded reason (a legacy cohort, a partner who sends nulls): scope the test with a where clause to exclude the known exception, then promote to error. A narrow error test beats a broad warn test.
  4. Fires on every run and nobody cares: delete it. A test that has been ignored for months is not documentation, it is noise that trains people to ignore the real ones.

For bucket 2, a time-boxed warn looks like this:

data_tests:
  - unique:
      config:
        severity: warn
        meta:
          owner: payments-data
          expires_on: 2026-12-15
          reason: duplicate webhook events, fix in ingestion

The meta block is the important part. It turns an anonymous downgrade into a decision with a name and a deadline against it.

For the highest-risk models, go one step further and separate the build from the publish step. Let the model materialise into a staging dataset, run its error tests, and only promote it to the dataset your BI tool reads when they pass. The same principle underpins good input controls generally, as we cover in manual data entry validation.

How Do You Stop Warn Tests Creeping Back In?

Downgrading a test is a one-line change, so it will keep happening unless something checks for it. Add a CI step after dbt parse that fails the pull request if any warn test has no expiry date:

dbt parse
jq -e '[.nodes[]
  | select(.resource_type=="test"
      and (.config.severity|ascii_downcase)=="warn"
      and (.config.meta.expires_on == null))]
  | length == 0' target/manifest.json

The -e flag makes jq exit non-zero when the result is false, which fails the CI job. It costs nothing, takes seconds, and means every warn test in the project has an owner and a date.

Three more habits keep the list short:

  • Review the warn list monthly. The first jq query above is the agenda. Anything past its expiry date is promoted or deleted.
  • Alert on warn status, not just errors. Parse run_results.json after each run and post any warn result to a channel the model owner watches.
  • Ban blanket severity defaults in dbt_project.yml through code review. Severity belongs on individual tests, where someone had to justify it.

In our work with growth-stage fintech and e-commerce teams, this audit tends to surface the same thing: a project that looks well tested on paper and has its most important checks switched to advisory. Migrating calculations out of spreadsheets and into tested dbt models removed a class of recurring manual errors for one client, but only because the tests were allowed to stop a bad build. A test that cannot fail is a comment, not a control.

Frequently Asked Questions

Q: What does severity warn do in a dbt test?

A: It makes dbt report the failure in the log and in run_results.json with a status of warn, but the run continues. In dbt build, downstream models still build. The test only becomes a gate if you set severity to error or a threshold that actually triggers an error.

Q: How do I find all dbt tests set to warn?

A: Run dbt parse, then filter target/manifest.json with jq for test nodes where config.severity equals warn. Also check dbt_project.yml for a project-wide +severity: warn, and look for custom warn_if and error_if thresholds that make a test tolerant.

Q: What is the difference between warn_if and error_if in dbt?

A: Both take a condition on the number of failing rows, such as > 10. With severity error, dbt checks error_if first and then warn_if. With severity warn, it ignores error_if and only evaluates warn_if, so the test can never fail the run.

Q: Does dbt build stop when a test warns?

A: No. A warn result does not stop dbt build, and models that depend on the tested model still run. Only an error-level test failure causes dbt to skip downstream nodes in the same build.

Q: When is it acceptable to use dbt test severity warn?

A: It is reasonable for a short, time-boxed period while an upstream defect is fixed, or for genuinely advisory checks like soft freshness expectations. It should always carry an owner and an expiry date, and it should never guard a primary key or a financial figure.

Fix the Tests That Cannot Fail

The challenge most teams face is not writing tests. It is knowing which of them actually protect the numbers leadership reads, and which ones were quietly downgraded and forgotten. A dbt test severity warn audit is the fastest way to find out, and the fix is usually a few hours of triage rather than a rebuild. At Fintel Analytics, we have helped fintech, payments and e-commerce teams audit their dbt and BigQuery projects, put real gates in front of their dashboards, and move critical logic from spreadsheets into tested SQL models. If you suspect your green builds are hiding problems, we can show you what is really failing, and that is far cheaper to learn from an audit than from a finance review.

← Back to all posts
dbt Test Severity Warn: Find Silent Test Failures (2026) — Fintel Analytics