Building reliable ML feature pipelines for startups means solving one specific problem above all others: ensuring the data your model trains on is identical in structure, logic, and freshness to the data it sees in production. When those two environments diverge — and they almost always do, silently — your model decays from the moment it ships. The good news is that this is a solvable engineering problem, and solving it does not require the infrastructure of Uber or Airbnb.
Most early-stage companies discover training-serving skew the hard way. A model that looked excellent in evaluation suddenly delivers poor predictions in production. The data science team assumes the model needs retraining. The engineering team assumes it is a data quality issue. In reality, it is both — and the root cause is almost always a feature pipeline that was never designed to serve two masters at once.
This post is a practitioner's guide to building ML feature pipelines that hold up in production — at the scale of a growth-stage company, with a realistic team and a real deadline.
What Is Training-Serving Skew and Why Does It Destroy ML Models?
Training-serving skew is the gap between how features are computed during model training and how they are computed at inference time. On paper, this sounds like a straightforward engineering mistake. In practice, it is one of the most persistent failure modes in production ML.
The mechanisms are varied. Your training pipeline runs overnight on a full historical dataset using a Python notebook. Your serving pipeline runs in real time via an API endpoint. Two engineers maintain these separately. A refactor in the serving stack quietly pins a feature value to a default. The model continues generating predictions — but with silently degraded accuracy. Nobody raises an alert because the model is not erroring; it is just wrong.
As one analysis of the problem notes, training-serving skew can quite easily crop up via a bug in your model's code and cause serious repercussions further down the line — impacting machine learning models by reducing performance over time as it gradually decays. What makes this so dangerous for growth-stage companies is that the decay is gradual. There is no crash, no exception, no failed deployment. The model just quietly becomes less useful while your team assumes it is working.
There are three common variants we see in early-stage ML work:
Transformation skew — The feature engineering logic (e.g. normalisation, encoding, aggregation) differs between training and serving. Often introduced when a data scientist writes training logic in pandas and a separate engineer re-implements it in a production language or framework.
Temporal leakage — A feature value at training time is computed from data that was not yet available at the prediction moment. Aggregates like a 7-day click count joined naively pull in data that happened after the event being predicted. This produces inflated model performance during evaluation that collapses in production.
Point-in-time incorrectness — Point-in-time joins prevent data leakage by ensuring that when generating training data, you join only the feature values that were actually available at the time of each event. If future data is included, the model shows unrealistically high performance during training but degrades significantly in production.
A pattern we see repeatedly in fast-growing fintech and payments companies: a fraud or credit-scoring model that scores 0.91 AUC in backtesting and 0.74 AUC in production. The delta is not a modelling failure — it is a feature pipeline failure. The training set was built with full history available; the serving environment only has access to features within a narrow real-time window. Nobody designed the pipeline to enforce consistency between the two.
📺 Watch: Feature Engineering for AI: Transforming Raw Data into Predictions
Why Do Startups Get This Wrong? The Real Root Causes
The standard narrative is that startups move fast and cut corners. That is true but insufficient. The more precise explanation is that startups build their ML training pipeline first, in isolation, optimised for iteration speed — and then bolt on a serving layer as an afterthought. By the time the model is in production, the two environments have diverged in ways that are difficult to audit.
Several structural problems drive this:
The notebook-to-production gap — Training is normally done in an analytical environment such as interactive notebooks or data lakes, but serving happens in the production environment via microservices or edge devices — and these models are usually trained and productionised by different people. When the data scientist hands a trained model to a backend engineer to deploy, they hand over a model file — not the feature logic. The engineer re-implements it from a spec. Subtle differences compound.
No shared feature definitions — Without a central place that defines what "7-day transaction volume" means — including nulls handling, timezone normalisation, and aggregation window — every team that needs that feature writes their own version. Finance calculates it differently from the ML pipeline. The ML pipeline calculates it differently from the serving endpoint. Three definitions, three numbers, no single source of truth.
Absence of feature logging — If you are not logging the actual feature values used at inference time, you cannot detect skew after the fact. You cannot retrain on what the model actually saw. You cannot debug degradation with any precision. Systematically recording features used at inference and using those logged features as the basis for retraining and validation is a discipline that most early-stage teams skip — until the first serious production incident forces the issue.
Around 85% of machine learning projects fail, and poor data quality is the #1 reason (Mindinventory, 2026). From our experience, "data quality" in this context is rarely a problem with raw source data — it is almost always a problem with how features are engineered, versioned, and served. The raw data is fine. The pipeline that processes it into model inputs is not.
How to Build an ML Feature Pipeline That Actually Holds Up
The goal is a single source of feature logic that is shared between training and serving. Here is the practical path to get there at startup scale — without building a distributed feature platform on day one.
Step 1: Centralise your feature definitions in SQL or dbt
If your data warehouse is already built on BigQuery or a similar platform, start by writing your feature transformations as SQL models in dbt. This achieves two things immediately: your transformations are version-controlled, and they are decoupled from notebook-specific Python logic. A user_7d_transaction_volume model defined in dbt can be materialised as a table for training and queried via an API for serving — with identical logic in both cases.
This is not a silver bullet for real-time serving latency, but for the majority of ML use cases in fintech and payments — credit risk, fraud scoring, customer health scoring — a feature freshness of 15–60 minutes is entirely acceptable and dramatically simpler to maintain.
Step 2: Enforce point-in-time correctness from the start
Every time you join feature data to your training labels, you must join on event timestamp, not processing timestamp. This is a discipline issue as much as a technical one. Build a templated approach to constructing training datasets that enforces this pattern — a jinja macro in dbt, a utility function in Python — so that no one can accidentally build a training set with lookahead bias.
Step 3: Log your serving features
A useful served-features log includes: model version, feature names with versions, raw feature values before defaulting, transformed feature values after preprocessing, prediction score, and the feature read timestamp. This log is not optional. It is the foundation for detecting skew, debugging production issues, and retraining on real-world data. Store it in your data warehouse. Query it in your BI layer. Build a dashboard that compares training-time feature distributions against serving-time distributions weekly.
Step 4: Monitor feature drift, not just model performance
Continuously comparing serving and training data distributions — for each feature and overall — is essential. Modern ML observability platforms provide automated tools to detect and notify when feature skew is detected, monitoring statistical properties, attribution scores, and distribution drift in real time. At early stage, you do not need a commercial observability platform to do this. A weekly SQL query that computes the mean and standard deviation of your top-10 features in your served-feature log and compares them against your training dataset statistics will catch the majority of drift events before they become incidents.
Step 5: Decide on a feature store when you need one — not before
Tecton, Feast, and Databricks Feature Store have achieved production maturity and are worth evaluating when your team has multiple data scientists, multiple models in production, and genuine sub-second serving latency requirements. Before that point, a well-structured dbt layer with disciplined SQL feature engineering and a serving log will take you further than a prematurely adopted feature platform that nobody has the bandwidth to operate properly.
If you are evaluating whether your current ML infrastructure is the right foundation for scaling further, explore how Fintel Analytics approaches ML data engineering — we work with growth-stage companies across fintech, payments, and e-commerce to design and deliver exactly this kind of production-ready ML infrastructure.
What Does a Production-Ready Feature Pipeline Look Like at Series A Scale?
A Series A fintech — typically 30–80 people, one or two data scientists, a small data engineering function — does not need Uber's Michelangelo. It needs something that can be maintained by two engineers, understood by a data scientist who did not build it, and extended without breaking existing models.
Here is the architecture pattern we implement for clients at this stage:
Offline store (training): BigQuery tables materialised by dbt, partitioned by event date, with a standardised as_of_date column used for point-in-time filtering. Training dataset generation is a parameterised SQL script that accepts a label table and a lookback window. Reproducible, auditable, fast.
Online store (serving): For features that need to be computed in real time, a lightweight Redis cache populated by a streaming job (Pub/Sub or Kafka) stores pre-computed feature vectors keyed on entity ID. For features that tolerate 15-minute freshness, a BigQuery API call with a materialised view is sufficient and removes an entire infrastructure component.
Feature registry: A dbt schema YAML file, extended with custom metadata fields for owner, freshness requirement, and model dependencies. Not glamorous, but it answers every question a new data scientist will ask when they join the team: what features exist, how are they computed, and which models depend on them.
Serving log: A BigQuery table, written to by the inference service, containing every prediction request's feature vector alongside the prediction output. Queried weekly by a Holistics BI dashboard that surfaces feature distribution drift and model score distribution over time.
This is not a feature store in the platform sense. But it solves the same problems — consistency, discoverability, lineage, and reusability — with a fraction of the operational overhead. We have delivered this pattern for a Series A payments company where the previous approach was a Jupyter notebook that re-computed features from scratch on demand, with transformation logic that differed meaningfully from the production serving code. Rebuilding this as a unified dbt-backed pipeline eliminated the skew entirely and gave the team confidence to retrain monthly rather than quarterly.
For teams also navigating how to keep deployed models healthy once they ship, our guide on ML model monitoring for startups covers the complementary problem: detecting degradation in production before your users notice it.
When Should You Consider a Dedicated Feature Store?
A feature store platform makes sense when the coordination cost of managing features across teams exceeds the cost of operating the platform itself. That threshold is higher than most vendors would have you believe, and lower than most early-stage companies assume.
Some concrete signals that you are ready:
- You have three or more models in production that share overlapping features but compute them independently
- Your serving latency requirement is sub-100ms and a BigQuery API call cannot meet it
- You have multiple data scientists who are duplicating feature engineering work across notebooks
- You have experienced a production incident directly caused by training-serving skew that cost measurable revenue
Organisations looking to adopt a feature store generally have three approaches: build, buy, or open source. Building a custom feature store in-house is the most flexible option but requires substantial engineering effort and expertise. Buying a managed service can accelerate adoption with less overhead but reduces control. Open source feature stores provide community-driven solutions that organisations can host and customise, and each option has tradeoffs in cost, maintenance, and integration effort.
For most pre-Series B companies, Feast (open-source, Python-native, integrates cleanly with BigQuery) is the right starting point if you decide a feature store platform is warranted. It imposes structure without locking you into a commercial platform, and it is compatible with the dbt-centric data stack that most growth-stage teams are already running.
The mistake to avoid is adopting a feature store to solve an organisational or governance problem that is better solved by documentation, code review standards, and a shared dbt project. Technology does not fix a process that was never defined.
Frequently Asked Questions
Q: What is training-serving skew in machine learning?
A: Training-serving skew is a discrepancy between how features are computed during model training versus how they are computed when the model runs in production. It causes models that perform well in evaluation to degrade silently after deployment, because the data the model sees at inference time differs structurally from the data it was trained on.
Q: Do early-stage startups need a feature store?
A: Not immediately. A well-structured dbt layer with shared feature definitions, point-in-time-correct training dataset generation, and a served-feature log will solve the core problems — consistency, discoverability, and reusability — with far less operational overhead than a feature store platform. Most pre-Series B teams should build the discipline before they build the platform.
Q: How do I detect training-serving skew in production?
A: Log the actual feature values used at inference time into your data warehouse. Then run weekly comparisons of feature distributions (mean, standard deviation, null rate) between your training dataset and your served-feature log. Significant divergence in any feature that has high model importance is a skew signal. More advanced approaches use statistical drift tests such as the Kolmogorov-Smirnov test or Population Stability Index (PSI) per feature.
Q: What causes data leakage in ML feature pipelines?
A: Data leakage occurs when features used in model training contain information that would not have been available at the time of the prediction — for example, joining a 7-day aggregate without filtering to data that existed before the prediction event timestamp. Point-in-time joins, enforced via a consistent dataset construction pattern, are the primary defence.
Q: How does a feature store help with ML feature pipelines?
A: A feature store provides a centralised repository for feature definitions, ensuring that the same transformation logic is used in both training and serving environments. It typically includes an offline store for training data retrieval, an online store for low-latency inference, and a feature registry for discovery and lineage. The core value is eliminating the duplication of feature engineering logic across teams and environments.
Training-serving skew is not a glamorous problem — it rarely comes up in funding pitches or product roadmaps — but it is one of the most common reasons production ML fails to deliver the value it promised in development. At Fintel Analytics, we have helped fintech, payments, and e-commerce companies untangle exactly this kind of infrastructure debt: rebuilding feature pipelines on a unified dbt and BigQuery foundation, introducing serving logs, and giving teams the visibility to detect drift before it becomes a business problem. If your models are underperforming in production and you are not sure why, that is a solvable engineering problem — and the sooner it is solved, the less revenue it costs you.