Rails for data science: the platform beneath a forecasting engine

Most write-ups of a forecasting project are about the winning model. This one is about the thing underneath it: the platform. It wasn't the whole story: the team's modelling did the forecasting, across a genuinely hard and varied catalogue, and did it well. But the platform is the part I owned, the part that made the team's work efficient and cohesive, and the part that generalises beyond this project.

The brief, in one line: forecast demand for Suzuki collision parts across a catalogue that behaves like several forecasting problems at once. Collision parts sell only after an accident, so demand is intermittent and volatile. A few high-volume parts move steadily; a long tail sits idle for months and then spikes. No single model wins across that spread, which means the real task is not "pick a model" but "compare many models honestly," with a team spread across a nine-hour time difference, where small inconsistencies quietly invalidate every comparison.

That reframing matters. Strong models still had to be built, and building them across such a varied catalogue was hard, genuinely collaborative work. But a model is only as trustworthy as the evaluation around it, and getting that right, consistently, across a distributed team, is an infrastructure problem.

The two failure modes worth engineering against

Work like this fails in two predictable ways.

The first is inconsistency: each person loads and shapes the data slightly differently, so results can't be trusted against one another. The comparison looks rigorous and isn't.

The second is leakage: information from the future slipping into training. It is the single most common reason a forecast looks superb in testing and then disappoints in production. It is especially easy to introduce when the rule that classifies each part is computed once across the full history rather than recomputed at each point in time.

Both are problems the infrastructure exists to solve. Handled once, centrally, they let the whole team move faster and trust each other's results. Left to each notebook, they compound.

What I built

My contribution to this project was the platform layer (the shared library and the Features API) rather than the models themselves, which the team implemented collaboratively. Three pieces did the load-bearing work.

A shared library. One way to load and prepare the data: scoping, deduplication, hygiene checks, and the reindexing that tells a genuine zero-sale month apart from a missing row. Written once, reused everywhere. Across a distributed team, a single source of truth is the difference between results you can compare and results that merely resemble each other.

A model contract. Every model, however different inside, wrapped in a function with the same shape: take data and parameters, return predictions in one agreed format. A naïve baseline, a classical intermittent-demand method, a gradient-boosted tree, and a zero-shot foundation model all present the same face to the evaluation harness. New models drop in with almost no boilerplate and get compared on identical terms, which is the point.

A leakage-safe harness. A rolling-origin backtest: step forward a month at a time, train only on history available at that point, forecast at the true operational lead time, pool the errors. Crucially, the classification that routes each part is recomputed at each step from past data only. Point-in-time correctness is a property of the harness, not a discipline anyone has to remember.

A Features API. Testing whether outside signals (economic indicators, fleet-size proxies, catalogue attributes) help a forecast is easy to do badly. The Features API turned exogenous data into a uniform interface: list what's available, pull it as a frame, join it on the right key without hand-written merges. That mattered less for any single join than for what it made possible next.

Automating the search

Once every model speaks one contract and every feature is reachable through one API, feature and model search stops being manual labour.

I ran an automated meta-study to measure what each exogenous feature was actually worth. Rather than trust one tuned model, it looped an Optuna search across every candidate feature and every demand class, comparing each feature's effect against a matched baseline. A controlled detail made the comparison unusually clean: the searches shared a fixed set of starting trials, so each feature was tested against an identical configuration, with the difference in error attributable to the feature, not to lucky tuning.

There's an honest lesson in it. The trial budget per study was initially set far too low for the search space. But because every study shared the same startup trials and seed, that error became an asset: it made the paired comparisons exactly controlled. Reporting that plainly is part of the method; a platform should make results legible enough to see both what worked and why.

Where this points

The endpoint is the interesting part. Once models speak one contract and features live behind one API, an agent can drive the search itself: selecting models, assembling inputs, evaluating and tuning them semi-autonomously in response to a prompt. Exposing the contract-compliant library and the Features API as tools an agent can call is a short step from where the platform already stands.

That's the thesis behind most of what I do. Good models matter enormously, and on this project they were the team's achievement. But the platform they run on is what lets a team, or an agent, build them efficiently, compare them fairly, and trust the answer. The models were the team's; the rails were mine, and the rails are what generalise.

The structured version of this engagement, section by section, is in the case study.