All work
case study
In progressEvaluating Pretrained Time Series Foundation Models for Smart City Sensor Forecasting
Do pretrained forecasting models beat trained baselines on messy city sensor data?
Problem
Pretrained foundation models promise forecasts without task-specific training. Real city sensor data has gaps, outages and short histories, so I test whether the models hold up and whether they beat models trained on the same data.
Approach
- Six pretrained models against statistical, boosting and deep learning baselines, zero-shot and fine-tuned.
- Quantile forecasts scored by quantile loss and 80 % interval coverage.
- Stress tests with gaps and outages: 1,000+ rolling forecasts per model.
- Diebold-Mariano tests with Holm correction, protocol frozen before the confirmatory test.
Results
Preliminary pilot results; the thesis is in progress and the numbers may change.
- All six beat seasonal-naive: 30 of 30 tests.
- Chronos-2: 24-hour error down 29 %. Best trained baseline: 18 %.
- 80 % interval coverage: 0.70–0.74 before recalibration, 0.80 after.
- FastAPI service: forecast and 80 % band in 0.5–1.7 s.
Worth noting
I found my own uncertainty estimates too narrow and fixed them.