Batch-to-Streaming Energy Forecasting with Spark ML and Kafka
I wanted to take a model beyond a training notebook. The project uses hourly meter readings, building data, and weather from several sites to predict energy use over six hours. The saved model then scores weather events as they arrive through a live stream.
Batch training and streaming create different problems. Energy use spans about eight orders of magnitude, from nearly empty buildings to whole campuses. Inputs mix numbers, circular wind direction, and large categories. The metric must also balance absolute and relative error. In streaming, events arrive out of order, window state needs a memory limit, and an offline model must score events fast enough for live use.
I defined Spark schemas for all three tables to avoid slow inference. The features followed the data analysis. Log1p scaled floor area, building age replaced construction year, and sine and cosine encoded wind direction so 359° stayed close to 1°. Each site's three hottest and coldest months defined peak seasons. I removed building and site IDs so the trees learned patterns that transfer to unseen buildings and prevent identity memorisation.
I trained Random Forest and GBT on the same features to isolate the model choice. RMSLE drove selection because it treats a 10% error equally across small and large buildings. Random Forest beat GBT at 1.97 versus 2.29 RMSLE. Tuning reduced it to 1.95. Training on log1p energy produced the best score, 1.30, by giving the trees a less skewed target. I saved that final pipeline for streaming.
A Kafka producer replays weather records in time order. Spark Structured Streaming applies a five-second watermark, which absorbs expected delay while bounding state. It repeats the training features and scores each event with the saved model. Separate Kafka topics receive two tumbling windows: seven seconds for six-hour building totals and one second for daily site totals. Each window matches how quickly its group gains enough events to be useful.
A consumer notebook drains the output topics, joins predictions against actual metered consumption, and builds the plots an operator would actually look at. A building x interval heatmap flags demand-response candidates. Per-site daily trend lines show where consumption is heading. A shortfall/excess chart flags which sites the model under- versus over-predicts, and the under-predicted ones are the operationally urgent failure mode. A predicted-vs-actual trace annotates the single largest error and traces it back to a concrete cause. The feature set has no way to see one-off building-level events like an equipment fault or an unscheduled occupancy change, so it necessarily smooths over them.
The live stream reuses the exact feature logic from offline training and keeps aggregation memory bounded. Model selection used the metric that fit the data. The result shows that one Spark pipeline can move cleanly from batch training to live scoring.
I built this to test the full path from training to a realistic deployment. Training decisions had to support the later stream. The streaming job had to bound its state, handle event time correctly, and reuse the saved model.
Batch decisions shaped the stream more than I expected. Removing raw building and site IDs prevented leakage and let the saved model score unseen buildings. Watermarks and windows also had to match the clocks of both producer and consumer. Tuning either side alone was insufficient.
Before deployment, I would add access control to Kafka because short-interval building forecasts can reveal identity or occupancy. The largest errors come from faults and occupancy changes missing from the features. The next model needs those signals. Another tuning pass on the same inputs cannot supply them.