Not Every Project Needs the Emergency Vendor 🏗️
The Right Latency Depends on the Decision
There's a belief installed in many data teams: migrating everything to real-time is a sign of technological maturity. They ask for every report to update by the minute, for every indicator to reflect what just happened. What doesn't get questioned is whether the business actually makes decisions at that pace.
Batch processing is not old technology to replace. It's the right tool for decisions that don't need to operate by the second.
Batch processing, known as batch, groups data and executes it in blocks at a scheduled time. Real-time processing, or streaming, acts on each piece of data the instant it arrives. They're different tools for different needs.
A construction company has two types of suppliers. One delivers any material in 24 hours: expensive, efficient for emergencies. The other consolidates orders week by week, groups by volume, and delivers in blocks: more economical, predictable, reliable. For the ceramic finishes being installed tomorrow, the fast supplier makes sense. For blocks, cement, and rebar that arrive in tons, waiting isn't any problem.
A project that demands the emergency supplier provide everything it needs, without distinguishing urgencies, ends up paying more without getting better construction. Not every project needs the emergency vendor.
A monthly portfolio report, an accounting closure, the inventory at end of day are processes where precision matters more than speed. Batch solves them with efficiency and lower cost. Real-time makes sense for anomaly detection, equipment monitoring, or critical inventory alerts.
IBM's comparison of stream and batch processing explains that modern architectures often combine both approaches based on the urgency of insight and the nature of analysis. The error is not keeping batch. It is migrating because of fashion without reviewing which business decision changes with that investment.
The team that replaces a reliable overnight closure with a stream that no one checks until the next day didn't gain agility. It spent infrastructure. Not every project needs the emergency vendor.
First make an inventory of your reports and classify them by decision frequency. Then identify which require data instantly and which function well with daily closures. Then don't replace what already operates with reliability without a clear business reason. Finally, before approving a migration to real-time, ask what concrete decision can't wait until end of day closure.
Which of your current reports would update to real-time without that changing any business decision? 🏗️