Ingestion
Two lanes in.
Some data cannot wait and some arrives overnight. The platform is designed with a lane for each.
-
Streaming
For data that changes by the second, such as vehicle positions. Records are handled as they arrive, a few at a time.
-
Batch
For data that arrives in files, such as timetables, fare uploads and work orders. Loaded on a schedule, all at once.
Both lanes end in the same place: the Bronze layer of the lakehouse, where records are stored exactly as received.
Nothing is cleaned or dropped on the way in, so any later step can be run again from the original.