Absolutely 100% of successful data engineering projects I've ever seen share 1 thing:

Leverage a datalake

Follow this simple 6 steps to build yours: 🧵

What's a datalake and why?

A datalake is just a place where you store all data that your system sees or generates

Why? Because it is easy, cheap and it serves a myriad use cases, from disaster recovery to experimentation by replaying history

Let's see the how:

Step 1: Where

You will store A LOT of data, and you want to make sure it never gets deleted so two requirements here:

Managed services such as Amazon S3, Google GCS are excellent choices

Step 2: Add an envelope

You'll store heterogeneous records, how will you make sense of them?

Metadata. Which attributes define your records: arrival time, source, entity, action,...
Define a container for your record with the record + above attributes

That's the envelope

Step 3: Define partitioning

Having all compressed files together is fine, but for better reading you want to organise them.

This depends on use cases, but minimum by datetime. Think of it as a directory tree: year/month/day/hour/...

Next a real component:

Step 4: The firehose

Queue-like component to:

Receive & buffer records from sources in your system
When buffer full or timeout: compress and drop the compressed file to the right partition

Depending on where (step 1), you have managed services too (Firehose, Dataflow,...)

Step 5: Connect the sources

Time to get your system's sources send records to the firehose and wait for the compressed files to appear on your datalake!

IMPORTANT: All messages sent need to comply with the envelope (step 2)!

Step 6: Monitoring

Monitoring are your eyes into the system.

This system has very little complexity so minimum of:

Alarms (when no data received in X time?)

Recap:

6 repeatable steps for building a successful datalake:

1) Pick a destination

2) Define metadata

3) Define partitioning

4) Plug the firehose

5) Connect the sources

6) Monitor and alarm

6 simple steps for a successful datalake

Absolutely 100% of successful data engineering projects I've ever seen share 1 thing: Lev