Maxime Naulleau
1y ago
Avoid these mistakes if you are new to Airbyte, want to use it with DBT on Kubernetes, and need customer data ingestion:
1. using Airbyte's DBT integration
2. using Airflow as a workaround
Here's why... š§µ
1/5
Maxime Naulleau
1y ago
1. using Airbyte's DBT integration won't work out of the box
It happens that Airbyte's team messed up the custom DBT transformation feature. They acknowledged this both in their doc and GitHub ticket 5091.
So you need a workaround, such as a scheduler. š
2/5
Maxime Naulleau
1y ago
2. using Airflow to schedule users' DBT steps is not handy
We need to sync all users' data periodically. When a user adds a new account, we want to sync only this one. Airflow can do this but isn't designed for it.
Instead, pick a flexible tool like Argo Workflow.
3/5
Maxime Naulleau
1y ago
If you liked this thread and are interested in SaaS, customer-facing integrations, and their business outcomes:
ā¢ Follow me at @MaximeNaulleau. I publish 4x per week.
ā¢ Send a DM for help in setting up Airbyte.
4/5
Maxime Naulleau
1y ago
The TL:DR:
1. using Airbyte's DBT integration on Kubernetes won't work out of the box
2. instead of Airflow, pick a flexible tool like Argo Workflow to schedule DBT on Kubernetes
Airbyte is a great project, but it has its caveats.
5/5