Our team is experiencing some growth pains as we have doubled in size in each of the last 2 years. As such we have been required to deliver insights fast, no matter the cost. What the cost is? Tech Debt! A lot of it. The last 2 hires who joined in 2023 got to feel all of the pain. To the question of where can we find the documentation of the data flows we had to shrug our shoulders. With that being said it is now time to clean up and hop on the Analytics Engineering train.

We use dbt in combination with Google bigquery which makes it very easy for data analysts to model their own data to their needs. This allows for fast delivery of many custom data needs. Analysts in my team can proudly refer to themselves as end-to-end analysts covering everything from the arrival of raw data in our DWH to finished, consumable and actionable dashboards and reports. As we have grown quite a bit since we introduced dbt, a lot of the agreed upon practices have been stretched.

The clean and clear structures we wanted to follow are gone. The number of CTEs in some models has reached double digits, the redundancies are uncountable and the documentation is lacking. It is nearly impossible to onboard new analysts because it is so vague of what the expectation is in terms of data modeling.

With that being said it is time to evaluate the status quo and tear it up. I have been tasked to come up with an Analytics Engineering strategy including documentation, data flows, testing and future role definition. I will consult the dbt community to start out with and find a strategy that works for us.

I hope I will be able to share updates soon.