Data Engineering in 2025

I’m a big fan of Airflow now, and can’t wait for my next opportunity to use it. The user interface is just – delicious. Airflow orchestrates python scripts. This project was copying data nightly from SQL Server into Snowflake.

This post started out as “Airflow Integration” and was going to start with setting up Airflow, then a breakdown of python into sections with explanations, before ending with alerts, and maybe I’ll still do that post … but I realised my main “take away” was … AI.

Sure I’ve used python before but this time seemed different. The code feels 100% mine – which is odd – as I didn’t type a single word of it.

I found that staying infront of the AI marching band can be tricky, and many times I found myself wheeling away from rabbit holes or just shouting ‘halt’.

My development experience helped. I found knowing how to get past roadblocks, frequent testing, when to be pragmatic, and how to use source-control, were pure gold.

Ultimately, I feel vibe coding shines for rapid prototyping or as an easy entry point for non-coders. Although debugging was painful, it remains a truly fascinating new tool.

Leave a comment