“Data Engineering” in 2025

I’m a big fan of Airflow now, and can’t wait for my next opportunity to use it. The user interface is just – delicious. Airflow orchestrates python scripts. This project was copying data nightly from SQL Server into Snowflake.

This post started out as “Airflow Integration” and was going to start with setting up Airflow, then a breakdown of the python into sections, before ending with alerts, and maybe I’ll still do that post … but when I started I realised my main “take away” was … A.I.

Sure I’ve used python before but this time seemed different. The code feels 100% mine – which is odd – as I didn’t type a single word of it.

Staying in front of the amalgamation engine marching band can be tricky, and many times I found myself wheeling away from rabbit holes or just shouting ‘halt’.

“Co-pilot” is absolutely the right relationship to maintain. As soon as I started asking open questions like “well, what do you think?” it was time to call it a day.

My development experience helped. I found knowing how to get past roadblocks, frequent testing, when to be pragmatic instead of perfect, and when to step back to the last working version were – pure gold.

Ultimately, I feel vibe coding shines for rapid prototyping or as an easy entry point for non-coders. Although debugging was painful, it remains a truly fascinating new tool.