Case Study: Data Engineering for a Major Financial Services Firm
In 2024, I joined Needles, providing data engineering expertise to a major New Zealand financial services firm. Like many organisations in this sector, the client’s data landscape was a blend of modern cloud tools and legacy systems, tied together by intricate business logic. My role was to help them modernise their platform while ensuring continuity for critical operations.
The work was an even split between refactoring existing pipelines and building new, cloud-native components. On the refactoring side, I improved fragile or inefficient processes, streamlining workflows without disrupting downstream consumers. On the greenfield side, I designed and deployed entirely new ETL pipelines in AWS, using Athena, RDS, S3, Airflow (MWAA), dbt, Python, and Terraform. This combination of skills allowed me to deliver both stability and innovation — improving the reliability of the old while accelerating the adoption of the new.
A major focus was data warehouse design and architecture. I worked closely with stakeholders — split roughly 50:50 between other engineers and business analysts — to ensure data models were not only technically sound but also usable by non-technical teams. In financial services, that means navigating a thicket of regulations, reporting requirements, and legacy conventions. By engaging directly with analysts, I was able to capture nuanced requirements and embed them in the architecture.
To support ongoing delivery, I introduced CI/CD pipelines in GitHub Actions and improved local developer environments. These changes reduced onboarding friction, improved reproducibility, and helped the client’s internal team deploy changes with more confidence.
The result was a more robust and flexible data platform — one that could accommodate the complex realities of financial services data while giving the client a clearer path forward to modernisation. This project showcased my ability to bridge old and new: stabilising legacy systems while laying the groundwork for future innovation.