Data Engineer – Databricks & AWS Lakehouse Solutions
Digital
Data
Bachelor, Master
Senior (7+)
- Location
- Gent, East Flanders
- Work type
- Consultancy
- Work model
- Fulltime, Hybrid
Our client is seeking a skilled professional to join their data organization. This role involves designing and building scalable, reliable, and cost-efficient data pipelines on a governed lakehouse platform using Databricks on AWS.
Responsibilities
- Design and maintain production-ready data pipelines using medallion architecture.
- Optimize ETL/ELT processes for large-scale data processing on Delta Lake.
- Implement data governance through Unity Catalog, managing access controls and data lineage.
- Build CI/CD pipelines for automated testing and deployment of data workloads.
- Configure Databricks workspaces and compute resources for optimal performance and cost.
- Collaborate with cross-functional teams in an Agile environment to translate requirements into technical solutions.
Requirements
- You have experience with Python, PySpark, and SQL for data engineering.
- You possess in-depth knowledge of Databricks, Delta Lake, and Unity Catalog.
- You bring experience with AWS services including S3, IAM, KMS, and VPC.
- You're proficient in CI/CD and Infrastructure as Code using GitHub Actions, Terraform, and Databricks Asset Bundles.
- You have knowledge of medallion architecture, ETL/ELT patterns, and CDC.
- You possess the ability to work effectively in an Agile/Scrum environment.
- You are fluent in English and preferably Dutch as well
Nice to Haves
- Experience with data quality frameworks such as Great Expectations.
- Familiarity with Lakehouse monitoring or CloudWatch dashboards.
Offer
- Start date: ASAP - long term engagement
- Location: Gent/Zwijnaarde - hybrid (minimum 2 days/week onsite)
- Contract: open to both permanent employees and freelancers
)