Senior Data Engineering

Senior Data Engineering

Job Description for Data Engineer – Databricks Data PlatformRole Overview

Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.

Key Responsibilities

· Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.

· Ingest data from databases, APIs, event streams, and flat files.

· Design and implement Medallion Architecture (Bronze / Silver / Gold layers).

· Write data transformation logic in PySpark, Spark SQL, and Python.

· Implement data quality checks, validation rules, and reconciliation processes.

· Optimise Spark workloads for performance, cost, and scalability.

· Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.

· Support CI/CD for pipelines using Git, DevOps tooling, and release processes.

· Monitor pipelines, triage failures, and restore services within agreed SLAs.

· Maintain technical documentation: data models, pipeline flows, runbooks.

· Adhere to data security, privacy, and regulatory compliance standards.

Must-Have Skills

· Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.

· Apache Spark: PySpark and Spark SQL for large-scale data processing.

· Delta Lake: ACID transactions, schema evolution, time travel, merge.

· Medallion Architecture or equivalent layered data platform design.

· Python for pipeline development and transformation logic.

· AWS: S3, Glue, IAM, EC2, Lambda.

· Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).

· Data modelling: star schema, snowflake schema, Data Vault.

· CI/CD, Git, version control, and Agile delivery practices.

Good-to-Have Skills

· Unity Catalog for data governance and lineage.

· AWS Kinesis for real-time streaming.

· Data Warehouse and Informatica PWC Knowledge

· dbt for SQL-based transformation workflows.

· AWS Redshift or Athena for cloud data warehousing.

· Terraform or AWS CloudFormation for infrastructure-as-code.

· AWS Step Functions for workflow orchestration.

· Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).

· Industry experience in insurance or financial services.

· Databricks or AWS Data Analytics certification.

·

Technical Environment

· Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.

· Languages: Python, PySpark, SQL, Spark SQL.

· Ingestion: Auto Loader, APIs, JDBC, CDC tools.

· Processing: Apache Spark, Structured Streaming, Delta Live Tables.

· Cloud: AWS.

· Orchestration: Databricks Workflows, AWS Step Functions.

· DevOps: Git, CI/CD, Terraform, GitHub Actions.

· Analytics: Power BI, Tableau, Looker.

Experience and Qualifications

· Bachelor's in Computer Science, Engineering, IT, or equivalent.

· 4–8 years in data engineering, ETL/ELT, and cloud data platforms.

· Production-grade pipeline delivery on AWS.

· Agile or DevOps-driven delivery environment.