Azure Databricks Developer

Area(s) of responsibility

Key Responsibilities
1. Data Solution Design & Architecture
•    Design end to end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.
•    Architect and implement scalable and secure Modern Data Warehouse (MDW) and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.
•    Develop data models, integration patterns, and reusable frameworks aligned with best practices and enterprise architecture standards.
•    Participate in requirement discussions, solution blueprinting, and technical feasibility assessments.
2. Data Pipeline Development
•    Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
•    Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
•    Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
•    Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
3. Databricks & PySpark Engineering
•    Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
•    Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
•    Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
•    Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.
4. Data Governance, Quality & Security
•    Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency.
•    Ensure compliance with organizational standards for data security, encryption, access control, and data lifecycle management.
•    Create and maintain technical documentation, data flow diagrams, and operational support guides.
5. Collaboration & Stakeholder Management
•    Collaborate closely with BI, analytics, and business teams to understand data requirements and deliver reliable, production-ready solutions.
•    Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
•    Provide guidance and mentoring to junior engineers when required.
________________________________________
Mandatory Skills & Experience
•    Minimum 7–10 years of experience in data engineering, with strong hands on exposure to Azure data ecosystem.
•    At least 2 years of real project experience in Azure Databricks (not POCs).
•    At least 2 years of hands-on experience building data pipelines using Azure Data Factory (ADF).
•    At least 2 years of experience developing PySpark-based transformations in Databricks.
•    Strong SQL programming experience, including writing complex queries, performance tuning, and handling large datasets.
•    Experience with Azure Data Lake Storage (ADLS Gen2), data partitioning strategies, and file formats like Parquet/Delta/JSON.
•    Knowledge of CI/CD pipelines (Azure DevOps preferred) for automated deployment of ADF/Databricks artifacts.