Databricks AI & Data Engineer

Databricks AI & Data Engineer

Day to Day Job Duties: (what this person will do on a daily/weekly basis)

Design, develop, and deploy Agentic AI solutions that automate data governance, data quality, metadata management, lineage analysis, policy enforcement, and stewardship workflows across enterprise data platforms.

Build scalable AI agents leveraging Large Language Models (LLMs), Retrieval Augmented Generation (RAG), orchestration frameworks, and enterprise governance tools to support data management and regulatory compliance initiatives.

Partner with Data Governance Engineers, Data Engineers, Architects, and business stakeholders to rapidly discover governance use cases, define agent capabilities, and align solution architecture with enterprise data strategy.

Develop and maintain autonomous and human-in-the-loop agents that assist with data cataloging, business glossary management, policy compliance validation, data ownership assignment, issue remediation, and governance workflow orchestration.

Integrate AI agents with enterprise platforms including data catalogs, metadata repositories, data quality tools, MDM platforms, cloud data platforms, and governance technologies such as Ataccama, Collibra, Informatica, Microsoft Purview, Databricks, Snowflake, AWS, and Azure.

Create proof-of-value demonstrations and pilot implementations to validate governance automation opportunities and accelerate stakeholder adoption.

Collaborate with internal delivery teams and client stakeholders to continuously refine governance backlogs, prioritize agent development initiatives, and incrementally onboard governance controls across the enterprise.

Implement monitoring, observability, guardrails, and governance controls for Agentic AI solutions to ensure security, compliance, transparency, and responsible AI practices.

Support proposal development, solution architecture discussions, effort estimation, and client presentations related to Data & AI governance modernization initiatives.

Contribute to reusable frameworks, accelerators, and best practices for enterprise-scale Agentic AI implementations focused on data governance and management.

Basic Qualifications: (what are the skills required to this job with minimum years of experience on each)

Minimum 8+ years of experience in Data Management, Data Engineering, Data Governance, Metadata Management, or Enterprise Data Architecture.

Minimum 5+ years of experience designing and implementing enterprise data governance, master data management (MDM), metadata management, data quality, or regulatory compliance solutions.

Minimum 3+ years of experience developing AI/ML, Generative AI, or Agentic AI solutions using LLMs and orchestration frameworks.

Minimum 3+ years of hands-on experience with Python and modern AI development frameworks such as LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, OpenAI, Azure AI Foundry, Amazon Bedrock, or equivalent platforms.

Minimum 3+ years of experience working with cloud platforms including Azure, AWS, or Google Cloud Platform.

Minimum 2+ years of experience integrating governance platforms such as Collibra, Databricks Unity Catalog, or related data management technologies.

Minimum 2+ years of experience designing APIs, workflow orchestration, event-driven architectures, or enterprise system integrations.

Experience defining governance policies, lineage workflows, data quality controls, stewardship processes, and compliance monitoring capabilities.

Experience working in Agile delivery environments with backlog-driven development, iterative releases, and stakeholder feedback cycles.

Strong client-facing communication skills with the ability to lead workshops, requirements gathering sessions, architecture reviews, and executive demonstrations.