Hadoop / PySpark

Hadoop / PySpark

Good to have skills: Spark SQL, YARN, HDFS, Oozie, Airflow Preferred Qualifications: • Experience designing reusable frameworks and standardized pipeline patterns to improve team productivity and consistency. • Strong expertise in optimizing Spark jobs (partitioning, caching, shuffles) and Hive performance (file formats, partitions, bucketing). • Experience implementing data quality checks, monitoring, and operational dashboards for production pipelines. • Ability to drive stakeholder communication, manage technical trade-offs, and lead solutioning for complex data use cases. • Demonstrated mentoring and leadership experience, enabling teams to deliver high-quality Big Data solutions at scale. Key Responsibilities: • Lead the design and development of scalable Big Data solutions using Hadoop and PySpark for batch and large-scale processing. • Architect and implement end-to-end data pipelines, ensuring reliability, performance tuning, and efficient resource utilization on Hadoop clusters. • Develop and optimize Hive data models, queries, and partitioning strategies to support analytics and downstream consumption. • Drive technical planning, estimation, and delivery for data engineering initiatives, ensuring timelines and quality standards are met. • Establish coding standards, review code, and enforce best practices for maintainability, testing, and production readiness. • Troubleshoot production issues, perform root-cause analysis, and implement preventive measures to improve stability and throughput. • Collaborate with product, analytics, and platform teams to translate requirements into scalable technical solutions. • Mentor team members, guide technical decisions, and support skill development across Hadoop, PySpark, Big Data, and Hive. Minimum Qualifications: • Education: BTECH, MTECH, MCA, MSC (or equivalent). • 5–9 years of overall experience with strong hands-on expertise in Hadoop and PySpark for large-scale data processing. • Proven experience building and maintaining Big Data pipelines and working with Hive for querying and data modeling. • Strong understanding of distributed processing concepts, performance optimization, and data reliability practices. • Experience leading technical execution through code reviews, design discussions, and delivery ownership.