Senior Consultant – AI and Data, AWS Databricks

Job Description

Engage in the development, optimization, and deployment of scalable ETL/ELT pipelines utilizing PySpark, SQL, and AWS services. Gain expertise in medallion architecture using Databricks lakehouse. Acquire strong knowledge in Databricks end-to-end platform management (Unity Catalog, Users/Groups, security, etc.), Databricks integrations (AWS/Azure/GCP.), DevOps – CI/CD pipelines (for notebooks, mlflows, etc). Build and manage data lakehouse solutions leveraging AWS S3, Glue, Iceberg, and other AWS-native components. Migrate on-premises ETL workloads to modern AWS-based architectures with a focus on performance, reliability, and cost efficiency. Implement metadata-driven ingestion frameworks and medallion/layered architecture (Bronze/Silver/Gold). Orchestrate frameworks such as Astronomer (Airflow), AWS Step Functions, or AWS-managed workflows. Apply data warehouse (DW) concepts and best practices for modeling and data integration. Collaborate with data analysts, data scientists, and BI engineers for seamless data delivery. Perform code reviews, troubleshoot issues, and ensure end-to-end data quality.

Qualifications

1. Possess 8+ years of total IT experience with at least 2+ years in AWS-based data engineering. 2. Demonstrate strong hands-on experience in Databricks Lakehouse Architecture. 3. Proficiency in PySpark, Scala, SQL, Python ETL pipeline development. 4. Familiarity with AWS services: S3, Glue, Lambda, Step Functions, CloudWatch. 5. Knowledge of Unix/Linux environments and Iceberg table format. 6. Experience with Astronomer (Airflow) or other orchestration tools. 7. Experience with structured and semi-structured data formats such as Parquet, JSON, CSV, XML. 8. Understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling. 9. Good knowledge of CI/CD, version control (GitHub, Azure DevOps, Jenkins). 10. Strong analytical, troubleshooting, and problem-solving skills. 11. Ability to work independently and collaborate with stakeholders to deliver high quality solutions.

Benefits

- Opportunities to work on diverse, meaningful, and industry-leading projects. - Coaching, learning programs, and a personalized growth and development plan. - Exposure to a collaborative, interdisciplinary work culture. - Flexibility to manage your work in the way that suits you best. - Supportive colleagues and a global environment for continuous knowledge exchange.