Associate Principal Engineer, Azure Data Factory Architect

Job Description

We're looking for an Associate Principal Engineer to architect and build scalable data integration solutions using Azure Data Factory. You will design, develop, and maintain ADF pipelines, triggers, datasets, and linked services. You'll also implement ETL / ELT workflows for batch and near real-time data processing, develop and optimize ADF Mapping Data Flows, and integrate ADF with Azure services such as Azure SQL, Synapse, Data Lake, and Blob Storage. Performance tuning, monitoring, and cost optimization of pipelines will be crucial, along with implementing and managing CI/CD pipelines for ADF using Azure DevOps. Troubleshooting production issues, performing root cause analysis, defining architecture standards, mentoring data engineers, and ensuring data reliability are all part of the role. Collaboration with stakeholders to understand data integration requirements is also essential.

Qualifications

1. Requires a total experience of 9 years or more. 2. Must have strong experience in Azure Data Engineering / Data Integration. 3. Deep hands-on expertise in Azure Data Factory is essential. 4. A solid understanding of ETL / ELT concepts is a must. 5. Experience with ADF pipelines, triggers, datasets, and linked services is required. 6. Hands-on experience with Mapping Data Flows and Integration Runtimes (Self-hosted & Azure) is needed. 7. Must have strong experience with Azure Data Lake Storage (ADLS Gen2). 8. Experience integrating on-premise and cloud data sources is important. 9. Strong SQL skills (Azure SQL / Synapse) are necessary. 10. Experience with Azure Synapse Analytics is expected. 11. Familiarity with CI/CD pipelines using Azure DevOps is needed. 12. Understanding of data orchestration, scheduling, and dependency management is required. 13. Experience with monitoring, logging, and error handling in ADF is important. 14. A good understanding of performance tuning and cost optimization is expected. 15. Strong problem-solving and communication skills are a must. 16. Java (Strong) is a plus.