.webp)
Senior Data Engineer
Senior
India
Full Time
CET Time Zone
A Senior Azure Data Engineer will lead the design, development, and optimization of scalable data pipelines and architectures on Azure. This role involves leveraging Azure Data Factory (ADF), Databricks, PySpark, Scala, and Python to drive efficient data ingestion, transformation, and orchestration. Additionally, the engineer will ensure secure, high-performance solutions aligned with best practices for AI-driven and analytics projects. They will also mentor junior engineers, oversee a small team, and implement robust security and compliance measures within the Azure ecosystem.
Join us to explore new possibilities with data and help shape our future with meaningful discoveries.
Responsibilities
- Develop basic data pipelines using Azure Data Factory, Azure Synapse Analytics, or Azure Databricks.
- Assist in ingesting structured/semi-structured data from sources (e.g., APIs, databases, files) into Azure Data Lake Storage (ADLS).
- Write simple SQL queries and scripts for data transformation and validation.
- Write simple Pyspark, scala and python code if required
- Monitor pipeline performance and troubleshoot basic issues.
- Collaborate with AI/ML teams to prepare datasets for model training.
- Document workflows and adhere to data governance standards.
- Architect end-to-end data solutions (lakehouses, medallion architectures) using Azure Purview, Fabric, and Delta Lake.
- Design cost-effective, scalable infrastructure for large-scale AI training/inference.
- Mentor junior/mid-level engineers and define best practices.
- Partner with AI teams to operationalize GenAI pipelines(fine-tuning, RAG architectures).
- Implement advanced security (private endpoints, managed identities, encryption).
- Optimize costs via resource scaling (serverless, auto-pause) and workload management.
- Evaluate emerging Azure services (e.g., Fabric, AI-driven analytics).
Education
- 6+ years of Azure data engineering experience.
- Mastery of Azure services (Synapse, Databricks, Event Hubs, Cosmos DB).
- Expertise in distributed computing, real-time analytics, and MLOps.
- Proven ability to lead projects and communicate with executives.
- Strong knowledge of AI/GenAI use cases and ethical data practices.
Does this sound like you?
