Job description
We are looking for a Senior Data Engineer - AI & Analytics with strong Data Engineering expertise, analytical thinking, and AI enablement capabilities to build scalable data solutions that power analytics, dashboards, recommendations, and AI-driven use cases. The role involves designing and evolving data products within a modern Azure + Databricks Lakehouse architecture , enabling business insights and AI solutions through curated, consumption-ready datasets. The ideal candidate will own the end-to-end data lifecycle and work closely with business, product, and engineering teams to deliver scalable and maintainable solutions. Qualification BE / B.Tech / MCA / ME / M.Tech 7+ years of experience in Data Engineering / Analytics Engineering Key Responsibilities Design, develop, and optimize scalable data pipelines using Databricks and Azure data services Integrate internal/external data sources and build reusable, modular data components Develop curated datasets and data products for analytics, dashboards, recommendations, and AI applications Design batch and streaming solutions; optimize Spark workloads, Delta tables, and low-latency data processing Drive data quality, governance, reliability, root-cause analysis, and exploratory data analysis (EDA) Collaborate with stakeholders to define KPIs, business metrics, and analytics-ready datasets Prepare and structure datasets for AI Agents / GenAI and support Azure AI Foundry integration patterns Design semantic and metadata-driven datasets and enable downstream AI and BI consumption Support Qlik / BI performance optimization and ensure consistency of business metrics Implement testing, CI/CD, version control, and engineering best practices using Azure DevOps Participate in agile delivery including planning, estimation, releases, and cross-functional collaboration Required Skills Data Engineering & Platform Spark 3.x (DataFrames, SQL, Batch & Structured Streaming) Databricks (Workflows, SQL Warehouses, DLT, Unity Catalog, Auto Loader, Pipelines) Azure Data Services and Lakehouse / Medallion Architecture Parquet / Delta, partitioning, compaction, and performance optimization Programming & Analytics Strong Python and SQL (Spark SQL, TSQL, HiveQL) Data quality, EDA, KPI-driven analytical modeling Understanding of statistical concepts and data readiness for analytics/recommendation use cases Experience building reusable, analytics-ready, and AI-ready datasets AI, BI & Delivery Azure AI Foundry integration and AI/Agent data preparation Experience supporting Qlik / Power BI / Tableau workloads Testing frameworks ( pytest, Great Expectations, Acceptance Testing ) CI/CD with Azure DevOps and YAML pipelines Agile/Scrum development practices Good to Know ADLS, Managed Identity, Azure AI Foundry Feature engineering concepts Airflow / ADF / Synapse Pipelines Scala or Java Data Catalogs (Purview, Unity Catalog, Apache Atlas) Healthcare domain experience (preferred)
This job post has been translated by AI and may contain minor differences or errors.
