We are looking for a Senior Data Engineer - AI & Analytics with strong Data Engineering expertise, analytical thinking, and AI enablement capabilities to build scalable data solutions that power analytics, dashboards, recommendations, and AI-driven use cases. The role involves designing and evolving data products within a modern Azure + Databricks Lakehouse architecture , enabling business insights and AI solutions through curated, consumption-ready datasets. The ideal candidate will own the end-to-end data lifecycle and work closely with business, product, and engineering teams to deliver scalable and maintainable solutions. Qualification BE / B.Tech / MCA / ME / M.Tech 7+ years of experience in Data Engineering / Analytics Engineering Key Responsibilities Design, develop, and optimize scalable data pipelines using Databricks and Azure data services Integrate internal/external data sources and build reusable, modular data components Develop curated datasets and data products for analytics, dashboards, recommendations, and AI applications Design batch and streaming solutions; optimize Spark workloads, Delta tables, and low-latency data processing Drive data quality, governance, reliability, root-cause analysis, and exploratory data analysis (EDA) Collaborate with stakeholders to define KPIs, business metrics, and analytics-ready datasets Prepare and structure datasets for AI Agents / GenAI and support Azure AI Foundry integration patterns Design semantic and metadata-driven datasets and enable downstream AI and BI consumption Support Qlik / BI performance optimization and ensure consistency of business metrics Implement testing, CI/CD, version control, and engineering best practices using Azure DevOps Participate in agile delivery including planning, estimation, releases, and cross-functional collaboration Required Skills Data Engineering & Platform Spark 3.x (DataFrames, SQL, Batch & Structured Streaming) Databricks (Workflows, SQL Warehouses, DLT, Unity Catalog, Auto Loader, Pipelines) Azure Data Services and Lakehouse / Medallion Architecture Parquet / Delta, partitioning, compaction, and performance optimization Programming & Analytics Strong Python and SQL (Spark SQL, TSQL, HiveQL) Data quality, EDA, KPI-driven analytical modeling Understanding of statistical concepts and data readiness for analytics/recommendation use cases Experience building reusable, analytics-ready, and AI-ready datasets AI, BI & Delivery Azure AI Foundry integration and AI/Agent data preparation Experience supporting Qlik / Power BI / Tableau workloads Testing frameworks ( pytest, Great Expectations, Acceptance Testing ) CI/CD with Azure DevOps and YAML pipelines Agile/Scrum development practices Good to Know ADLS, Managed Identity, Azure AI Foundry Feature engineering concepts Airflow / ADF / Synapse Pipelines Scala or Java Data Catalogs (Purview, Unity Catalog, Apache Atlas) Healthcare domain experience (preferred)
