Data Engineer

Job Purpose To design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie, Semantic Models, and Metric Views on top of a robust end-to-end data engineering stack.
The role focuses on enabling self-service analytics using natural language (GenAI), ensuring business-friendly data consumption, and building governed, scalable, and high-performance data ecosystems across batch and real-time pipelines.
Duties and Responsibilities KEY ROLES / PRINCIPAL ACCOUNTABILITIES AI for BI & GenAI Enablement • Design and implement GenAI-powered BI solutions using Databricks Genie or MS Fabric • Create and manage Data Rooms for business users with curated datasets • Define and maintain Instructions Layer (prompt engineering for business context) • Enable natural language to SQL/insights workflows for self-service analytics • Drive adoption of Databricks One & other platforms as a unified analytics interface Semantic Layer & Metrics Engineering • Design and manage Semantic Models for business abstraction • Develop reusable and governed Metric Views (KPIs, aggregations, business definitions) • Ensure consistency across BI tools (Power BI / Genie) • Align semantic layer with business glossary and data governance policies Data Engineering & Platform Development • Build scalable pipelines using Azure Databricks (PySpark, SQL) • Develop and orchestrate ETL workflows using Azure Data Factory • Work with Delta Lake architecture (Bronze–Silver–Gold layers) • Enable real-time and batch data processing pipelines • Enable data exposure via APIs for BI and downstream systems CI/CD & DevOps • Implement CI/CD pipelines for data and AI workflows • Automate deployments across environments (Dev, QA, Prod) • Ensure version control and reproducibility ________________________________________ ?
KEY RESPONSIBILITIES • Translate business problems into AI-driven BI solutions • Own end-to-end delivery: ingestion ?
transformation ?
semantic layer ?
AI consumption • Design scalable data architecture aligned with lakehouse principles • Ensure data quality, governance, and metric consistency • Collaborate with business teams to onboard them onto Genie-based analytics • Optimize performance of queries, pipelines, and AI responses • Establish best practices for AI in BI (prompting, semantic tuning, governance) • Drive adoption of self-service BI with minimal dependency on tech teams Key Decisions / Dimensions KEY DECISIONS / DIMENSIONS • Define semantic layer design and metric definitions • Decide GenAI prompting strategies and instruction frameworks • Prioritize data vs AI optimization trade-offs • Handle production issues with RCA and long-term fixes • Drive architectural decisions for lakehouse + AI integration Major Challenges MAJOR CHALLENGES • Ensuring accuracy and trust in AI-generated insights • Driving adoption of GenAI-based BI over traditional dashboards • Maintaining semantic consistency across multiple tools • Balancing performance, cost, and scalability • Managing dependencies across data engineering, AI, and business teams Required Qualifications and Experience REQUIRED SKILLS & EXPERIENCE Must Have • Azure Databricks – PySpark, SQL, Delta Lake • Strong experience in Semantic Modeling & Metrics Layer design • Hands-on with Databricks Genie / GenAI-based BI workflows • Databricks One (Unified BI Experience) • Prompt Engineering / Instruction tuning for AI systems • Python (Pandas, PySpark, FastAPI) • Azure Data Factory (ADF) for ETL pipelines • Strong SQL and data modeling skills Good to Have • Cosmos DB / MongoDB (NoSQL concepts) • Azure Data Explorer (KQL) ________________________________________ DATA STACK (MANDATORY FOR SCREENING) SNo Data Platform / Concepts Associated Technologies 1 Databricks Lakehouse PySpark, SQL, Delta Lake 2 AI for BI Databricks Genie, Genie Rooms, Instructions, Agents 3 Semantic Layer Semantic Models, Metric Views 4 ETL & Orchestration Azure Data Factory 5 Programming Python, C#/.
NET 6 Cloud Platform Azure (Preferred) 10 DevOps CI/CD Pipelines, Git