Job description
Role Overview We are looking for a Data Engineer to support and enhance the CS-AT platform, which processes large-scale machine log data (15–20 years) to enable predictive and proactive maintenance solutions. The role involves building and optimizing data pipelines, log processing systems, and analytics platforms on Azure. Key Responsibilities • Design, build, and maintain scalable data pipelines • Handle ingestion and processing of large-scale log data • Monitor and optimize pipeline performance and reliability • Work on log parsing and pattern extraction • Collaborate with data scientists on predictive and proactive maintenance use cases • Develop and optimize queries using KQL (Kusto Query Language) • Build and support dashboards using Azure Data Explorer • Ensure data quality, monitoring, and operational stability • Support integration with downstream systems such as OneAI and MI Log Interpreter Required Skills Core Skills • Strong experience in data engineering (5–6 years) • Expertise in writing KQL • Expertise in building data pipelines and ETL/ELT processes • Experience with log data processing and analysis • Python and C# language skills Azure Technologies • Azure Functions • Azure Event Grid • Azure Data Explorer (Kusto)+KQL • Azure Data Factory • Databricks/Spark • Delta Lake Data & Engineering Skills • Pipeline monitoring and optimization • Performance tuning of large data systems • Handling high-volume historical datasets • Experience in distributed data processing • Documentation according to CS standards Good to Have • Experience with predictive maintenance use cases • Exposure to machine logs / IoT / telemetry data • Understanding of data science workflows Soft Skills • Strong problem-solving ability • Data Product Ownership mindset (Development + Operations) • Ability to work in cross-functional teams • Proactive approach to optimization and operations Team Structure Part of a (4 – 6) member cross-functional team including Development and Operations. Key Success Metrics • Stable and optimized data pipelines • Improved processing efficiency • Reliable log ingestion and analysis • Smooth integration with AI systems
This job post has been translated by AI and may contain minor differences or errors.
