Job description
This role is for one of the Weekday's clients Min Experience: 3+ years Location: Bengaluru JobType: full-time We are looking for a highly skilled Big Data Engineer with 3+ years of experience in building scalable data pipelines and distributed systems.
The ideal candidate will have strong expertise in Apache Spark (Scala), experience working across on-premise and AWS environments, and a solid understanding of large-scale data processing in AdTech ecosystems.
This role involves working on high-volume datasets (billions of records), optimizing distributed jobs, and contributing to the design of robust data infrastructure powering analytics and identity-driven use cases.
Key Responsibilities: Design, develop, and optimize large-scale batch data pipelines using Apache Spark (Scala) Process and transform high-volume datasets (TBs of data) in distributed environments Build and maintain data pipelines across hybrid infrastructure (On-Prem + AWS) Work with object storage systems such as S3 and MinIO for efficient data access and storage Develop reusable and scalable data processing frameworks Optimize Spark jobs for performance (memory tuning, partitioning, shuffling, etc.
) Manage and orchestrate workloads using HashiCorp Nomad Integrate data pipelines with PostgreSQL and other downstream systems Ensure data quality, consistency, and reliability across pipelines Troubleshoot production issues and perform root cause analysis Contribute to system design discussions, especially for high-scale AdTech use cases (identity resolution, user profiling, etc.
) Required Skills: Strong programming experience in Scala Good working knowledge of Python (for auxiliary tasks, scripting, or ML integration) Deep expertise in Apache Spark (Core and SQL) Strong understanding of distributed data processing and large-scale systems Experience working with AWS (S3, EMR or equivalent ecosystem) and on-prem clusters Hands-on experience with object storage systems (S3 / MinIO) Experience with HashiCorp Nomad or similar orchestration tools Solid understanding of data modeling and ETL pipeline design Experience working with PostgreSQL or similar relational databases Strong debugging and performance tuning skills for Spark jobs Familiarity with Unix/Linux environments and shell scripting Good to Have: Experience in AdTech, Identity Graph, or User Profiling systems Exposure to machine learning pipelines or feature engineering workflows Experience with data lake architectures Understanding of cost optimization and resource management in AWS Tech Stack Summary: Languages: Scala (Primary), Python (Secondary) Processing: Apache Spark (Core and SQL) Infrastructure: AWS and On-Prem Storage: S3, MinIO Orchestration: HashiCorp Nomad Database: PostgreSQL Must-have skills Spark, SQL, Scala Good-to-have skills Python, AWS, Big Data
This job post has been translated by AI and may contain minor differences or errors.
