We are looking for an experienced Python PySpark Developer to design, develop, and optimize large-scale data processing systems. The ideal candidate will work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python.
Key Responsibilities
Data Engineering & Development
Develop and maintain data pipelines using Python and PySpark
Process and transform large datasets in distributed environments
Build scalable ETL/ELT workflows
Big Data Processing
Work with Apache Spark (PySpark) for batch and real-time processing
Optimize Spark jobs for performance and efficiency
Handle structured and unstructured data
Data Integration
Ingest data from multiple sources:
Databases (SQL/NoSQL)
APIs
Files (CSV, JSON, Parquet)
Integrate with data platforms like:
Hadoop (HDFS)
Cloud (AWS, Azure, GCP)
Performance Optimization
Tune Spark jobs (partitioning, caching, parallelism)
Optimize SQL queries and transformations
Improve data processing efficiency and cost
Collaboration & Support
Work with data engineers, data scientists, and analysts
Translate business requirements into technical solutions
Participate in code reviews and agile development practices
Monitoring & Troubleshooting
Debug and resolve issues in data pipelines
Monitor job execution and data quality
Ensure reliability and availability of data workflows

* Primary skills:Python, Pyspark