Job Brief: We are seeking a highly skilled and motivated Data Engineer to join our team. As a Data Engineer, you will play a critical role in designing, developing, and maintaining our data infrastructure, ensuring the availability, scalability, and reliability of our data systems. You will collaborate closely with cross-functional teams, including data scientists, analysts, and software engineers, to support data-driven decision-making and enable the efficient processing of large-scale data. VentureDive Overview: Founded in 2012 by veteran technology entrepreneurs from MIT and Stanford, VentureDive is the fastest growing technology company in the region that develops and invests in products and solutions that simplify and improve the lives of people world-wide. We aspire to create a technology organization and an entrepreneurial ecosystem in the region that is recognized as second to none in the world. Key Responsibilities:
Design and develop robust, scalable, and high-performance data pipelines and car processes to extract, transform, and load data from various sources into our data warehouse or data lake.
Collaborate with stakeholders to understand their data requirements, and design and implement appropriate data models and database schemas to support their needs.
Optimize data pipelines and ETL processes for performance and efficiency, ensuring timely and accurate data delivery to end-users.
Monitor, troubleshoot, and resolve issues related to data quality, data consistency, and data integrity, ensuring the reliability and correctness of our data systems.
Implement and maintain data governance practices and policies, ensuring compliance with data privacy and security regulations.
Collaborate with data scientists and analysts to provide them with the necessary data infrastructure and tools for conducting advanced analytics and deriving insights.
Stay up to date with the latest trends and technologies in data engineering and recommend innovative solutions to improve data engineering processes and systems.
Document data engineering processes, data flows, and system architectures to ensure knowledge sharing and maintain an up-to-date repository of technical documentation.
Work closely with cross-functional teams, including software engineers and infrastructure teams, to optimize data infrastructure and ensure its seamless integration with other systems.
Qualification Requirements:
Bachelor’s degree in computer science, Engineering, or a related field.
Proven experience as a Data Engineer or in a similar role, working with large-scale data processing and ETL pipelines.
Strong programming skills in languages such as Python, Java, or Scala, with experience in data manipulation and processing frameworks like Apache Spark.
Experience with SQL and database technologies (e.g., relational databases, SQL queries, data modeling).
Proficiency in working with big data technologies such as Hadoop, Hive and knowledge of distributed systems and cloud computing platforms (e.g., AWS, Azure, GCP).
Familiarity with data integration and workflow management tools such as Apache Airflow
Knowledge of data warehousing concepts and experience with data warehousing solutions is highly desirable.
Strong analytical and problem-solving skills, with the ability to analyze complex data-related issues and propose effective solutions.
Excellent communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.
Attention to detail and a strong commitment to delivering high-quality work within established timelines.
In order to thrive at VentureDive, you …are intellectually smart and curious …have the passion for and take pride in your work …deeply believe in VentureDive’s mission, vision, and values …have a no-frills attitude …are a collaborative team player …are ethical and honest Are you ready to put your ideas into products and solutions that will be used by millions? You will find VentureDive to be a quick pace, high standards, fun and a rewarding place to work at. Not only will your work reach millions of users world-wide, you will also be rewarded with competitive salaries and benefits. If you think you have what it takes to be a VenDian, come join us ... we're having a ball!
