Job description
Valleysoft is a leading IT services provider offering innovative solutions to clients worldwide.
As a Data Engineer at Valleysoft, you will be responsible for designing, building, and managing scalable data pipelines and architectures that enable data-driven decision making.
You will collaborate with data scientists, analysts, and business stakeholders to ensure the availability, reliability, and quality of data.
Key responsibilities include: Develop, construct, test, and maintain data architectures such as databases and large-scale processing systems.
Ensure data quality, integrity, and security throughout the data lifecycle.
Design and implement ETL/ELT processes to gather and prepare data for analysis.
Collaborate with cross-functional teams to understand data needs and deliver solutions.
Monitor and optimize the performance of data systems and pipelines.
Private Health Insurance Training & Development Provide second and third-line application support for the enterprise data platform, covering data warehouses, data marts, ETL pipelines, and data integration layers across production and non-production environments.
Monitor, triage, and resolve incidents related to ETL job failures, data pipeline breaks, data quality anomalies, and performance degradation — ensuring SLA adherence and minimal business disruption.
Investigate root causes of recurring data issues, pipeline failures, and feed delays; implement permanent fixes and preventive measures through proper change control channels.
Perform day-to-day operational tasks including job reruns, data corrections, parameter adjustments, and table refreshes in coordination with data owners and business stakeholders.
Collaborate with data engineering, DBA, middleware, and infrastructure teams to resolve cross-system issues affecting data availability and pipeline integrity.
Maintain and update ETL code, mappings, workflows, and configuration files in response to source system changes, schema evolution, or business rule updates.
Support onboarding of new data feeds and sources by assisting in integration testing, environment validation, and go-live stabilization.
Maintain accurate and up-to-date support documentation: runbooks, known error records, escalation procedures, and post-incident reports.
Participate in change advisory board (CAB) reviews for data platform changes, providing impact assessments and rollback plans.
Proactively identify opportunities to improve platform reliability, automate repetitive operational tasks, and reduce manual intervention in pipeline management.
Technical Requirements Hands-on experience supporting ETL platforms such as Informatica PowerCenter, IBM DataStage, Talend, or equivalent enterprise integration tools.
Strong SQL skills (Oracle, SQL Server, or DB2) — capable of writing diagnostic queries, tracing data lineage manually, and performing targeted data fixes under change control.
Solid understanding of data warehouse architecture : ODS, staging layers, data marts, and EDW — sufficient to trace and isolate issues across pipeline stages.
Experience with enterprise job scheduling tools such as Control-M, TWS, or Autosys — managing job dependencies, calendars, and failure alerts.
Familiarity with Unix/Linux shell scripting for log analysis, file transfer monitoring, and lightweight automation of support tasks.
Understanding of database performance concepts : execution plans, index usage, partition pruning, and statistics — to diagnose slow-running jobs and queries.
Exposure to data quality monitoring frameworks and the ability to validate pipeline outputs against expected row counts, aggregates, and business rules.
Experience with ITSM platforms (ServiceNow, Remedy, or equivalent) for incident logging, change requests, and problem management workflows.
Familiarity with cloud data platform components (Azure Data Factory, AWS Glue, Snowflake) is a plus.
Banking Sector Requirements (Preferred) Prior experience supporting data platforms in a bank or financial institution, with exposure to time-sensitive regulatory and financial reporting pipelines.
Familiarity with banking source systems feeding the data warehouse — core banking, GL, loan origination, cards, and payment systems.
Understanding of data reconciliation and balancing requirements: ensuring ETL outputs align with source system totals and downstream report figures.
Experience supporting pipelines that feed regulatory reports (CBE submissions, IFRS 9 staging, AML transaction monitoring) where data accuracy and timeliness are non-negotiable.
Awareness of data classification and access control requirements for sensitive banking data, including PII handling and audit trail obligations.
Experience operating within ITIL-aligned support models with strict change control, incident prioritization, and escalation procedures applicable to regulated environments.
This job post has been translated by AI and may contain minor differences or errors.
