Singapore-HQ financial services group

Our client is a financial services conglomerate headquartered in Singapore, with presence across multiple countries in Asia.

Lack of observability into the ETL pipeline, and manual job management led to increased losses due to failure of critical jobs. Our client wanted to build an automated ETL monitoring and management solution.

• A centralised dashboard to track all jobs, and gain insight into 100+ attributes.
• Built-in workflows to generate failure reports and successfully completed jobs.
• Capabilities to automate job retries and escalate issues when a job failure is detected.

Solution

Beyondsoft’s data engineering team iterated through multiple MVPs and built a centralised ETL observability tool within half a year. The solution uses Apache Kafka to stream events from multiple services constituting the ETL pipeline, and deploys a lookup mechanism to validate errors. Errors and exceptions are escalated to iChamp ticketing tool and the concerned SREs and stakeholders.

• Centralised ETL monitoring tool implements a dashboard to visually monitor jobs from multiple pipelines.
• Automated triaging generates failure reports which offer insights into the root cause of failure, along with other key attributes.
• The solution schedules retries for failed jobs, generates a ticket and escalates to SREs when the retry routine fails.