Singapore-HQ financial services group

Business challenge

Our client is a financial services conglomerate headquartered in Singapore, with presence across multiple countries in Asia.

Lack of observability into the ETL pipeline, and manual job management led to increased losses due to failure of critical jobs. Our client wanted to build an automated ETL monitoring and management solution.

  • A centralised dashboard to track all jobs, and gain insight into 100+ attributes.
  • Built-in workflows to generate failure reports and successfully completed jobs.
  • Capabilities to automate job retries and escalate issues when a job failure is detected.


Beyondsoft’s data engineering team iterated through multiple MVPs and built a centralised ETL observability tool within half a year. The solution uses Apache Kafka to stream events from multiple services constituting the ETL pipeline, and deploys a lookup mechanism to validate errors. Errors and exceptions are escalated to iChamp ticketing tool and the concerned SREs and stakeholders.

  • Centralised ETL monitoring tool implements a dashboard to visually monitor jobs from multiple pipelines.
  • Automated triaging generates failure reports which offer insights into the root cause of failure, along with other key attributes.
  • The solution schedules retries for failed jobs, generates a ticket and escalates to SREs when the retry routine fails.

Project outcome metrics

Reduced losses associated with job failure by 76%
Increased productivity
Increased observability
ETL management now requires half of previous headcount

Summary of the services used in the project

  • Data Engineering 
  • Apache Kafka for Event Streaming 
  • ETL Dashboard 
  • Automated Defect Triaging

Build reliability into your data pipelines with observability and automated triaging.

Get in touch to discuss how superlative IT can transform your business.

Insights & resources

Beyondsoft is a leading global provider of IT solutions par excellence. Explore our insights on the transformative power of IT.   

Global IT provider Beyondsoft expands footprint into the UK
Demystifying data migrations: Understanding the lift and shift.
A better CX starts with automated call centre operations.
6 tactics to engage B2B consumers online.
Accelerate cloud migration and modernisation.
Optimise your contact centre knowledge base to drive efficiency.
Discover how AI is transforming retail mortgage.
How a data mesh architecture can help your organisation.
The SaaS economy: Reduce costs and maximize your cloud investments.

Read more on how we’ve helped businesses with their IT transformation

A global telecommunications major
AI customer care executive reducing tickets by half

Read now

A large media company in North America
Beyond compliance: building consumer trust and operationalising privacy

Read now

Public Agency – UK
Consulting, advisory and implementation of Microsoft D365 to meet their business goals as part of digital transformation journey to improve public engagement

Read now

A global automotive manufacturer
Modernising monolithic applications and migrating to AWS

Read now

A global cloud computing and virtualisation technology company
Enabling workplace risk mitigation and business continuity during pandemic with PowerBI

Read now

US-based Ed-Tech major
Empowering cloud-native development with cloud, modern development tooling and DevSecOps

Read now

Quality engineering for a leading insurer
Improving data visualisation and BI for a leading reinsurance company

Read now

Large global insurance and asset management company
Modernising core insurance applications

Read now

Leading global P & C insurer
Modernising data platform for improving branch performance

Read now

Developing your future

With a focus on precision engineering, we’re dedicated to providing state-of-the-art technology that is tailored to meet the specific needs of our clients. Trust Beyondsoft’s precision-engineered IT to take your business to the next level and to thrive in today’s fast-paced digital landscape.

Are you ready to go beyond?

Contact us today.

    cf7captchaRegenerate Captcha

    Share this study