ETL monitoring and management solution increases visibility by 87% and productivity by 43%

THE CHALLENGE

At the time, our banking customer had no central ETL monitoring and management solution for their data warehouse. They did not have visibility into the jobs that were scheduled or underway or which jobs had failed. They had no method for automatically reinitiating jobs that had failed. They were also unable to pinpoint the reason a job failed.

 They needed a centralized dashboard to handle their business side as well as the ability to monitor and manage hundreds of attributes around the various jobs, including job status, job flows, errors and failures, successful completions, failure reports, ticket handling, and more. Our customer also needed the ability to retrigger failed jobs on an ad-hoc basis along with process lineage visibility.

THE SOLUTION

Beyondsoft created a centralized ETL monitoring and management solution, including multiple MVPs within just seven months. The solution enables our customer to monitor incoming jobs from several data pipelines from multiple source systems in different formats. The solution delivers insights on job status including run time, success or failure status, reason of failure, and other key measures.

Beyondsoft also created a tool to automatically handle job failures. When a job fails, the tool creates a failure report with in-depth details including who initiated the job, how long the job had been running, the reason for failure, and other details. The solution then automatically generates a ticket that is escalated to the appropriate party.

ETL monitoring and management solution architecture overview
ETL monitoring and management solution architecture
THE RESULTS

The ETL monitoring and management solution provides our customer with a central dashboard for monitoring as well as in-depth visibility and control through error detection, error enrichment, event triggering services, unified monitoring service, unified metadata, SLA impact, and recovery. Our customer also benefits from enhancements in their existing components like Airflow, Job Server, Ingestion, Computes, and DaaS. The solution from Beyondsoft delivered the following benefits:

  • Increased productivity by 43%.
  • Reduced losses from job failures by 76%.
  • Cut required headcount in half, from 16 to 8.
  • Increased observability of the entire process by 87%.
TECHNOLOGIES USED

Java 1.8, Spring Boot, Spring Cloud, Spring MVP, Spring Data, Spring Security, Kafka, ELK, Filebeat, Healthbeat, Kubernetes, Docker, Jenkins, React js, and Kibana.

How we do it

Our success factors over the years are a testament to driving your return on investment. Singapore is our global head office and we have 15 regional offices around the world.

Nearly 3 decades of strong IT consulting and services

Global presence across four continents

Certifications* in CMMI 5, ISO 9001, ISO 45001, and ISO 27001

~30,000 global experts

Microsoft Azure Expert MSP

ISO 9001 and 45001 (certificates issued to Beyondsoft International (Singapore) Pte Ltd). ISO 27001 (certificates issued to Beyondsoft International (Singapore) Pte Ltd, Beyondsoft (Malaysia) Sdn. Bhd., and Beyondsoft Consulting Inc., Bellevue, WA, USA)