Enhancing Cloud Infrastructure Management System with Automation, AI and Microservices

About the Client

Our client offers a cutting-edge software platform and methodology for data collection and analysis, providing insights into internet-connected infrastructure through publicly available data. Their platform uses advanced DevOps technology to provide corporate clients with actionable intelligence, standing out from traditional survey-based data providers.

Challenge

In 2017, our client partnered with Edvantis to develop a system that would enable non-technical managers to efficiently manage IT cloud infrastructure. At the time, the client was in the midst of transitioning their monolithic application to a microservices architecture to improve scalability and prepare the platform for future growth.

Our team stepped in to take over an unfinished project that included 50 separate APIs. It was a significant challenge to integrate and optimize these components into a cohesive, functional system and add essential features such as error handling, logging, and monitoring.

Another key challenge was optimizing data acquisition and processing. This required ensuring fast, reliable, and consistent data collection from targeted sources, implementing robust quality control measures, and presenting the data in an accessible format for seamless analysis.

Project Scope

This project aimed to optimize data acquisition and processing while ensuring adaptability, scalability, and quality control, with AI-driven solutions playing a key role in streamlining operations. The key tasks included:

• Transition from a monolithic architecture to microservices.
• Automate accounting processes for improved efficiency.
• Migrate from Kubernetes hosting to a serverless infrastructure.
• Develop and maintain data scrapers to collect information from various sources at different intervals (e.g., daily, every three days, bi-monthly).
• Design and manage databases alongside data management tools to store raw data and enable further processing.
• Implement multi-stage data processing pipelines for certain sources, incorporating quality control measures at each stage to ensure accuracy and preserve intermediate results.
• While not the primary focus, the project occasionally included tasks related to data analysis and visualization, performed on an ad hoc basis.
• Provide ongoing support for existing solutions, including resolving issues caused by source updates and identifying new data sources to enhance the dataset as necessary.
• Integrate AI technologies to automate specific processes, such as discovering new data sources, identifying correlations, making forecasts, and more.

Team Size and Composition

Edvantis provided a flexible solution with an adaptable team structure to ensure efficient task execution at every stage of the project. The team size varied between 2 and 4 experts, depending on the workload and project requirements. At the peak of development cycle, it included a Lead Software Engineer, two Software Engineers, and one QA Engineer, ensuring a well-rounded approach to achieving project goals.

Solution

By augmenting the client’s in-house team, Edvantis got the project back on track and led it to successful closure.

The client’s previous infrastructure proved difficult to manage and did not meet their evolving business needs. To deliver better results, we focused on transitioning them to a cloud-native, serverless architecture. This shift made their operations more efficient, scalable, and cost-effective, streamlining management and enabling the business to adapt quickly to future demands.

As part of the migration from monolithic to microservices, our team also helped the client improve their continuous integration/continuous deployment (CI/CD) process. To speed up delivery, we switched from Jenkins to Bitbucket Pipelines and incorporated additional integration pre-deployment testing.

The Edvantis team designed and implemented a comprehensive data monitoring system tailored to the client’s needs. Key features included:

• A fully automated system for collecting data from multiple external sources on a defined schedule (daily, weekly, or monthly).
• Rigorous data quality checks to ensure accuracy and reliability.
• An intuitive user interface for seamless data visualization.
• Ongoing functionality assessments and enhancements to maintain optimal performance.
• Gradual integration of AI-driven solutions, such as forecasting and data analysis, into specific processes.

Results

Edvantis supported the client in successfully transitioning from a monolithic system to microservices, and eventually to a serverless architecture, achieving a 60% reduction in infrastructure hosting costs. Additionally, we streamlined their data collection processes and optimized their data management system for greater efficiency and performance.

The implemented solution delivered significant and measurable results across key performance areas. By automating data collection and conducting stringent quality checks, the system ensured consistent accuracy and reliability, reducing manual errors.

The intuitive user interface empowered stakeholders to quickly access and interpret insights, streamlining decision-making processes. The introduction of the new billing solution to the 20+ end users resulted in a significant decrease in procurement time and costs.

Additionally, the integration of AI-driven capabilities provided enhanced forecasting accuracy and actionable analytics, driving more informed strategic planning. Continuous performance enhancements have ensured the system remains adaptable, scalable, and aligned with evolving organizational needs, delivering long-term value and operational efficiency.