Speaker
Description
This paper explores the strategic implementation of the Elastic Stack (ELK) for enhancing observability and logging capabilities within the Secure Web Portal (SWP). The ELK stack, comprising Elasticsearch, Logstash, and Kibana, was integrated within SWP to create a comprehensive logging system. By connecting all application logs to a central Syslog instance, data was stored in Elasticsearch. This approach facilitated the aggregation and analysis of logs from multiple components, providing real time insights into the system’s operational status. Elasticsearch itself functions as a powerful search engine and analytics tool, using a unique data store optimized for real time indexing and full-text search. This project demonstrates how observability, achieved through the ELK stack, can significantly enhance the performance and resilience of the SWP. The ability to measure a system's internal states by looking at its outputs enabled faster resolution when issues occurred. In conclusion, applied observability, using the ELK stack, offers a significant advantage for securing and managing complex systems, aligning with the trend of using AI to analyse and optimize data use. The system was designed to monitor data flows, schedules, and dependencies, incorporating key dimensions of data observability, including freshness, quality, volume, schema and origin.
| [email protected] |
