Centralised Log Management for the Public Sector with Splunk Enterprise
Key takeaways
- A public-sector organisation needed to replace its ageing observability platform while managing hundreds of gigabytes of log data daily.
- WeAre implemented a scalable, centralised log management solution using Splunk Enterprise and Amazon S3.
- The solution achieved approximately 15× log data compression, reducing long-term storage requirements while keeping logs searchable.
- The platform supports compliance requirements and can expand with new data sources, dashboards and monitoring needs.
The Challenge
Our client is a public-sector organisation that needs to monitor user activity across multiple systems as part of its daily operations. To meet legal and compliance requirements, the organisation must collect, retain and retrieve large volumes of user activity logs reliably.
The scale of the data presented a significant challenge. The environment generates hundreds of gigabytes of log data every day, with additional data sources expected to be integrated over time. This data needs to remain searchable and readily available when required for audits, investigations and other operational needs.
At the same time, the client’s existing observability solution was approaching end of life. The only replacement offered by the existing supplier was a cloud-based solution, creating a need to evaluate a more suitable long-term alternative.
The objective
The goal was to design and implement a modern observability platform and log management solution that could:
Meet legal and compliance requirements for data retention and access
Centralise log data from a growing number of sources
Enable fast search and retrieval of large datasets
Reduce long-term operating and licensing costs
Provide a scalable foundation for future monitoring and analytics use cases
The solution
We designed a new observability solution using Splunk Enterprise and Amazon S3, delivered through a Managed Service Provider (MSP) model.
The architecture was designed for high-volume data ingestion while separating data required for fast access from data retained in long-term storage. This tiered storage approach helps maintain search performance while keeping storage costs predictable as data volumes increase.
In production, the architecture achieves approximately 15× compression of stored log data, significantly reducing the storage footprint of long-term retention while keeping the data available for search and retrieval.
Starting With a Proof of Concept
The project began with a proof of concept (PoC) designed to validate the proposed architecture before moving to full implementation.
The proof of concept demonstrated whether the platform could meet the client’s requirements for performance, scalability, and compliance under realistic conditions.
Following a successful evaluation by the client, the project progressed to a full implementation phase planned to run for approximately one year.
Integrating Data Sources Incrementally
Rather than migrating every source at once, data sources were integrated into the platform incrementally.
The integration method depends on the individual source. The solution uses dedicated Splunk add-ons where appropriate, alongside other suitable integration methods depending on the technical requirements of each system.
This approach allows new sources to be introduced systematically while giving the project team the flexibility to address the technical requirements of each system.
Following the initial production launch, additional data sources and dashboards can be introduced as the client’s monitoring requirements evolve.
Delivery Approach and Team
The proof of concept was delivered by a project manager and a development team with strong Splunk expertise.
For the full implementation, the core delivery team consisted of a Project Manager, AWS Architect and Splunk Architect, bringing together project delivery, cloud infrastructure and Splunk expertise.
As the project moved into the dashboard development phase, the team expanded to support reporting and visualisation work with Splunk experts.
This combination of architecture, platform and implementation expertise enabled the team to manage both the underlying infrastructure and the practical observability requirements of the client.
The Results
The new solution gives the client a scalable observability and log management platform capable of processing very large volumes of data while maintaining fast access to critical information.
Log data from multiple systems can now be centralised within the same environment, supporting both day-to-day operational monitoring and legal and compliance-related use cases.
1. Efficient Long-Term Data Retention
One of the key benefits of the architecture is its ability to balance performance with the cost of retaining large amounts of data.
By combining Splunk Enterprise with Amazon S3 in a tiered architecture, the solution can keep frequently accessed information readily available while storing large volumes of historical log data more efficiently.
With approximately 15× compression of stored log data in production, the platform substantially reduces the storage required for long-term retention.
2. Faster, Automated Security Response
The platform also supports automated security operations.
Infrastructure threats can be detected and contained within seconds through automated playbooks, while continuous vulnerability monitoring helps identify security issues without relying entirely on manual checks.
This automation reduces the manual security workload for both the client and the service team while enabling a faster response when potential threats are detected.
3. A Platform Designed to Grow
The new architecture gives the client a foundation that can expand as its monitoring needs change.
Additional data sources, dashboards, and monitoring touchpoints can be introduced over time without replacing the underlying platform. This creates opportunities to extend the environment beyond its initial log management requirements and support new observability and analytics use cases in the future.
Compared with the previous system, the client expects lower long-term costs through more efficient storage and a modern, scalable architecture that is easier to operate and develop further.
Despite a demanding implementation schedule, the project was delivered on time, allowing the previous system to be successfully retired.
Building Scalable Observability With Splunk Enterprise
For organisations handling hundreds of gigabytes of log data every day, observability is not simply a matter of collecting more information. The underlying architecture needs to make that data accessible when it matters while keeping storage, performance and operational costs manageable.
For this public-sector organisation, Splunk Enterprise and Amazon S3 created a scalable foundation for centralised log management, long-term retention, security monitoring and future observability use cases.
The result is an environment designed not only to replace an end-of-life system, but also to support the client’s evolving monitoring requirements over the long term.
About WeAre Solutions Oy
WeAre Solutions Oy is a Finnish observability-focused consultancy and a leading Splunk Elite Partner in the Nordics. We specialize in observability and monitoring (using Splunk), Atlassian services (Jira), and software development. Founded in 2016 and headquartered in Helsinki, our mission is to turn observability into a competitive advantage for organizations.
At WeAre, we help organizations assess their Splunk environments, identify improvement opportunities, and align performance with real business needs. You can start with an observability assessment to understand your current state, or contact our team for a free consultation.