Last week saw Splunk .conf 2020 take place and instead of a Las Vegas venue it swapped to a virtual event, as with so much as an effect of Covid-19. This was my first Splunk .conf event and for the un-initiated there are a huge variety of talks, in all 230 this year and obviously the focus for myself and CND colleagues was on cyber security related presentations, that said I did enjoy the cognitive rest sessions with celebrities Gwen Stefani (who gave a private .conf performance of her brand of ska pop hits) and another session with the actor Will Farrell.
As a registered Splunk partner it is important for CND to keep up with current developments and just filtering on security there are 68 unique talks this year. Plus Splunk released a new minor version 8.1.0 (release notes) which for the first time supports data ingestion in conjunction with network load balancers (LTM).
The talk which really grabbed my attention was 'TRU1537 – Hardened Splunk: A Crash Course in Making Splunk Environments More Secure' (click here to access the video and handouts). The talk started with a recap on fundamentals and components such as the CIA triad (Confidentiality, Integrity, Availability) and the Cyber Attack Kill Chain (SANS blog URL). It also covered basic best practices such as regular software and OS patching, applying security benchmarks such as CIS (URL) among other essentials. This was all important and interesting as a baseline security recap but where it got particularly interesting was focusing in on the Splunk Deployment Server.
Before I delve into the security threat surface potential that this particular Splunk component presents let me back up a bit on Splunk. Splunk is the market leading Data-to-Everything platform, by ingesting every possible log source across an infrastructure it leads to very powerful security data being rapidly available to SOC analysts and other cyber professionals protecting an organisation. To get data form all your endpoints including end-user workstations, VPN logs etc. you need to have a UF or Universal Forwarder running (other methods are available but the UF is the preferred method). This is a very small software package available for *nix, windows and various other OS distros that uses minimal resources and it simply sends back to the core data ingestion layer (indexers) all desired monitored files and telemetry metrics. With me so far? I've drawn a simple diagram above to help illustrate these components.
The Deployment Server (DS) is where Splunk Administrators configure the various desired applications (literally made up of a series of text files) per the given organisation's monitoring strategy, this is then sent for distribution to all nodes that send data back to the core thus allowing low effort scalable management. The premise of .conf presentation was that an inadequately protected and hardened DS would allow a Threat Actor to distribute malware across your entire estate if they manage to compromise the DS and gain persistent access to it. This solves their pivot and malware distribution issue with just a single malicious 'hack'.
This really got me thinking. Often in IT organisations the server admin team, network team, security operations and firewall admins all operate in silo's which is a shame and a server admin juggling all their responsibilities may not have the background or latitude to adequately protect through a layered model a Splunk Deployment Server. Another scenario is that an organisation grows and where it was initially placed and sized for is no longer appropriate but there is never quite the time or impetus to migrate to a better practice solution behind a firewall, perhaps on a dedicated VLAN segment (or following the micro-segmentation trend) perhaps if there is a firewall it is a legacy layer 3,4 packet filter and doesn't have layer 7 application scanning capabilities or full stateful inspection.
I'd recommend watching the video and evaluate if your Splunk environment is adequately protected. There were some of the other great talks such as the McLaren racing cars (SFU1666A) briefing which was fascinating to listen through the plethora and dependence on machine data.
If you're concerned by the issues raised here get in touch, we can help. CND has Splunk certified Consultant and Architects on hand with a cyber professional background to support and critically protect your business.