Splunk Enterprise 10.0

I was very surprised on Tuesday to receive a nudge from a fellow Splunk Professional Services (PS) consultant, saying that Splunk 10.0 had been released overnight with almost no fanfare or mention. Splunk first released their Data-to-Everything platform in 2004 and has been ingesting data logs as the leading SIEM tool, trusted by corporations, governments and other organisations globally since.

As a PS consultant, my colleagues and I are always eager to try the new features which hopefully makes our job easier and improves the usability and security posture of the organisation we are working to support. New major Splunk versions are typically released every few years and version 9.0 was released back in June 2022 and version 8.0 in October 2019. Patch releases and minor features are released incrementally, typically every 6-10 weeks approximately.

So this morning I set about one of our lab instances running 9.4.3 and set to upgrade it to 10.0.0,

Engineers and admins can be lazy or blasé when it comes to reading manuals; I like to think I'm very diligent in this matter and always check the docs before an upgrade particularly at a client site. However mistakes do happen, and I was caught out in Splunk 9.4.0 at the start of this year when support for linux changed from being kernel version based to Linux OS based. The 'so what' being that Red Hat Linux (RHEL) version 7.0 was quietly dropped, and only by digging into the sub-page of system requirements was this noticeable. 

Lesson learned without pain thankfully, and I always recommend clients upgrade their standalone or test environments first so that we can adequately assess the changes alongside their hardware, storage and OS. 

Did I mention, Read the Docs: Welcome to Splunk Enterprise 10.0

Depending on the size of your organisations' Splunk deployment and the maturity of it, will probably influence what really matters to you in terms of features and changes. There are several new Federated search elements which is not something many of my clients have made use of, but MSP's or multi-national organisations may have particular use cases for federated search. 

Features I found notable: 

• mTLS (mutual Transport Layer Security) - Useful for organisations looking to secure logs data transiting to the Splunk indexer tier.
  
  
• OpenSSL 3.0 - Transit security enhancement, replacing older OpenSSL versions.
  
  
• RBAC changes, fine-grained access - Previously when decided on role based access, admin_all_objects was the sledgehammer option in some circumstances. This can provide junior admins or users which a greater level of access than they need. 
  
  
• Python 3.9 - This is a major change and it has been a gripe for some app developers that 3.7 (which is quite niche and deprecated) was still being used. When preparing apps for cloud and using a virtual environment for development this can be a pain, so I'm pleased to see this and I hope this will see the start of more regular alignment with newer Python versions.
  
  
• Forwarders (Agents), Effective configuration - Forwarders (endpoints with the UF agent package) tend to be under the administrative remit of different engineering teams in most organisations. This means that you more often than not, need to pickup the phone and ask a favour to check the config on an endpoint or ask for some code to be pushed out to an endpoint. These new changes should mean that config is available to inspect right from the Splunk Enterprise UI. This could be a real timesaver and reduce admin effort. Note that Splunk has renamed forwarders to Agents in the settings menu, see below. 
  
  
• Bulk Data Move - This is somewhat interesting but constrained to S1 (standalone) Splunk instances, and allows data to be moved from one index to another to live in a more logically index. Perhaps this will be expanded to clusters in future, which may prove useful as organisations grow their Splunk environment.

​Splunk has an aggressive software support policy, I really admire this and at 24 months it means admins must stay on top of their patching through a little and often approach. Too often in the technology industry we are forced to deal with legacy products and operating systems that are well past their death date. I did support a client last year that was in a real mess with Splunk Enterprise version 7.x on their servers, and Splunk UF 6.x on their endpoints. This lead to a recursive dependency cycle problem between apps, platform and OS and we had to adopt a rapid upgrade and fix-forward approach to get them in a good position after an MSP had effectively let them down. The point of this is that I encourage my clients to be upgrading or reviewing the patch releases every 4-6 weeks and keeping a beady eye on the advisory page for any CVE's that need patching, note a series of critical >9 CVSS scores mainly  related to GoLang in June this year.

The big changes this cycle appear to be:

• Removal of support for Ubuntu 20.04
• Removal of RHEL 7 (I mentioned this being less obvious in 9.4.x)
• Windows, there are probably some changes. But please, don't run Splunk Enterprise on Windows, you're just losing out on performance potential.

I'm keen to explore some of the changes discussed, notably Forwarder / Agent management and visibility, this could be significant for admin teams. I'll also be reporting back from Splunk .Conf in Boston this September where CND will be sending at least two Professional Services consultants, do get in touch if we can help you with the items raised here.