Splunk .conf 23

TLDR:

Excellent technical conference with great opportunities for meeting other Splunkers, sound in-depth technical workshops with a good deal of content to follow up on and study. Splunk as a tech company manages an effortlessly relaxed corporate culture that is very enjoyable to be a part of.

Intro:

This was to be my first global scale tech conference, I felt a real sense of privilege to be attending and having watched many of the recorded sessions from previous years, I hoped to gain a lot more insight by attending in person. It's a bit of an effort to get to Las Vegas, a 9.5hr flight and then 8 time zones worth of jet lag to deal with, so I went about a day and half early arriving on the Saturday night which worked well as a plan.

After having the Sunday to acclimatise (which isn't easy in 48°C heat) and attending a Splunk UK evening social meetup, I went to registration in the Venetian hotel conference centre around midday on Monday. There was a stampede for the Splunk T-Shirt shop and free hoodie, there was certainly a buzz of excitement and Splunkers proudly sporting jovial slogan t-shirts everywhere. I indulged in a photo with the ButterCup games pony and it was great to meet others in the Splunk community and attend the restricted Innovation lab (INN2160).

The Keynote presentation was set for 1700hrs with the CEO Gary Steele opening the conference at that time. I made sure to go early for a seat in the cavernous hall which seemed to have capacity for many thousands of people; pink and orange lights alongside upbeat music filled the space and all the stewards were putting an effort into dancing and it genuinely created an air of relaxed fun which can be a rare thing at corporate events! Acrobats dashed round the stage and flew off trampolines and to a raucous cheer the event opened up. Mr Steele certainly gave a positive opening speech and it felt good to be a part of this community, I was pleased to hear the message toned down on cloud migration. Apparently they've listened to clients that many are not necessarily ready (or dare I say interested in migrating to someone else's computer in AWS). After this engaging opener the rear cargo doors opened up to the conference arena which was again vast. Attendees spilled out and drinks and networking started in earnest.

I spent time in the customer success zone talking to colleagues in the professional services org and with the training team feeding back on some of the Enterprise Security content. It was a really great opportunity to meet people and very positive. I learned of a brand new industry focused certification 'Cybersecurity Defence Analyst' (URL), exam ID SPLK-5001 (URL) which I'd be keen to study for and take. I also saw the new hardware appliance: Splunk Edge Hub up close, it resembles a NUC or single board computer with a display on the front showing, temperature, x,y,z axis movement and a variety of sensors ideal for assembly lines or data centre monitoring. I find these sort of IoT solutions very interesting and I think have huge potential for detailed metrics and observability that can't otherwise be readily captured (see Edge Hub Central). The event wound down at 2030hrs and I was keen to get my head down for an early start the following day.

Tuesday .conf sessions

Having come all this way I was eager to attend as many sessions as possible and had booked up about a week in advance, this was just in time as some of the hands on labs were already fully booked, so a tip to others make sure you book up early to avoid disappointment if you attend in future. The majority of the slides should be available to download in the coming week via:

https://conf.splunk.com/watch/conf-online.html#/

I've included the session reference name to help others find the key interesting content that I enjoyed.

The recording for this session can be found online already (URL), a lot of focus on real world industry making use of Splunk to increase their uptime and availability, good to listen to how others are using the platform inc. Ikea and a cruise ship company.

I really enjoyed this session hosted by the Splunk SURGe security research team focused on Blue Team techniques, they have some detailed white papers on ransomware and adversary techniques using the MITRE ATT&CK framework (URL). Worthwhile study material.

Technical session on efficient use of this subsearch method that can iterate through multi-value fields and JSON arrays. I have used the command and inherited searches making use of it but it isn't one I'd readily reach for, so good to review and how it is highly efficient. Apparently there will be 30 useful examples in the slidedeck to download and reference.

A deep dive on scaling up to very large scale ES instances, a focus on starting with a small number of data-models and accelerated searches, RBA and increasing slowly after quality controlling and reviewing the impact on performance. Also some useful data on optimizing scheduled searches using Skew allowance. They also referred to the Splunkbase Performance insights app (URL) which seems worth exploring.

Great demonstration session on some of the challenges with client data at ingest, how to overcome multiple complex requirements. The demo showed both CLI and GUI solutions, I did ask a question and challenge the GUI approach as there is an obvious downside in terms of application context and packaging KO's for distribution. The slides should be very useful and I will certainly be downloading them once available and practising a few of the techniques.

I'm often engaged with clients who use a version of Git code control, it becomes really important especially with scale and multiple engineers and users working on an app. This class is a useful basic overview on how backflushing code to Git and ensuring a single source of control is adhered to. Also, a very useful control on not making errors with Robot checks so that you don't accidentally reduce frozenTimePeriodInSecs and wipe your data for example.

Wednesday .conf sessions
PLA1641B – Splunk Ingest Action and Rulesets: Advanced Pipeline Configurations

Great class on the ingest-time eval actions (URL), this wasn't previously taught in the admin training as a newer command and as such is valuable to review the new options that regex alone can't deal with for index time fields. Caveat on possible performance impact.

I don't really know what SAP software is or does, but I have a good friend who tells me it is vitally important to large organisations and contains all their stock management and business operations processes. He was explaining to me how challenging it is to migrate ~15 instances to a single instance; this class was demonstrating how Splunk can be a huge advantage to allow simple observability of the overall system health and so much better than the SAP alternative allows. It was a good show case of how Splunk can simplify observability via intuitive dashboard GUI. See the PowerConnect certified SAP app on SplunkBase (URL).

100 million clients give or take have an NTT account, as an organisation they face a range of cyber attacks and this class provided an explanation of how a very large Splunk instance on premise handles cyber attack using premium Splunk apps.

I upgrade Splunk clusters regularly and the docs give a very clear guidance (URL) on what should be undertaken first and the preparation required in a cluster. This class gave some value add detail on preparation and where clients often go wrong inc. Kvstore upgrades with long durations. I was surprised that they recommended upgrading a License Manager and Monitoring Console ahead of the Indexer tier and asked a question to that effect. I'll be keen to review the slides slowly and I'd recommend it to intermediate level administrators.

SEC1728C – Robotic Threat Hunting using ChatGPT

AI, it gets a lot of attention and the Key Note speech also gave a new NLP example of how Splunk may be used to create SPL from simple human text input. This talk by a Saudi Oil company team showed how a ChatGPT integration (URL) could allow queries and 'AI' answers. I'm not convinced on the value here and I think perhaps this is a bit of AI hype?

Organised fun often turns out to be dull, this was not that. I was really impressed with the laid back, easy going atmosphere that Splunk as a tech company has managed to create with its social scene. Previously I worked as a tech in the finance and banking industry and that became a bland social scene with reputation and cultural sensitivity concerns over-riding any possibility of a good time. It was a really fun party with DJ Jazzy Jeff and Columbian DJ Esther Anaya. Great fun, good chat and brilliant to spend time with some other Splunkers.

A slightly sore head the following morning but a lot of smiles for this first session of the day. This was presented by a Japanese company (Rakuten) and focused on the MFA fatigue attack that can see end users just press approve after numerous push requests for access. It'll be worth reviewing the TTP for Splunk SPL and how organisations can replicate some of this novel approach.

I think perhaps the most engaging talk of the conference for me, it focused almost entirely on Splunk SOAR / Phantom and cyber attack scanning automation. The way that they approach Portable Executables (.exe files) malware scanning and automate hash checking with a feedback loop means a highly efficient lifecycle reducing compute cycles and making it hard for even customised malware to break-out and persist without detection.

This was very niche and explained how a developer can create a custom Splunk command function within an app and the requirements and method to do so. I understood the principle and it is impressive that Splunk is flexible enough for this. I couldn't help but think if you're creating custom commands you might be building your platform into an unsupportable corner or are really undertaking cutting edge work (probably the former). There were some benefits to data ingestion highlighted though where non-log based data may need to be imported.

By Thursday afternoon .conf was over with a lot of detail to absorb and it felt a bit like a boot camp training course, all the content up front and many hours now required to really extract the value. I was glad to collect my bags and go, I'd had enough of walking through casino floors just to get a meal and the smell of narcotics and mid-day drinkers on the Vegas strip was grim. I'd very much enjoyed my first conference and there was a lot gained and I believe .conf represents good value in terms of awareness, upskilling, networking and wider product understanding and strategy. Would I come back despite the travel friction etc, yes definitely!