In this post I’d like to share my personal insights into threat hunting automation. As with many topics in security, there is some debate as to whether hunting can be automated, or whether it remains the commonwealth of the people. I’ve seen some marketing suggesting that hunting can be fully automated using machine learning, artificial intelligence and other sophisticated-sounding, buzz-wordy techniques. Of course, that’s just marketing, and most of us in this industry are familiar with the growing chasm that separates some (not all) of the marketing out there from the reality of what products can actually do.
For the past 5 years, I’ve spent the majority of my time automating certain aspects of threat hunting. I’m a huge fan of automation and derive much satisfaction from having a machine take over a mundane and repetitive task that a human was formerly doing day after day. That being said, I firmly believe that hunting wouldn’t be hunting if a human being wasn’t at the helm. The entire reason we automate is to allow people to do what they’re good at. And in 2017, people are still vastly superior hunters to machines. Don’t get me wrong, I think machine learning is really cool and has a rightful place in your portfolio of security tools, I just don’t know any data scientists capable of creating a system that hunts as well as the (human) hunters I’ve worked with over the years.
I use a really simple and binary approach to this problem. Start by looking at what people are good at that machines are typically not so good at. Then look at the things that people are able to do, but that tend to be repetitive, boring and lead to low job satisfaction. Then automate the latter. The added bonus is that every time you automate the latter, you end up enabling more opportunity for the former.
Yes, machines can be taught to do some of the above, but only with much work, normalized datasets and a fairly slow feedback loop and development lifecycle.
Automating the repetitive is the primary way that we can liberate people to do what people do best. That’s why we put the above phrase on our T-shirts. We love automating the mundane and we love watching a human analyst go to town hunting with tools that let them do what they’re naturally good at. I’ve witnessed people with little cybersecurity background, but with strong analytical skills, run circles around seasoned analysts simply because they were provided with an environment and toolset that enabled them to just be themselves. It sounds so simple, I know. But the converse is far too often the norm. So many analysts are found suffering from alert fatigue or wasting hours a day wrangling data just to get it in a suitable format for consumption.
I think automation is key and should absolutely be a large part of your hunting program. I think the trap people often fall into is trying to build systems that mimic a human analyst, because, of course, it just sounds way cooler to say that you’re building a machine that is intelligent and learns. It also potentially attracts more budget. The reason that this tends to be a trap is that most people skip over the easy stuff because their ego and curiosity persuade them to tackle the hardest problems first. But there is so much low-hanging fruit that can be automated that is going unaddressed. What we end up with is a whole slew of miserable analysts doing repetitive tasks, and a pile of software attempting to emulate a human and failing miserably. I say, liberate your humans to do what they’re good at and use your machines to do the things that your people despise doing or that a machine is better doing. Once you’ve created an environment like that, then spend some time on the harder problems; just be aware of the potential pitfalls along the way that can end up leading to flip-flopped priorities.
These are just my thoughts based on my experiences. Drop me a line at brian at (this website domain) if you're interested in chatting further on this topic.