Proof of Concept: Automating Threat Hunting in Elastic with Tines and AI

Unbalanced Game

Cybersecurity is an unbalanced game, and the defenders are always at a disadvantage. By nature, their infrastructure is static and their intelligence is frequently outdated. With AI tools becoming more efficient and popular for discovering vulnerabilities, the industry is seeing 0-days being dropped left and right for some of the most popular systems in the world, with a sizeable number of them being exploited before any patch is available. In addition to that, attackers can now simply have new infrastructure generated and stood up, along with fresh variants of malware in a matter of hours.

This isn’t a new problem; attackers have long been focused on quickly deployable and disposable architecture. Unlike defenders, they have no need to remain static and gain significant advantage when they choose not to. To help combat this behavior, the industry has made use of several different systems for keeping up with the latest emerging threats using cyber threat intelligence (CTI). CVEs, STIX/TAXII, MISP platforms, Suricata and Yara rules, etc. There is an overwhelming abundance of available feeds for CTI. Still, as attackers have sped up, so must defenders.

No matter what feelings anyone holds about AI, it is undeniable that defenders must create new methods to maintain their defensive posture against malicious actors. It's clear these new methods will need to revolve around AI, but the “how” is the industry's largest question. With the simple phrase “AI” holding so much buzz word power, vendors shove the technology into every tool and crevice they can find, often regardless of if the use case makes sense. This blog post describes my personal stab at the problem: a proof of concept using AI integrated through a SOAR to automate threat hunting against the latest emerging threats.

The Bash Build

I’ve previously worked with Splunk SOAR, and though it’s an industry standard, I’ve found I prefer another platform: Tines. Designing and creating automations with Tines is more straightforward than Splunk SOAR. The flowchart formatting lets me take complex operations without needing to stop and wrap my brain around how my data is flowing. However,  it’s important to know that Tines free community edition can be severely limiting. If I was to use it to build my tool, I needed to make sure that the idea even worked. So, I decided to use the language I know best: Bash. With the assistance of AI to speed up this project, I created a series of Bash scripts to act as stand-ins for the various components of Tines. This includes integration with Elastic, a local LLM, and both gathering and storing the threat intelligence and extracted IOCs. Over time, this grew into a strong directory of scripts, all pipelined to flow together and run on a 6-hour schedule via a cron job. 

This was the general workflow and purpose of the scripts:

1. Collectors pulled indicators from a dozen CTI feeds into a store of tens of thousands of IOCs.
2. A hunter queried those indicators across five types (IP address, domain, URL, file hash, and TLS certificate fingerprint) against logs planted in Elastic.
3. A converter translated SigmaHQ rules into further Elastic queries deterministically. It was tempting to have an LLM handle these and more advanced rules than what I could program, but I found my weak local model was too poor at that task to be of any use, even in a POC.
4. A correlation engine, this component written in jq, decided what rose to the level of a case. And additionally, integration into that local model which generated new IOCs from freshly published reports, which had moderate success at finding activity that the large sets of IOCs and SIGMA rules otherwise would not have. Finally, for the detected incidents of IOCs and malicious activity, the pipeline automatically generated a case in Elastic, with AI writing a description of the threat and what/how the system detected it.

Eventually, this system started working reliably. I would not expect a system like this to catch everything, but with a process of IOCs and activity into Elastic by hand, plus using AI to generate more IOCs free of context of the project and tens of thousands of benign logs, my average detection rate was about 55~65% of the planted IOCs and behaviors (beaconing, lateral movements, and attack chains from recent reports). The behavioral detection wasn’t perfect, there were a small number of false positives, but false positives on the IOC hunts never occurred after the earliest of my tests. After a couple weeks of building, breaking, testing, and tweaking the project, the system was generating cases like these fully autonomously every time I injected another batch of IOCs:

By this point, I had proven the idea of ingesting IOCs, using AI to extract more from reports, perform threat hunting, and have AI draft reports on the activity found was doable and more effective than I initially expected. Now it was time to begin trying to recreate as much of this as I could with my severely limited free Tines account. 

Tines To Do It Again

Before I begin digging into this, there are a few items of basic Tines terminology that are important to know:

Actions – The smallest unit of automation in Tines. This could be a webhook, an HTTP request, an event transform, sending/receiving an email, etc.

Flow – Actions connected together to create logical chains.

Stories – One or more flows that work to complete a task. These can be likened to individual scripts, whereas flows are functions and actions are commands.

There’s plenty more to Tines, but for the purposes of this blog post, that’s all anyone needs to know. The translation of Bash to Tines wasn’t the smoothest task I’ve ported from one technology to another, but it was less complex than initially feared. The goal of this implementation wasn’t seeing how effective I could make the threat detection, but to see how much I could fit in the community edition of Tines to integrate actual SOAR technology.

With a free-tier limit of only three flows, it will surprise no one that I had to choose what I wanted to implement very carefully. To that end, after a small redesign of the architecture, I decided on three components: Hunt Detectors, Correlation Gate, and Make. 

Starting with Hunt Detectors, this is the story that actually runs the hunts. A webhook triggers it, and from there it fires two Elastic queries over Cloudflare tunnels. One for standard IOCs, and another for behavioral indicators. Whenever a query returns results, Hunt Detectors shapes that information into a finding and ships it over to the Correlation Gate story. Said finding follows this format:

{ "host": "...", "sources": ["ioc:feed" | "get:sigma" | "make:llm"],
  "signals": [...], "severity": "...", "dst": "...", "allowlisted": false }

Once these findings arrive, the Correlation Gate classifies the type of indicator found and breaks it down into a token, which is just the hostname joined with the kind of detection (not to be confused with an AI token). The gate then compares this against whatever activity, if any, it has previously detected on the host. I labeled this as the "state" of a host, which generally tracks just how compromised a machine may be. This is tracked to keep things organized for the event that multiple malicious indicators are found on the same machine. The IOCs routinely had no false positives, so I considered a matched one likely to be a true positive. Therefore, the gate labels any case opened on a detected IOC as high priority. 

If the hunts later match and report a behavioral signal on the same machine, the gate updates the state and upgrades it from high to critical. Behavioral indicators on their own are not always reliable indicators of malicious activity. To account for known, benign activity, I added an allow list so I can whitelist non-malicious indicators and keep them out of the reports. Finally, if only a behavioral indicator appears, the gate marks the activity for review and stores it in case future indicators surface, although it creates no case.

Finally, we get to the third and last Tines story my account had resources to use: Make. This story introduces AI and ties it directly into the workflow. This works by sending Tines a report's URL via a webhook, having it fetch the article, and then feeding it to the LLM. From there, the model creates a new detection token, which Tines uses to create a hunt query from. If this hunt turns up potentially malicious activity, the story doesn't open a case; it puts the finding into review, much like lone behavioral indicators.

There are two decisions worth noting about the Make flow:

1) the AI is not configured to create the hunting queries. With access to a powerful, frontier model and a simple validation flow backed by context of data sources, it is likely possible to have an AI consistently make it's own advanced hunting queries. However, my small 14b model is not ever going to be one-such example. So, I've made a few key decisions on where to and not to include it in the intel gathering and threat hunting workflows.  

2) It is personal preference that a case is not automatically created if one of the reports leads to a finding. I'm sure setting it otherwise would increase the number of true positives detected, but since AI is in charge of the extraction of IOC and behavioral indicators, there stands a chance that it could add a too-vague IOC and generate hundreds to thousands of cases of benign activity. It's my theory that this would remain true, and even probable to happen, in an enterprise environment with top-of-the-line models available. Therefore, I will accept it as indicators supporting the creation of a case, but never cause enough alone. 

Performance Compared to Bash

So, with the Tines workflows built, how does this perform compared to the Bash setup? Surprisingly well in smaller hunts. I expected there to be significant performance loss from the migration, even with downsizing the number of IOCs I threw at it. The Community Edition of Tines has low limits on the number of runs available, and I expected to face timeout issues between waiting for my LLM to respond and with the hunts. Nonetheless, it holds a detection rate similar to that of the pure Bash version.

However, this isn’t a completely fair comparison. As mentioned earlier, I wasn’t looking to obtain a similar level of performance with the Tines migration; I wanted to prove that a recreation in Tines was perfectly feasible. That run limit was a blocker I didn’t want to spend much effort dancing around. So, instead of throwing the tens of thousands of IOCs across five major categories that I used for Bash (domains, IPs, URLs, SHA256, and TLS cert SHA1), I only used a few thousand IOCs split across two categories: domains and IPs.  And instead of gathering a generous supply of SigmaHQ rules, Tines used several simple LOLbin detections I manually placed:

This narrower scope led to results more impressive than they actually are, but I still accomplished my goal. This is definitely doable on a SOAR platform like Tines, and with tuning, I suspect could be quite effective. Further, Tines is quicker and easier to work with than Bash, and if needed, can still run Python for enhanced programmability. There’s no doubt a full recreation would be a stronger end-product than my first Bash POC. Worth noting is that it continued the pattern of few false positives. In this case, there were none.

Limitations and Deciding the Tines Components

I’m pleased to see this project be successful as a POC, but that isn’t to say I’m satisfied with how far I was able to go with the Tines integration. I’ve already discussed some of the limitations I faced, but it’s time to explain them more fully, as well as the reasoning behind why I chose to migrate the components I did to Tines.

With a community license, there are more limitations to consider than just the three flows and low number of runs. There also is no persistent datastore, no Tines case management, and no built-in AI functions (I worked around this through HTTP requests fed through a Cloudflare tunnel to my local Ollama installation). All these features could have been major functionality to design my architecture for, but instead they were challenges to build around. Couple this with my relatively weak local AI, holding a wet rag’s worth of 14 billion parameters, and getting more in-depth with the AI integration didn’t seem to hold much backing as a proof of concept. 

So, with those limitations in mind, I chose the components I converted simply because they were the most SOAR-esque, and they proved that the individual programs and pieces of intel gathering, IOC formatting, and threat hunting could be integrated together into a single, complete automation.  Something like gathering threat intelligence is a common SOAR use-case and would’ve been fine for the project, but that’s not a concept that needs proving. Nor are the other individual actions Tines roped together, including the AI use-cases. I kept the remaining components as Bash, generally as stand-ins to emulate what I would expect from further Tines flows, which could replace almost all of my Bash. 

What Would I Do Without License Limitations?

Looking beyond a proof of concept, I want to postulate about what an enterprise implementation would look like. I could convert most of my work to Tines flows, but certain matters, like a persistent database of IOCs and a SIEM, would be poor ways to use Tines. If an enterprise, licensed version of Elastic was also available, then there are some major features that could bring a lot more value from this concept. One of those licensed features is Elastic Threat Intelligence. Another feature, this one not licensed but important for this design, is the Elastic Detection Engine. For the purposes of this write-up, I will refer to these as ETI and EDE, respectively.

ETI can ingest IOCs from threat intel sources directly into Elastic, storing them with expiration dates and confidence scores.  Additionally, ETI can match these ingested IOCs against incoming logs. This doesn’t quite remove the need for the IOC threat hunt, as any previous activity will not be detected by ETI, but it takes a great portion of work and scope off of the autonomous threat hunting. 

What remains that must be hunted is both well-defined and relatively small in scope. When the pipeline ingests new IOCs (either from a feed or extracted from a report) Tines can run a hunt for these indicators against recent activity from, for example, the last month. ETI’s detections can handle everything from there, and there’s no concern of an IOC hitting the network after the automatic hunt runs, which is a flaw in the POC. 

EDE covers the behavioral format of what ETI does. It already comes with hundreds of prebuilt detections, and an AI can still find or create additional Sigma rules. Tines can then automate the conversion of those Sigma rules to an NDJSON format, which Elastic can read and ingest into EDE. It’s not a perfect conversion, but in more cases than not, it would get the job done. EDE runs these detection rules on a schedule, covering everything going forward. And since Tines already back-tests each new rule against historical logs before a human approves it, the retrospective check happens as part of the vetting, fulfilling the role of the POC's behavioral detection hunts. 

Already, with just two features, we’ve strayed far from the design and workflow of my POC and the usage of LLMs is noticeably lower, despite having hypothetical access to the leading frontier models. The principle behind this is simple. LLMs are fluid; the same prompt can generate two different results from similar data. Due to that, it’s better to minimize AI usage as much as possible in favor of automated, static systems that will return consistent results. Where such systems struggle and where AI shines in this design is still the same initial job it performed with my Bash version. AI excels at research, extracting data and patterns, and triaging. It offers an opportunity to supplement existing automated threat hunts with information that threat feeds may have missed.

Tines can automate pulling reports from various cybersecurity news sources and feed those reports to an LLM with their built-in AI integration. The LLM scours through them, extracting indicators, behavioral patterns/Sigma rules, and even documenting potential variants of existing IOCs and exploits. Tines packages these findings for Elastic and checks them against the SIEM to confirm there are no duplicates before pushing them to ETI and EDE. Converted Sigma rules for EDE could have sources attached and wait for human review in a development workspace before a person pushes them to any production environments. 

Here is a graph visualizing what this might look like:

Put together, this lessens the use of Tines and AI as much as possible, limiting potential bottlenecks and resource consumption compared to my more LLM-heavy POC model. This also simplifies the role Tines plays, making it easier to maintain and easier to potentially integrate other desired technologies and sources into the automated threat hunting workflow. AI still gathers IOCs and additional information related to them, while the Elastic EDE rule and alert pages can natively handle the descriptions of the vulnerabilities and where they were detected. 

The only loss compared to the POC is that AI is not automatically supplementing campaign and vulnerability information about individual IOCs found, such as hashes or domains. This feature can be rebuilt using Tines’ licensed Cases feature. Alerts based on ETI detections can be automatically ingested to Tines, where AI can review the information and triage both the IOC and any additional details, such as details on the source host/user involved.   

Final Outcomes 

Wrapping things up, this project sought to explore how defenders can use AI to keep up with the growing pace attackers will be introducing new attacks as LLMs continue to improve. For the goal of introducing a new method of integrating AI into defensive technologies, I believe I’ve succeeded with this proof of concept. Both my Bash and current Tines implementation leave room for improvement and are imperfect at detection, but it’s a promising start to what may be a long line of research. I believe the drafted enterprise solution is a stronger approach, but only time will tell. The available public models, both frontier and self-hosted, continue to grow stronger and more efficient. There are multiple new, ground-breaking releases and announcements every month.  

Based on their current capabilities, this project takes care to limit LLM involvement where it is not necessary or significantly beneficial. In one or two years' time, this entire model will likely be defunct, and reliable AI integration more powerful and embedded. Nonetheless, I believe this is a step in the right direction. Attackers have been far faster in researching and utilizing AI’s capabilities than the majority of defenders, and they’re swiftly becoming more than a few steps ahead as a result. Unfortunately, that pace is only going to speed up over time as jailbroken (abliterated) models increase in capability and decrease in compute requirements. 

No matter if the premise of this project becomes outdated in the next 18 months, these sorts of implementations deserve to be investigated and researched. Defenders need to start investigating clear use-cases and integrations of AI to cover ground as attacks grow more common. And importantly, these uses need to be realistic and reasonable. The hard part of integrating it as a defender comes back to the unbalanced game of defenders remaining static. We cannot afford to overextend AI usage and make a large mistake. Attackers lack that disadvantage, and it’s becoming clear they know this. Where will ideas like mine and AI as a whole take the industry? I’m not sure there’s anyone in the field who knows, I sure don't!