SOAR tools: how to evaluate and choose the right security orchestration platform

Buying decisions for SOAR tools, meaning security orchestration, automation and response platforms, tend to be made on the wrong criteria. Vendors demo a polished phishing playbook, the integration catalog shows four hundred logos, and the evaluation team leaves impressed. Eighteen months later the same team is staffing a full-time engineer to keep thirty playbooks running against APIs that keep moving underneath them.

The gap between the demo and the operating reality is predictable, and it is measurable during evaluation if you know what to test. What follows is a framework for doing that, plus an honest section on when the right call is not to buy a SOAR platform at all.

Integration breadth is a vanity metric

Every vendor advertises an integration count. The number tells you almost nothing, because integrations vary enormously in depth. A "supported" integration might expose three read-only actions or it might expose the vendor's full API surface with bidirectional sync.

A practical way to scope this is to map your intended playbooks against the techniques you care most about in MITRE ATT&CK, then check that each required response action exists in the connectors you would rely on.

Test this concretely. Pick the five systems your playbooks will touch most, then ask the vendor to enumerate every action available for each one and demonstrate two of them live against your tenant during the proof of concept. Ask specifically about the actions you need to write, not just read, since containment depends on them. Then ask what happens when you need an action the integration does not cover, because the answer determines whether your team writes custom code and how painful that is.

The second question is who maintains the integration. Vendor-maintained connectors get updated when upstream APIs change. Community-maintained ones sometimes do. Anything you build yourself is yours forever. Ask for the vendor's track record on connector updates following a major API version change at one of the large platform providers, and ask how customers were notified.

The authoring model in SOAR tools determines your maintenance bill

SOAR tools generally offer a visual drag-and-drop builder, a code-first interface, or both. This choice looks cosmetic during evaluation and turns out to be the largest single driver of long-term cost.

Visual builders lower the barrier for analysts who are not developers, which is a genuine benefit when you have three playbooks. At thirty playbooks, a visual graph stored as proprietary JSON becomes very hard to diff, review, test, or reason about in aggregate. Code-first authoring gives you version control, code review, and unit tests, at the cost of requiring people who can write and maintain Python.

The questions worth asking are whether playbooks can be exported to a text format that a human can read in a pull request, whether there is a testing harness that lets you run a playbook against a recorded alert without touching production, and whether you can promote changes from a staging environment. Vendors that answer all three well are rare, and the answers correlate strongly with how the platform holds up in year two.

What happens when an upstream API changes

This is the single best diagnostic question in a SOAR platform evaluation, and most evaluation teams never ask it.

When a vendor deprecates an endpoint or renames a field, three things determine your exposure. Whether the connector fails loudly or silently, whether the platform surfaces per-step failure rates in a dashboard you actually look at, and how quickly the vendor ships an updated connector. Silent failure is the dangerous case, because a playbook that catches the exception and continues will close cases as benign using enrichment that never ran.

Ask to see the execution monitoring interface. Ask whether you can alert on a playbook's failure rate rather than only on total executions. Ask whether failed steps halt the workflow by default or fall through. If the vendor cannot show you this in a live environment, assume it does not exist in a usable form.

Case management, approvals, and the analyst experience

The orchestration engine gets the attention, but analysts spend their time in case management. Evaluate it as its own product. Look at whether enrichment artifacts land in the case in a readable form or as a wall of raw JSON, whether the timeline shows what the automation did and why, and whether an analyst can take a manual action from inside the case without switching consoles.

Approval workflows deserve equal scrutiny. Map them against the response phases in NIST SP 800-61 so containment, eradication, and recovery actions each have a defined authority. Check that approvals can route by action type rather than only by severity, that they reach people where they already work, and that a pending approval has a timeout with a defined fallback. A gate with no timeout is a playbook that stalls forever at 3 a.m.

There is also the question of what the platform does with the alerts nobody wrote a playbook for, which in most deployments is the majority. If those simply queue up for a human, your SOC challenges around analyst capacity have not moved.

Pricing models and where costs hide

The three common models are per-user, per-action or per-execution, and flat platform licensing. Per-execution pricing sounds fair and creates a perverse incentive, since the team starts rationing automation to control spend, which is the opposite of the goal. Per-user pricing penalizes bringing more of the SOC into the tool.

Whatever the model, the license is rarely the largest line item. Budget for the implementation partner most teams end up hiring, the engineering time to build and maintain playbooks, and the connector work for the tools in your environment the vendor does not support. A useful sanity check is to estimate the fully loaded cost of one engineer's time spent on playbook maintenance and compare it against the license. If the ratio surprises you, the ratio is the point.

Frame the return on the spend against containment speed rather than against playbook count. IBM's Cost of a Data Breach research has repeatedly found that faster identification and containment correlates with materially lower breach cost, which is the outcome the budget is really buying.

Buy versus build, and when SOAR is the wrong answer

Building on general-purpose workflow tooling is a real option, and some large teams do it well. It makes sense when you have platform engineers already, when your environment is unusual enough that commercial connectors will not fit, and when you can commit to maintaining it as a product with an owner. It fails when it becomes one person's side project.

There are also situations where none of the SOAR tools on your shortlist is the right purchase.

• Your alert volume is low enough that a handful of documented manual runbooks and a good ticketing system will serve you better.
• Your detection quality is poor, in which case automating the response to bad alerts scales the noise rather than reducing it.
• Your underlying tooling lacks the APIs required for the actions you want, which means the orchestration layer has nothing to orchestrate.
• Your actual bottleneck is investigation rather than execution, which is the most common case and the one SOAR is least equipped to fix.

That last point is critial. If your analysts are not slow because they are clicking through consoles but slow because they are reasoning about ambiguous evidence, a deterministic playbook engine will not help. This is a different question from SIEM versus SOAR, which is about detection versus response. This is about whether the work you need automated can be expressed as rules.

Verizon's Data Breach Investigations Report has documented year after year how large a share of breaches involve the human element and credential misuse, categories where the investigative judgment is exactly the part that resists encoding.

Where agentic platforms fit in the evaluation

If investigation is the bottleneck, the comparison set widens beyond traditional SOAR solutions. Exaforce is an Agentic SOC Platform built on a Multi-Model AI Engine, with a Semantic Data Model that interprets logs, cloud configuration, code, identity, and threat feeds, a Behavioral Model that learns baselines for identities, assets, and locations, and a Knowledge Model that supplies reasoning and dynamic workflows.

The evaluation criteria shift accordingly. Instead of asking how many connectors exist, you ask whether the system can show its reasoning on an alert you bring to the proof of concept, whether Exaforce's Exabots can run in copilot mode until you trust them enough for autopilot, and whether the Advanced Data Explorer lets your analysts interrogate conclusions in natural language. Reported outcomes include false positives down up to 80 percent, mean time to respond improved up to 70 percent, and SIEM costs reduced by 40 percent. Our comparison of SOC automation platforms covers how these categories line up against each other.

Conclusion

Evaluating SOAR tools well means testing the things that only show up later. Integration depth on the five systems you actually use, an authoring model your team can maintain in year two, visible failure handling when upstream APIs move, approval routing tied to blast radius, and a pricing model that does not tax you for automating more.

Run those tests and you will either find a SOAR platform that fits or discover that your constraint was never orchestration. If it is the latter, the more useful exercise is working out how to choose a SOC automation tool against the investigation problem you actually have.

‍