Seven Log Problems and the SecDataOps Fix for Each

Walk into enough client engagements and the same complaint shows up in different clothes: the security team says the logs are not good enough, the platform team says they are shipping everything, and the finance lead says the bill is out of control. All three are right. The cause is rarely a missing tool. It is that nobody owns the path between the machine that writes an event and the person who needs it. That path is the territory of SecDataOps, and the quickest way to understand it is to treat it as a diagnosis problem: symptom, root cause, fix.

Symptom one: the bill keeps climbing and coverage does not

The usual story is a SIEM licence priced by ingest, a pile of chatty sources, and a steady stream of exceptions to keep the invoice down. Teams respond by switching off whatever looks noisy, and then discover later that the switched-off data was exactly what an investigation needed.

Root cause. Everything lands in the one expensive tier, and there is no record of which detections depend on which streams, so cuts are made blind.

Fix. Tier the data and price each stream. The Ops Atlas, which hosts a reference page on SecDataOps as a discipline of telemetry engineering, gives a worked example: 400 GB a day of firewall flows held entirely in the hot tier costs roughly 400 units daily, whereas keeping just 8 GB of daily summaries hot and shipping the untouched stream to a deep archive for 400 days lands near 164, so raw history grows from about three months to more than thirteen. Do the same sum for your top five streams before anyone cuts anything.

Symptom two: a rule that looked fine never fires

The team imports a detection written for one log format and points it at the data they have. Nothing triggers, and nobody is sure whether that means no attacks or no match. The Atlas quotes the observation that a mismatch between a detection's expected log source and the real one can render the rule ineffective, which is a polite way of saying it silently does nothing.

Root cause. Required fields were never written down, so no one noticed they were missing or renamed.

Fix. Write a short contract per source and keep a list of the fields each rule needs. Add fixture events and run them through the parser in your build so a renamed field breaks a test, not an investigation. The Atlas calls this parser-as-code, and it is as unglamorous as it sounds.

Symptom three: nobody noticed the feed died

Feeds go quiet for plain mechanical reasons, long before any attacker is involved. The Atlas gives a small but telling runbook example: a routine OS upgrade turned off DNS query logging on a pair of hosts, and client-certificate rotation broke forwarding on another pair, with neither change passing through the change board.

Root cause. Health was judged by the absence of complaints.

Fix. Put a health monitor on each important source and page on silence. One sample target in the Atlas reads: nineteen in twenty events arrive within five minutes, parse success at or above 99.5 percent, completeness of at least 98 percent, and an alert after fifteen minutes with no events. Then, every month, run a silence test: mute one lab source and see how long the alert takes to appear. If the answer is measured in days, fix the monitor.

Symptom four: we cannot look back far enough

An incident gets reported, and the earliest logs you hold begin three months after the trouble started. Retention defaulted to ninety days when the system was installed and nobody revisited it.

Root cause. Retention was a vendor default instead of a decision.

Fix. Choose a retention period per source based on real obligations. The Atlas lists several external anchors you can cite in a design review: a federal US memo with a twelve-month active and eighteen-month cold floor, joint guidance led by Australia that treats a year as the starting point for admin and security records with just ninety days for informational material, and a 2022 Indian directive that sets half a year. It also cites a 2026 Mandiant report with a median dwell of 14 days globally but 122 for some cases and about 400 for one malware family. Pick your numbers deliberately and write down why.

Symptom five: the SIEM people say it is not their job

A subtle one. Everyone assumes the SIEM vendor handles collection, parsing and storage because the licence says so. In practice the product searches what arrives, and the arrival is somebody else's problem.

Root cause. No named owner for the pipeline.

Fix. Assign an owner to every link of the supply chain: the producer, the collector, the pipeline, the tiers and the consumers. If you are scoping this kind of engagement, our blueprint for taking a client from first assessment to self-sufficiency lines up neatly with the register-then-contract sequence below.

Symptom six: the normalised data looks great and the answers are wrong

Cleaned data feels trustworthy, so teams delete the raw originals to save money. Months later a parser bug or a schema deprecation is discovered, and the evidence to repair it is gone.

Root cause. Normalisation was treated as a replacement for raw storage.

Fix. Park the originals in the cheapest storage and do your cleaning on the way out. The Atlas also recommends keeping both event time and processing time in UTC with millisecond precision, and using a stable unique identifier so replays do not double-count.

Symptom seven: collecting everything did not make us safer

A few teams take the opposite approach and ingest every log they can find, planning to sort it out later. Official guidance, as relayed by the Atlas, warns that excess collection makes sharp rules harder to craft because the useful signal gets lost in bulk.

Root cause. Collection was driven by what is available, not by what detections and audits require.

Fix. Work backwards from consumers. List the detections and obligations, mark the fields they need, and collect to that list. Also check platform specifics before relying on a source: one cloud audit service makes object-level events an extra-cost opt-in, one warehouse view trails reality by as much as three hours and skips failed queries, and a rendered-text Windows forwarding format can double each event.

A practical sequence that works

If you only remember one thing, remember the order of operations. It follows the Atlas's thirty, sixty, ninety day outline.

1. Days 1 to 30. Create a coverage register, grade each source zero to five, agree a retention policy and attach health monitors to the ten sources that matter most.
2. Days 31 to 60. Draft contracts, sample fixtures, automated build tests and canary events for the same ten sources, and attach a cost to every stream.
3. Final stretch, roughly days 61 to 90. Re-home your three heaviest streams, proving by replaying one week of originals that nothing downstream breaks, then hold the first silence drill and share a baseline report with stakeholders.

For teams worried about folding this into security workstreams, our consulting primer on building security into release pipelines covers the adjacent territory without overlapping it.

A short example of the sequence in action

Imagine, purely as an illustration, a mid-sized company whose identity provider, firewalls and cloud audit trail all feed one search platform. During the register stage the team discovers that two firewall clusters were never connected, a gap nobody had flagged. During the contract stage they find that the identity feed renamed a field after an update, which explains why one imported rule had never triggered. During the tiering stage they realise most of their spend goes to flow records that no rule reads in raw form. None of these discoveries needed new software. They needed someone to ask who owns each link and write the answer down.

Warning signs that you need help sooner

Some signals mean the problem is already costing you. A bill nobody can explain per stream. An incident where the first question was whether the logs even exist. A change that silently altered a source. Detection rules written by one team and run by another with no shared field list. Any two of these together justify an assessment now rather than after the next audit.

A last point on expectations. SecDataOps is not a product you buy and finish. The Atlas describes a ladder, from reactive through inventoried, contracted and measured to optimising, and most organisations spend a long time on the first two rungs. Progress shows up as boring facts: fewer surprises, shorter answers to hard questions, and a bill you can explain.

Kabir Malhotra