Why Your APM Is Missing the Credential-Stuffing Attack Happening Right Now

Your APM dashboard, right now, has the data to show you whether someone is currently attempting credential stuffing against your login endpoint. It's not telling you because the dashboard is plotting count(*) and p99(latency) instead of the queries that would surface it.

This is fixable in roughly an afternoon. Here's how, and why.

For the broader frame on combining APM and security observability, see the APM + security in one tool page.

What credential stuffing looks like in trace data

When an attacker runs credential stuffing against /auth/login, the spans look like:

2026-05-09T14:23:01Z  POST /auth/login  401  18ms  client_ip=185.220.101.42
2026-05-09T14:23:02Z  POST /auth/login  401  21ms  client_ip=185.220.101.42
2026-05-09T14:23:02Z  POST /auth/login  401  19ms  client_ip=185.220.101.42
2026-05-09T14:23:03Z  POST /auth/login  401  17ms  client_ip=185.220.101.42

Hundreds or thousands of these per minute from one IP, all 401, all with normal latency. Sometimes one returns 200 — the attacker found a working credential.

Your APM aggregates these into "POST /auth/login: 401 rate is 12% of requests, p99 latency is 22ms." Both numbers look fine until you realize 80% of the requests are from one IP that should never have hit /auth/login more than 3 times in its life.

Why APM misses it

Three structural reasons:

Aggregation hides per-IP patterns. The default APM dashboard groups by endpoint, not by source. A 4xx-rate spike across the whole endpoint is invisible if it's coming from one IP that's a tiny fraction of overall traffic.

Latency-focused metrics ignore success/failure semantics. A successful credential stuffing attempt looks like every other 200. Failure looks like every other 401. APM rarely flags these as anomalous because they're, individually, normal-looking responses.

Time windows are too coarse. APM dashboards typically show 15-minute, 1-hour, or 24-hour windows. Credential stuffing happens at much finer granularity — bursts of 200 requests in 30 seconds. Coarse windows smooth this out into invisibility.

What detection actually requires

Three additions to a standard APM stack:

1. Per-source aggregation. Group by client.address (or by ASN, or by header-derived fingerprint), not just by endpoint. The query is one GROUP BY away.

2. Time-windowed thresholds. "More than N events in M minutes" type rules, where M is small (1–10 minutes). Most APM alerting systems support this; it's just rarely configured for security cases.

3. Multi-IP rollup. Sophisticated attackers use rotating residential proxies — no single IP exceeds your threshold, but the aggregate behavior is anomalous. Roll up by ASN, by user agent class, by header pattern.

The simplest credential-stuffing detection rule is:

COUNT(spans WHERE
  http.target = '/auth/login' AND
  http.status_code = 401 AND
  timestamp > now() - 5min
) GROUP BY client.address
HAVING count > 50

This alone catches per-IP credential stuffing. Add distributed detection on top:

COUNT(spans WHERE
  http.target = '/auth/login' AND
  http.status_code = 401 AND
  timestamp > now() - 5min
) GROUP BY ASN(client.address)
HAVING count > 200 AND distinct(client.address) > 20

This catches distributed attacks — many IPs in the same ASN, all hitting /auth/login.

The data layer matters

Both queries assume your trace data is in a backend that supports flexible aggregation. ClickHouse does. Elasticsearch does. Datadog's proprietary backend does, but the query language is awkward and per-trace queries can be expensive.

If your APM doesn't expose raw spans for ad-hoc querying — or charges per query — security detection becomes impractical. This is one of the reasons OpenTelemetry-native tools have an advantage: the data is in your control, queryable in SQL, and the cost is fixed regardless of how creative your detection rules get.

Beyond credential stuffing

Once you have the data layer set up, the same approach catches:

• Account takeover sequences: password reset → MFA enroll → device add, all from a new ASN
• Cart cycling / coupon abuse: repeated checkout attempts with different coupon codes
• API scraping: high-volume GETs to listing endpoints with rotating user agents
• Vulnerability scanners: sequential probing of /wp-admin, /.env, /admin, /.git/config

Each is a different SQL query against the same span data. The overhead of each new detection rule is one query plus one alert config — not a new product.

What this looks like in practice

If you're using OpenTelemetry-native tools, the rules ship pre-configured. The SecureNow alert rules library has the credential-stuffing detection above as a one-click enable, plus 20 more rules for the patterns listed.

If you're rolling your own, the queries above are the starting point. Schedule them on a 1-minute cron, post matches to Slack, and you have functional detection in an afternoon.

What to do tomorrow

Open your APM. Go to the trace explorer (or whatever your tool calls the raw-data view). Filter to /auth/login with status 4xx in the last hour. Group by client IP.

If any IP shows up with more than 100 failed auth attempts, you have credential stuffing happening right now. The fact that your dashboard didn't surface it is the bug, not the lack of attacks.

The fix is detection rules. The data is already there.

Related

• APM + security in one tool
• Application security monitoring with OpenTelemetry
• Detecting business-logic abuse with spans