Web Application Security Monitoring with OpenTelemetry

The fields a tracing tool collects — HTTP method, path, IP, status code, latency, headers, payload — are precisely the fields a security monitoring tool needs. Traditional security products built their own data pipelines because OpenTelemetry didn't exist yet. It does now, and the duplication is no longer required.

This post walks through how to use OpenTelemetry as the foundation for web application security monitoring. For the broader framing, see the ASM pillar page.

The data model

Every request that hits your application produces a trace span with these standard OpenTelemetry attributes:

• http.method — GET, POST, PUT, DELETE
• http.target — /api/users/123
• http.status_code — 200, 401, 500
• client.address — the requester's IP
• http.user_agent — Mozilla/5.0 ... or sqlmap/1.5.7
• http.request.body.size and (optionally) http.request.body itself
• http.response.status_code — same as above, redundant in some semconv versions

Plus the span timing fields (start time, duration). That's already enough for ~80% of application security detection rules.

For deeper coverage, OpenTelemetry semantic conventions also define attributes for authentication outcomes (enduser.id, enduser.role), database queries (db.statement, db.system), and exceptions (exception.type, exception.message, exception.stacktrace). All standard, all auto-captured by modern instrumentation libraries.

What detection looks like in queries

If your traces are in ClickHouse (which is true for SigNoz, SecureNow, Uptrace, and most OTel-native tools), security detection is just SQL.

Authentication failure spike from one IP (credential stuffing):

SELECT
  resource_attributes['client.address'] AS ip,
  COUNT() AS failures,
  COUNT(DISTINCT span_attributes['enduser.id']) AS distinct_usernames
FROM otel_traces
WHERE
  span_attributes['http.target'] LIKE '/auth/%' AND
  span_attributes['http.status_code'] = '401' AND
  timestamp > now() - INTERVAL 5 MINUTE
GROUP BY ip
HAVING failures > 50
ORDER BY failures DESC;

Scanner activity (admin path probing):

SELECT
  resource_attributes['client.address'] AS ip,
  COUNT() AS hits,
  groupArray(span_attributes['http.target']) AS paths
FROM otel_traces
WHERE
  span_attributes['http.target'] IN (
    '/wp-admin', '/.env', '/.git/config',
    '/phpinfo.php', '/admin', '/.aws/credentials'
  ) AND
  timestamp > now() - INTERVAL 10 MINUTE
GROUP BY ip
HAVING hits > 3;

Sudden 5xx spike on one endpoint (potential exploit attempt):

SELECT
  span_attributes['http.target'] AS path,
  COUNT() AS errors,
  COUNT(DISTINCT resource_attributes['client.address']) AS ips
FROM otel_traces
WHERE
  span_attributes['http.status_code'] LIKE '5%' AND
  timestamp > now() - INTERVAL 5 MINUTE
GROUP BY path
HAVING errors > 10
ORDER BY errors DESC;

These run in milliseconds against a properly-indexed ClickHouse table. The same logic in a traditional WAF is harder to express, harder to debug, and harder to retrofit when your detection needs evolve.

What needs custom instrumentation

The default OTel auto-instrumentation captures status codes and paths but not necessarily request bodies or full headers. Three patterns add the depth that security needs:

Capture request body for sensitive endpoints.

import { trace } from '@opentelemetry/api';

app.post('/api/login', (req, res) => {
  const span = trace.getActiveSpan();
  span?.setAttribute('http.request.body', JSON.stringify(req.body));
  // ... actual login logic
});

Be careful with PII and credentials. The standard practice is redaction before capture — replace passwords with [REDACTED], mask credit card numbers, etc.

Tag user identity on authenticated requests.

const span = trace.getActiveSpan();
span?.setAttribute('enduser.id', req.user.id);
span?.setAttribute('enduser.role', req.user.role);

This makes per-user investigation trivial. "Show me everything user X did in the last hour" becomes a one-line query.

Tag tenant identity for multi-tenant apps.

span?.setAttribute('tenant.id', req.tenant.id);

Critical for SaaS. Per-tenant SLOs and per-tenant security investigations both need this attribute. See the SaaS observability page for more on multi-tenant patterns.

The IP intelligence layer

OTel doesn't ship IP reputation; you add it. Two approaches:

Enrich at ingestion. Run a span processor that looks up each IP against an intel feed (AbuseIPDB, IPinfo, MaxMind) and adds attributes like client.country, client.asn, client.is_known_threat. This is cheap if you cache lookups; the major IP intel services rate-limit at 1000 lookups/day on free tiers.

Enrich at query time. Join against an IP intel table when running security queries. Slightly more flexible but slower for ad-hoc investigation.

Most production setups do both — enrich at ingestion for the hot fields, enrich at query for everything else.

What you give up vs a traditional WAF

Honest tradeoffs:

• No inline blocking. OTel-based monitoring detects; a WAF prevents. For blocking you need a separate product (the SecureNow firewall is one option, free).
• Latency to detection. WAF blocks in microseconds; OTel-based detection runs on a cron or streaming query, with seconds-to-minutes latency.
• Bot-specific detection. WAFs have evolved bot fingerprinting (TLS, JA3) that OTel traces don't capture by default.

What you gain:

• Forensic depth. Every request stored with full context. Investigations are SQL queries, not vendor-specific syntax.
• Cost. OTel + ClickHouse is 10× cheaper than a per-host ASM product at the same scale.
• Flexibility. New detection rule? Write a SQL query, schedule it. No vendor product release cycle.

The hybrid pattern

Mature teams combine layers:

1. Edge IP firewall — block known-bad IPs and bots before they hit the app. Free, low-overhead.
2. WAF — managed rule sets for obvious attack signatures. Cloudflare's free tier is sufficient for most.
3. OTel-based ASM — long-tail detection, forensic depth, custom queries.
4. Optional RASP — for high-stakes endpoints where inline blocking is required.

Most teams under 200 engineers do layers 1, 2, and 3. Layer 4 is for regulated industries.

Practical setup

If you're starting fresh:

npm install securenow
node -r securenow/register app.js

The package wraps OpenTelemetry, adds the firewall (layer 1), captures bodies and headers with redaction, ships pre-configured ASM detection rules, and routes signals to a free dashboard. 5-minute setup. See the APM + security in one tool writeup for the architecture.

If you're integrating with an existing OTel pipeline:

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
# point your collector at SecureNow's OTLP endpoint, or any OTel destination

Either way, the data model is the same. The destination determines the depth of detection and dashboard.

Related

• Application security monitoring guide
• Continuous ASM on a $0 budget
• APM + security in one tool