AI Security Auditing, Part 5: Turn AI Findings Into Runtime Detections

AI Security Auditing series — Part 5 of 5. Part 1: Setup · Part 2: Authentication · Part 3: API & OWASP · Part 4: Injection & XSS · Part 5: Runtime detections

Through Parts 2–4 you found and fixed vulnerabilities in your code. This final part is what turns a one-time audit into lasting security posture: taking the other report — the detection-and-mitigation runbook — and making it real.

Because here's the thing every audit eventually runs into: you can't fix your way to safety. You'll miss an endpoint. You'll ship a new one next week. An attacker will find something before you re-run the audit. Code fixes close known holes; runtime detection catches anyone trying the doors — including the ones you haven't patched yet.

You'll need the CLI connected for this part, so if you skipped it in Part 1:

npm install -g securenow@latest
securenow login

The report you've been skimming

Every audit wrote two files. So far we've focused on <category>-code-findings.html. Now open <category>-detection-mitigation.html. It contains, for each threat, the detection logic in plain terms and the mitigation — rate-limit, challenge, block, or revoke.

Your job in this part is to translate that logic into running rules.

Detection as code

A SecureNow detection rule is a SQL query that runs on a schedule against your trace telemetry. The query returns the offending IPs; SecureNow aggregates them, raises a notification, and — if you've wired remediation — acts.

The reason to do this from the CLI rather than a dashboard is simple: detection rules are configuration, and configuration belongs in version control. When a rule lives only in a dashboard, you lose code review on the query, the diff when someone changes a threshold, and the ability to rebuild your posture in a fresh environment.

Let's build the magic-link brute-force rule the authentication audit (Part 2) recommended. Save the query as rule.sql:

SELECT
  coalesce(
    nullIf(attributes_string['http.client_ip'], ''),
    nullIf(attributes_string['net.peer.ip'], ''),
    nullIf(attributes_string['network.peer.address'], ''),
    nullIf(attributes_string['client.address'], '')
  ) AS ip,
  count() AS request_count
FROM signoz_traces.distributed_signoz_index_v3
WHERE resource_string_service$$name IN (__USER_APP_KEYS__)
  AND timestamp >= now() - INTERVAL 15 MINUTE
  AND (
    attributes_string['http.target'] LIKE '/api/auth/signin%'
    OR attributes_string['http.target'] LIKE '/api/auth/callback%'
  )
GROUP BY ip
HAVING ip != '' AND request_count >= 20
ORDER BY request_count DESC
LIMIT 100

Two conventions make it portable: __USER_APP_KEYS__ (SecureNow swaps in your scoped app keys, so the rule only ever sees your traffic) and an ip column (so matches can be grouped per IP and handed to remediation). Then create the rule:

securenow alerts rules create \
  --name "Auth: magic-link brute force" \
  --description "Single IP flooding magic-link signin or token callback" \
  --nlp "single IP sending too many /api/auth/signin or /api/auth/callback requests in 15 minutes" \
  --category auth \
  --severity high \
  --apps <your-app-key> \
  --sql @rule.sql

Validate it before trusting it:

securenow alerts rules test <rule-id> --app <your-app-key> --mode dry_run --wait

A complete status with no error means the query runs cleanly against your live data — even if it returns zero rows today, which is exactly what you want from a preventive rule.

From detection to response

Detection is half the loop. Wire the response, also from the CLI:

# Soft: rate-limit the abused route inline at the firewall
securenow ratelimit add --app <key> --route /api/auth/signin \
  --mode prefix --method POST --limit 10 --window 15m

# Hard: auto-block IPs your detection flags as high risk
securenow automation create --name "Block high-risk auth abusers" \
  --conditions '[{"field":"riskScore","operator":"gte","value":90}]' \
  --actions '[{"type":"addToBlocklist","config":{"ttlHours":72,"method":"ALL"}}]'

Now the vulnerability the AI found in Part 2 is fixed in code and watched in production — with a graduated response from rate-limit to block.

Blocking exploit traffic fast

For the loud, exploit-shaped threats from Part 4 — injection payloads, SSRF probes, scanner fingerprints — a scheduled rule that runs every 15 minutes is too slow. Those are the cases for synchronous blocking: an instant rule returns a 403 to the probing request in a couple of seconds rather than waiting for the next evaluation window. That's what protects the endpoint you haven't patched yet.

One rule per detection, one category at a time

Work through your audit reports the same way you ran the audits — one category at a time:

1. Open <category>-detection-mitigation.html.
2. For each detection, write the SQL (the report gives you the logic and the fields).
3. alerts rules create, then alerts rules test --mode dry_run.
4. Add the matching rate-limit or automation for response.
5. Commit rule.sql and the create command into your deploy scripts.

By the end, every threat category you audited has both a code fix and a live detection — the complete loop.

The complete picture

Step back and look at what you've built across this series:

• Part 1 — an AI agent set up to audit your code locally, nothing uploaded.
• Parts 2–4 — the vulnerabilities in your highest-risk categories found and fixed, each with a file:line.
• Part 5 — every finding backed by an always-on runtime detection, versioned as code, with graduated response from rate-limit to instant block.

That's what "auditing your app's security with AI" means when you do it completely: not a report you read once, but a posture you can rebuild from your repo — code hardened, attacks watched, and your source code never having left your machine to get there.

Where to go from here

• Re-read the overview: How to audit your app's security with AI.
• Go deeper on writing detections: Create custom alert rules from the command line.
• Audit the categories you haven't yet — browse all 25 free prompts. Cloud, secrets/IAM, supply chain, and AI/LLM features are common next steps.

You've closed the loop. Now keep it closed: re-run the relevant category audit on every release, and let the runtime detections cover the gap in between.