Investigate an attack by asking questions

Dana can't sign in. Her password stopped working this afternoon, and she thinks someone else got into her account. The whole story is in your telemetry, but it's spread across tables, and no single request looks like a break-in. Here we find out exactly what happened by asking questions in plain English: in the dashboard, from the terminal and through a coding agent, all in one conversation. Everything below is from a real run on tutorial/ep10-forensics, with securenow@8.15.

Evidence in four places

They don't share one key. Events know the user but not the request; requests know the address but not the user. So an investigation is a chain: a user leads to an address, the address to its requests, the requests to the data they touched. Each step needs the answer to the last one.

How the forensics chat works

1. It decides where the answer lives: events for accounts and sign-ins, traces for requests, logs for what your code wrote.
2. It learns your data when it needs to: small discovery queries first (which attribute keys exist, which event types).
3. It writes one read-only query, scoped to one app and one environment, and shows it to you. Only SELECT runs, and every query is checked before it runs. One that looks expensive waits for your OK.
4. It reads the rows back. Under every result, What this shows is written from the rows the query returned.
5. It remembers. Each answer, with its query, its findings, and the addresses, user agents and trace IDs in its rows, goes into the next question. A follow-up can simply say "that address".

The incident

The lab app, Northstar: five colleagues at work, Dana writing notes. ep10/incident.mjs plays a scripted account takeover against it, with ordinary traffic all around, and records the ground truth, so every answer below can be checked. The accounts are fictional and every address is from the documentation ranges.

All we start with is the support ticket and Dana's user ID.

Step 1 — The whole timeline, in one question

Open Forensics in the dashboard, pick the app and the environment, and ask:

A customer with user id d265e1af-55ab-40e0-9eea-86d7e00dc31e says she can't sign in and thinks someone else got
into her account. Show her sign-ins, failed sign-ins and account changes (password, role, invites), oldest first:
time, event, IP, user agent and reason.

One query, straight to the security events. Twelve rows:

A normal sign-in from her Mac, five failures from five addresses, then a sign-in from a new network. Twenty-six seconds later, from that same address, a password change and an invitation. Then Dana, three times, failing with the password that used to work. One question, and we have the shape of it.

Step 2 — Follow-ups that remember

That successful sign-in from the new address: had this account ever been used from that IP or that browser
before it? And did that IP ever fail a sign-in, on any account?

We never repeated the address: the chat took it, and the browser, from the last answer. Her account had never been used from either, and that address never failed a sign-in on any account. It got in on the first try, with her real password.

Those five failed sign-ins on her account: were they part of something bigger? Across the whole app, group
failed sign-ins by user agent: attempts, distinct IPs, distinct accounts, first and last time, and how many
sign-ins succeeded with that user agent.

One Windows Chrome 118 user agent: 120 failed sign-ins from 40 addresses against 6 accounts in two and a half minutes. Zero successes. That's credential stuffing, and it got nothing. The loud attack wasn't the one that got in.

Save to library, under the answer, keeps that query for the next time.

Step 3 — Continue from the terminal

The on-call engineer picks up the same conversation:

npx securenow forensics conversations
#  ID                         Title                                              App        Env       Msgs
#  6ae138668f1b2299a136c8f9   A customer with user id d265e1af-55ab-40e0-9eea…   lab-auth   preview   6

npx securenow forensics chat --conversation <id> "List every request from that new address, in order: time, method, route, status. How long did it all take?"

The terminal gets the same plan, SQL, rows and findings as the dashboard:

  Results (40 rows)
  time                            method   route   status
  2026-10-27 22:31:51.455000000   GET              200
  2026-10-27 22:31:51.457000000   POST             200
  …
  findings > The new address 203.0.113.252 made 40 requests, from 2026-10-27 22:31:51.455000000 to 2026-10-27 22:32:17.577000000, taking 26.122 seconds total. …

The route column came back empty: this app's spans don't fill it. So say so:

npx securenow forensics chat --conversation <id> "The route column came back empty. Show the request path instead (http.target), same order."

It edited its own query, and the whole sequence appeared. Condensed from the 40 rows:

Sign in, look around, try the admin pages, walk through note numbers, export, change the password, plant an invite. Forty requests in 26 seconds: that's a script, not a person clicking.

Step 4 — What it touched

npx securenow forensics chat --conversation <id> "Did any of it reach something her account isn't allowed to? From the permission checks for that address, count denied and granted by route and reason."

It went to the permission-check events and showed its query:

SELECT route, decision, reason, count() AS checks
FROM (
  SELECT
    attributes_string['route'] AS route,
    attributes_string['decision'] AS decision,
    attributes_string['reason'] AS reason
  FROM signoz_logs.distributed_logs_v2
  WHERE resources_string['service.name'] IN ('<app-key>')
    AND resources_string['deployment.environment'] IN ('preview')
    AND scope_name = 'securenow.events'
    AND attributes_string['event.type'] IN ('authz.denied', 'authz.granted')
    AND attributes_string['http.client_ip'] = '203.0.113.252'
)
GROUP BY route, decision, reason
ORDER BY checks DESC, route ASC, decision ASC, reason ASC
LIMIT 100

Three tries at admin routes, all denied. Nineteen note IDs came back not found. That's deliberate: the app answers "not found" for a note that isn't yours, so it never confirms the note exists. The Episode 6 fix held.

Now read the findings under it: they said 10 denials on admin routes. The rows say 3. The findings are a reading of the rows, and when they disagree, the rows win.

npx securenow forensics chat --conversation <id> "The 7 granted note reads: which note ids, and who owns those notes?"

Notes 739, 741, 742, 744, 746, 747 and 749, every one owned by Dana. One discovery query ran first, to find where note and owner IDs live: the permission-check events carry object_id and owner_id.

npx securenow forensics chat --conversation <id> "From the app's notes.export log lines: each export's time, its userId and note count from the log line, and the client IP of the request that wrote it."

That log line is the app's own:

console.info(JSON.stringify({ level: 'info', event: 'notes.export', userId: req.user.id, notes: rows.length, requestId: req.requestId }));

It has no IP, so the chat joined each line to its request by trace_id:

Her notes left, three times. The colleague exported their own notes from their own address. Nobody else's notes left.

Step 5 — Hand it to your coding agent

claude mcp add securenow -- npx securenow mcp

The instruction (ep10/agent-prompt.txt on the branch; bash ep10/agent-run.sh <id> runs it headless and read-only):

We're investigating an account takeover on our lab-auth app (SecureNow, preview environment). A teammate
started a forensics conversation in the SecureNow dashboard and the on-call engineer continued it from the
terminal; its id is <id>. Read it first, then continue THAT conversation with securenow_forensics_chat (pass
the conversationId) to finish the investigation:

1. Did the attacker leave anything behind that would let them back in after Dana's password is reset?
2. Anything we haven't covered yet that matters for the response?

Don't change anything: this is read-only. Ask the chat precise follow-up questions, check its findings
against the rows it returns, and when you're done write a short incident report for the ticket: how they
got in, from where, the timeline, what they touched, what left, what they left behind, and the response
you recommend (each action with the exact evidence behind it).

Claude Code read the conversation with securenow_forensics_conversation_get, read the app's code to understand its events, then asked the chat 13 more questions in the same thread. Precise ones, like:

Did any defence fire? Count securenow.events of type api.ratelimit.exceeded (by scope and client IP) app-wide,
no time window. And for the 120 Chrome/118 failures, the maximum number of failed sign-ins from any single IP
and the maximum against any single account within one 15-minute window.

What it found:

• A way back in that a password reset doesn't close. From Dana's account the attacker created an org and an admin invitation to an outside address. The app returns the invite link to whoever creates it, so the attacker has it. Still pending: no accept request, no membership.invite.accepted event. It expires seven days after it was created.
• Two more things a reset may not clear: the session issued to the attacker at the password change (probably expired, not proven), and a device-trust cookie for Dana's email, set at the attacker's sign-in and valid for 400 days.
• No defence fired: zero rate-limit events. The stuffing wave stayed under both limits: 3 attempts per address (the limit is 15 per 15 minutes) and at most 24 per account (lockout at 30).
• The other five targeted accounts had no sign-in after their own, and no account changes.
• Three findings that didn't match their rows, the "10 denials" among them. It went with the rows.

Then it wrote the incident report: how they got in, from where, the timeline, what left, what was left behind, and seven actions, each with its evidence. Revoke the invite and delete the org first. Then reset her password in a way that ends every session, invalidate the device cookie, tell Dana which notes were read and where else to change that password, block the addresses, warn the five other targeted users, and add detections, notifications and MFA for this shape of attack.

The MCP tools behind it: securenow_forensics_chat (pass conversationId to continue a conversation; it waits for the answer), securenow_forensics_chat_status, securenow_forensics_chat_confirm (runs a query the chat held back as large), securenow_forensics_conversations_list and securenow_forensics_conversation_get.

Step 6 — One conversation, three places

Back in the dashboard, the agent's questions sit in the same conversation, under the ones from the dashboard and the terminal. Ask for the trace IDs of the password change and the invite, and each one opens the request it came from: route, status, timing, and even the request body, with both passwords redacted before they ever left the app. The stuffing query is waiting in your saved library.

Reproduce it

1. Connect the lab to SecureNow (Episode 2) and give the run its own environment, for example a run folder with deploymentEnvironment: "preview" on port 3104 (Episode 7 shows the setup), so the investigation only sees this incident.
2. Run the incident: LAB_BASE=http://localhost:3104 node ep10/incident.mjs. It takes about 12 minutes and writes the ground truth to ep10/run/incident-summary.json.
3. Ask the questions above in the dashboard, or from the terminal with npx securenow forensics chat --app <app-key> --env preview "<question>", then --conversation <id> for each follow-up. --conversation <id> without a question opens an interactive session.
4. Hand it to your agent: claude mcp add securenow -- npx securenow mcp, then bash ep10/agent-run.sh <id>.

Check every answer against the ground truth. Ours matched on every row: the 120 failures, the 40 addresses, the 26 seconds, the 7 notes.

Asking good questions

• Name the columns and the order ("time, event, IP, user agent and reason, oldest first"). That's the table you get back.
• Point back at earlier answers: "that address", "those five failed sign-ins", "the 7 granted note reads". The chat carries them.
• Say what's wrong. An empty column or an odd count is one follow-up away from a fixed query.
• Go wide when something looks small. "Across the whole app, group by user agent" turned five failures into a 120-attempt wave.
• Check the rows. Every answer shows its query and its rows, and every query only reads. When the findings and the rows disagree, the rows win.

What you just did

You went from a support ticket to an incident report by asking questions: how they got in, from where, what they touched, what left, and the invite that would have let them back in after a password reset. One conversation, from the dashboard, the terminal and a coding agent. Ask the question you have. Then ask the next one.