> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcargo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Scheduled threshold checks over your span telemetry that fire actions when a metric breaches. Define them with defineAlert.

An **alert** is a scheduled check over your [span telemetry](/deploy/monitoring#spans-and-traces). Every tool run, play batch, and agent message emits spans, and an alert watches a slice of them: on each tick it computes one metric over the spans since it last ran, compares it to a threshold, and — when the threshold is breached — fires its actions, each as its own run. You define one with `defineAlert`.

## Define an alert

```ts alerts/error-spike.ts theme={null}
import { defineAlert } from "@cargo-ai/cdk";

import { sentinel } from "../agents/sentinel";
import { slack } from "../connectors/slack";
import { enrich } from "../tools/enrich";

export const errorSpike = defineAlert("error-spike", {
  description: "Error rate of the enrich workflow",
  schedule: { type: "cron", cron: "@every 5m" }, // when the check runs
  scope: { kind: "spans", workflow: enrich }, // which spans it watches
  threshold: { metric: "errorRate", operator: "gte", value: 10 }, // when it breaches
  actions: [
    {
      ref: slack.actions.postMessage,
      config: {
        channelId: "C0123456789",
        format: "markdown",
        body: "Enrich error rate hit {{event.value}}% — {{event.spansUrl}}",
      },
    },
    { ref: sentinel, config: { prompt: "Investigate {{alert.url}}" } },
  ],
});
```

`schedule` takes a 5-field cron expression or an `@every` interval (`@every 5m`, `@every 1h30m`), evaluated in UTC. An alert evaluates **at most once a minute** — every tick scans your spans, so sub-minute intervals are rejected. In the CDK `enabled` defaults to `true`, so a deployed alert is armed; set it to `false` to deploy one without arming it. `folder` files the alert under an `"alert"`-kind [folder](/folders/overview).

## Scope: what it watches

The scope's `kind` names the data source. Each source pairs with its own metric menu, so a metric can never be asked of a source that can't compute it.

### Spans scope

`kind: "spans"` watches span telemetry directly. Use any subset of these filters; omitted fields don't narrow. They mirror the **Spans** view.

| Field                          | Narrows to                                                              |
| ------------------------------ | ----------------------------------------------------------------------- |
| `workflow`                     | one play or tool's workflow — a handle, or `workflowRef(uuid)`          |
| `parentAgent`                  | the spans of runs one agent spawned                                     |
| `nodeKind`                     | `native`, `connector`, `tool`, or `agent` nodes                         |
| `integration`                  | one integration slug                                                    |
| `connector`                    | one connector                                                           |
| `action`                       | one action slug                                                         |
| `tool`                         | one tool *node*                                                         |
| `agent`                        | one agent *node*                                                        |
| `executionTitleOrErrorMessage` | spans whose title or error message contains the text (case-insensitive) |
| `executionStatuses`            | `pending`, `success`, and/or `error`                                    |
| `userUuid`                     | runs started by one user                                                |

<Note>
  `parentAgent` and `agent` are different filters. `parentAgent` matches spans
  of runs an agent *started* — the agent-trigger case. `agent` matches an agent
  *node* running inside the watched spans.
</Note>

<Warning>
  `workflow` takes a play/tool handle or `workflowRef(uuid)` — not `toolRef` or
  `agentRef`. Spans are keyed by the workflow behind a tool, not by the tool's
  own uuid, so passing the wrong kind of reference is rejected at deploy time
  rather than producing an alert that silently matches nothing.
</Warning>

### SQL scope

`kind: "orchestrationSql"` runs read-only SQL over your orchestration data — the `spans`, `runs`, `batches`, and `records` tables, automatically scoped to your workspace. The query computes the value itself, so the threshold carries only the comparison:

```ts theme={null}
export const slowNights = defineAlert("slow-nights", {
  schedule: { type: "cron", cron: "0 * * * *" },
  scope: {
    kind: "orchestrationSql",
    sql: `select count(*) from spans
          where execution_status = 'error'
            and execution_started_at > now() - interval 1 hour`,
  },
  threshold: { operator: "gte", value: 50 },
  actions: [
    { ref: sentinel, config: { prompt: "Error spike: {{event.value}}" } },
  ],
});
```

<Warning>
  SQL scopes are **not** windowed for you — Cargo runs the query exactly as
  written. Make it self-windowing (as the `interval 1 hour` above does),
  otherwise every tick evaluates your whole history.
</Warning>

The value is the **first column of the first row**, and it must be numeric. A query returning no rows, a `NULL`, or a non-number produces an `error` event rather than a value — so an empty aggregate can't quietly read as `0` and breach an `lte` threshold.

## Threshold: when it breaches

`operator` is `gte` (breach at or above `value`) or `lte` (breach at or below). For a spans scope, `metric` says what is measured — it is required, since it decides what the value means. A metric that supports aggregations requires one too:

| `metric`    | Value                                                                   | `aggregation`              |
| ----------- | ----------------------------------------------------------------------- | -------------------------- |
| `errorRate` | failed spans as a percentage (0–100) of the window's **finished** spans | —                          |
| `duration`  | span duration in seconds, over finished spans only                      | `avg`, `p50`, `p95`, `p99` |
| `credits`   | credits consumed by the window's spans                                  | `sum`, `avg`, `p95`        |
| `count`     | number of spans in the window                                           | —                          |

<Tip>
  `count` with `lte` is a dead-man's switch: an empty window really evaluates to
  `0`, so **silence breaches**. `{ metric: "count", operator: "lte", value: 0 }`
  tells you a workflow stopped running at all. The other metrics treat an empty
  window as nothing to judge, not as zero — and for `errorRate` and `duration`,
  "empty" means no *finished* spans, so a window of runs that are all still
  going is not judged yet either.
</Tip>

## Actions: what fires on breach

Each action becomes its own run, exactly like a play's `healthAlertActions`. An action is a connector action, an agent, or a tool, and its `config` is the input it runs with:

```ts theme={null}
actions: [
  { ref: slack.actions.postMessage, config: { channelId: "C0…", body: "…" } },
  { ref: sentinel, config: { prompt: "…" }, waitUntilFinished: true },
  { ref: enrich, config: {} },
]
```

An **agent**'s `config` is typed already — every agent takes the same `{ prompt, output? }`, so a misspelled key is an editor error in the literal above with nothing to import.

A **connector action** or a **tool** has an input of its own, and a bare object literal leaves it unchecked. Wrap it in `alertConnectorAction` / `alertToolAction` to have TypeScript check `config` against the real thing:

```ts theme={null}
import { alertConnectorAction, alertToolAction, defineAlert } from "@cargo-ai/cdk";

actions: [
  alertConnectorAction({
    ref: slack.actions.postMessage,
    config: { channelId: "C0…", body: "Error rate {{event.value}}%" },
  }),
  alertToolAction({ ref: enrich, config: { domain: "acme.com" } }),
]
```

Misspelled and missing fields become editor errors instead of deploy failures, and every field still accepts a `{{ … }}` template string — so a numeric input can be bound to `{{event.value}}`. Where the two differ is the source of the schema: a connector action's comes from [`cargo-ai cdk types`](/get-started/project-layout), so an integration you haven't synced keeps the loose object, while a tool's comes from its own `defineWorkflow` input and needs no sync — but a `toolRef(uuid)` names a tool you didn't author here, so that one stays loose. Both are helpers rather than the type of `actions` because TypeScript can't infer a per-element type through an array literal, the same reason `agentConnectorTrigger` is one.

`config` is interpolated against the firing under two roots — `alert` is what you configured, `event` is what this firing measured — so an action can say what happened:

| Variable                                              | Value                                       |
| ----------------------------------------------------- | ------------------------------------------- |
| `{{alert.name}}` / `{{alert.uuid}}` / `{{alert.url}}` | the alert that fired                        |
| `{{event.value}}`                                     | the computed value, rounded to two decimals |
| `{{event.threshold}}`                                 | the threshold it was compared against       |
| `{{event.operator}}`                                  | `gte` or `lte`                              |
| `{{event.windowStart}}` / `{{event.windowEnd}}`       | the evaluated window, as ISO timestamps     |
| `{{event.spansUrl}}`                                  | link to the workspace's Spans view          |

## The evaluation window

An alert does **not** re-scan a fixed lookback on every tick. Each evaluation covers the time since the previous one, so windows are contiguous and never overlap and every span is judged exactly once:

* **Window start** — where the last evaluation ended. The very first evaluation starts from the moment the alert was last saved.
* **Window end** — slightly behind now, by an allowance for span indexing lag. Spans that land late are picked up by the next tick instead of being missed.

This is why the cron is the window size: `@every 5m` means each evaluation judges roughly the last five minutes.

<Note>
  Actions fire **at most once** per window. Before firing, an alert atomically
  claims its window; if a retry or an overlapping tick already claimed it,
  nothing is recorded and nothing fires. Actions spawn runs that spend credits
  and can take real action, so a duplicate is worse than a rare miss — and a
  sustained breach is detected again on the next tick anyway.
</Note>

Disabled alerts are skipped, and every evaluation — breach or not — records an [event](/alerts/events).

## From the CLI

```bash theme={null}
cargo-ai observability alert list
cargo-ai observability alert get <alert-uuid>

cargo-ai observability alert create \
  --name "Enrich error rate" \
  --cron "@every 5m" \
  --scope '{"kind":"spans","workflowUuid":"<uuid>"}' \
  --threshold '{"metric":"errorRate","operator":"gte","value":10}'

cargo-ai observability alert update --uuid <alert-uuid> --enabled false
cargo-ai observability alert remove <alert-uuid>
```

`create` also takes `--actions` (a JSON array), `--description` and `--folder`.
On `update`, passing `none` to `--description` or `--folder` clears the field,
while omitting the flag leaves it untouched.

<Note>
  Unlike `defineAlert`, an alert created through the API or the CLI starts
  **disabled** — arm it with `alert update --uuid <uuid> --enabled true` once
  its scope and threshold look right.
</Note>

Try a scope and threshold before you commit to it — `preview` computes the value
now and reports whether it would breach, without firing anything:

```bash theme={null}
cargo-ai observability alert preview \
  --scope '{"kind":"spans","workflowUuid":"<uuid>"}' \
  --threshold '{"metric":"duration","aggregation":"p95","operator":"gte","value":30}' \
  --window-minutes 60
```

`--window-minutes` (default `60`) is the lookback for spans scopes; a SQL scope
windows itself, so the flag doesn't affect it.

## Using the UI

See [Using the UI](/alerts/using-ui) to build an alert visually, with a live preview of the spans it matches, and [Events](/alerts/events) for reading an alert's history.

An alert built in the UI can be brought back into code: `cargo-ai cdk pull` writes it as a `defineAlert` under `alerts/`, with its scope, threshold and actions referencing the other resources it was pulled alongside.