Skip to content

Frame, view & page schema

A route is authored as two files at the same id: a frame, *.frame.json, whose view says what the data means, and a page document, *.page.json, whose page says how to display it and whose title names the route. This is the schema reference for both shapes; for how to think in views and pages, see Views and Pages & templates.

Every authored field is snake_case. Editors autocomplete and validate frames from the generated JSON Schema at .data/schemas/frame.json. The CLI writes the schema path into installed frames. Use editor autocomplete for the complete list of nested fields.

Frame

Here is a whole, minimal frame — a view over a table, defining a metric by name:

{
"view": {
"tables": {
"traces": {
"table": {
"from": "otel_traces",
"timestamp": "Timestamp",
"with": {
"ErrorRate": "countIf(lower(StatusCode) = 'error') / nullif(count(), 0)"
}
}
}
}
}
}

And the page beside it, reading that metric by name:

{
"title": "Services",
"page": {
"@block/stat": {
"title": "Error rate",
"value": {
"@expr/query": "SELECT ErrorRate FROM traces"
},
"format": "percent"
}
}
}

The top-level fields of a *.frame.json:

FieldRequiredPurpose
viewyesThe data half — tables, imports, and named expressions. See View.
scopenoThe route params this frame introduces, and a filter for its route and every route beneath it. See Scope.
settingsnoHow the route renders — today the default time window. See Settings.
templatesnoReusable blocks and expressions, split into blocks and expressions. #/block/… references resolve in the document that contains them. Put page templates in that page or in a shared *.templates.json file. See Pages & templates.
workflowsnoNamed workflows over the frame’s view. A workflow uses { title, schedule, execute } or { title, schedule, use }, with optional enabled, timeout, and version fields. See Workflows, Run deadlines, and Workflow state.
$schemanoPath to the generated frame JSON Schema; auto-written into installed frames.

A frame holds neither a page nor a title: the *.page.json beside it carries both — see Page — and its title is what names the route. A frame with no page beside it renders nothing: it exists to be imported or borrowed by others, and doesn’t appear in frame search.

Settings

settings holds how a route renders, as opposed to what it means (view) or which entity it addresses (scope). Today it has one field:

FieldPurpose
default_timerangeThe route’s initial time window, relative to now.
{
"title": "OpenTelemetry",
"settings": { "default_timerange": { "from": "now-24h" } },
"view": { "tables": {} }
}

Settings are inherited down the route. A frame at @opentelemetry covers every route beneath it — @opentelemetry/services/checkout, @opentelemetry/traces/{TraceId}, and any *.page.json or *.md file that lives under them — so a pack declares its window once at the root. A deeper frame that declares its own wins for its own subtree, and a *.page.json may declare settings too (applied over the frame chain it borrows its view from). Merging is per key, nearest-wins.

default_timerange is only the initial value: it seeds the URL-synced from/to, so a link that already carries a window wins, the picker stays writable, and following a link from one frame to another keeps the window you were looking at. With nothing declared anywhere in the chain, the window is the last 15 minutes.

View

The view says where rows come from and what they mean. It has one required field, tables, plus optional imports, view-wide where, and with:

{
"view": {
"tables": {
"traces": {
/* one table definition, keyed by its alias */
}
},
"imports": {
"@opentelemetry/views/combined": {
/* one import, keyed by frame name */
}
},
"where": "IsServerSpan",
"with": { "ErrorRate": "ErrorCount / nullif(RequestCount, 0)" }
}
}
  • tables — direct database tables, keyed by the CTE alias blocks reference (required; use {} when a frame only imports). See Table.
  • imports — other frames’ views spliced in, keyed by frame name. See Import.
  • where — a SQL predicate applied to every source, scoping the whole frame.
  • with — named scalar expressions (derived columns and metrics) every block can select by name, applied across all sources.

There is no view-level select, group_by, or scalars — selecting and grouping happen in the blocks’ @expr/query; the view only names data. A with entry is a bare SQL string, or one of the tagged forms: { "value": { "expression", "title?", "format?" } } for SQL with presentation metadata, or { "counter" | "level" | "rate": … } for a measure — a metric reading declared by what it is, whose SQL the engine derives.

Table

Each entry in tables is keyed by its alias — the handle blocks reference in their from. All fields are optional:

  • The entry is tagged by kind: table for a table of rows, timeseries for a table whose rows are samples of series (the only kind that accepts a measure).
  • from — what this alias reads from, resolved alias-first: a name matching another alias in the view (this frame’s own or one an import contributed) derives from it, inheriting that alias’s where, timestamp, and scalars and refining them ("api_requests": { "table": { "from": "events", "where": "…" } } — see Derived tables); any other name is a physical database table. Defaults to the alias key, but prefer a semantic alias distinct from the physical name — the shipped packs all use one, so a physical name like otel_traces always means the raw table.
  • where — a predicate for this source alone, combined with the view-level where and, on a derived table, with every ancestor’s.
  • with — scalars scoped to this source. On a derived table these are added to the inherited ones; a repeated name replaces the inherited definition for this alias only.
  • timestamp — the column used for time bucketing. Inherited from the parent by a derived table.
  • On a timeseries entry only: identity (what makes two rows samples of the same series) and timestamp are required, plus an optional accumulation_start (when a series began accumulating, which also marks resets). Inherited by derived tables.
  • disable_auto_scope — set true to opt this table out of the route scope (the param comparisons and every scope.where) when it doesn’t carry the columns they read.

Alias keys must start with a letter or underscore and may contain letters, digits, _, -, and . (so system.parts is a legal defaulted alias).

Import

Each entry in imports is keyed by the imported frame’s name. An import builds on an installed semantic layer — { "imports": { "@opentelemetry/views/combined": {} }, "tables": {} } — inheriting that view’s scalars and scoping and AND-composing your where onto it. An import accepts:

  • tables — a list of the imported frame’s aliases to splice in; omit to import all. Derivations resolve before the filter applies, so selecting a derived alias without its parent still carries the parent’s scoping and scalars — the parent is simply not exposed.
  • params — values for the imported frame’s own scope.params, keyed by param name: { "params": { "ServiceName": "checkout" } }. Every declared param is required. Each imported table receives the imported frame’s scope unless it sets disable_auto_scope. The imported scope adds param comparisons and its scope.where. Ancestor scopes do not apply to the import, so an imported scope.where must not name an ancestor’s param.
  • where / with — a predicate / scalars folded into every table spliced from this import.
  • disable_auto_scope — propagate the route-scope opt-out to every spliced table.

Page

A *.page.json is what makes a route render. Its page is a tree of blocks, each node a single-key object like { "@block/stack": { … } }, typically rooted at @block/page. Any block is accepted: an @block/page renders as-is, and any other block is wrapped in @block/page automatically. A frame with no page beside it renders nothing — it exists to be imported or borrowed.

The top-level fields of a *.page.json:

FieldRequiredPurpose
titleyesRoute title. Supports Handlebars, e.g. "Logs: {{ServiceName}}".
pageyesThe block tree to render.
routesnoThe sections this page can show inside its @block/outlet, and which one its own URL shows. See Sections.
settingsnoHow this route renders, applied over the frame chain it borrows its view from. See Settings.
templatesnoReusable blocks and expressions this page’s #/block/… refs resolve against. A ref never reaches the frame’s own templates.
$schemanoPath to the generated page JSON Schema; auto-written into installed pages.
{
"title": "Latency",
"page": {
"@block/stat": {
"title": "p95 latency",
"value": {
"@expr/query": "SELECT DurationP95 FROM traces"
},
"format": "duration"
}
}
}

A block queries the view rather than the raw table, so the frame’s scope, parameters, and the page’s time range fold into every query automatically:

{
"@expr/query": {
"from": "traces",
"select": ["RequestCount", "ErrorRate"],
"where": "IsServerSpan"
}
}

Sections

A page’s routes splits it into sections, each with its own URL under the page’s. children maps a section key to a title and a block; default names the one the page shows at its own URL. The page puts an @block/outlet where the section renders, and everything around that slot — a header, a tab strip, a filter bar — stays on screen across sections.

{
"title": "Checkout",
"routes": {
"default": "overview",
"children": {
"overview": { "title": "Overview", "block": { "@block/use": "#/block/overview" } },
"traces": { "title": "Traces", "block": { "@block/use": "#/block/traces" } }
}
},
"page": { "@block/tabs": { "children": ["overview", "traces"] } }
}

@block/tabs’ children arm reads the titles and URLs out of routes.children and renders the outlet beneath itself, so the sections are declared once. A page that wants its own content above the sections, or the outlet somewhere the strip is not, writes an @block/outlet where it wants the section and navigates to it however it likes.

A section key is a URL segment: letters, digits, _ and -, and no /. checkout/traces renders the page with the traces section in its outlet; checkout renders it with overview, so the two URLs show the same screen. A file-defined route wins a path a section also names, so adding checkout/traces/index.page.json takes that URL over.

A section belongs to its page:

  • One view, one runtime. A section renders against the view its page already compiled, so every section’s queries run against one runtime and a filter set in the page’s header reaches the section below it.
  • One file. A section is authored inside its page’s document, so editing any part of it reloads the whole route, and @block/use refs in a section resolve against the page’s templates.
  • No nesting. A section is a block, so it declares no routes of its own. An @block/outlet inside a section renders nothing.

Every surface that resolves a route resolves sections: noemata run checkout/traces, noemata screenshot checkout/traces, and the app’s URL all render the same tree, and noemata validate --online renders every section a page declares rather than only the default one.

Scope

scope.params names the route params a frame introduces. scope.where sets a filter for the frame’s route and its descendants. A frame at services/{ServiceName}/index.frame.json can render any service. Its {ServiceName} path segment binds the param, which scope.params lists by name. Noemata compares the param value to the view scalar or column with the same name:

{
"scope": { "params": ["ServiceName"] }
}

To compare a param’s value to a column with a different name, map the segment name to that column. The map can be the whole scope.params or an entry of the list:

{ "scope": { "params": { "Database": "database" } } }
{ "scope": { "params": ["HostName", { "Namespace": "k8s_namespace", "Service": "service_name" }] } }
  • Every param is a required string.
  • scope.params must name each {Segment} introduced by the frame’s path exactly once. Validation reports an extra name as param_not_in_path, an unnamed segment as segment_not_declared, and a duplicate name as param_duplicate_name.
  • Noemata adds a comparison to every table in the rendered view for each route param, including params from ancestor frames. The comparison uses the column declared in scope.params.
  • The URL or a parent frame supplies each param value.
  • A where can reference a param with a {Name:String} placeholder. The page context also exposes the value under the param name.
  • Set disable_auto_scope: true on a table that does not contain the columns used by the scope.

scope.where is added to each auto-scoped table on the route and its descendants. The filter also applies to child frames with their own views. A {Name:String} placeholder binds the route param Name. A frame can declare scope.where without scope.params; the filter then applies to its descendants.

For example, the following scope.where on services/ limits every service page beneath it to the shop namespace:

{ "scope": { "where": "ServiceNamespace = 'shop'" } }

Entity search and validate --online param sampling apply the scope.where of frames whose route binds no param. The services/ filter above also limits the services listed for services/{ServiceName}.

You can restate a param comparison in scope.where to help the database skip data. The filter must match every row selected by the param comparison. Otherwise the filter removes valid rows from the page.

view.where and scope.where apply to different tables:

  • view.where filters every table of this frame’s own view, including tables that set disable_auto_scope. A child route that declares its own frame doesn’t receive it.
  • scope.where filters the views of this frame and its descendants. Tables with disable_auto_scope skip this filter.

Overlays (*.frame.local.json, *.frame.overrides.json) cannot change scope, because it defines which URLs the frame serves.

A concrete route shadows its parameterized sibling. Put services/redis/index.frame.json next to services/{ServiceName}/index.frame.json and every route to services/redis — URL, @block/link, @block/frame, the CLI — renders the concrete frame instead. Static segments are matched before the {Param} one, with backtracking, so deeper paths the override doesn’t define (services/redis/traces/{TraceId}) still fall through to the parameterized subtree. This is how you give one entity a hand-built page while the rest keep the generic one, and it works inside a managed integration pack — your file is yours, and version-controlled, even though the pack’s own files are not.

Two rules follow from the route still being parameterized underneath:

  • The override declares no scope.params — the segment is a literal, so declaring one is a param_not_in_path error.
  • The shadowed parameterized frame still binds ServiceName = "redis" and applies its scope to the concrete route. The scope includes the declared column and scope.where. A scheduled workflow beside the concrete route uses the same scope as its page. A table without the declared column should set disable_auto_scope: true.

Give every entity page a listing above it. A parameterized frame at services/{ServiceName}/ leaves services/ a bare namespace: it appears in the navigation as a folder you cannot open, and there is no URL for “all services”. Add services/index.frame.json — an unparameterized frame whose page is the inventory table, each row linking on to services/{ServiceName} — and the route becomes navigable, linkable, and the natural parent of the entity page. Its settings are inherited by every route beneath it, and the entity page keeps its own view (the deepest frame on the route is always the view source).

Standalone views and pages

The same building blocks exist as standalone files:

  • *.view.json — a view with no page, referenced by other frames via imports.
  • *.page.json — a page on its own; it borrows the nearest frame’s view for its data model.
  • *.workflows.json — a bundle of named workflows ({ "workflows": { "<name>": … } }), addressable as <path>#<name>. Each definition has execute or use (a workflow template reference plus parameters). Inline execute is { steps: … }, the general checkpointed pipeline, or { detect: … }, a declarative detection. Validated and shipped in packs; never a route. An inline workflow first selects a frame with the same filename stem (foo.workflows.json → foo.frame.json, index.workflows.json → index.frame.json), then falls back to the nearest frame on its containing folder’s route. A frame-embedded workflow uses its own frame, so its queries speak the view’s aliases and scalars, and the run’s evaluation window bounds every table through the view’s timestamp. The view is resolved whenever a run is dispatched or resumed (the definition is pinned at start; the view is not); with no frame at or above the file, the workflow runs on a standalone runtime (raw SQL, no time scoping). That frame’s route must bind every param it has: a scheduled run has no URL to bind a {ServiceName} segment from, so validation rejects a scheduled workflow under one (group the evaluation by the column instead), while a concrete override such as services/redis binds its value and the run evaluates scoped to it. The server runs scheduled definitions on their cadence, catching up per catch_up after downtime.

Workflow state

@expr/workflow_state is the durable member of the state family (@expr/state, @expr/local_state, @expr/url_state): its medium is the workflow’s stored key-value map. As an expression it reads — the value stored for key as of run start, or defaults when absent — and as a pipeline step ({ "@expr/workflow_state": "key" }) it writes: the incoming value becomes the pending write for that key, and the flow passes through. Keys under noemata.* are reserved for the runtime (a detection’s alert machine carries its state at noemata.alert, an identity detection’s seen set at noemata.seen) and rejected by validation.

{
"execute": {
"steps": [
{
"@expr/query": {
"from": "services",
"select": { "as": { "ServiceName": "ServiceName", "value": "ErrorRate" } }
}
},
{
"@expr/query": {
"as": "c",
"full_join": {
"from": {
"@expr/workflow_state": {
"key": "state",
"defaults": { "@expr/empty_table": { "ServiceName": "string", "firing": "number" } }
}
},
"as": "p",
"on": "c.ServiceName = p.ServiceName"
},
"select": {
"as": {
"ServiceName": "coalesce(c.ServiceName, p.ServiceName)",
"firing": "if(coalesce(c.value, 0) > 0.05, 1, 0)",
"was_firing": "coalesce(p.firing, 0)"
}
}
}
},
{ "@expr/workflow_state": "state" },
{ "@expr/filter": { "where": "firing != was_firing" } }
]
}
}

The semantics that make read-modify-write across runs loop-free:

  • Reads are frozen for the whole run. Every read sees the map as it stood at run start; this run’s writes never feed back into it. A run is a pure function from the stored map to the stored map.
  • Writes accumulate and merge at completion. Last write per key wins; keys never written carry forward unchanged. Reading and writing the same key — as state above — is the intended idiom: the read is the previous occurrence’s value, the write is this one’s.
  • A table-valued default is spelled @expr/empty_table (column name → type), so a join against a first run’s empty state still sees typed columns.

State is exactly-once durable: the seed pins with the run like the definition, each write is a run checkpoint, and a resumed run replays recorded writes instead of recomputing. A completed run publishes its exported map as the workflow’s carried state (state.json under the defining file’s folder at .shared/workflows/<filename>/<workflow-name>/), which the next run seeds from; it expires with schedule_state_retention_days once a workflow stops running. The state belongs to the definition’s version (an integer, default 1): a run seeds only from state published under its own version, so bumping it starts the next occurrence from empty — the reset for an edit that changes what the state means (a new group_by, a renamed key) — and a run still in flight under the old version is abandoned on resume instead of completing into the old chain. See Versioning a workflow’s state. Writes are workflow-only and must be top-level steps; a read anywhere else in the pipeline (a join’s from, an effect’s message) is ordinary expression vocabulary. @expr/set_context is pure scoping in workflows exactly as in frames: nothing it publishes is persisted, and entry names may not start with _, which the runtime reserves for its own scope entries.

Emitting events

@expr/emit in step position turns each surviving input row into an OTel record on the deployment’s export pipeline — collector, the otel_logs/otel_metrics_* tables, and every consumer downstream. The kind key says what each row becomes: log (the default record type below), or one of the metric kinds. Like @expr/log it is a tap (the flow passes through) and an effect (workflow-only, with best-effort delivery and possible duplicates across a crash-resume). The row-set properties (where, order_by, limit) are SQL over the input table — that is where a query is being written. A value position (an attribute, time, body, service_name) states its meaning: { "literal": "fired" } is the word on every record, { "sql": "ServiceName" } reads or computes per row, and bare numbers and booleans are literals (they have nothing to disambiguate) — a bare string is invalid there, so the quoting trap ("'fired'") cannot be written. One emit is one query over the input: literals never touch it, order_by/limit, plain column reads, and supported where comparisons run in memory. Unsupported predicates or computed SQL values send the table to the database, once per emit per occurrence. See query steps for the supported predicates.

{
"execute": {
"steps": [
{
"@expr/query": {
"from": "services",
"select": { "as": { "ServiceName": "ServiceName", "value": "ErrorRate" } }
}
},
{
"@expr/emit": {
"log": {
"event_name": "acme.error_rate.breach",
"where": "value > 0.05",
"severity": "warn",
"entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } },
"attributes": {
"kind": { "literal": "fired" },
"value_pct": { "sql": "value * 100" }
}
}
}
}
]
}
}

The row set and values:

  • where filters the input rows; order_by and limit cap them after it.
  • time is the per-row record time and defaults to the run’s occurrence window end. A literal is a fixed instant (an ISO { "literal": … } string or bare epoch milliseconds); a SQL value reads type-aware — a temporal column as epoch milliseconds, a bare DateTime (uint32 on the wire) as epoch seconds, a string as a parsed date. A time that cannot be read fails the emit.
  • severity defaults to info. A bare string must be one of debug/info/warn/error (schema-checked); a per-row severity is spelled { "sql": "if(value > 0.1, 'error', 'warn')" } and must evaluate to one of the four.
  • body is a literal, { "sql": … } per row (a map or list keeps its structure), or { "handlebars": "{{service}} exceeded {{value}}" } — rendered per row, its variables reading the input columns by name, in memory.
  • id gives each record its dedup identity, stamped as noemata.event.id — what a consumer deduplicates on, since replay can duplicate delivered records. It is { "sql": … } only (one literal id would collapse every record into one identity) and defaults to a hash of the emitted values and time plus an ordinal among identical rows, so a crash-replay re-sends identical ids and genuine duplicates stay distinct.
  • event_name defaults to noemata.extract, so a free-standing emit’s records land on the shipped extractions view.
  • Records carry the emitting server’s resource; an authored resource_attributes map (or the service_name sugar for its service.name entry) replaces that identity per row, and rows with distinct evaluated maps group into distinct resources.

multi shares one evaluation across several records per row: the shared where/order_by/limit/time/attributes/resource_attributes apply first, and each child in events narrows the row set with its own where (ANDed) or overlays time and the maps per key — one query, one upload of the input table, however many children:

{
"@expr/emit": {
"multi": {
"entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } },
"events": [
{ "log": { "event_name": "acme.breach", "where": "value > 0.05" } },
{ "log": { "event_name": "acme.evaluated", "attributes": { "value": { "sql": "value" } } } }
]
}
}
}

The runtime adds the record identity: noemata.event.id, noemata.workflow.run_id, and noemata.workflow.occurrence as record attributes, with the workflow identity (noemata.workflow.ref, noemata.workflow.step, and noemata.workflow.hash) on the instrumentation scope — see the record conventions. Authored attributes may not use event.name or noemata.* (validation rejects them), and authored resource attributes may not use telemetry.sdk.*. Records that follow the conventions land on the shipped alerts, changes, and extractions views.

Metric kinds

Three sibling kinds emit OTLP metric data points on the same pipeline: metric_sum (a monotonic delta sum), metric_gauge (an instantaneous reading), and metric_histogram (a pre-aggregated count and sum with no bucket boundaries — one +Inf bucket, no distribution claim). Each carries the metric identity — a required name plus optional description and unit; points sharing all three merge under one metric entry per batch — and its readings as bare SQL text over the input table: a metric reading is inherently per row, so there is no literal/SQL ambiguity to state ("value": "errors" reads the column, "value": "1" is the constant). The row context (where, time, attributes, resource_attributes, service_name) works as on log, and a metric kind can stand alone or sit beside log children in a multi.

{
"@expr/emit": {
"metric_sum": {
"name": "acme.request.errors",
"unit": "1",
"value": "errors",
"attributes": { "deployment": { "literal": "prod" } }
}
}
}
  • A point’s time defaults to the occurrence window’s end, and on the delta kinds (metric_sum, metric_histogram) start_time defaults to the window’s start — the evaluation bucket. Authoring a per-row time on a delta kind requires start_time too, so backfilled points carry their own bucket bounds and no two delta intervals overlap.
  • On a detection-hosted emit, a metric child gets the detection’s group_by columns stamped as data-point attributes by column name — the identity of a recorded metric series — and only those (the log fold’s every-output-column default would fork the series per reading). Authored maps overlay per key, and a numeric attribute always encodes as a double.
  • Metric data points carry none of the runtime record attributes — noemata.event.id, noemata.workflow.run_id, and noemata.workflow.occurrence would each fork the series identity (per point, per run, per occurrence). The workflow and detection identity travels on the instrumentation scope, and metrics need no dedup id: a replayed batch lands as identical rows, which the max-per-(identity, time)-then-sum read absorbs.

Detections

execute: { detect: … } is the declarative sibling of steps: one variant object describing what to evaluate each occurrence, which the executor evaluates as one step: query, then emission and successful state publication. Failed or crashed occurrences are not replayed; subsequent occurrences use the last committed detector state. A detection requires a schedule (the occurrence window is what it evaluates) and a configured OTLP collector (its records are its entire output — a run without one fails before evaluating anything). Every workflow record carries noemata.workflow.hash on its scope: a content hash of the authored execute definition, referenced block/expression helper definitions, literal template parameters, and evaluation schedule. Unused templates, the configured workflow reference, state version, and runtime context values resolved while executing do not enter the hash. Changing the hash does not reset carried state; version controls that reset.

Run deadlines

A scheduled run has an execution deadline equal to its current cadence interval. A run that consumes that interval reaches its next occurrence, and a stateful workflow starts accumulating lag. Set timeout to a shorter duration when that runtime already indicates a pathological state:

{
"title": "Check service health",
"schedule": { "every": "15m" },
"timeout": "2m",
"execute": { "threshold": { "from": "traces", "value": "count()" } }
}

The effective deadline is the shorter of timeout and the occurrence’s cadence interval. An unscheduled run uses timeout directly and otherwise defaults to 60 seconds. Exceeding the deadline during evaluation fails the current step with its pending node labels and cancels its in-flight work. Finalization overruns also fail the run; already-committed effects and carried state remain intact. Storage operations settle before an overrun can be recorded, so the deadline controls success rather than guaranteeing an exact return time. See Run execution and persistence for failure-persistence limits. Database request and statement timeouts are separate, nested limits; a stricter database timeout can fail an individual query before the workflow deadline.

Seven variants exist: extract copies matching source rows out as records; threshold alerts while an aggregate crosses a comparator; appears/disappears track identity sightings; value_change records exact observed value transitions per identity; changes detects statistical shifts in bucketed series; and burn_rate alerts on SLO error-budget consumption. Each variant is also a top-level execute key: execute: { extract: … } is sugar for execute: { detect: { extract: … } }, and both spellings run — and hash — identically.

extract

{
"execute": {
"extract": {
"from": "logs",
"where": "K8sClusterName != '' AND SeverityLevel >= 13",
"select": ["EventName", "Body", "SeverityLevel", "ServiceName"],
"limit": 100,
"alert": {
"levels": { "critical": "SeverityLevel >= 17", "warning": "SeverityLevel >= 13" }
},
"entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } },
"emit": { "log": {} }
}
}
}
  • from is a view table name, a structured sub-query, or a query expression. Query expressions include reusable @expr/use references that produce a table. where, select, and group_by shape the copied rows (with group_by, one summarized row per group). order_by defaults to source time (_ts) descending when limit caps per-row copies (a summarizing group_by select gets no order default — _ts does not survive the aggregation). A view-table source names its time column, so an authored select that leaves it out gets _ts added — the example above still records each row at its source time (a summarizing group_by select is left alone).
  • alert.levels is a stateless per-row classification: SQL predicates evaluated over the output row, highest level first (critical > warning > info), the winning level appended as the level column. The predicates enter the evaluation query as it stands — the one statement the detection already runs — and the level itself is computed in memory from their results. No lifecycle and no transition events: the copy is stamped, nothing resolves.
  • emit is optional. Omitted or true emits source-row events named noemata.detection.extract; false disables delivery. Custom emits replace defaults. Selected attributes, source time, Body, and source severity classification are preserved. In extraction SQL, selected source columns take precedence over generated severity and detection_kind aliases. Source EventName becomes noemata.extract.source.event_name; OTel severity carries the classification. Matched rows also receive noemata.alert.level and noemata.alert.condition; extraction has no lifecycle transition, persistent state, recovery event, or default alert gauge.
  • entity names the OTel entity the detection fires on — { type, id }, where type is a semconv entity type (service, host, k8s.pod, k8s.deployment, k8s.namespace, k8s.node, k8s.container, container, process, session, user, and the other workload kinds) and id maps that type’s semconv identifying attributes to the per-row values the detection reads. The emit path stamps the row’s resource with the identifying attributes under their semconv keys plus entity.type and a stable entity.id hash, so records land in their subject’s own OTel resource and otel_logs.ServiceName / HostName / K8sNamespaceName read as the subject. attributes / resource_attributes on the detection carry additional stamps in the same value model as emit attributes; on a threshold, an attribute’s SQL can read source columns and aggregates, which project into the evaluation query at the group grain ("run.error": { "sql": "any(LatestFailure)" }); every emit inherits entity, attributes, and resource_attributes, emit-level maps overriding per key — and the extract.* stamps overriding everything. Subjects without a semconv resource key (Noemata-native agent, database, mcp_server, pipeline, …) stay on attributes: { 'subject.type', 'subject.*' }; the identity chain has a subject.* fallback for them.
  • id is a per-row SQL dedup identity; omitted, each record’s id is a hash of its emitted values and time, so re-runs re-emit the same ids and the sink deduplicates.

threshold

{
"execute": {
"threshold": {
"from": "traces",
"group_by": "ServiceName",
"value": "EntryErrorRate",
"require": "EntryRequestCount >= 20",
"alert": {
"for": { "up": 2, "down": "10m" },
"levels": {
"warning": { "enter": { "above": 0.05 }, "leave": { "below": 0.03 } },
"critical": { "above": 0.2 }
}
},
"entity": {
"type": "service",
"id": { "service.name": { "sql": "ServiceName" } }
}
}
}
}
  • value is a SQL reading evaluated under group_by for each occurrence. Emit and recommendation SQL can read value, grouping dimensions, detection_kind, and generated severity. Alerting also exposes level, alert_transition, has_data, and the existing internal lifecycle columns. Use detection_kind for detector identity and alert_transition for lifecycle changes. Group dimensions cannot use detection_kind, alert_transition, severity, or has_data.
  • alert.levels are comparators as data, not SQL: { "above": n }, { "below": n }, { "between": [a, b] } (alert while inside the band), { "outside": [a, b] }. A bare comparator is the enter condition with leave defaulting to its negation; spelling both is hysteresis. A NULL reading satisfies no comparator: no level enters on it, the default leave treats it as the condition clearing, and an authored leave holds through it (its own comparator is not satisfied either). require is a guard ANDed into every level’s enter, never leave — a volume guard a comparator cannot hold.
  • The alert block is the stateful lifecycle: for gates persistence (up gates a quiet instance entering, down a firing one leaving; a count of occurrences or a duration judged against occurrence timestamps), level changes of an already-firing instance are immediate, and critical > warning > info. no_data says what a previously-seen instance absent from an evaluation means: resolve (default) treats absence as clear, hold freezes the level and every counter. limit caps tracked instances (admitted worst-first; rows it rejects still reach emits), and body overrides the generated record body.
  • alert.emit defaults to transition events named noemata.detection.threshold and the noemata.alert.level gauge. The gauge emits held level ranks 1–3 while firing and one final zero on resolution. Quiet instances produce no further points. Grouping and detection identity remain stable across level changes. alert.emit: false or [] disables these outputs.
  • An authored alert.emit replaces its defaults. Log children receive transitions, including missing-data resolution; metric children receive settled alert state. Authored names, body, severity, IDs, time, attributes, and resource attributes override their defaults. A shared multi context under alert.emit remains disallowed; place context on individual children.
  • Detector emit defaults to events plus threshold-value measurements without alerting, and measurements alone with alerting. false disables detector output; custom emission replaces its defaults. Custom detector logs receive transition rows when alerting is configured. Overriding event_name with a custom name excludes events from the default detection views; query logs with an explicit EventName filter. Custom detector metrics receive present evaluation rows, including healthy and admission-rejected groups.
  • order_by and limit bound the evaluation itself (deterministic top-k, a cost guard). A row trimmed here never reaches the machine: its instance reads as absent and follows no_data. Use alert.limit to cap tracked instances while preserving all groups for admission and recovery evaluation.
  • A threshold with an alert block is stateful: occurrences run in order, each seeding the machine from the previous occurrence’s carried state (readable at the noemata.alert state key). Bump the definition’s version when an edit changes what that state means (a new group_by, a different level set).

One-shot detectors (appears, changes, and value_change) preserve unmatched findings without alert attributes, using detected severity for changes and info for appears and value_change. Matching findings receive the alert classification once. Detector emit controls unmatched finding delivery; alert.emit controls classified delivery.

All seven detectors accept recommend: { body, name?, where?, attributes? }, with or without alerting. where evaluates after detection and alerting; false or NULL preserves the event without advice. A match adds noemata.recommendation.name and noemata.recommendation.body. Name defaults to the workflow reference. Recommendations preserve event identity, severity, and body.

body accepts a literal, { "sql": … }, or { "handlebars": … }. Threshold SQL can combine source aggregates with output columns: source dependencies project before grouping discards them, and enrichment finishes after alerting. Handlebars reads output columns such as {{value}}, {{level}}, and {{alert_transition}}. Missing recovery values are not filled from previous evaluations. attributes uses emit literal/SQL conventions; matching advice overrides detector attributes, and emit attributes override advice. Use custom attributes for estimates and units.

Identity detectors accept change: { name, of, from, to }. The optional name is a string; of, from, and to accept literal or SQL values. These fields decorate emitted records without changing identity or transition state.

appears and disappears

{
"execute": {
"disappears": {
"from": "hostmetrics",
"where": "MetricName = 'system.cpu.time' AND HostName != ''",
"group_by": "HostName",
"forget": "7d",
"alert": { "levels": { "warning": "misses >= 3" } },
"entity": {
"type": "host",
"id": { "host.name": { "sql": "HostName" } }
}
}
}
}

The identity pair shares one mechanism: a carried seen set of every identity sighted, per group_by combination (required, with at least one dimension: the identity is the mechanism), with sighting times read from the source’s _ts. Both are always stateful; the first run seeds the set with a single lookback over the horizon and registers every existing identity silently, so enabling an identity detection workflow on a live system records nothing. Neither takes order_by/limit: a truncated sighting set would read as absence.

  • appears records a noemata.detection.appears event when a new identity concludes its appearance, timestamped at its first sighting. remember (occurrences or a duration judged against occurrence timestamps; default 7d) is how long an absent identity stays known before its return counts as new again, and the seed horizon. The alert block is optional and one-shot — nothing resolves an appearance: for is a consecutive-sightings debounce the appearance must survive before it records (no_data: hold lets the count survive a missed occurrence), and levels are SQL predicates over the appearance row (the group columns, first_seen, last_seen) that classify it into a single fired record on alerts.
  • disappears alerts when a known identity stops being sighted — the heartbeat shape. Its rows expose misses (consecutive absent occurrences, available to SQL predicates) and last_seen_at to the level predicates; the alert’s for is the grace, and reappearance resolves. The detection’s conclusion of absence — the fire when it alerts, the first miss when it does not — records a noemata.detection.disappears event with the last sighting in noemata.disappears.last_seen, timestamped at the concluding occurrence. forget (default 7d) drops an absent identity from the seen set; under the default no_data: resolve the drop also resolves a firing alert, while no_data: hold keeps it firing past the drop until the identity returns. forget must outlast the alert’s for.up on the occurrence grid, or the detection could never fire — checked against the schedule before anything evaluates.
  • Level predicates lower to the same machine columns as threshold comparators, with the same NULL-safe default leave; the entered predicate lands whole under alert.condition on enter-kind records.
  • Top-level emit is optional and defaults to finding events without alerting. With alerting, alert.emit owns transition events. Custom detector logs also receive transition rows.

Identity initialization batches appears, disappears, and value_change by default. Batch duration follows the source frame’s inherited settings.default_timerange, resolved against the occurrence end; an absent or non-positive duration uses the fifteen-minute built-in default. Set initialization: { "max_source_range": "6h" } for a shorter or longer duration; the duration can be an expression. Set initialization: { "batch": false } for whole-horizon evaluation of derived sources. Current inventories, cross-boundary joins, ranking, and whole-window aggregation can require this opt-out. The runtime does not inspect SQL to choose. The detector checkpoints the contiguous processed range, accumulated identities, and chosen width, then yields between queries. Committed initialization progress retains the width and the original remember or forget horizon even if frame defaults change. Initialization emits no records and does not count slices as presence or absence occurrences. The completed seed preserves each identity’s earliest and latest sightings. This setting bounds the query time range, not its row count or memory.

value_change

{
"execute": {
"value_change": {
"from": "host_configuration",
"group_by": "Host",
"value": "tuple(Cores, Memory)",
"require": "Cores > 0 AND Memory > 0",
"remember": "7d",
"change": { "of": { "literal": "size" } }
}
}
}

The source must expose _ts. group_by identifies the entity; value is its watched scalar or tuple. The latest non-null, qualifying observation in each occurrence is compared with the retained value. require filters incomplete observations before selecting the latest value. Missing observations preserve state until remember expires (default 7d). Null tuple members require an explicit completeness predicate.

Initialization silently retains the latest observations over the remember horizon. A newly observed or expired identity seeds silently. Subsequent exact transitions, including A → B → A, emit noemata.detection.value_change with detector kind value_change, and typed noemata.value_change.from/noemata.value_change.to. Multiple changes within one occurrence collapse to its latest observation. The source timestamp is the record time. Equal source timestamps use the value’s string representation as a deterministic query tie-break; a conflicting value at an already retained timestamp fails the run. Older observations do not replace newer state.

Optional alert.levels classify transitions into one-shot alerts. Predicates can read group columns and the quoted identifiers "noemata.internal.value_change.value_before", "noemata.internal.value_change.value_after", and "noemata.internal.value_change.last_seen". change.name and change.of customize record labels. Initialization uses the shared batching defaults and opt-out described above; each batch retains the latest value per identity without emitting historical transitions.

changes

Use value_change for exact configuration transitions; use changes for statistically confirmed shifts in noisy numeric series.

{
"execute": {
"changes": {
"from": "traces",
"where": "IsEntrySpan AND ServiceName != ''",
"group_by": ["ServiceName", "ServiceVersion"],
"value": "quantile(0.95)(DurationMs)",
"window": "24h",
"kinds": ["step"],
"min_magnitude": 1,
"alert": {
"levels": {
"critical": "change_direction = 'up' AND change_magnitude >= 3",
"warning": "change_direction = 'up'"
}
},
"entity": {
"type": "service",
"id": { "service.name": { "sql": "ServiceName" } }
}
}
}
}

Records when a series changes regime. Each occurrence aggregates value per interval bucket (default: the schedule’s every) over the lookback window (required; at most 10,000 buckets), one series per group_by combination, and runs the @expr/changes detector over each series with the same knobs a chart’s changes annotations take (kinds, min_magnitude, min_segment_size, seasonality). Buckets are calendar-aligned in UTC, so a chart over the same view and interval shows the same detections. interval must divide a day evenly or be a whole number of days, as for a schedule’s every. fill says what a bucket with no source rows reads as: null (default) leaves it out of the series, 0 counts it as a zero reading — use 0 for counts and sums.

  • A change is confirmed once min_segment_size complete buckets follow its boundary, so a step confirms min_segment_size × interval after it happens (plus the schedule’s delay and phase). Each confirmed change records one noemata.detection.changes event timestamped at its boundary, with noemata.changes.of the authored value, noemata.changes.from/noemata.changes.to the levels before and after, noemata.changes.magnitude in the series’ own standard deviations, noemata.changes.direction, and noemata.changes.type (step, spike, dip, trend, distribution, non_stationary).
  • The detection’s output rows are the confirmed changes of the occurrence: the group columns plus the finding columns change_boundary (ISO), change_type, change_direction, change_magnitude, change_severity, change_value_before, and change_value_after, named after the change.* attributes they record as. A dimension may not use one of those names.
  • A cursor per series retains its last recorded boundary: a change already recorded never records again, and a boundary the segmentation moves by a bucket or two once more points follow it (within min_segment_size buckets of the cursor) is the same change. Bump the definition’s version to reset the cursors.
  • The alert block is optional and one-shot, as for appears: levels are SQL predicates over the output row ("change_direction = 'up' AND change_magnitude >= 3"), and each confirmed change that matches a level fires a single fired record on alerts with noemata.changes.magnitude, noemata.changes.direction, and noemata.changes.type. The instance is the series plus change_boundary, so two changes in one series are two alerts. There is no for and no no_data; limit, order_by, and body are the machine’s own.
  • Top-level emit is optional when alert is absent (a recording variant). With alert, alert-lifecycle emit lives under alert.emit — same shorthand as threshold. A top-level emit alongside alert composes with alert.emit.

Enable single-resolution materialization with source_retention and component_store:

Only component_store.catch_up.max_source_range is required for changes. repair_window defaults to five minutes, rounded up to a whole bucket. Set it explicitly when ingestion delays or backfills require longer repair. Finalized buckets do not incorporate later source arrivals. Burn rate uses the same defaults and additionally requires component_store.intervals.

Optional resource defaults for both detectors are retention_margin: "1h", max_pending_bytes: 16777216, and, inside catch_up, max_batches: 32, max_duration: "20s", max_rows_to_read: 10000000, max_bytes_to_read: 268435456, max_result_rows: 50000, and max_result_bytes: 8388608. Explicit values override these defaults.

max_duration controls admission of maintenance work. An admitted query has a separate 30-second deadline and may finish beyond that budget; its completed progress is checkpointed before further maintenance stops. Publication and parent-rollup phases also stop admitting work at their phase budgets. Evaluation reads use the independent query deadline. Workflow cancellation and deadlines still apply, and query errors remain failures.

{
"source_retention": "7d",
"component_store": {
"catch_up": {
"max_source_range": "1h"
}
}
}
  • interval is the stored resolution. Averages and quantiles are stored as bucket values, without combining buckets into wider aggregates. Values must fit finite Float64 readings; integer readings beyond exact Float64 precision are rejected. Group identities retain their source types.
  • Initialization queries bounded source ranges and checkpoints each admitted batch before yielding. Subsequent runs repair recent buckets and fill remaining historical gaps. source_retention must cover the repair window, one bucket, and the greater of the maintenance duration and 30-second query deadline. Choose it from the source’s actual retention.
  • repair_window and max_source_range must be multiples of interval. The lookback must leave at least 2 × min_segment_size + 1 finalized buckets outside repair.
  • Detection uses only the contiguous finalized prefix of the lookback. Missing coverage or repairable buckets stop that prefix. Confirmation therefore includes the repair delay as well as the detector’s evidence requirements. Too little finalized history holds the cursors and emits no findings.
  • Sparse bucket values are retained as workflow data during repair and publication. Include the metrics_gauge relation in the workflow’s view. Finalized values publish there as noemata.changes.component; payload hashes and row-count markers protect readback. Payloads are released only after matching readback. Missing published data later blocks coverage; it is not interpreted as a zero.
  • fill applies after assembling history. Missing source rows in a successfully queried bucket are distinct from missing materialization coverage. The checkpoint retains repair payloads, unpublished finalized payloads, publication manifests, coverage, and the source column schema alongside the change cursors.

burn_rate

{
"execute": {
"burn_rate": {
"from": "traces",
"where": "ServiceName != ''",
"group_by": "ServiceName",
"bad": "countIf(IsEntrySpan AND IsError)",
"total": "countIf(IsEntrySpan)",
"objective": 99.9,
"window": "30d",
"source_retention": "7d",
"component_store": {
"intervals": ["1m", "5m", "1h"],
"repair_window": "10m",
"retention_margin": "1h",
"max_pending_bytes": 50000000,
"catch_up": {
"max_source_range": "1h",
"max_batches": 4,
"max_duration": "30s",
"max_rows_to_read": 1000000,
"max_bytes_to_read": 100000000,
"max_result_rows": 100000,
"max_result_bytes": 10000000
}
},
"entity": {
"type": "service",
"id": { "service.name": { "sql": "ServiceName" } }
}
}
}
}

Alerts while an error budget burns faster than its objective allows. from is a view table name or a structured sub-query; a query expression is rejected, since it would evaluate once under the occurrence window and the trailing windows could not read past it. bad and total are count aggregates under the grouping (one SLO instance per group_by combination, or one ungrouped instance); objective is the target percentage, strictly between 0 and 100; window is the SLO window the budget spans, at least one day. A window’s burn is its error fraction divided by the budget (1 - objective / 100): burn 1 spends the budget exactly over the SLO window, burn 14.4 spends it in 1/14.4 of the window.

  • Without component_store, every occurrence reads bad and total over each trailing window of the lattice and over the SLO window, all ending at the occurrence end, in one statement. The output row per instance carries burn_long_1/burn_short_1 … burn_long_4/burn_short_4 (the four window pairs, fastest first), value (the highest long-window burn, the noemata.burn_rate.value on records), budget_remaining (one minus the SLO window’s burn; 1 is untouched, 0 exhausted, negative overspent), and exhausts_at (the timestamp the budget runs out at the current burn: the occurrence end once it is spent, NULL while nothing burns or when the projection lies past the year 2299). A dimension may not use one of those names.
  • The default lattice is the SRE workbook’s at a 30-day window, each long window scaled linearly to window and each short window a twelfth of its long: critical at 14.4 times the budget over 1h and 5m or 6 times over 6h and 30m, warning at 3 times over 1d and 2h or 1 times over 3d and 6h. A level enters while any of its pairs has both windows at or above the pair’s factor, so a short sharp burn and a long slow burn both fire and a blip does not.
  • Default gauges are noemata.burn_rate with numeric noemata.burn_rate.window_seconds for each distinct trailing window, noemata.burn_rate.max for the maximum long-window burn, and noemata.burn_rate.budget_remaining for the unspent budget fraction. All use unit 1.
  • alert is optional; omitted or false, the detector emits evaluation events and measurements. alert: true enables the default lattice’s levels. Authored levels are SQL predicates over the output row ("budget_remaining < 0.2", "burn_long_1 >= 10 AND burn_short_1 >= 10"), with the machine’s for, no_data, limit, and body as on threshold. Fires and resolves land on alerts with noemata.burn_rate.budget_remaining and noemata.burn_rate.exhausts_at (the ISO string of exhausts_at) beside noemata.burn_rate.value. There is no order_by or limit on the evaluation: it runs once per window, and a per-window cap would select a different instance set per window.
  • component_store replaces repeated long source scans with exact additive buckets. bad and total must each be one count() or countIf() aggregate. intervals are ascending fixed durations that each divide a day evenly or are a whole number of days; each interval must divide the next, and every lattice window must be divisible by the first. A 1m/5m/1h store sums final 1m buckets into 5m buckets, then sums twelve final 5m buckets into 1h buckets. Window evaluation selects the largest final buckets that fit and uses smaller buckets at its boundaries, so an unfinished hour does not delay current readings. All windows use the last base-interval boundary at or before the requested occurrence end; exhausts_at uses that same effective endpoint.
  • source_retention is required with component_store and declares the oldest source data maintenance may query. It must cover repair_window, one base bucket, and the greater of catch_up.max_duration and the 30-second query deadline. Every occurrence re-queries the repair window plus the bucket receiving its final read. A successful empty bucket records coverage without storing group rows, while a missing coverage range remains unknown. Final buckets are immutable and are the only buckets emitted as noemata.slo.bad, noemata.slo.total, and noemata.slo.complete metrics.
  • Catch-up reserves progress for publication, parent rollups, and source queries in that order. max_source_range bounds one source query; max_batches independently bounds publication partitions, parent jobs, and source batches; max_duration assigns the first third to publication, the next third to parents, and the remaining time to source work, with unused time available to later phases. Each query receives its remaining phase timeout. The four ClickHouse row, byte, and result limits reject work that exceeds its configured cost. A failed query keeps its range pending. The next occurrence uses the last deliberately committed batch; intermediate progress from a failed or crashed occurrence is discarded. schedule.catch_up remains independent: one current alert evaluation can fill a six-hour component gap across several occurrences without replaying 360 alert evaluations.
  • Final metric publication is at least once. The workflow keeps each finalized Arrow payload until bounded read-back sees its completion marker and both count points for every nonempty group; collector acceptance is not treated as a receipt. max_pending_bytes caps all currently referenced component payloads, including repair buckets, published children awaiting parents, and partial parents. Serialized sizes are checked before writes; only complete source buckets that fit are admitted. Publication and rollups continue to release space. Temporary query buffers and unreachable file versions awaiting garbage collection are outside this limit. While any required window has unknown coverage, the detector emits no partial reading and forces no_data: hold for that occurrence, preserving alerts and excluding the held interval from elapsed for timing.
  • Stored component state records source availability, the requested effective endpoint, per-window coverage bounds, oldest backlog age, source-expiry status, repair lag, and pending-publication bytes in the workflow’s carried state. Component generations hash the resolved view snapshot, source, filter, grouping, count expressions, and base interval. Objective, alert, emit, and SLO-window edits reuse compatible buckets; a source or view edit starts a new generation.
  • Finalized history is read from the database. Workflow data normally retains the repair window plus bounded publication, parent-rollup, and interrupted work. In the dense 10,000-group, 10-minute repair benchmark, workflow data used 4.00 MB and the carried checkpoint used 5.68 kB. The 30-day database query read 14.84 million component points into 10,000 grouped rows in 523 ms on local chDB; database CPU was 6.78 seconds across cores. Size database retention and query concurrency for the detector count and active group density.
  • burn_rate.alert accepts a boolean shorthand alongside the object form: alert: true opts into the SRE-workbook default lattice (14.4×/6×/3×/1× paired long+short windows scaled to window), alert: false or absent runs recording-only (SLO metrics and change records still emit; no lifecycle machine), and an authored AlertStateDefinition replaces the default. When alerting, the emit lives under alert.emit — same shorthand as threshold. A top-level emit alongside alert composes with alert.emit.

Workflow execution identity

Workflow queries do not run with the author’s database privileges. Scheduled runs execute under the deployment’s standing identity: the service_account/hybrid principal’s credential, or, in a deployment with authentication: "none", the anonymous identity every request already runs as (ClickHouse’s default user). Only a user principal has no standing identity, and runs no scheduler.

This makes workflow authorship a privileged operation: anyone who can commit a workflow file gets its queries executed with the standing identity’s grants. Two consequences for operating a shared deployment:

  • Treat workspace write access as the control point — review workflow changes the way you review code, because merging one is permission to run it.
  • Scope the service account’s ClickHouse grants narrowly: read on the telemetry databases plus write on Noemata’s own tables is enough. The service account does not need — and should not have — the union of your users’ privileges.

A per-workflow run_as (an explicitly declared identity resolved against stored credentials) is planned; execution identity will always be declared in the definition rather than inferred from who committed it.

See also