Frame, view & page schema
A route is authored as two files at the same id: a frame, *.frame.json, whose view says what the data means, and a page document, *.page.json, whose page says how to display it and whose title names the route. This is the schema reference for both shapes; for how to think in views and pages, see Views and Pages & templates.
Every authored field is snake_case. Editors autocomplete and validate frames from the generated JSON Schema at .data/schemas/frame.json. The CLI writes the schema path into installed frames. Use editor autocomplete for the complete list of nested fields.
Frame
Here is a whole, minimal frame — a view over a table, defining a metric by name:
{ "view": { "tables": { "traces": { "table": { "from": "otel_traces", "timestamp": "Timestamp", "with": { "ErrorRate": "countIf(lower(StatusCode) = 'error') / nullif(count(), 0)" } } } } }}And the page beside it, reading that metric by name:
{ "title": "Services", "page": { "@block/stat": { "title": "Error rate", "value": { "@expr/query": "SELECT ErrorRate FROM traces" }, "format": "percent" } }}The top-level fields of a *.frame.json:
| Field | Required | Purpose |
|---|---|---|
view | yes | The data half — tables, imports, and named expressions. See View. |
scope | no | The route params this frame introduces, and a filter for its route and every route beneath it. See Scope. |
settings | no | How the route renders — today the default time window. See Settings. |
templates | no | Reusable blocks and expressions, split into blocks and expressions. #/block/… references resolve in the document that contains them. Put page templates in that page or in a shared *.templates.json file. See Pages & templates. |
workflows | no | Named workflows over the frame’s view. A workflow uses { title, schedule, execute } or { title, schedule, use }, with optional enabled, timeout, and version fields. See Workflows, Run deadlines, and Workflow state. |
$schema | no | Path to the generated frame JSON Schema; auto-written into installed frames. |
A frame holds neither a page nor a title: the *.page.json beside it carries both — see Page — and its title is what names the route. A frame with no page beside it renders nothing: it exists to be imported or borrowed by others, and doesn’t appear in frame search.
Settings
settings holds how a route renders, as opposed to what it means (view) or which entity it addresses (scope). Today it has one field:
| Field | Purpose |
|---|---|
default_timerange | The route’s initial time window, relative to now. |
{ "title": "OpenTelemetry", "settings": { "default_timerange": { "from": "now-24h" } }, "view": { "tables": {} }}Settings are inherited down the route. A frame at @opentelemetry covers every route beneath it — @opentelemetry/services/checkout, @opentelemetry/traces/{TraceId}, and any *.page.json or *.md file that lives under them — so a pack declares its window once at the root. A deeper frame that declares its own wins for its own subtree, and a *.page.json may declare settings too (applied over the frame chain it borrows its view from). Merging is per key, nearest-wins.
default_timerange is only the initial value: it seeds the URL-synced from/to, so a link that already carries a window wins, the picker stays writable, and following a link from one frame to another keeps the window you were looking at. With nothing declared anywhere in the chain, the window is the last 15 minutes.
View
The view says where rows come from and what they mean. It has one required field, tables, plus optional imports, view-wide where, and with:
{ "view": { "tables": { "traces": { /* one table definition, keyed by its alias */ } }, "imports": { "@opentelemetry/views/combined": { /* one import, keyed by frame name */ } }, "where": "IsServerSpan", "with": { "ErrorRate": "ErrorCount / nullif(RequestCount, 0)" } }}tables— direct database tables, keyed by the CTE alias blocks reference (required; use{}when a frame only imports). See Table.imports— other frames’ views spliced in, keyed by frame name. See Import.where— a SQL predicate applied to every source, scoping the whole frame.with— named scalar expressions (derived columns and metrics) every block can select by name, applied across all sources.
There is no view-level select, group_by, or scalars — selecting and grouping happen in the blocks’ @expr/query; the view only names data. A with entry is a bare SQL string, or one of the tagged forms: { "value": { "expression", "title?", "format?" } } for SQL with presentation metadata, or { "counter" | "level" | "rate": … } for a measure — a metric reading declared by what it is, whose SQL the engine derives.
Table
Each entry in tables is keyed by its alias — the handle blocks reference in their from. All fields are optional:
- The entry is tagged by kind:
tablefor a table of rows,timeseriesfor a table whose rows are samples of series (the only kind that accepts a measure). from— what this alias reads from, resolved alias-first: a name matching another alias in the view (this frame’s own or one an import contributed) derives from it, inheriting that alias’swhere,timestamp, and scalars and refining them ("api_requests": { "table": { "from": "events", "where": "…" } }— see Derived tables); any other name is a physical database table. Defaults to the alias key, but prefer a semantic alias distinct from the physical name — the shipped packs all use one, so a physical name likeotel_tracesalways means the raw table.where— a predicate for this source alone, combined with the view-levelwhereand, on a derived table, with every ancestor’s.with— scalars scoped to this source. On a derived table these are added to the inherited ones; a repeated name replaces the inherited definition for this alias only.timestamp— the column used for time bucketing. Inherited from the parent by a derived table.- On a
timeseriesentry only:identity(what makes two rows samples of the same series) andtimestampare required, plus an optionalaccumulation_start(when a series began accumulating, which also marks resets). Inherited by derived tables. disable_auto_scope— settrueto opt this table out of the route scope (the param comparisons and everyscope.where) when it doesn’t carry the columns they read.
Alias keys must start with a letter or underscore and may contain letters, digits, _, -, and . (so system.parts is a legal defaulted alias).
Import
Each entry in imports is keyed by the imported frame’s name. An import builds on an installed semantic layer — { "imports": { "@opentelemetry/views/combined": {} }, "tables": {} } — inheriting that view’s scalars and scoping and AND-composing your where onto it. An import accepts:
tables— a list of the imported frame’s aliases to splice in; omit to import all. Derivations resolve before the filter applies, so selecting a derived alias without its parent still carries the parent’s scoping and scalars — the parent is simply not exposed.params— values for the imported frame’s ownscope.params, keyed by param name:{ "params": { "ServiceName": "checkout" } }. Every declared param is required. Each imported table receives the imported frame’s scope unless it setsdisable_auto_scope. The imported scope adds param comparisons and itsscope.where. Ancestor scopes do not apply to the import, so an importedscope.wheremust not name an ancestor’s param.where/with— a predicate / scalars folded into every table spliced from this import.disable_auto_scope— propagate the route-scope opt-out to every spliced table.
Page
A *.page.json is what makes a route render. Its page is a tree of blocks, each node a single-key object like { "@block/stack": { … } }, typically rooted at @block/page. Any block is accepted: an @block/page renders as-is, and any other block is wrapped in @block/page automatically. A frame with no page beside it renders nothing — it exists to be imported or borrowed.
The top-level fields of a *.page.json:
| Field | Required | Purpose |
|---|---|---|
title | yes | Route title. Supports Handlebars, e.g. "Logs: {{ServiceName}}". |
page | yes | The block tree to render. |
routes | no | The sections this page can show inside its @block/outlet, and which one its own URL shows. See Sections. |
settings | no | How this route renders, applied over the frame chain it borrows its view from. See Settings. |
templates | no | Reusable blocks and expressions this page’s #/block/… refs resolve against. A ref never reaches the frame’s own templates. |
$schema | no | Path to the generated page JSON Schema; auto-written into installed pages. |
{ "title": "Latency", "page": { "@block/stat": { "title": "p95 latency", "value": { "@expr/query": "SELECT DurationP95 FROM traces" }, "format": "duration" } }}A block queries the view rather than the raw table, so the frame’s scope, parameters, and the page’s time range fold into every query automatically:
{ "@expr/query": { "from": "traces", "select": ["RequestCount", "ErrorRate"], "where": "IsServerSpan" }}Sections
A page’s routes splits it into sections, each with its own URL under the page’s. children maps a section key to a title and a block; default names the one the page shows at its own URL. The page puts an @block/outlet where the section renders, and everything around that slot — a header, a tab strip, a filter bar — stays on screen across sections.
{ "title": "Checkout", "routes": { "default": "overview", "children": { "overview": { "title": "Overview", "block": { "@block/use": "#/block/overview" } }, "traces": { "title": "Traces", "block": { "@block/use": "#/block/traces" } } } }, "page": { "@block/tabs": { "children": ["overview", "traces"] } }}@block/tabs’ children arm reads the titles and URLs out of routes.children and renders the outlet beneath itself, so the sections are declared once. A page that wants its own content above the sections, or the outlet somewhere the strip is not, writes an @block/outlet where it wants the section and navigates to it however it likes.
A section key is a URL segment: letters, digits, _ and -, and no /. checkout/traces renders the page with the traces section in its outlet; checkout renders it with overview, so the two URLs show the same screen. A file-defined route wins a path a section also names, so adding checkout/traces/index.page.json takes that URL over.
A section belongs to its page:
- One view, one runtime. A section renders against the view its page already compiled, so every section’s queries run against one runtime and a filter set in the page’s header reaches the section below it.
- One file. A section is authored inside its page’s document, so editing any part of it reloads the whole route, and
@block/userefs in a section resolve against the page’stemplates. - No nesting. A section is a block, so it declares no
routesof its own. An@block/outletinside a section renders nothing.
Every surface that resolves a route resolves sections: noemata run checkout/traces, noemata screenshot checkout/traces, and the app’s URL all render the same tree, and noemata validate --online renders every section a page declares rather than only the default one.
Scope
scope.params names the route params a frame introduces. scope.where sets a
filter for the frame’s route and its descendants. A frame at
services/{ServiceName}/index.frame.json can render any service. Its
{ServiceName} path segment binds the param, which scope.params lists by
name. Noemata compares the param value to the view scalar or column with the
same name:
{ "scope": { "params": ["ServiceName"] }}To compare a param’s value to a column with a different name, map the segment name to that column. The map can be the whole scope.params or an entry of the list:
{ "scope": { "params": { "Database": "database" } } }{ "scope": { "params": ["HostName", { "Namespace": "k8s_namespace", "Service": "service_name" }] } }- Every param is a required string.
scope.paramsmust name each{Segment}introduced by the frame’s path exactly once. Validation reports an extra name asparam_not_in_path, an unnamed segment assegment_not_declared, and a duplicate name asparam_duplicate_name.- Noemata adds a comparison to every table in the rendered view for each route param, including params from ancestor frames. The comparison uses the column declared in
scope.params. - The URL or a parent frame supplies each param value.
- A
wherecan reference a param with a{Name:String}placeholder. The page context also exposes the value under the param name. - Set
disable_auto_scope: trueon a table that does not contain the columns used by the scope.
scope.where is added to each auto-scoped table on the route and its
descendants. The filter also applies to child frames with their own views. A
{Name:String} placeholder binds the route param Name. A
frame can declare scope.where without scope.params; the filter then applies
to its descendants.
For example, the following scope.where on services/ limits every service page beneath it to the shop namespace:
{ "scope": { "where": "ServiceNamespace = 'shop'" } }Entity search and validate --online param sampling apply the scope.where of
frames whose route binds no param. The services/ filter above also limits the
services listed for services/{ServiceName}.
You can restate a param comparison in scope.where to help the database skip
data. The filter must match every row selected by the param comparison.
Otherwise the filter removes valid rows from the page.
view.where and scope.where apply to different tables:
view.wherefilters every table of this frame’s own view, including tables that setdisable_auto_scope. A child route that declares its own frame doesn’t receive it.scope.wherefilters the views of this frame and its descendants. Tables withdisable_auto_scopeskip this filter.
Overlays (*.frame.local.json, *.frame.overrides.json) cannot change scope, because it defines which URLs the frame serves.
A concrete route shadows its parameterized sibling. Put services/redis/index.frame.json next to services/{ServiceName}/index.frame.json and every route to services/redis — URL, @block/link, @block/frame, the CLI — renders the concrete frame instead. Static segments are matched before the {Param} one, with backtracking, so deeper paths the override doesn’t define (services/redis/traces/{TraceId}) still fall through to the parameterized subtree. This is how you give one entity a hand-built page while the rest keep the generic one, and it works inside a managed integration pack — your file is yours, and version-controlled, even though the pack’s own files are not.
Two rules follow from the route still being parameterized underneath:
- The override declares no
scope.params— the segment is a literal, so declaring one is aparam_not_in_patherror. - The shadowed parameterized frame still binds
ServiceName = "redis"and applies its scope to the concrete route. The scope includes the declared column andscope.where. A scheduled workflow beside the concrete route uses the same scope as its page. A table without the declared column should setdisable_auto_scope: true.
Give every entity page a listing above it. A parameterized frame at services/{ServiceName}/ leaves services/ a bare namespace: it appears in the navigation as a folder you cannot open, and there is no URL for “all services”. Add services/index.frame.json — an unparameterized frame whose page is the inventory table, each row linking on to services/{ServiceName} — and the route becomes navigable, linkable, and the natural parent of the entity page. Its settings are inherited by every route beneath it, and the entity page keeps its own view (the deepest frame on the route is always the view source).
Standalone views and pages
The same building blocks exist as standalone files:
*.view.json— a view with no page, referenced by other frames viaimports.*.page.json— a page on its own; it borrows the nearest frame’s view for its data model.*.workflows.json— a bundle of named workflows ({ "workflows": { "<name>": … } }), addressable as<path>#<name>. Each definition hasexecuteoruse(a workflow template reference plus parameters). Inlineexecuteis{ steps: … }, the general checkpointed pipeline, or{ detect: … }, a declarative detection. Validated and shipped in packs; never a route. An inline workflow first selects a frame with the same filename stem (foo.workflows.json→foo.frame.json,index.workflows.json→index.frame.json), then falls back to the nearest frame on its containing folder’s route. A frame-embedded workflow uses its own frame, so its queries speak the view’s aliases and scalars, and the run’s evaluation window bounds every table through the view’stimestamp. The view is resolved whenever a run is dispatched or resumed (the definition is pinned at start; the view is not); with no frame at or above the file, the workflow runs on a standalone runtime (raw SQL, no time scoping). That frame’s route must bind every param it has: a scheduled run has no URL to bind a{ServiceName}segment from, so validation rejects a scheduled workflow under one (group the evaluation by the column instead), while a concrete override such asservices/redisbinds its value and the run evaluates scoped to it. The server runs scheduled definitions on their cadence, catching up percatch_upafter downtime.
Workflow state
@expr/workflow_state is the durable member of the state family (@expr/state, @expr/local_state, @expr/url_state): its medium is the workflow’s stored key-value map. As an expression it reads — the value stored for key as of run start, or defaults when absent — and as a pipeline step ({ "@expr/workflow_state": "key" }) it writes: the incoming value becomes the pending write for that key, and the flow passes through. Keys under noemata.* are reserved for the runtime (a detection’s alert machine carries its state at noemata.alert, an identity detection’s seen set at noemata.seen) and rejected by validation.
{ "execute": { "steps": [ { "@expr/query": { "from": "services", "select": { "as": { "ServiceName": "ServiceName", "value": "ErrorRate" } } } }, { "@expr/query": { "as": "c", "full_join": { "from": { "@expr/workflow_state": { "key": "state", "defaults": { "@expr/empty_table": { "ServiceName": "string", "firing": "number" } } } }, "as": "p", "on": "c.ServiceName = p.ServiceName" }, "select": { "as": { "ServiceName": "coalesce(c.ServiceName, p.ServiceName)", "firing": "if(coalesce(c.value, 0) > 0.05, 1, 0)", "was_firing": "coalesce(p.firing, 0)" } } } }, { "@expr/workflow_state": "state" }, { "@expr/filter": { "where": "firing != was_firing" } } ] }}The semantics that make read-modify-write across runs loop-free:
- Reads are frozen for the whole run. Every read sees the map as it stood at run start; this run’s writes never feed back into it. A run is a pure function from the stored map to the stored map.
- Writes accumulate and merge at completion. Last write per key wins; keys never written carry forward unchanged. Reading and writing the same key — as
stateabove — is the intended idiom: the read is the previous occurrence’s value, the write is this one’s. - A table-valued default is spelled
@expr/empty_table(column name → type), so a join against a first run’s empty state still sees typed columns.
State is exactly-once durable: the seed pins with the run like the definition, each write is a run checkpoint, and a resumed run replays recorded writes instead of recomputing. A completed run publishes its exported map as the workflow’s carried state (state.json under the defining file’s folder at .shared/workflows/<filename>/<workflow-name>/), which the next run seeds from; it expires with schedule_state_retention_days once a workflow stops running. The state belongs to the definition’s version (an integer, default 1): a run seeds only from state published under its own version, so bumping it starts the next occurrence from empty — the reset for an edit that changes what the state means (a new group_by, a renamed key) — and a run still in flight under the old version is abandoned on resume instead of completing into the old chain. See Versioning a workflow’s state. Writes are workflow-only and must be top-level steps; a read anywhere else in the pipeline (a join’s from, an effect’s message) is ordinary expression vocabulary. @expr/set_context is pure scoping in workflows exactly as in frames: nothing it publishes is persisted, and entry names may not start with _, which the runtime reserves for its own scope entries.
Emitting events
@expr/emit in step position turns each surviving input row into an OTel record on the deployment’s export pipeline — collector, the otel_logs/otel_metrics_* tables, and every consumer downstream. The kind key says what each row becomes: log (the default record type below), or one of the metric kinds. Like @expr/log it is a tap (the flow passes through) and an effect (workflow-only, with best-effort delivery and possible duplicates across a crash-resume). The row-set properties (where, order_by, limit) are SQL over the input table — that is where a query is being written. A value position (an attribute, time, body, service_name) states its meaning: { "literal": "fired" } is the word on every record, { "sql": "ServiceName" } reads or computes per row, and bare numbers and booleans are literals (they have nothing to disambiguate) — a bare string is invalid there, so the quoting trap ("'fired'") cannot be written. One emit is one query over the input: literals never touch it, order_by/limit, plain column reads, and supported where comparisons run in memory. Unsupported predicates or computed SQL values send the table to the database, once per emit per occurrence. See query steps for the supported predicates.
{ "execute": { "steps": [ { "@expr/query": { "from": "services", "select": { "as": { "ServiceName": "ServiceName", "value": "ErrorRate" } } } }, { "@expr/emit": { "log": { "event_name": "acme.error_rate.breach", "where": "value > 0.05", "severity": "warn", "entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } }, "attributes": { "kind": { "literal": "fired" }, "value_pct": { "sql": "value * 100" } } } } } ] }}The row set and values:
wherefilters the input rows;order_byandlimitcap them after it.timeis the per-row record time and defaults to the run’s occurrence window end. A literal is a fixed instant (an ISO{ "literal": … }string or bare epoch milliseconds); a SQL value reads type-aware — a temporal column as epoch milliseconds, a bareDateTime(uint32on the wire) as epoch seconds, a string as a parsed date. A time that cannot be read fails the emit.severitydefaults toinfo. A bare string must be one ofdebug/info/warn/error(schema-checked); a per-row severity is spelled{ "sql": "if(value > 0.1, 'error', 'warn')" }and must evaluate to one of the four.bodyis a literal,{ "sql": … }per row (a map or list keeps its structure), or{ "handlebars": "{{service}} exceeded {{value}}" }— rendered per row, its variables reading the input columns by name, in memory.idgives each record its dedup identity, stamped asnoemata.event.id— what a consumer deduplicates on, since replay can duplicate delivered records. It is{ "sql": … }only (one literal id would collapse every record into one identity) and defaults to a hash of the emitted values and time plus an ordinal among identical rows, so a crash-replay re-sends identical ids and genuine duplicates stay distinct.event_namedefaults tonoemata.extract, so a free-standing emit’s records land on the shippedextractionsview.- Records carry the emitting server’s resource; an authored
resource_attributesmap (or theservice_namesugar for itsservice.nameentry) replaces that identity per row, and rows with distinct evaluated maps group into distinct resources.
multi shares one evaluation across several records per row: the shared where/order_by/limit/time/attributes/resource_attributes apply first, and each child in events narrows the row set with its own where (ANDed) or overlays time and the maps per key — one query, one upload of the input table, however many children:
{ "@expr/emit": { "multi": { "entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } }, "events": [ { "log": { "event_name": "acme.breach", "where": "value > 0.05" } }, { "log": { "event_name": "acme.evaluated", "attributes": { "value": { "sql": "value" } } } } ] } }}The runtime adds the record identity: noemata.event.id, noemata.workflow.run_id, and noemata.workflow.occurrence as record attributes, with the workflow identity (noemata.workflow.ref, noemata.workflow.step, and noemata.workflow.hash) on the instrumentation scope — see the record conventions. Authored attributes may not use event.name or noemata.* (validation rejects them), and authored resource attributes may not use telemetry.sdk.*. Records that follow the conventions land on the shipped alerts, changes, and extractions views.
Metric kinds
Three sibling kinds emit OTLP metric data points on the same pipeline: metric_sum (a monotonic delta sum), metric_gauge (an instantaneous reading), and metric_histogram (a pre-aggregated count and sum with no bucket boundaries — one +Inf bucket, no distribution claim). Each carries the metric identity — a required name plus optional description and unit; points sharing all three merge under one metric entry per batch — and its readings as bare SQL text over the input table: a metric reading is inherently per row, so there is no literal/SQL ambiguity to state ("value": "errors" reads the column, "value": "1" is the constant). The row context (where, time, attributes, resource_attributes, service_name) works as on log, and a metric kind can stand alone or sit beside log children in a multi.
{ "@expr/emit": { "metric_sum": { "name": "acme.request.errors", "unit": "1", "value": "errors", "attributes": { "deployment": { "literal": "prod" } } } }}- A point’s
timedefaults to the occurrence window’s end, and on the delta kinds (metric_sum,metric_histogram)start_timedefaults to the window’s start — the evaluation bucket. Authoring a per-rowtimeon a delta kind requiresstart_timetoo, so backfilled points carry their own bucket bounds and no two delta intervals overlap. - On a detection-hosted emit, a metric child gets the detection’s
group_bycolumns stamped as data-point attributes by column name — the identity of a recorded metric series — and only those (the log fold’s every-output-column default would fork the series per reading). Authored maps overlay per key, and a numeric attribute always encodes as a double. - Metric data points carry none of the runtime record attributes —
noemata.event.id,noemata.workflow.run_id, andnoemata.workflow.occurrencewould each fork the series identity (per point, per run, per occurrence). The workflow and detection identity travels on the instrumentation scope, and metrics need no dedup id: a replayed batch lands as identical rows, which the max-per-(identity, time)-then-sum read absorbs.
Detections
execute: { detect: … } is the declarative sibling of steps: one variant object describing what to evaluate each occurrence, which the executor evaluates as one step: query, then emission and successful state publication. Failed or crashed occurrences are not replayed; subsequent occurrences use the last committed detector state. A detection requires a schedule (the occurrence window is what it evaluates) and a configured OTLP collector (its records are its entire output — a run without one fails before evaluating anything). Every workflow record carries noemata.workflow.hash on its scope: a content hash of the authored execute definition, referenced block/expression helper definitions, literal template parameters, and evaluation schedule. Unused templates, the configured workflow reference, state version, and runtime context values resolved while executing do not enter the hash. Changing the hash does not reset carried state; version controls that reset.
Run deadlines
A scheduled run has an execution deadline equal to its current cadence interval. A run that consumes that interval reaches its next occurrence, and a stateful workflow starts accumulating lag. Set timeout to a shorter duration when that runtime already indicates a pathological state:
{ "title": "Check service health", "schedule": { "every": "15m" }, "timeout": "2m", "execute": { "threshold": { "from": "traces", "value": "count()" } }}The effective deadline is the shorter of timeout and the occurrence’s cadence interval. An unscheduled run uses timeout directly and otherwise defaults to 60 seconds. Exceeding the deadline during evaluation fails the current step with its pending node labels and cancels its in-flight work. Finalization overruns also fail the run; already-committed effects and carried state remain intact. Storage operations settle before an overrun can be recorded, so the deadline controls success rather than guaranteeing an exact return time. See Run execution and persistence for failure-persistence limits. Database request and statement timeouts are separate, nested limits; a stricter database timeout can fail an individual query before the workflow deadline.
Seven variants exist: extract copies matching source rows out as records; threshold alerts while an aggregate crosses a comparator; appears/disappears track identity sightings; value_change records exact observed value transitions per identity; changes detects statistical shifts in bucketed series; and burn_rate alerts on SLO error-budget consumption. Each variant is also a top-level execute key: execute: { extract: … } is sugar for execute: { detect: { extract: … } }, and both spellings run — and hash — identically.
extract
{ "execute": { "extract": { "from": "logs", "where": "K8sClusterName != '' AND SeverityLevel >= 13", "select": ["EventName", "Body", "SeverityLevel", "ServiceName"], "limit": 100, "alert": { "levels": { "critical": "SeverityLevel >= 17", "warning": "SeverityLevel >= 13" } }, "entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } }, "emit": { "log": {} } } }}fromis a view table name, a structured sub-query, or a query expression. Query expressions include reusable@expr/usereferences that produce a table.where,select, andgroup_byshape the copied rows (withgroup_by, one summarized row per group).order_bydefaults to source time (_ts) descending whenlimitcaps per-row copies (a summarizinggroup_byselect gets no order default —_tsdoes not survive the aggregation). A view-table source names its time column, so an authoredselectthat leaves it out gets_tsadded — the example above still records each row at its source time (a summarizinggroup_byselect is left alone).alert.levelsis a stateless per-row classification: SQL predicates evaluated over the output row, highest level first (critical>warning>info), the winning level appended as thelevelcolumn. The predicates enter the evaluation query as it stands — the one statement the detection already runs — and the level itself is computed in memory from their results. No lifecycle and no transition events: the copy is stamped, nothing resolves.emitis optional. Omitted ortrueemits source-row events namednoemata.detection.extract;falsedisables delivery. Custom emits replace defaults. Selected attributes, source time,Body, and source severity classification are preserved. In extraction SQL, selected source columns take precedence over generatedseverityanddetection_kindaliases. SourceEventNamebecomesnoemata.extract.source.event_name; OTel severity carries the classification. Matched rows also receivenoemata.alert.levelandnoemata.alert.condition; extraction has no lifecycle transition, persistent state, recovery event, or default alert gauge.entitynames the OTel entity the detection fires on —{ type, id }, wheretypeis a semconv entity type (service,host,k8s.pod,k8s.deployment,k8s.namespace,k8s.node,k8s.container,container,process,session,user, and the other workload kinds) andidmaps that type’s semconv identifying attributes to the per-row values the detection reads. The emit path stamps the row’s resource with the identifying attributes under their semconv keys plusentity.typeand a stableentity.idhash, so records land in their subject’s own OTel resource andotel_logs.ServiceName/HostName/K8sNamespaceNameread as the subject.attributes/resource_attributeson the detection carry additional stamps in the same value model as emit attributes; on athreshold, an attribute’s SQL can read source columns and aggregates, which project into the evaluation query at the group grain ("run.error": { "sql": "any(LatestFailure)" }); every emit inheritsentity,attributes, andresource_attributes, emit-level maps overriding per key — and theextract.*stamps overriding everything. Subjects without a semconv resource key (Noemata-nativeagent,database,mcp_server,pipeline, …) stay onattributes: { 'subject.type', 'subject.*' }; the identity chain has asubject.*fallback for them.idis a per-row SQL dedup identity; omitted, each record’s id is a hash of its emitted values and time, so re-runs re-emit the same ids and the sink deduplicates.
threshold
{ "execute": { "threshold": { "from": "traces", "group_by": "ServiceName", "value": "EntryErrorRate", "require": "EntryRequestCount >= 20", "alert": { "for": { "up": 2, "down": "10m" }, "levels": { "warning": { "enter": { "above": 0.05 }, "leave": { "below": 0.03 } }, "critical": { "above": 0.2 } } }, "entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } } } }}valueis a SQL reading evaluated undergroup_byfor each occurrence. Emit and recommendation SQL can readvalue, grouping dimensions,detection_kind, and generatedseverity. Alerting also exposeslevel,alert_transition,has_data, and the existing internal lifecycle columns. Usedetection_kindfor detector identity andalert_transitionfor lifecycle changes. Group dimensions cannot usedetection_kind,alert_transition,severity, orhas_data.alert.levelsare comparators as data, not SQL:{ "above": n },{ "below": n },{ "between": [a, b] }(alert while inside the band),{ "outside": [a, b] }. A bare comparator is theentercondition withleavedefaulting to its negation; spelling both is hysteresis. ANULLreading satisfies no comparator: no level enters on it, the default leave treats it as the condition clearing, and an authoredleaveholds through it (its own comparator is not satisfied either).requireis a guard ANDed into every level’senter, neverleave— a volume guard a comparator cannot hold.- The
alertblock is the stateful lifecycle:forgates persistence (upgates a quiet instance entering,downa firing one leaving; a count of occurrences or a duration judged against occurrence timestamps), level changes of an already-firing instance are immediate, andcritical>warning>info.no_datasays what a previously-seen instance absent from an evaluation means:resolve(default) treats absence as clear,holdfreezes the level and every counter.limitcaps tracked instances (admitted worst-first; rows it rejects still reach emits), andbodyoverrides the generated record body. alert.emitdefaults to transition events namednoemata.detection.thresholdand thenoemata.alert.levelgauge. The gauge emits held level ranks 1–3 while firing and one final zero on resolution. Quiet instances produce no further points. Grouping and detection identity remain stable across level changes.alert.emit: falseor[]disables these outputs.- An authored
alert.emitreplaces its defaults. Log children receive transitions, including missing-data resolution; metric children receive settled alert state. Authored names, body, severity, IDs, time, attributes, and resource attributes override their defaults. A sharedmulticontext underalert.emitremains disallowed; place context on individual children. - Detector
emitdefaults to events plus threshold-value measurements without alerting, and measurements alone with alerting.falsedisables detector output; custom emission replaces its defaults. Custom detector logs receive transition rows when alerting is configured. Overridingevent_namewith a custom name excludes events from the default detection views; querylogswith an explicitEventNamefilter. Custom detector metrics receive present evaluation rows, including healthy and admission-rejected groups. order_byandlimitbound the evaluation itself (deterministic top-k, a cost guard). A row trimmed here never reaches the machine: its instance reads as absent and followsno_data. Usealert.limitto cap tracked instances while preserving all groups for admission and recovery evaluation.- A threshold with an
alertblock is stateful: occurrences run in order, each seeding the machine from the previous occurrence’s carried state (readable at thenoemata.alertstate key). Bump the definition’sversionwhen an edit changes what that state means (a newgroup_by, a different level set).
One-shot detectors (appears, changes, and value_change) preserve unmatched findings without alert attributes, using detected severity for changes and info for appears and value_change. Matching findings receive the alert classification once. Detector emit controls unmatched finding delivery; alert.emit controls classified delivery.
All seven detectors accept recommend: { body, name?, where?, attributes? }, with or without alerting. where evaluates after detection and alerting; false or NULL preserves the event without advice. A match adds noemata.recommendation.name and noemata.recommendation.body. Name defaults to the workflow reference. Recommendations preserve event identity, severity, and body.
body accepts a literal, { "sql": … }, or { "handlebars": … }. Threshold SQL can combine source aggregates with output columns: source dependencies project before grouping discards them, and enrichment finishes after alerting. Handlebars reads output columns such as {{value}}, {{level}}, and {{alert_transition}}. Missing recovery values are not filled from previous evaluations. attributes uses emit literal/SQL conventions; matching advice overrides detector attributes, and emit attributes override advice. Use custom attributes for estimates and units.
Identity detectors accept change: { name, of, from, to }. The optional name is a string; of, from, and to accept literal or SQL values. These fields decorate emitted records without changing identity or transition state.
appears and disappears
{ "execute": { "disappears": { "from": "hostmetrics", "where": "MetricName = 'system.cpu.time' AND HostName != ''", "group_by": "HostName", "forget": "7d", "alert": { "levels": { "warning": "misses >= 3" } }, "entity": { "type": "host", "id": { "host.name": { "sql": "HostName" } } } } }}The identity pair shares one mechanism: a carried seen set of every identity sighted, per group_by combination (required, with at least one dimension: the identity is the mechanism), with sighting times read from the source’s _ts. Both are always stateful; the first run seeds the set with a single lookback over the horizon and registers every existing identity silently, so enabling an identity detection workflow on a live system records nothing. Neither takes order_by/limit: a truncated sighting set would read as absence.
appearsrecords anoemata.detection.appearsevent when a new identity concludes its appearance, timestamped at its first sighting.remember(occurrences or a duration judged against occurrence timestamps; default7d) is how long an absent identity stays known before its return counts as new again, and the seed horizon. The alert block is optional and one-shot — nothing resolves an appearance:foris a consecutive-sightings debounce the appearance must survive before it records (no_data: holdlets the count survive a missed occurrence), andlevelsare SQL predicates over the appearance row (the group columns,first_seen,last_seen) that classify it into a singlefiredrecord onalerts.disappearsalerts when a known identity stops being sighted — the heartbeat shape. Its rows exposemisses(consecutive absent occurrences, available to SQL predicates) andlast_seen_atto the level predicates; the alert’sforis the grace, and reappearance resolves. The detection’s conclusion of absence — the fire when it alerts, the first miss when it does not — records anoemata.detection.disappearsevent with the last sighting innoemata.disappears.last_seen, timestamped at the concluding occurrence.forget(default7d) drops an absent identity from the seen set; under the defaultno_data: resolvethe drop also resolves a firing alert, whileno_data: holdkeeps it firing past the drop until the identity returns.forgetmust outlast the alert’sfor.upon the occurrence grid, or the detection could never fire — checked against the schedule before anything evaluates.- Level predicates lower to the same machine columns as threshold comparators, with the same NULL-safe default
leave; the entered predicate lands whole underalert.conditionon enter-kind records. - Top-level
emitis optional and defaults to finding events without alerting. With alerting,alert.emitowns transition events. Custom detector logs also receive transition rows.
Identity initialization batches appears, disappears, and value_change by
default. Batch duration follows the source frame’s inherited
settings.default_timerange, resolved against the occurrence end; an absent or
non-positive duration uses the fifteen-minute built-in default.
Set initialization: { "max_source_range": "6h" } for a shorter or longer
duration; the duration can be an expression. Set
initialization: { "batch": false } for whole-horizon evaluation of derived
sources. Current inventories, cross-boundary joins, ranking, and whole-window
aggregation can require this opt-out. The runtime does not inspect SQL to choose.
The detector checkpoints the contiguous processed range, accumulated identities,
and chosen width, then yields between queries. Committed initialization progress retains the width and
the original remember or forget horizon even if frame defaults change.
Initialization emits no records and does not count slices as presence or absence
occurrences. The completed seed preserves each identity’s earliest and latest
sightings. This setting bounds the query time range, not its row count or memory.
value_change
{ "execute": { "value_change": { "from": "host_configuration", "group_by": "Host", "value": "tuple(Cores, Memory)", "require": "Cores > 0 AND Memory > 0", "remember": "7d", "change": { "of": { "literal": "size" } } } }}The source must expose _ts. group_by identifies the entity; value is its
watched scalar or tuple. The latest non-null, qualifying observation in each
occurrence is compared with the retained value. require filters incomplete
observations before selecting the latest value. Missing observations preserve
state until remember expires (default 7d). Null tuple members require an
explicit completeness predicate.
Initialization silently retains the latest observations over the remember
horizon. A newly observed or expired identity seeds silently. Subsequent exact
transitions, including A → B → A, emit noemata.detection.value_change with detector kind value_change,
and typed noemata.value_change.from/noemata.value_change.to. Multiple changes within
one occurrence collapse to its latest observation. The source timestamp is the
record time. Equal source timestamps use the value’s string representation as a
deterministic query tie-break; a conflicting value at an already retained timestamp
fails the run. Older observations do not replace newer state.
Optional alert.levels classify transitions into one-shot alerts. Predicates can
read group columns and the quoted identifiers
"noemata.internal.value_change.value_before",
"noemata.internal.value_change.value_after", and
"noemata.internal.value_change.last_seen". change.name
and change.of customize record labels. Initialization uses the shared batching
defaults and opt-out described above; each batch retains the latest value per
identity without emitting historical transitions.
changes
Use value_change for exact configuration transitions; use changes for statistically
confirmed shifts in noisy numeric series.
{ "execute": { "changes": { "from": "traces", "where": "IsEntrySpan AND ServiceName != ''", "group_by": ["ServiceName", "ServiceVersion"], "value": "quantile(0.95)(DurationMs)", "window": "24h", "kinds": ["step"], "min_magnitude": 1, "alert": { "levels": { "critical": "change_direction = 'up' AND change_magnitude >= 3", "warning": "change_direction = 'up'" } }, "entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } } } }}Records when a series changes regime. Each occurrence aggregates value per interval bucket (default: the schedule’s every) over the lookback window (required; at most 10,000 buckets), one series per group_by combination, and runs the @expr/changes detector over each series with the same knobs a chart’s changes annotations take (kinds, min_magnitude, min_segment_size, seasonality). Buckets are calendar-aligned in UTC, so a chart over the same view and interval shows the same detections. interval must divide a day evenly or be a whole number of days, as for a schedule’s every. fill says what a bucket with no source rows reads as: null (default) leaves it out of the series, 0 counts it as a zero reading — use 0 for counts and sums.
- A change is confirmed once
min_segment_sizecomplete buckets follow its boundary, so a step confirmsmin_segment_size × intervalafter it happens (plus the schedule’sdelayand phase). Each confirmed change records onenoemata.detection.changesevent timestamped at its boundary, withnoemata.changes.ofthe authoredvalue,noemata.changes.from/noemata.changes.tothe levels before and after,noemata.changes.magnitudein the series’ own standard deviations,noemata.changes.direction, andnoemata.changes.type(step,spike,dip,trend,distribution,non_stationary). - The detection’s output rows are the confirmed changes of the occurrence: the group columns plus the finding columns
change_boundary(ISO),change_type,change_direction,change_magnitude,change_severity,change_value_before, andchange_value_after, named after thechange.*attributes they record as. A dimension may not use one of those names. - A cursor per series retains its last recorded boundary: a change already recorded never records again, and a boundary the segmentation moves by a bucket or two once more points follow it (within
min_segment_sizebuckets of the cursor) is the same change. Bump the definition’sversionto reset the cursors. - The
alertblock is optional and one-shot, as forappears:levelsare SQL predicates over the output row ("change_direction = 'up' AND change_magnitude >= 3"), and each confirmed change that matches a level fires a singlefiredrecord onalertswithnoemata.changes.magnitude,noemata.changes.direction, andnoemata.changes.type. The instance is the series pluschange_boundary, so two changes in one series are two alerts. There is noforand nono_data;limit,order_by, andbodyare the machine’s own. - Top-level
emitis optional whenalertis absent (a recording variant). Withalert, alert-lifecycle emit lives underalert.emit— same shorthand as threshold. A top-levelemitalongsidealertcomposes withalert.emit.
Enable single-resolution materialization with source_retention and component_store:
Only component_store.catch_up.max_source_range is required for changes.
repair_window defaults to five minutes, rounded up to a whole bucket. Set it
explicitly when ingestion delays or backfills require longer repair. Finalized
buckets do not incorporate later source arrivals. Burn rate uses the same
defaults and additionally requires component_store.intervals.
Optional resource defaults for both detectors are retention_margin: "1h",
max_pending_bytes: 16777216, and, inside catch_up, max_batches: 32,
max_duration: "20s", max_rows_to_read: 10000000,
max_bytes_to_read: 268435456, max_result_rows: 50000, and
max_result_bytes: 8388608. Explicit values override these defaults.
max_duration controls admission of maintenance work. An admitted query has a
separate 30-second deadline and may finish beyond that budget; its completed
progress is checkpointed before further maintenance stops. Publication and
parent-rollup phases also stop admitting work at their phase budgets. Evaluation
reads use the independent query deadline. Workflow cancellation and deadlines
still apply, and query errors remain failures.
{ "source_retention": "7d", "component_store": { "catch_up": { "max_source_range": "1h" } }}intervalis the stored resolution. Averages and quantiles are stored as bucket values, without combining buckets into wider aggregates. Values must fit finite Float64 readings; integer readings beyond exact Float64 precision are rejected. Group identities retain their source types.- Initialization queries bounded source ranges and checkpoints each admitted batch before yielding. Subsequent runs repair recent buckets and fill remaining historical gaps.
source_retentionmust cover the repair window, one bucket, and the greater of the maintenance duration and 30-second query deadline. Choose it from the source’s actual retention. repair_windowandmax_source_rangemust be multiples ofinterval. The lookback must leave at least2 × min_segment_size + 1finalized buckets outside repair.- Detection uses only the contiguous finalized prefix of the lookback. Missing coverage or repairable buckets stop that prefix. Confirmation therefore includes the repair delay as well as the detector’s evidence requirements. Too little finalized history holds the cursors and emits no findings.
- Sparse bucket values are retained as workflow data during repair and publication. Include the
metrics_gaugerelation in the workflow’s view. Finalized values publish there asnoemata.changes.component; payload hashes and row-count markers protect readback. Payloads are released only after matching readback. Missing published data later blocks coverage; it is not interpreted as a zero. fillapplies after assembling history. Missing source rows in a successfully queried bucket are distinct from missing materialization coverage. The checkpoint retains repair payloads, unpublished finalized payloads, publication manifests, coverage, and the source column schema alongside the change cursors.
burn_rate
{ "execute": { "burn_rate": { "from": "traces", "where": "ServiceName != ''", "group_by": "ServiceName", "bad": "countIf(IsEntrySpan AND IsError)", "total": "countIf(IsEntrySpan)", "objective": 99.9, "window": "30d", "source_retention": "7d", "component_store": { "intervals": ["1m", "5m", "1h"], "repair_window": "10m", "retention_margin": "1h", "max_pending_bytes": 50000000, "catch_up": { "max_source_range": "1h", "max_batches": 4, "max_duration": "30s", "max_rows_to_read": 1000000, "max_bytes_to_read": 100000000, "max_result_rows": 100000, "max_result_bytes": 10000000 } }, "entity": { "type": "service", "id": { "service.name": { "sql": "ServiceName" } } } } }}Alerts while an error budget burns faster than its objective allows. from is a view table name or a structured sub-query; a query expression is rejected, since it would evaluate once under the occurrence window and the trailing windows could not read past it. bad and total are count aggregates under the grouping (one SLO instance per group_by combination, or one ungrouped instance); objective is the target percentage, strictly between 0 and 100; window is the SLO window the budget spans, at least one day. A window’s burn is its error fraction divided by the budget (1 - objective / 100): burn 1 spends the budget exactly over the SLO window, burn 14.4 spends it in 1/14.4 of the window.
- Without
component_store, every occurrence readsbadandtotalover each trailing window of the lattice and over the SLO window, all ending at the occurrence end, in one statement. The output row per instance carriesburn_long_1/burn_short_1…burn_long_4/burn_short_4(the four window pairs, fastest first),value(the highest long-window burn, thenoemata.burn_rate.valueon records),budget_remaining(one minus the SLO window’s burn; 1 is untouched, 0 exhausted, negative overspent), andexhausts_at(the timestamp the budget runs out at the current burn: the occurrence end once it is spent, NULL while nothing burns or when the projection lies past the year 2299). A dimension may not use one of those names. - The default lattice is the SRE workbook’s at a 30-day window, each long window scaled linearly to
windowand each short window a twelfth of its long:criticalat 14.4 times the budget over 1h and 5m or 6 times over 6h and 30m,warningat 3 times over 1d and 2h or 1 times over 3d and 6h. A level enters while any of its pairs has both windows at or above the pair’s factor, so a short sharp burn and a long slow burn both fire and a blip does not. - Default gauges are
noemata.burn_ratewith numericnoemata.burn_rate.window_secondsfor each distinct trailing window,noemata.burn_rate.maxfor the maximum long-window burn, andnoemata.burn_rate.budget_remainingfor the unspent budget fraction. All use unit1. alertis optional; omitted orfalse, the detector emits evaluation events and measurements.alert: trueenables the default lattice’s levels. Authoredlevelsare SQL predicates over the output row ("budget_remaining < 0.2","burn_long_1 >= 10 AND burn_short_1 >= 10"), with the machine’sfor,no_data,limit, andbodyas on threshold. Fires and resolves land onalertswithnoemata.burn_rate.budget_remainingandnoemata.burn_rate.exhausts_at(the ISO string ofexhausts_at) besidenoemata.burn_rate.value. There is noorder_byorlimiton the evaluation: it runs once per window, and a per-window cap would select a different instance set per window.component_storereplaces repeated long source scans with exact additive buckets.badandtotalmust each be onecount()orcountIf()aggregate.intervalsare ascending fixed durations that each divide a day evenly or are a whole number of days; each interval must divide the next, and every lattice window must be divisible by the first. A 1m/5m/1h store sums final 1m buckets into 5m buckets, then sums twelve final 5m buckets into 1h buckets. Window evaluation selects the largest final buckets that fit and uses smaller buckets at its boundaries, so an unfinished hour does not delay current readings. All windows use the last base-interval boundary at or before the requested occurrence end;exhausts_atuses that same effective endpoint.source_retentionis required withcomponent_storeand declares the oldest source data maintenance may query. It must coverrepair_window, one base bucket, and the greater ofcatch_up.max_durationand the 30-second query deadline. Every occurrence re-queries the repair window plus the bucket receiving its final read. A successful empty bucket records coverage without storing group rows, while a missing coverage range remains unknown. Final buckets are immutable and are the only buckets emitted asnoemata.slo.bad,noemata.slo.total, andnoemata.slo.completemetrics.- Catch-up reserves progress for publication, parent rollups, and source queries in that order.
max_source_rangebounds one source query;max_batchesindependently bounds publication partitions, parent jobs, and source batches;max_durationassigns the first third to publication, the next third to parents, and the remaining time to source work, with unused time available to later phases. Each query receives its remaining phase timeout. The four ClickHouse row, byte, and result limits reject work that exceeds its configured cost. A failed query keeps its range pending. The next occurrence uses the last deliberately committed batch; intermediate progress from a failed or crashed occurrence is discarded.schedule.catch_upremains independent: one current alert evaluation can fill a six-hour component gap across several occurrences without replaying 360 alert evaluations. - Final metric publication is at least once. The workflow keeps each finalized Arrow payload until bounded read-back sees its completion marker and both count points for every nonempty group; collector acceptance is not treated as a receipt.
max_pending_bytescaps all currently referenced component payloads, including repair buckets, published children awaiting parents, and partial parents. Serialized sizes are checked before writes; only complete source buckets that fit are admitted. Publication and rollups continue to release space. Temporary query buffers and unreachable file versions awaiting garbage collection are outside this limit. While any required window has unknown coverage, the detector emits no partial reading and forcesno_data: holdfor that occurrence, preserving alerts and excluding the held interval from elapsedfortiming. - Stored component state records source availability, the requested effective endpoint, per-window coverage bounds, oldest backlog age, source-expiry status, repair lag, and pending-publication bytes in the workflow’s carried state. Component generations hash the resolved view snapshot, source, filter, grouping, count expressions, and base interval. Objective, alert, emit, and SLO-window edits reuse compatible buckets; a source or view edit starts a new generation.
- Finalized history is read from the database. Workflow data normally retains the repair window plus bounded publication, parent-rollup, and interrupted work. In the dense 10,000-group, 10-minute repair benchmark, workflow data used 4.00 MB and the carried checkpoint used 5.68 kB. The 30-day database query read 14.84 million component points into 10,000 grouped rows in 523 ms on local chDB; database CPU was 6.78 seconds across cores. Size database retention and query concurrency for the detector count and active group density.
burn_rate.alertaccepts a boolean shorthand alongside the object form:alert: trueopts into the SRE-workbook default lattice (14.4×/6×/3×/1× paired long+short windows scaled towindow),alert: falseor absent runs recording-only (SLO metrics and change records still emit; no lifecycle machine), and an authoredAlertStateDefinitionreplaces the default. When alerting, the emit lives underalert.emit— same shorthand as threshold. A top-levelemitalongsidealertcomposes withalert.emit.
Workflow execution identity
Workflow queries do not run with the author’s database privileges. Scheduled runs execute under the deployment’s standing identity: the service_account/hybrid principal’s credential, or, in a deployment with authentication: "none", the anonymous identity every request already runs as (ClickHouse’s default user). Only a user principal has no standing identity, and runs no scheduler.
This makes workflow authorship a privileged operation: anyone who can commit a workflow file gets its queries executed with the standing identity’s grants. Two consequences for operating a shared deployment:
- Treat workspace write access as the control point — review workflow changes the way you review code, because merging one is permission to run it.
- Scope the service account’s ClickHouse grants narrowly: read on the telemetry databases plus write on Noemata’s own tables is enough. The service account does not need — and should not have — the union of your users’ privileges.
A per-workflow run_as (an explicitly declared identity resolved against stored credentials) is planned; execution identity will always be declared in the definition rather than inferred from who committed it.
See also
- Views and Pages & templates — how to author the two halves.
- Authoring frames — the authoring guide.
- Frames — the concept behind the schema.