Skip to content
NOEMATA

Database resources

A *.resources.json file under @frames/ declares physical objects used by a frame’s view: stored columns, skip indexes, projections, and rollups.

noemata validate and the publish gate check every resources file. noemata publish installs the resources before the new revision is served; see Installing.

{
  "resources": {
    "pod_name": {
      "column": {
        "table": "@opentelemetry/views/logs#logs",
        "expression": "ResourceAttributes['k8s.pod.name']"
      }
    },
    "pod_name_filter": {
      "index": {
        "table": "@opentelemetry/views/logs#logs",
        "column": "pod_name",
        "kind": "bloom_filter"
      }
    },
    "by_pod": {
      "projection": {
        "table": "@opentelemetry/views/logs#logs",
        "order_by": ["pod_name", "Timestamp"]
      }
    }
  }
}

Entries

Each key under resources names one entry. Keys start with a letter or _ and contain letters, digits, _ or -. Each entry has one kind: column, index, projection or rollup. Every kind also accepts these fields:

  • table — view table in <frame id>#<alias> form, such as @opentelemetry/views/logs#logs. Validation resolves the alias through the compiled view to its physical table.
  • async — optional, defaults to false. When true, publication serves the revision before the resource mutations and backfill finish.

An index, projection or rollup can name a column entry from the same file and physical table, or a physical table column. A view scalar is not a physical column. Validation warns when a name refers to both.

column

  • expression — row expression over physical table columns in database SQL, such as ResourceAttributes['k8s.pod.name']. It cannot call aggregate or window functions, use qualified names, or read other resource entries.
  • type — optional, the stored column’s type.

On ClickHouse the column is a MATERIALIZED column named __nm_<hash>, compressed with ZSTD(1):

  • A Map subscript is stored as LowCardinality(String), the value type of the OTel exporter’s attribute maps. Declare type for a map with other values.
  • A string literal or a string function (lower, concat, toString, replaceAll, …) is stored as String.
  • Any other expression needs type, such as "DateTime" for toStartOfHour(Timestamp). Without it the entry cannot be installed, and validation warns.
  • An existing column for the same expression satisfies the entry. The OTel exporter’s __otel_materialized_k8s.* columns satisfy ResourceAttributes['k8s.*'] entries. m['k'] and arrayElement(m, 'k') are the same expression.

index

  • column — the column to index.
  • kind — bloom_filter, minmax, set or text.

On ClickHouse the kinds install as bloom_filter(0.01), minmax, set(1000) (each GRANULARITY 1) and text(tokenizer = 'splitByNonAlpha'). ClickHouse allows one text index per column.

projection

  • order_by — the columns the projection sorts by, at least one.
  • columns — optional, the columns it stores. The order_by columns are always stored.

Without columns, the projection stores ordinary table columns and the declared sort columns. ClickHouse SELECT * projections omit materialized columns. Smaller projections use less storage and serve only queries that read their stored columns.

rollup

  • grain — the bucket width, such as 1m or 1h.
  • dimensions — the columns a row is keyed by.
  • measures — measures and aggregate scalars of the view table.
  • retention — optional, such as 400d. Without it, the table’s retention applies. A publication updates the rollup TTL when this value changes. Queries use only buckets within the retained range.
  • from — optional, a finer rollup entry in the same file. Its grain must divide this grain. It must store each dimension and measure this rollup uses.

Rollups can store count, sum, min, max, avg, argMax, argMin, maxMap, minMap, any, uniq, approximate quantile, and quantiles aggregates. The If and OrNull combinators are supported where ClickHouse permits them. maxMap and minMap do not support OrNull.

argMax and argMin choose an arbitrary row when ordering values tie. OrNull aggregates return NULL when no value was aggregated; quantiles returns an empty array. countIfOrNull, uniqIfOrNull, and quantilesIfOrNull are refused because they return NULL only for groups with no rows. Use countOrNullIf, uniqOrNullIf, or quantilesOrNullIf instead.

On ClickHouse a rollup installs as four objects in the database of the physical table, named with the deployment’s table namespace (noemata by default):

  • <namespace>_rollup_<alias>_<key>__<hash8> — an AggregatingMergeTree target partitioned by day and sorted by dimensions and bucket. The hash identifies the stored shape. A changed shape gets a new target.
  • <namespace>_rollup_<alias>_<key>__<hash8>_mv — a materialized view that writes source rows from a cutover about 30 seconds after the plan.
  • A backfill of source rows before the cutover. It starts after the cutover and processes one-day chunks from newest to oldest, within retention.
  • <namespace>_rollup_<alias>_<key> — a stable view for queries and other tools. Publication points it at the target before serving the revision.

The entry cannot be installed, and validation warns, when a measure cannot be stored (such as uniqExact or an exact quantile), the grain does not divide the bucket ladder, or a per-series measure needs dimensions the rollup lacks.

Installing

noemata publish installs what the revision declares, in this order:

  1. Plan. The installer compares declarations with each physical table. Unsupported entries are reported as warnings and omitted. Entries that refer to them are omitted too.
  2. Grants. The ClickHouse user needs SELECT, INSERT, ALTER TABLE, CREATE TABLE, CREATE VIEW, DROP TABLE, and DROP VIEW on each database in the plan. Missing grants fail publication before installation starts. The error lists each missing grant.
  3. Install and backfill. The installer adds columns, indexes, projections, rollup targets, and views. It waits for mutations that write columns and indexes into existing parts, plus rollup backfills. For async entries, publication waits only for the initial steps.
  4. Serve. Publication points the stable views at their targets, then serves the revision.
  5. Retire and converge. Resources the revision no longer declares are retired for seven days. Rollup materialized views keep writing during this period. A later revision can declare an accelerator again and cancel its retirement. Publication then finishes asynchronous entries.

noemata publish --dry-run prints the file changes and planned resource steps. It runs no installation steps.

The CLI uses the operator’s NOEMATA_CH_* credentials, as noemata init does. The app uses the signed-in user’s credentials. Missing grants are listed in the publication error.

A failed installation stops publication before the revision is served. The ledger records rollup progress. Run noemata publish again to resume installation or repair drift. A publish with no file changes installs resources missing from the served revision and re-points its stable views without creating a revision.

An installation holds a namespace lock until it finishes. A second publication or prune fails and names the lock holder. If a process crashes, the lock expires within a minute. The next installation resumes from the ledger.

Set resources.install_on_publish: false in noemata.json to disable installation, as projects/dev-ws does. --install-resources enables it for one publish. --no-install-resources disables it for one publish.

The ledger

The <namespace>_resource_ledger table sits beside rollup targets. It records each rollup’s declaration, hash, state, cutover, and backfilled range. Retired accelerators also have ledger rows with retirement and prune times. noemata resources status prints the ledger.

Commands

  • noemata resources plan [--revision <id>] — print the steps for the head revision or another revision. noemata publish --dry-run includes the draft’s resource plan.
  • noemata resources prune [--dry-run] — retire undeclared rollups and __nm_* objects. Drop retired objects after seven days. The command checks tables named by the head revision or resource ledger, plus tables with __nm_* objects in those databases. Shared tables may contain another deployment’s accelerators. Review --dry-run before pruning shared tables.
  • noemata resources status — print the ledger.

How queries read resources

The compiler reads what the database reports as installed on a physical table, whether a resources file declared it or not.

  • Stored columns. A Map subscript that a stored column holds is read from the column: in the view table’s scalars, where and aggregates, and in the query’s own select, where, group by, having and order by when the query reads the table alone from its FROM. A subscript over a name the query or a scalar binds, such as an output alias ResourceAttributes, is left as written. The subscript stays when reading the column would lose the only projection that serves the query while the query filters on that projection’s leading sort column.
  • Rollups. A query level reads the coarsest ready rollup that answers it. Otherwise it reads the raw table. A rollup answers when:
    • the view table as declared now derives the rollup that was installed (the same content hash);
    • its grain divides the query’s bucket, and the window starts and ends on rollup bucket boundaries;
    • the range the rollup covers holds the window, and the query has a window;
    • every measure the query reads is stored with the definition the query reads, so quantileExact over a quantileDD sketch is not answered;
    • every column the query groups, filters or selects by is a dimension, or an expression over dimensions;
    • the query reads the table alone, and every inline aggregate can read the states stored by the rollup. For example, count() can use the count stored by a RequestCount measure. sum(Duration) can use the sum stored by an avg(Duration) measure. If no stored column has the required state, the query reads raw data. An unaliased select item that contains an inline aggregate reads a rollup only when it is a function call over column names, such as count() or toFloat64(count()), because rewriting any other item, such as count() / 60, would change the column name ClickHouse gives it. Alias such an item to let it read a rollup.
    • the query holds no SQL text the compiler cannot parse as one expression. An order_by string that lists several columns is such text; list them with order_by: { by: { … } } instead.
  • Provenance. The result’s Table.source lists each use under resources: substituted and kept for stored columns, rollup for the rollup a level read, and skipped_rollup with the reason for every rollup it did not read.

noemata validate --online compares each ready rollup with the raw table. It checks every measure per rollup bucket over the most recent complete buckets. The window ends at least five minutes before now and is capped by --rollup-window (default 1h). Exact measures allow a relative difference of 1e-9. uniqCombined and quantileDD allow 3%. Measures that use any must have matching buckets. A mismatch fails validation. Rollups that are unready, expired, or no longer match the view are reported as skipped.

Ownership and local files

  • Resources files belong to the deployment’s operator. A pack may ship one as a default; views and measures stay in the pack’s frames.
  • Files under a .local path (x.local.resources.json, x.resources.local.json, or a .local/ directory) cannot declare database objects because local files are excluded from revisions.
  • Resources files take no .overrides or .local overlays.

Validation

Errors, reported under the file’s path:

  • the frame or the alias of table does not exist, or the frame’s view does not compile;
  • a column expression does not parse, calls an aggregate or window function, or reads a measure, an aggregate scalar, or another entry;
  • a name is an entry of another kind, a column entry of another physical table, or a measure;
  • a rollup measure is missing or is a row scalar;
  • a rollup from is not a rollup of the file, forms a cycle, or does not divide the grain;
  • a key appears twice in the file, or two column entries store the same expression on one table.

Warnings: the database cannot install the entry (with its reason), or a name is a view scalar and is read as the physical column of that name.