Transforms

Transforms reshape a table after it comes back from a query — adding columns, filtering rows, parsing, filling gaps. As a rule, do as much as you can in SQL (it’s pushed down to the database); reach for a transform when the reshape is awkward in SQL, needs to react to client-side state, or operates on data the database returned opaquely (a JSON column, an array).

They chain. Standalone, each takes a [source, options] tuple; the natural way to apply several is an @expr/pipeline, which feeds each step’s result into the next. The per-operator options are in the expression reference.

Table factories

Table factories create typed Arrow tables from authored data or workspace files:

@expr/from_rows takes an array of row objects. Missing fields become null.
@expr/from_arrays takes a map of column names to equal-length arrays.
@expr/from_columns takes a map of typed column definitions.
@expr/from_csv reads a CSV file and infers unspecified column types.
@expr/from_json reads a JSON file as one row with one json cell holding the whole document.
@expr/from_arrow reads an Arrow IPC stream or file.

Use @expr/from_rows for small inline datasets:

{
  "@expr/from_rows": [
    { "service": "checkout", "status": 200 },
    { "service": "payment", "status": 503 }
  ]
}

@expr/from_arrays infers types in its short form. Its expanded form accepts a partial types map; columns absent from the map are inferred.

{
  "@expr/from_arrays": {
    "arrays": {
      "service": ["checkout", "payment"],
      "status": [200, 503]
    },
    "types": { "status": "int32" }
  }
}

@expr/from_columns declares every column’s Arrow type beside its values. Nested types use field definitions with type and optional nullable fields.

{
  "@expr/from_columns": {
    "foo": {
      "type": "int32",
      "values": [1]
    },
    "bar": {
      "type": {
        "struct": {
          "foo": {
            "type": {
              "list": {
                "type": { "utf8": { "nullable": true } }
              }
            }
          }
        }
      },
      "values": [{ "foo": ["bar", null] }]
    }
  }
}

Primitive type names follow Flechette’s Arrow vocabulary: null, bool, signed and unsigned int8/int16/int32/int64, float16/float32/float64, binary, utf8, date and time units, and large binary/string types. Parameterized types include lists, structs, maps, fixed-size values, timestamps, durations, intervals, decimals, and dictionaries. The generated JSON Schema completes the available variants and their required parameters.

File factories accept a path string as their short form. The expanded form uses file and adds format options. Paths follow the same rules as a template ref: a path that starts with @ (for example @frames/@opentelemetry/cloud_pricing.arrow) is a workspace path, and any other path resolves against the directory of the file that authors the expression, including a *.overrides.*, *.local.*, or templates file. A path computed by an expression, such as a template parameter, must start with @.

{ "@expr/from_csv": "./my_file.csv" }

{
  "@expr/from_csv": {
    "file": "./my_file.csv",
    "columns": [{ "type": "int32" }, { "type": { "struct": { "foo": "utf8" } } }]
  }
}

The columns entries apply by CSV position. An optional name renames the output column. Unspecified entries are inferred from CSV values. CSV columns cannot use binary Arrow types.

@expr/from_json reads a whole JSON document into one cell, so a block that takes a table can show or pick from a workspace file — a @block/kv over a workflow file, an @expr/get into a settings document. The column is value unless column names it; an array document is one cell holding an array, not one row per element.

{ "@expr/from_json": "./detectors.workflows.json" }

{ "@expr/from_json": { "file": "./detectors.workflows.json", "column": "definition" } }

{ "@expr/from_arrow": "./snapshot.arrow" }

Every factory reads 64-bit integers as bigint, the same as a database result, so a factory table is compared, joined and sent to an SQL worker like a queried one.

Every factory input is expressible. Use an expression as the whole input, or wrap a structure in @expr/resolve when individual fields depend on expressions. @expr/data_table remains available as a broad runtime coercion boundary; authored frames should use the factory whose input shape they intend.

`@expr/derive` and `@expr/map`

Add or rewrite columns. Use @expr/derive to add computed columns — a CEL expression per new column, seeing the whole row. Use @expr/map to replace existing columns in place (for example, parse a column through another operator). @expr/derive is the one you’ll reach for most; it’s the client-side equivalent of adding a SELECT … AS you couldn’t push into the query.

{
  "@expr/derive": [
    { "@expr/query": "SELECT Duration FROM spans" },
    { "duration_ms": "Duration / 1000000" }
  ]
}

`@expr/merge_map`

Builds a map-valued column from named entries, layering them over the column named by into when it already holds one. Entries use the @expr/derive vocabulary — a CEL expression or an operator per key — and the column comes out as Map(String, String), the shape an OTel collector writes attributes in. An entry that evaluates to null removes its key rather than leaving the value underneath in place.

{
  "@expr/merge_map": [
    { "@expr/query": "SELECT ServiceName, Region FROM spans" },
    { "into": "attributes", "entries": { "service.name": "ServiceName", "region": "Region" } }
  ]
}

Chain steps to build one map up from several sources — a later step’s entries win on a repeated key.

`@expr/filter`

Keeps rows matching a CEL predicate. Prefer a SQL WHERE when you can; use this when the condition depends on something only available client-side (a piece of state, a derived column).

{
  "@expr/filter": [{ "@expr/query": "SELECT StatusCode FROM spans" }, { "where": "StatusCode > 0" }]
}

`@expr/grouping_set`

Returns the rows of one set from a GROUPING SETS result, without the label column. Use it when a @block/context runs one grouping-sets query and the blocks below each show a different set. set names the set; the other options are a query body that runs over the set’s rows. Grouping sets describes the query that produces the input.

{
  "@expr/grouping_set": {
    "from": { "@expr/get": "runs" },
    "set": "by_workflow",
    "select": ["WorkflowRef", "runs"]
  }
}

`@expr/cel`

Evaluates a CEL expression per row against the table’s columns plus context. It’s the building block under derive / filter; reach for it directly when you need a single computed column expressed in CEL rather than SQL.

{
  "@expr/cel": [
    { "@expr/query": "SELECT StatusCode FROM spans" },
    { "expression": "StatusCode >= 400" }
  ]
}

An identifier in a CEL expression names a column, falling back to a context key. When the name isn’t spellable as an identifier — it holds spaces, dots, or dashes — write it as id("…"): id("http.status_code") >= 400 reads the column of that name, where a bare http.status_code would parse as a field access on http. The argument must be a string literal (the name is resolved before any row is evaluated); everything else about it is identical to writing the identifier inline. This applies wherever CEL does, derive and filter included.

`@expr/json_parse`

Parses a string column as JSON — the way to crack open a log body or an attributes blob the database handed back as text, so downstream steps can read its fields.

{
  "@expr/json_parse": [{ "@expr/query": "SELECT Body FROM logs" }, {}]
}

`@expr/unnest`

Expands an array column into one row per element — turn a list of tags or a repeated attribute into rows you can group or chart. On a nested column produced by @expr/nest it unpacks instead: each group’s rows come back as flat columns, restoring the pre-nest shape.

{
  "@expr/unnest": [{ "@expr/query": "SELECT Tags FROM events" }, { "column": "Tags" }]
}

`@expr/nest`

The inverse of @expr/unnest: groups by key column(s) and folds each group’s remaining rows into the sub-table-valued into column, leaving one row per distinct key. Each into cell reads as a table in its own right, everywhere the column is consumed. The main use is per-row mini charts: a @block/table defer entry can load one timeseries query for every row, @expr/nest it by the join key, and on attaches each row’s sub-table — which a block-column plot then reads by column name, with no per-row queries.

A nested column is opaque to value-rewriting transforms — @expr/densify, @expr/derive/@expr/map over it, and CEL expressions referencing it refuse with a clear error rather than corrupting the sub-tables. Reshape the flat series before nesting (densify, then nest), or @expr/unnest the column to get the flat rows back.

{
  "@expr/nest": [
    {
      "@expr/timeseries_query": {
        "from": "events",
        "select": { "as": { "Events": "count()" } },
        "group_by": "Sid"
      }
    },
    { "by": "Sid", "into": "series" }
  ]
}

`@expr/densify`

Fills the gaps in a table so every cell of its along × by grid has a row, with a fill value (0 or null) where nothing was measured. A @block/plot gap-fills its own marks already, so use this where something else needs the dense rows: a @block/table that should list every bucket, or a transform that runs over them.

Run along the bucket column of a @expr/timeseries_query result and the grid is the query’s own: every bucket of the window, whether or not a row came back for it. Run it along any other column, or over a table that never ran a windowed query, and the grid is every combination the rows already carry. An empty result stays empty either way, as does one whose rows a limit or a where shaped — a bucket missing there is one that shaping dropped.

It builds every along × by combination, so by must name a column with a bounded set of values — a series per user or per trace id is refused rather than materialized. by also cannot name the along axis itself.

fill defaults to null — a cell nothing was measured in, which a chart draws as a break. Only such a cell is filled: a value the query returned is kept as it is, a returned NULL included. A 0 only reaches a numeric column; anywhere else the inserted cell is null.

Over a bucket column, one series needs no by at all:

{
  "@expr/densify": [
    { "@expr/timeseries_query": { "from": "metrics", "select": "count()" } },
    { "along": "_time_bucket", "fill": 0 }
  ]
}

Grouped by a series column, over rows the database already returned:

{
  "@expr/densify": [
    { "@expr/query": "SELECT bucket, service, count FROM metrics" },
    { "along": "bucket", "by": ["service"], "fill": 0 }
  ]
}

`@expr/changes`

Detects changes in a series — spikes, dips, steps, trend changes, distribution changes, and a series-level non_stationary verdict — and annotates the source table with them. Every input row passes through unchanged, and seven columns are appended: kind, direction (up/down), magnitude, severity, provisional, and the level value_before/value_after the change — null on every row except the one a change lands on. One pipeline therefore feeds the series line, the annotation marks, and a table; @expr/filter with !is_null(kind) keeps only the change rows. A row carries at most one annotation, and a stationary series appends all-null columns.

value names the series to scan: a column of the source table, or a SQL value expression the source query computes — the computed column is named by its text, so "value": "count()/60" scans a column named count()/60. A paired column (@expr/paired_query) is scanned on its foreground, and a grain struct on its per-bucket half. along defaults to the source’s time column (a @expr/timeseries_query result works with no along at all); by detects per series. as renames the appended columns ({ "kind": "change_kind" }) — the way out when the source already uses one of their names. kinds narrows to specific kinds of change, min_magnitude (default 0.2) drops changes too small to matter, and max_changes keeps the largest. min_segment_size (default 5) sets how many points a regime needs before it counts. Each of these props also accepts an expression, so the scanned measure or a threshold can follow a control.

Where the series comes from

from takes the same vocabulary as a block’s from:

A view table name ("from": "spans") runs one implicit @expr/timeseries_query: value becomes a select entry, by becomes select entries plus group_by dimensions, and the query brings the window binding, the bucket grid, and view measures. "from": "spans", "value": "count()" is the whole spelling of “scan the request rate”.
A tagged query ({ "timeseries_query": { … } }) keeps the authored body as the statement; a value / along / by its select does not already project is appended as a column named by its text, and a timeseries_query body also groups by the by / along dimensions. Reach for it when the query needs a where, its own resolution, or a shape the implicit form cannot spell.
Anything else — an expression, a context key bound to a query result, inline rows — resolves to a table first. Plain column names annotate that table directly, with no query; a computed value wraps it in SELECT *, <expression>, shipping the rows along as an external table.

Detection is PELT segmentation (Killick et al. 2012) with segmented-regression classification, plus a robust outlier scan for spikes and dips; multiple changes in one window are found in a single pass. Statistical significance gates internally at p ≤ 0.05, and the p-values are not reported: they rank within one series only, while magnitude compares across series and kinds.

`magnitude` and `severity`

magnitude is how much the series moved, in units of its own standard deviation: the level difference for a step or a transient, the displacement the new slope accumulates for a trend change, the shift in spread for a distribution change. Because it is size rather than evidence, it stays meaningful on a near-noiseless series where every change is overwhelmingly significant — over a 1173-series production corpus the median step moved the level by 0.06 standard deviations while testing below 1e-4, which is what the min_magnitude floor of 0.2 exists to drop. Set it to 0 to see every significant change whatever its size.

severity bands magnitude for attention: info below 1, warning from 1 (the change is as large as the series’ own noise), critical from 3 (the classical three-sigma bar). Rank and correlate with magnitude; color and triage with severity.

Two spikes in the same buckets of two different series are candidates for a common cause, and so are a spike in one and a step in another — kind, the shared axis value, and magnitude together are the correlation vocabulary.

`provisional`

A change is flagged provisional: true when its remaining evidence lies in the future: fewer than min_segment_size points follow it, and the buckets that would settle it have not passed yet on the wall clock. Such a change is real in the current window, but its kind and scores can still shift as points arrive — a step entering a live window first reads as a dip, and a partial trailing bucket reads as an excursion. Treat provisional rows as “happening right now” and expect them to be re-labelled on the next refresh. A historical window reports nothing provisional: a change close to a crop edge may be re-scored by querying a wider window, but no amount of waiting will change it. Detection on a non-time axis (or a table without a time context) has no wall clock, so there the flag marks edge truncation on either side. Inside a workflow run the flag is judged at the occurrence window’s end instead of the wall clock: it asks whether the buckets that settle the change fall inside this occurrence’s series, so a resumed or replayed occurrence leaves the same changes for the next occurrence as an on-time run would.

Give it a bucketed series

The detector is calibrated for a bucketed timeseries — a @expr/timeseries_query result, or any query with one row per interval. Shapes to avoid:

Raw event rows. Point it at unbucketed rows (one per span or log line) and it over-reports heavily: 5000 raw span durations from a healthy service produce dozens of changes, because durations are lognormal and many rows share a timestamp. Aggregate to buckets first. A series is capped at 10 000 points and refuses beyond that.
A window that does not land on bucket boundaries. A partial bucket at either edge is a real low value in the data, and it is reported as a dip at the first or last point (flagged provisional when the window ends at now). @expr/timeseries_query aligns the window for you; a hand-written GROUP BY toStartOfInterval(…) over a now-relative range does not.
A smooth curve is reported as one thing. Segments are straight lines, so acceleration can only be expressed as a trend change at every bend — and when every bend goes the same way (the slope only ever increased, or only ever decreased), that staircase collapses to a single non_stationary row: located at the first point of the window, pointed in the drift’s net direction, and sized by how far the drift departs from the best straight line. On the smoothly accelerating CO₂ benchmark series this replaces 5 trend changes with one verdict; a staircase that bends both ways keeps its trend changes, because alternating slope changes are regime changes (the benchmark series with human-agreed knees keeps all of them). A steady linear drift is a single segment and reports nothing at all.

A repeating cycle is handled for you: seasonality defaults to "auto", which tries an hour, a day and a week against the time axis, falls back to autocorrelation, and takes the cycle out before scanning when it explains enough of the variance — on the Numenta benchmark’s anomaly-free daily series that is the difference between 56 reported changes and zero. It needs at least four full cycles in the window to estimate a profile, and it leaves the series alone when nothing repeats strongly enough. Pass a number to name the period in points yourself, or "none" to scan the series as given. A cycle that does not follow the clock — a GC sawtooth follows allocation rate — is not found; scan a derived signal instead (e.g. a per-15-minute peak envelope, or jvm.memory.used_after_last_gc).

The number of reported changes grows with the number of points, at roughly 3–7 per thousand on real operational metrics. A window of a few hundred buckets reports a handful; a window of several thousand reports dozens. Scan a series at the resolution you would look at it, not the finest one available.

Two more things worth knowing. min_segment_size counts points, not time, so a gap in the data does not widen a segment. A null cell means “no observation here” and the point is skipped; a 0 is a real zero. @expr/densify’s fill chooses between them, and the choice changes the result — filling an idle period with zeros turns it into a regime the detector can find a boundary against.

{
  "@expr/changes": {
    "from": "spans",
    "value": "countIf(HttpStatusCode >= 400)",
    "by": ["ServiceName"],
    "kinds": ["step", "spike"]
  }
}

`@expr/baseline`

Models what a series is expected to be and annotates the source table with the expectation: every input row passes through unchanged, and five columns are appended — expected, the corridor band_low/band_high, residual (value − expected, positive above expectation), and model, which names the model that actually produced the expectation for that row’s series. as renames the appended columns. One pipeline feeds the series line, the corridor band, and anything that reads the residual — including @expr/changes over residual, which detects changes relative to expectation instead of relative to level.

from, value, along, and by work as they do on @expr/changes — the same source vocabulary, computed value expressions and paired/grain columns included; the model is fitted per by series, from this window’s own rows only. model picks what expected is:

{ "cycle": { "period": "auto" } } (the default) — a repeating cycle’s per-phase median profile. auto tries an hour, a day and a week against the time axis, then autocorrelation, and uses the period only when the cycle explains enough of the window’s variance; a named period ("day") or one in points (96) skips that gate and needs only two full cycles of data. When no usable period exists the fit falls back to level, and the model column says so.
{ "trend": {} } — a robust straight line: least squares with one refit that excludes residuals past three robust σ, so a transient spike does not tilt the expectation it is judged against.
{ "level": {} } — the median.

band (default 2) is the corridor’s half-width, in robust standard deviations of the residuals around the expectation — the median absolute deviation scaled to read as a σ, so an outlier widens the corridor no more than any other point, and a strong cycle’s own amplitude does not count as spread. Rows with a null axis or value cell take no part in the fit and keep null appended cells; run @expr/densify first when empty buckets should pull the expectation toward zero.

{
  "@expr/baseline": {
    "from": "spans",
    "value": "count()",
    "by": ["ServiceName"],
    "band": 3
  }
}

`@expr/handlebars`

Renders a Handlebars template against context — the simple way to build a string out of state and values (a title, a label, a URL). Not a table transform, but it lives here as the everyday string-templating tool.

{ "@expr/handlebars": { "template": "{{method}} {{path}}" } }

Transforms

Table factories

@expr/derive and @expr/map

@expr/merge_map

@expr/filter

@expr/grouping_set

@expr/cel

@expr/json_parse

@expr/unnest

@expr/nest

@expr/densify

@expr/changes