Skip to content
NOEMATA

Operating workflows

What the scheduler reads

The scheduler starts only when workflows.enabled: true is set in the deployment config (noemata.json for CLI-managed instances). The default is false. Setup offers true, or false for a checkout that drafts on disk against a deployment in the store. Setup keeps a previous decline, and under fs: local it asks which workflows to run. Changing either setting requires a server restart. Individual workflow enablement still applies.

workflows.selection decides which workflows the scheduler runs:

  • non_local — the shared workflows of the published revision. An edit takes effect once noemata publish makes it part of a revision.
  • local — the workflows with a local source or override (a *.local.* file or a .local directory), read from the draft of the scheduler’s principal: the workspace directory under fs: local. A saved edit takes effect at the next occurrence. Each runs under a ref scoped to the instance, with its own state, so the local variant of a shared workflow never shares state with the shared one.
  • all — both.

The default is all under fs: local and non_local under fs: remote. fs: remote runs only non_local: a server with scheduling enabled refuses to start with another selection, because the instance has no disk for the state of local workflows. The scheduler lists the definitions again before each tick.

Remote schedulers retain file contents by path and hash in a service-account cache, with a 32 MiB content budget and a 4 MiB per-file limit. Unchanged definitions reuse their listing within a tick and their contents across ticks. Current-state reads refresh metadata before reusing contents. Conditional writes and workflow-data durability checks consult storage. Cache budgets exclude metadata, in-flight responses, and parsed frame objects. API and browser reads do not use this scheduler cache.

A checkout that drafts against a shared deployment should not schedule workflows. noemata up --edit turns scheduling off for that run. A checkout of a production config then never claims production occurrences or sends deliveries from a developer’s machine, and the deployment’s own servers run the workflows.

enabled: false is configuration and stops scheduling. A workflow with schedule: false never runs today; UI and API execution triggers are planned.

Execution deadlines

In workflow SQL, use fromUnixTimestamp64Milli(_time_end) for comparisons relative to the evaluation window. now() reads the database clock and does not follow a historical --to bound. Current system tables still describe current state regardless of the requested window.

The execution budget starts at dispatch admission, before coordination and current-state preparation. Preparation time reduces the budget available for pool admission, runtime preparation, and execution. A run whose preparation exhausts its budget fails before evaluating a step. Coordination and storage operations are charged after they settle. Successful state publication is included in the same budget; an overrun reports failure without undoing already published state or emitted records. Run-history export is best-effort and is not awaited.

Cooperative continuation

Detectors can finish a bounded initialization batch and yield their worker slot. The scheduler publishes that batch’s progress as current workflow state before continuing. Intermediate progress updates and completed step outputs remain in memory; a failed batch does not replace the last committed batch.

Admission rotates between eligible workflows. A yielded stateful workflow retains its lease while waiting for its next slice. A live continuation keeps the same occurrence and frozen initialization horizon. After a crash, that occurrence is not resumed. The next scheduled occurrence uses the committed initialization progress under a new run ID. A definition or borrowed-view change discards incompatible initialization progress. A yield emits an info-level run record with run.outcome: yielded.

appears, disappears, and value_change seed in batches whose duration defaults to the source frame’s inherited settings.default_timerange, with a fifteen-minute fallback. Set initialization.max_source_range for a shorter or longer batch. Set initialization: { "batch": false } when a derived source requires its whole horizon: current inventories, cross-boundary joins, ranking, or whole-window aggregates. The runtime does not infer batchability from SQL. Each unfinished batch persists its seed, processed boundary, and chosen batch width before yielding. Continuation retains that width and the original horizon. Counters and emitted records do not advance during initialization. Sighting seeds preserve earliest/latest timestamps; value seeds preserve the latest value. Batch duration limits the source time range, not query CPU, row count, or memory.

Each scheduled continuation receives a fresh execution timeout at admission. Burn-rate component maintenance also preserves its cumulative catch_up.max_duration across source-batch continuations, excluding the time waiting for admission. The timeout bounds an execution slice, not total initialization time. Online validation follows continuations within its original overall timeout and does not publish their state or emitted records.

Workflow views

A workflow first borrows the frame with the same filename stem: foo.workflows.json uses foo.frame.json, and index.workflows.json uses index.frame.json. If that frame is absent, lookup follows the containing folder’s route to its nearest frame. A workflow-level use applies this lookup to the template file instead; foo.templates.json first looks for foo.frame.json.

Scheduled workflows need values for every param in their source frame. A workflow under a dynamic route such as services/{ServiceName} has no URL value for ServiceName, so validation rejects the workflow and the scheduler skips it. Place the workflow beside an unparameterized frame and group its evaluation by the entity column (group_by: "ServiceName"), or beside a concrete route such as services/redis. A concrete route supplies the param value. Workflows with schedule: false can use a dynamic route because a later trigger can provide the value. For template-based workflows, validation uses the template’s source frame. Moving the configured instance does not change the source frame.

Workflow state locations

Each workflow has one current record containing its last attempted occurrence, state version, and a pointer to committed detector state. With ClickHouse Keeper coordination, that record is stored atomically in the coordination table under r:<path of schedule.json>. Without Keeper, a single scheduler process stores it as a file at .data/coordination/<database>/<namespace>/<path of schedule.json>, where <database> is the database that holds the file store. Multiple scheduler processes require Keeper.

For @frames/payments/checks.workflows.json#errors, state payloads are stored at:

@frames/payments/.shared/workflows/checks.workflows.json/errors/
  schedule.json                 # only without Keeper
  state-<timestamp>-<hash>.bin   # immutable current-state payloads

State payloads use the existing workflow_state area and schedule_state_retention_days TTL. Each successful publication writes a new payload. Expired state starts empty; an expired idle cursor discards catch-up debt. Keeper current records remain small, one per workflow, and are logically expired on read. Payload files are excluded from version control, publication, and full-text indexing. File-store access policies still apply to their owning folder.

Renaming or moving a definition changes its workflow reference and starts a separate state chain. Moving a referenced template does not move that state.

Run history

Run history is server telemetry: noemata.workflow.run records and workflow.run spans. There are no durable run directories, definition snapshots, or completed-step journals. History export does not block execution. A killed process may emit no terminal record; inferred crash reporting is not implemented.

Admission consumes the occurrence before execution starts. Successful completion publishes detector state; failure preserves the last committed state. Publication writes the immutable payload first, then conditionally replaces the current pointer. A stale runner cannot overwrite a newer admission’s state with Keeper coordination. A crash between those writes leaves an unreferenced payload that expires under the state retention policy.

Each run uses the live resolved definition and borrowed frame view supplied at admission. Threshold recommendations retain their counters and open instances in current workflow state.

Versioning a workflow’s state

Most edits keep the carried state meaningful — a loosened threshold resolves its firing instances at the next run, a new attributes entry decorates the next event. An edit that changes what the state means does not: a different group_by makes every stored row an orphan the pipeline can never reconcile, and a renamed @expr/workflow_state key leaves the old one carried forever. For those, bump the definition’s version (an integer, default 1):

{
  "title": "Service error rate over 5%",
  "version": 2,
  "schedule": { "every": "1m" },
  "execute": { "…": "…" }
}

The next occurrence starts from empty state. State also expires under the configured retention policy. Definition edits do not bump the version for you.

Delivery semantics

  • Failed and crashed occurrences are not retried. Admission advances the cursor before execution. A subsequent occurrence starts from the last committed detector state. Watch run telemetry and server logs for repeated failures.
  • Emission is best-effort and non-blocking. Queue overflow or a process crash can lose batches. State publication and emission are not atomic, so subsequent occurrences may emit an already delivered finding. Consumers should deduplicate on noemata.event.id. Per-row projection and delivery failures are reported asynchronously without failing the workflow. Online validation still reports invalid projections. Configure capacity and overflow through workflows.delivery.
  • An invalid definition is skipped without advancing the schedule. A workflow that fails validation (or whose borrowed view no longer compiles) is skipped each tick with a warning until fixed; the schedule then plans catch-up as if it had been offline.

Schema upgrades

Stop all old scheduler instances before deploying this execution model. Mixed-version scheduling is unsupported: old coordination cleanup does not preserve current-state records. Existing deployment state must be migrated once out of band or explicitly reset before starting this version; the runtime does not import older formats. Old run journals are not read or resumed. Keep their configured TTL until they expire.

The workflow_state area pattern is **/.shared/workflows/*/*/state*. Existing ClickHouse stores must update their materialized area expressions before new state payloads are written. No new area ID or sorting key is required. Do not run the new scheduler against an old area expression: its payloads would miss the state TTL. Apply this deployment migration out of band; startup does not rebuild area expressions.

ClickHouse file-version tables use a materialized area ID and are ordered by area, path, namespace, and version. Blob tables use the same area classification, are ordered by area, path, and hash, and have a primary key of area and path with a 1 MiB byte-granularity target. Blob reads constrain both area and exact path/hash identities. Every client must use area declarations matching the tables’ materialized expressions. Existing pre-release stores must be migrated out of band or recreated before using this schema; startup does not rebuild populated tables or support legacy layouts.

The file store carries the tables and retention policies (run_retention_days, schedule_state_retention_days) workflow state and retained legacy run files use. A record of what the deployment’s tables carry is kept in the store itself, at .shared/schema/file_store.json. noemata init, noemata config, and noemata up apply the schema when that record does not match the binary or the retention config: they claim the upgrade through the deployment’s coordinator (the local coordinator, which stores its values under .data/coordination/, or ClickHouse Keeper when configured) and apply additive, idempotent DDL, so an initialization that dies mid-way is replayed by the next, and a concurrent initialization waits for the record to match. Under ClickHouse Keeper, the same commands first create the coordination table, which holds that claim.

The server applies no DDL. It refuses to start while the coordination or sessions table is missing or the record does not match its binary and retention config, and its log names the command to run; an instance whose binary is older than the record refuses to start until you roll it forward. The commands run with the operator’s ClickHouse credentials, which need CREATE DATABASE, CREATE TABLE, and ALTER on the file store’s database and, under Keeper, on the coordination table’s database, and CREATE TABLE in the data database when the server stores sessions there. The server’s standing identity needs no DDL grant, and it reads and writes every session, so users who log in need no grant on the sessions table. Under fs: local there is no file store and nothing to upgrade.