Dataset

The Dataset node is where Solaris AI Flow remembers outcomes. Where the Storage node is a small per-workflow dedup cache, a Dataset is an append-heavy table scoped to your whole account, so many workflows can write into one dataset and a separate workflow can read across all of them.

This is the primitive behind deterministic scoring. A trading workflow appends each closed trade's outcome; a nightly scoring workflow aggregates the last 30 days to rank strategies. Because every workflow you own can write to the same dataset, and rows stick around for weeks rather than minutes, you can score across your entire account, not just one flow. Rows do expire, though: see Retention before you build anything that assumes a row is still there months later.

Storage vs Dataset

Pick the right tool:

• Storage is a key/value + list cache scoped to a single workflow, with mandatory short TTLs. Use it for dedup ("have I seen this mint?") and small per-workflow state.
• Dataset is a structured, queryable table scoped to your account, holding rows for weeks (30 days by default, 90 at most). Use it to capture outputs over time and compute scores, counts, and averages across workflows.

If you find yourself trying to log a growing history into Storage, or trying to read one workflow's Storage from another, you want a Dataset.

Prerequisites

• No API key required
• Manual dataset operations (append, query, and so on) are available to everyone
• Auto-capture (passive logging) is available to every account

Operations

Append Row

Adds a row to the dataset, creating the dataset on first write. The row is a JSON object of the small, queryable fields you want to keep, for example {"mint":"{token.mint}","pnl":1.2,"strategy":"sniper"}. Each row is automatically stamped with a creation time and the ID of the workflow that wrote it, so a single dataset can attribute rows back to multiple source workflows.

Keep rows to scalars (strings, numbers, booleans) you actually filter or aggregate on. The row is what lives in the queryable store, so smaller rows mean more history under the per-dataset size cap.

Upsert Row

Like Append, but first looks for an existing row whose match field equals the resolved match value. If found, that row is overwritten; otherwise a new row is appended. Use this for "one row per token" style datasets where re-processing the same token should update rather than duplicate.

To guarantee a unique match, upsert scans every row, so it is only available while a dataset stays within the Query / aggregate scan window (3,000 rows). That is below the per-dataset row cap (20,000), so upsert works until a dataset grows past the scan window; beyond that it errors rather than risk silently inserting a duplicate it couldn't detect; switch to Append, lower the dataset's retention, or split it. (Indexed keys for unbounded upsert are planned.)

Query

Returns rows matching an optional filter array, newest first by default. Filters are a JSON array like [{"field":"pnl","op":"gt","value":0}]. Supported ops are eq, ne, gt, gte, lt, lte, and contains. Ordering and comparison filters operate on numeric fields. Results are capped by the limit field (default 200, max 1000) and the scan window.

Field names are top-level only. Filter field, group by, and the aggregate field all read a row's top-level keys. A dot path like trade.pnl is treated as the literal key "trade.pnl", not a nested lookup, so a nested value silently won't match. Keep the fields you score on flat (the way auto-capture writes them), e.g. store pnl rather than {"trade":{"pnl":1.2}}.

Aggregate

The scoring engine. Computes count, sum, avg, min, or max over the rows matching your filters. With a group by field, it returns one bucket per distinct value, for example average PnL grouped by strategy. count needs no field; the other ops require a numeric field. This is how you turn a long log of outcomes into a single deterministic score.

Delete Rows

Removes rows matching the filters, operating over the same recent window as Query and Aggregate (the most recent rows, newest first) and deleting up to a batch per run. The hasMore flag in the response indicates more matches remain; re-run to keep draining. For routine expiry use retention, and to wipe an entire dataset use the Delete button in the Datasets manager rather than this node.

Configuration

Dataset, Row, Match Value, and Filters all accept template expressions, so you can build rows directly from upstream node output:

tradeOutcomes
{"mint":"{token.mint}","pnl":{trade.pnl},"strategy":"sniper"}

Auto-capture: passive logging on any node

Manually wiring an Append node into every workflow is tedious. Instead, most nodes can passively capture their own output. Open a node's config and turn on Save output to Dataset (an optional, off-by-default panel in node settings), pick a dataset, and map fields:

Dataset:  signalOutcomes
score   ← {output.score}
mint    ← {token.mint}

After every successful run, that node's output is appended to the dataset automatically, with no extra node on the canvas. This turns every workflow into a passive data generator for scoring. In capture field templates, the node's own output is available as {output...} and under its response name.

The toggle isn't offered on triggers or on the Dataset node itself (a trigger has no scored output, and the Dataset node already writes rows), so they never capture.

Auto-capture is available to every account. Toggle it per node via Capture output → Dataset. Captured values are coerced to numbers and booleans where possible so they aggregate cleanly. Bulky raw outputs are archived to R2 via raw-blob offload.

Templating against the response

Each Dataset operation writes its result to datasetResponse (override via the Response Name field):

All outputs include success: true, operation, and dataset alongside the operation-specific fields.

Query and aggregate return two completeness flags, and they mean different things:

• scanTruncated: true means the dataset is larger than the scan window, so the result was computed over only a partial slice: the most-recent rows (or, for an oldest-first query, the oldest rows). This is a correctness signal: a score over part of the dataset, not all of it.
• truncated: true is a superset of scanTruncated. It is also true for benign output caps even when the whole dataset was scanned: a Query returned fewer rows than matched (the Limit or the response-size budget), or a grouped Aggregate hit the distinct-group cap. When truncated is true but scanTruncated is false, the computed value is complete; only the returned list was capped.

Use scanned (rows actually read) against rowCount (the dataset's live total) to gauge coverage. Treat scanTruncated as the flag for "is this score over the full dataset?"; treat the bare truncated as "is there more output I'm not seeing?".

The Datasets manager

Datasets in the sidebar lists every dataset your account owns, with its row count, stored size, retention window, and when it was last written to. A storage bar tracks the account total against your cap, and warns as it approaches it.

Click a dataset to open its own page. The header carries the name (the string your nodes address it by), an editable description, and the authoritative row count and size read straight off the dataset record. From here you can export or delete it.

Three tabs sit underneath:

Rows shows the most recent captured rows, newest first. Each row gives you when it was captured, a preview of its data, when it expires, and which workflow wrote it. Search filters across the loaded rows. Click any row to open the full JSON with its provenance: the source workflow (linked, so you can jump back to the flow) and the source node. If a row's raw output was archived to R2, you can open the blob from here.

Schema infers the shape of your rows: each top-level field, the types seen for it, the percentage of rows it appears on, and a sample value. A field below 100% is optional, which is usually what breaks a downstream consumer that assumed it was always there.

Sources groups the rows by the workflow that wrote them, with a row count, how many distinct nodes wrote, and the last write time. This is how you find out that a dataset you thought one flow owned is actually being written to by three.

Rows, Schema and Sources describe a sample, not the whole dataset. They are computed from the latest 200 rows loaded into the page, and each says so inline. The row count and size in the header are the real, whole-dataset figures.

Exporting

Export a dataset as JSON or CSV. The export streams the dataset page by page, so it only reads what you actually download.

• JSON carries an envelope (dataset, exportedAt, rowCount, partial) around a rows array, each row with its full provenance.
• CSV flattens top-level row fields into data.* columns when most rows are flat objects; otherwise it emits data as a single JSON column.

Exports are capped at 25,000 rows or 10 MB, whichever comes first. A capped export is not silent: the file is marked -partial in its filename and partial: true in the JSON envelope. Rows whose raw output was archived to R2 are flagged (rawOutputArchived), not inlined, so nothing downstream mistakes the stored row for the full payload.

Retention: what expires, and what survives

Every row is deleted a fixed number of days after it was captured, not after the dataset was last touched. The default is 30 days; a dataset can set its own via Retention Days, up to the 90-day maximum. An explicit value is always clamped back down to the current maximum, so a longer window set under a higher tier can't outlive a downgrade.

An hourly background sweep does the deleting, so a row can outlive its cutoff by up to an hour before it actually disappears.

Retention never deletes the dataset itself. The name, description, and settings survive; only the rows go. So a dataset that nothing has written to in over 30 days will read 0 rows · 0 B while still existing, and it will fill straight back up the next time a workflow writes to it. An empty dataset is almost always an idle one, not a broken one.

Two places show you this before it bites: the Rows tab has an Expires column giving each row's remaining life (amber in its final day), and the dataset header names the retention window.

If you need a row to outlive its retention, export it.

Limits

Caps are enforced server-side and apply to every account (universal, while datasets move to credit metering):

The scan window is the number of most-recent rows a single Query or Aggregate folds in. For large datasets, narrow the result with filters; whole-dataset scoring at scale is on the roadmap (precomputed rollups). Retention is enforced by an hourly sweep over rows, not datasets, as described in Retention. Auto-capture and raw-blob offload (archiving bulky raw outputs to R2) are available to every account; these may move to credit metering later.

Marketplace and ownership

Datasets are scoped to your account, and nodes reference them by name. A marketplace clone resolves that name against the buyer's own datasets (created on first write), so an imported template never reads or writes the author's data. Rows are never shared between accounts.

Common use cases

• Strategy scoring: append each trade's PnL and strategy, then aggregate avg PnL grouped by strategy over a rolling window
• Signal quality tracking: capture every AI signal's score and later outcome, then score precision over time
• Cross-workflow counters and outcome logs that survive long enough to analyze

Example: score strategies from captured trades

Two workflows share one dataset:

1. Trading workflow closes a trade, then a Dataset (Append) node writes {"strategy":"{cfg.strategy}","pnl":{trade.pnl}} to tradeOutcomes. (Or turn on auto-capture on the trade node and skip the explicit node.)
2. Scoring workflow (Cron Trigger) runs Dataset (Aggregate): op: avg, field: pnl, group by: strategy. Output: {buckets: [{key:"sniper", value:1.8, count:120}, ...]}.
3. Condition / AI ranks the buckets and adjusts which strategies stay active.

Next steps

• Storage: per-workflow dedup and small state
• Condition: branch on {datasetResponse.value} or a bucket
• Code: post-process query rows or compute a custom score