Data Set Management

Quick Navigation

What is a dataset?

After you have connected data sources and set up a functional data catalog, a dataset is how you bundle catalog data for projects and workflows: a display label, optional longer descriptor, a primary entity as the starting point, optional related entities, and per-field write rules — without changing the schema at the source. Workflow steps use that bundle when they need to query records, write values back, or show the right fields.

In the API, label is the main name shown for the dataset; descriptor is extra descriptive text. For click-by-click setup in the product (including starting from the Data Catalog or adjusting field classifications), see How to Create and Manage Datasets in Streamline on the Help Center.

The Data Search Step takes a datasetId so the run queries the right records; the Deliver Data Step uses the same id when writing values back. Create and maintain datasets with the endpoints below (base URL and authentication match the rest of the Streamline API).

Core Concepts

Dataset resource

Responses include the full dataset shape (identifiers, traversal, permissions, audit timestamps, and enriched fields). For create and full replace (PUT), you must provide primaryEntityId and at least one of integrationId or entityGraphId. Since entityGraphId is deprecated, new integrations should always use integrationId. organizationId is set from the authenticated organization on create; you do not need to send it.

name	type	create / PUT	constraints	description
`id`	string (UUID)	returned after create; send on PUT	—	Stable identifier. Use as `datasetId` in workflow steps.
`label`	string	optional	—	Primary display name; recommended for operators and UIs.
`descriptor`	string	optional	—	Longer description.
`organizationId`	string (UUID)	set by service on create	—	Owning organization (from auth).
`integrationId`	string (UUID)	recommended	—	Primary integration; required if `entityGraphId` is not provided.
`entityGraphId`	string (UUID)	deprecated	—	Deprecated. Use `integrationId` instead.
`primaryEntityId`	string (UUID)	required	—	Root entity for queries and writes.
`primaryEntityAlias`	string	optional	—	Friendly alias for the primary entity.
`integrationIds`	array of UUID	response / enrichment	unique items	Integration ids for entities in the computed allow list; not used to satisfy create validation.
`capabilities`	object	response / enrichment	—	Field capabilities (for example `searchOperators`: map of field type to allowed operator names).
`createdAt`	string (date-time)	response	ISO 8601	Creation time (set on create).
`createdByUserId`	string (UUID)	response	—	Creating user when recorded.
`integrations`	array of object	response with `include`	—	Expanded integration records when using `include=integrations`.
`invalidated`	boolean	response	—	When true, treat the dataset as stale; see Invalidation.
`deletedAt`	string (date-time)	response (soft delete)	ISO 8601	Set on soft delete; omitted or null while active.
`dataTags`	array of string	computed on save	—	Derived from catalog tags for entities in the allow list (and related rules).
`entityAllowList`	array of UUID	computed on save	—	Derived from `primaryEntityId` and `traversalRoute`.
`entityFieldPermissions`	array of object	optional	—	Per-entity field ids and `writeMode` (`FULL` or `NONE`).
`traversalRoute`	array of object	optional	—	Route through relationship links (link id, entity id, alias, mode, nested links).
`linkAliases`	array of object	response (derived)	—	Convenience aliases derived from `traversalRoute` for clients.

capabilities is an envelope: searchOperators is an object whose keys are field types (for example scalar categories) and whose values are arrays of operator names the integration supports for search and filter UIs.

Invalidation

When invalidated is true, the dataset should not be treated as current: the underlying data source schema changed, and the existing definition may no longer match reality. Create a new dataset or adjust configuration so workflows and integrations use a valid definition.

📘
If a dataset stays invalidated, steps that rely on it may fail or return unexpected results until you fix or replace the dataset.

Related entities, fields, and traversal

entityAllowList on responses reflects entities reached from primaryEntityId and traversalRoute after save; you do not supply a separate allow list that overrides that computation on create or PUT.
dataTags on responses are derived from the catalog for those entities when the dataset is saved.
entityFieldPermissions controls which fields can be written (FULL) versus read-only (NONE) for deliver operations.
traversalRoute describes how to walk from the primary entity through links (for example, which relationship to follow and how nested segments are ordered).
linkAliases on responses are derived from the traversal route for presentation.

The JSON below shows a typical response shape (including enriched fields). Create and PUT bodies must include primaryEntityId and at least one of integrationId or entityGraphId (prefer integrationId), plus optional label, descriptor, traversalRoute, entityFieldPermissions, and so on — omit organizationId if you rely on the token's organization.

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Example dataset",
  "descriptor": "Optional longer description",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityAlias": "customers",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-01-15T10:00:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "invalidated": false,
  "dataTags": ["analytics"],
  "entityAllowList": ["e1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ],
  "traversalRoute": [],
  "linkAliases": []
}

How-To Guides

Let's explore common scenarios for managing datasets.

List datasets

Use list organization datasets to retrieve a paginated list of datasets for your organization.

GET /v1/activation/dataset

name	type	required	constraints	description
`entityId`	array of string (query)	no	UUID	Filter datasets by one or more entity IDs.
`label`	string (query)	no	—	Filter datasets by label text.
`pageSize`	integer (query)	no	1–100, default 50	Number of items per page.
`pageNumber`	integer (query)	no	1-based, default 1	Page number to return.
`offset`	integer (query)	no	min 0, default 0	Records to skip before pagination. Can combine with `pageNumber`.
`include`	array of string (query)	no	e.g. `integrations`	Expand related records inline (e.g. integration details).

{
  "content": [
    {
      "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
      "label": "Customer analytics",
      "descriptor": "Profiles, orders, and engagement",
      "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
      "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
      "createdAt": "2025-01-15T10:00:00Z",
      "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
      "capabilities": {
        "searchOperators": {
          "STRING": ["EQUALS", "CONTAINS"]
        }
      },
      "primaryEntityAlias": "customers",
      "invalidated": false,
      "dataTags": ["analytics", "pii"],
      "entityAllowList": [
        "e1b2c3d4-e5f6-7890-1234-567890abcdef",
        "e2b2c3d4-e5f6-7890-1234-567890abcdef"
      ],
      "entityFieldPermissions": [
        {
          "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
          "fieldPermissions": [
            {
              "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
              "writeMode": "FULL"
            }
          ]
        }
      ],
      "traversalRoute": [],
      "linkAliases": []
    }
  ],
  "page": {
    "size": 1,
    "pageSize": 50,
    "currentPage": 1,
    "totalElements": 1,
    "totalPages": 1
  }
}

Create a dataset

Use create dataset to define a new dataset for data activation.

POST /v1/activation/dataset

Required fields:

primaryEntityId — root entity for queries and writes
At least one of integrationId or entityGraphId — identifies the catalog layout. Since entityGraphId is deprecated, new integrations should use integrationId.

Optional fields:

label, descriptor, primaryEntityAlias, entityFieldPermissions, traversalRoute

📘
organizationId is set from the authenticated organization — you do not need to send it. entityAllowList and dataTags are computed on save from primaryEntityId and traversalRoute.

{
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
  "label": "Marketing campaigns",
  "descriptor": "Campaign effectiveness and customer engagement",
  "primaryEntityAlias": "campaigns",
  "entityFieldPermissions": [
    {
      "entityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f5b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ],
  "traversalRoute": [
    {
      "linkId": "l3b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "campaign_metrics",
      "traversalMode": "GRAPH",
      "links": []
    }
  ]
}

{
  "id": "ds3b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Marketing campaigns",
  "descriptor": "Campaign effectiveness and customer engagement",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg3b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-03-10T14:30:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "primaryEntityAlias": "campaigns",
  "invalidated": false,
  "dataTags": ["marketing", "engagement"],
  "entityAllowList": [
    "e5b2c3d4-e5f6-7890-1234-567890abcdef",
    "e6b2c3d4-e5f6-7890-1234-567890abcdef"
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f5b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ],
  "traversalRoute": [
    {
      "linkId": "l3b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "campaign_metrics",
      "traversalMode": "GRAPH",
      "links": []
    }
  ],
  "linkAliases": [
    {
      "linkId": "l3b2c3d4-e5f6-7890-1234-567890abcdef",
      "alias": "campaign_metrics"
    }
  ]
}

Batch lookup by entity

Use get datasets by entity IDs to find datasets grouped by entity ID for multiple entities at once. The response is keyed by entity ID, and whose values are arrays of dataset objects (possibly empty).

POST /v1/activation/dataset/batch

name	type	required	constraints	description
`include`	array of string (query)	no	e.g. `integrations`	Expand related records inline (e.g. integration details).

["e1b2c3d4-e5f6-7890-1234-567890abcdef", "e2b2c3d4-e5f6-7890-1234-567890abcdef"]

{
  "e1b2c3d4-e5f6-7890-1234-567890abcdef": [
    {
      "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
      "label": "Customer analytics",
      "descriptor": "Profiles and orders",
      "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
      "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
      "createdAt": "2025-01-15T10:00:00Z",
      "primaryEntityAlias": "customers",
      "invalidated": false,
      "dataTags": ["analytics"]
    }
  ],
  "e2b2c3d4-e5f6-7890-1234-567890abcdef": [
    {
      "id": "ds4b2c3d-e5f6-7890-1234-567890abcdef",
      "label": "Product catalog",
      "descriptor": "Catalog and inventory",
      "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationId": "i2b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityGraphId": "eg4b2c3d-e5f6-7890-1234-567890abcdef",
      "primaryEntityId": "e2b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationIds": ["i2b2c3d4-e5f6-7890-1234-567890abcdef"],
      "createdAt": "2025-01-18T11:20:00Z",
      "primaryEntityAlias": "products",
      "invalidated": false,
      "dataTags": ["catalog"]
    }
  ]
}

Get a dataset by ID

Use get dataset by id to retrieve a single dataset.

GET /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to retrieve.
`include`	array of string (query)	no	e.g. `integrations`	Optional expanded `integrations` (same as list).

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Customer analytics",
  "descriptor": "Profiles, orders, and engagement",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-01-15T10:00:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "primaryEntityAlias": "customers",
  "invalidated": false,
  "dataTags": ["analytics", "pii"],
  "entityAllowList": [
    "e1b2c3d4-e5f6-7890-1234-567890abcdef",
    "e2b2c3d4-e5f6-7890-1234-567890abcdef"
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        },
        {
          "fieldId": "f2b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "NONE"
        }
      ]
    }
  ],
  "traversalRoute": [
    {
      "linkId": "l1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "customer_orders",
      "traversalMode": "GRAPH",
      "links": []
    }
  ],
  "linkAliases": [
    {
      "linkId": "l1b2c3d4-e5f6-7890-1234-567890abcdef",
      "alias": "customer_orders"
    }
  ]
}

Replace a dataset

Use update dataset to replace the mutable definition. The same rules apply as create: primaryEntityId is required and at least one of integrationId or entityGraphId must be provided. entityAllowList and dataTags are recomputed from primaryEntityId and traversalRoute on save. Include traversalRoute when relationships change.

PUT /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to replace.

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Customer analytics (revised)",
  "descriptor": "Expanded metrics and segments",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityAlias": "customers_v2",
  "traversalRoute": [
    {
      "linkId": "l1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "customer_orders",
      "traversalMode": "GRAPH",
      "links": []
    }
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        },
        {
          "fieldId": "f2b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ]
}

The response body matches the saved dataset (same general shape as GET).

Update label or description

Use patch dataset when you only need to change label and/or descriptor. Other fields are rejected with 400 (see Error responses).

In the product you can also adjust field classifications while editing a dataset; this API operation does not update classifications or field definitions — use the Data Catalog (and the field endpoints described in step docs such as Data Search) for those. Structural changes still require PUT.

PATCH /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to patch.

{
  "label": "Customer analytics (renamed)",
  "descriptor": "Updated description for operators"
}

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Customer analytics (renamed)",
  "descriptor": "Updated description for operators",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-01-15T10:00:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "primaryEntityAlias": "customers",
  "invalidated": false,
  "dataTags": ["analytics", "pii"],
  "entityAllowList": [
    "e1b2c3d4-e5f6-7890-1234-567890abcdef",
    "e2b2c3d4-e5f6-7890-1234-567890abcdef"
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ]
}

A few things to note:

PATCH accepts only label and descriptor. Use PUT to change integrationId, primaryEntityId, traversalRoute, entityFieldPermissions, or other structural fields.
The API returns a 400 when unsupported fields are sent on PATCH.

Delete a dataset

Use delete dataset to remove a dataset. By default the row is soft-deleted (deletedAt set). Pass hard=true to remove the record from storage.

DELETE /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to delete.
`hard`	boolean (query)	no	default `false`	When `true`, hard delete; when `false` or omitted, soft delete.

A successful call returns 200 with no body.

Error responses

Responses follow a shared error shape (code, title, detail, traceId). Exact codes depend on the operation and input; the examples below are illustrative.

PATCH returns a 400 when the body includes fields other than label and descriptor:

{
  "code": "INVALID_PATCH_DATA",
  "title": "Invalid Patch Data",
  "detail": "Unsupported fields provided in patch request. Only label and descriptor are allowed.",
  "traceId": "req_12345678-abcd-4321-9876-fedcba098765"
}

When a dataset id does not exist or you cannot access it, you may receive 404:

{
  "code": "DATASET_NOT_FOUND",
  "title": "Dataset Not Found",
  "detail": "No dataset found with the specified ID",
  "traceId": "req_12345678-abcd-4321-9876-fedcba098765"
}