Data Set Management

Quick Navigation

What is a dataset?

After you have connected data sources and set up a functional data catalog, a dataset is how you bundle catalog data for projects and workflows: a display label, optional longer descriptor, a primary entity as the starting point, optional related entities, and per-field write rules — without changing the schema at the source. Workflow steps use that bundle when they need to query records, write values back, or show the right fields.

In the API, label is the main name shown for the dataset; descriptor is additional descriptive text. For click-by-click setup in the product (including starting from the Data Catalog or adjusting field classifications), see How to Create and Manage Datasets in Streamline on the Help Center.

The Data Search Step takes a datasetId so the run queries the right records; the Deliver Data Step uses the same id when writing values back. Create and maintain datasets with the endpoints below (base URL and authentication match the rest of the Streamline API).

Core Concepts

Dataset resource

Responses include the full dataset shape (identifiers, traversal, permissions, audit timestamps, and enriched fields). For create and full replace (PUT), you must provide primaryEntityId and at least one of integrationId or entityGraphId. Since entityGraphId is deprecated, new integrations should always use integrationId. organizationId is set from the authenticated organization on create; you do not need to send it.

name	type	create / PUT	constraints	description
`id`	string (UUID)	returned after create; send on PUT	—	Stable identifier. Use as `datasetId` in workflow steps.
`label`	string	optional	—	Primary display name; recommended for operators and UIs.
`descriptor`	string	optional	—	Longer description.
`organizationId`	string (UUID)	set by service on create	—	Owning organization (from auth).
`integrationId`	string (UUID)	recommended	—	Primary integration; required if `entityGraphId` is not provided.
`entityGraphId`	string (UUID)	deprecated	—	Deprecated. Use `integrationId` instead.
`primaryEntityId`	string (UUID)	required	—	Root entity for queries and writes.
`primaryEntityAlias`	string	optional	—	Friendly alias for the primary entity.
`integrationIds`	array of UUID	response / enrichment	unique items	Integration ids for entities in the computed allow list; not used to satisfy create validation.
`capabilities`	object	response / enrichment	—	Field capabilities (for example `searchOperators`: map of field type to allowed operator names).
`createdAt`	string (date-time)	response	ISO 8601	Creation time (set on create).
`createdByUserId`	string (UUID)	response	—	Creating user when recorded.
`integrations`	array of object	response with `include`	—	Expanded integration records when using `include=integrations`.
`invalidated`	boolean	response	—	When true, treat the dataset as stale; see Invalidation.
`deletedAt`	string (date-time)	response (soft delete)	ISO 8601	Set on soft delete; omitted or null while active.
`dataTags`	array of string	computed on save	—	Derived from catalog tags for entities in the allow list (and related rules).
`entityAllowList`	array of UUID	computed on save	—	Derived from `primaryEntityId` and `traversalRoute`.
`entityFieldPermissions`	array of object	optional	—	Per-entity field ids and `writeMode` (`FULL` or `NONE`).
`entityFieldSelections`	array of object	optional on create and PUT; also on PATCH	—	Catalog fields Streamline should use for each entity. Other fields are ignored and left out of UI field selections. On PUT, omit to keep selections for entities that remain; send `[]` to include every catalog field again.
`entityFieldOverrides`	array of object	optional on create, PUT, and PATCH	—	Pins for writable, searchable, sortable, and required-on-insert. Scalar type cannot be pinned. On PUT, sending `entityFieldPermissions` without this field rebuilds writable pins from permissions and drops writable pins set here. Searchable, sortable, and required-on-insert pins stay.
`hasExplicitFieldSelection`	boolean	response	—	True when an explicit field subset is saved.
`issues`	array of object	response with `include`	—	Unresolved dataset-repair issues when get-by-id uses `include=issues`.
`openIssueCount`	integer	response with `include`	—	Unresolved-issue count on list and batch when `include=issueSummary`. Null when that include is omitted.
`traversalRoute`	array of object	optional	—	Route through relationship links (link id, entity id, alias, mode, nested links).
`linkAliases`	array of object	response (derived)	—	Convenience aliases derived from `traversalRoute` for clients.

capabilities is an envelope: searchOperators is an object whose keys are field types (for example scalar categories) and whose values are arrays of operator names the integration supports for search and filter UIs.

Invalidation

When invalidated is true, the dataset should not be treated as current: the underlying data source schema changed, and the existing definition may no longer match reality. Create a new dataset or adjust configuration so workflows and integrations use a valid definition.

📘
If a dataset stays invalidated, steps that rely on it may fail or return unexpected results until you fix or replace the dataset.

Related entities, fields, and traversal

entityAllowList on responses reflects entities reached from primaryEntityId and traversalRoute after save; you do not supply a separate allow list that overrides that computation on create or PUT.
dataTags on responses are derived from the catalog for those entities when the dataset is saved.
entityFieldSelections lists the catalog fields you intend Streamline to use, one group of field ids per entity. Streamline will use and track only these fields. Fields you leave out are ignored and not shown in field selections in project builder UI. Adding or deleting an ignored field at the source does not open a field-deleted issue and does not break an existing Search. With no selection saved, every catalog field is included, and deleting any of them does open that issue. Send an empty array to clear a saved selection and return to that behavior. hasExplicitFieldSelection is true once a subset is saved. File entities still include every catalog field.
entityFieldPermissions controls which fields can be written (FULL) versus read-only (NONE) for deliver operations.
traversalRoute describes how to walk from the primary entity through links (for example, which relationship to follow and how nested segments are ordered).
linkAliases on responses are derived from the traversal route for presentation.

The JSON below shows a typical response shape (including enriched fields). Create and PUT bodies must include primaryEntityId and at least one of integrationId or entityGraphId (prefer integrationId), plus optional label, descriptor, traversalRoute, entityFieldPermissions, and so on — omit organizationId if you rely on the token's organization.

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Example dataset",
  "descriptor": "Optional longer description",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityAlias": "customers",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-01-15T10:00:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "invalidated": false,
  "dataTags": ["analytics"],
  "entityAllowList": ["e1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ],
  "traversalRoute": [],
  "linkAliases": []
}

How-To Guides

Let's explore common scenarios for managing datasets.

List datasets

Use list organization datasets to retrieve a paginated list of datasets for your organization.

GET /v1/activation/dataset

name	type	required	constraints	description
`entityId`	array of string (query)	no	UUID	Filter datasets by one or more entity IDs.
`label`	string (query)	no	—	Filter datasets by label text.
`pageSize`	integer (query)	no	1–100, default 50	Number of items per page.
`pageNumber`	integer (query)	no	1-based, default 1	Page number to return.
`offset`	integer (query)	no	min 0, default 0	Records to skip before pagination. Can combine with `pageNumber`.
`include`	array of string (query)	no	`integrations`, `issueSummary`	`integrations` expands integration records. `issueSummary` adds `openIssueCount`.

{
  "content": [
    {
      "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
      "label": "Customer analytics",
      "descriptor": "Profiles, orders, and engagement",
      "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
      "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
      "createdAt": "2025-01-15T10:00:00Z",
      "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
      "capabilities": {
        "searchOperators": {
          "STRING": ["EQUALS", "CONTAINS"]
        }
      },
      "primaryEntityAlias": "customers",
      "invalidated": false,
      "dataTags": ["analytics", "pii"],
      "entityAllowList": ["e1b2c3d4-e5f6-7890-1234-567890abcdef"],
      "entityFieldPermissions": [
        {
          "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
          "fieldPermissions": [
            {
              "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
              "writeMode": "FULL"
            }
          ]
        }
      ],
      "traversalRoute": [],
      "linkAliases": []
    }
  ],
  "page": {
    "size": 1,
    "pageSize": 50,
    "pageNumber": 1,
    "totalElements": 1,
    "totalPages": 1
  }
}

Create a dataset

Use create dataset to define a new dataset for data activation.

POST /v1/activation/dataset

Required fields:

primaryEntityId — root entity for queries and writes
At least one of integrationId or entityGraphId — identifies the catalog layout. Since entityGraphId is deprecated, new integrations should use integrationId.

Optional fields:

label, descriptor, primaryEntityAlias, entityFieldPermissions, traversalRoute

📘
organizationId is set from the authenticated organization — you do not need to send it. entityAllowList and dataTags are computed on save from primaryEntityId and traversalRoute.

{
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
  "label": "Marketing campaigns",
  "descriptor": "Campaign effectiveness and customer engagement",
  "primaryEntityAlias": "campaigns",
  "entityFieldPermissions": [
    {
      "entityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f5b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ],
  "traversalRoute": [
    {
      "linkId": "l3b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e6b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "campaign_metrics",
      "traversalMode": "GRAPH",
      "links": []
    }
  ]
}

{
  "id": "ds3b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Marketing campaigns",
  "descriptor": "Campaign effectiveness and customer engagement",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg3b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-03-10T14:30:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "primaryEntityAlias": "campaigns",
  "invalidated": false,
  "dataTags": ["marketing", "engagement"],
  "entityAllowList": [
    "e5b2c3d4-e5f6-7890-1234-567890abcdef",
    "e6b2c3d4-e5f6-7890-1234-567890abcdef"
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e5b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f5b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ],
  "traversalRoute": [
    {
      "linkId": "l3b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e6b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "campaign_metrics",
      "traversalMode": "GRAPH",
      "links": []
    }
  ],
  "linkAliases": [
    {
      "linkId": "l3b2c3d4-e5f6-7890-1234-567890abcdef",
      "alias": "campaign_metrics"
    }
  ]
}

Batch lookup by entity

Use get datasets by entity IDs to find datasets grouped by entity ID for multiple entities at once. The response is an object keyed by entity ID. Each value is an array of dataset objects, and may be empty.

POST /v1/activation/dataset/batch

name	type	required	constraints	description
`include`	array of string (query)	no	`integrations`, `issueSummary`	`integrations` expands integration records. `issueSummary` adds `openIssueCount`.

["e1b2c3d4-e5f6-7890-1234-567890abcdef", "e2b2c3d4-e5f6-7890-1234-567890abcdef"]

{
  "e1b2c3d4-e5f6-7890-1234-567890abcdef": [
    {
      "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
      "label": "Customer analytics",
      "descriptor": "Profiles and orders",
      "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
      "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
      "createdAt": "2025-01-15T10:00:00Z",
      "primaryEntityAlias": "customers",
      "invalidated": false,
      "dataTags": ["analytics"]
    }
  ],
  "e2b2c3d4-e5f6-7890-1234-567890abcdef": [
    {
      "id": "ds4b2c3d-e5f6-7890-1234-567890abcdef",
      "label": "Product catalog",
      "descriptor": "Catalog and inventory",
      "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationId": "i2b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityGraphId": "eg4b2c3d-e5f6-7890-1234-567890abcdef",
      "primaryEntityId": "e2b2c3d4-e5f6-7890-1234-567890abcdef",
      "integrationIds": ["i2b2c3d4-e5f6-7890-1234-567890abcdef"],
      "createdAt": "2025-01-18T11:20:00Z",
      "primaryEntityAlias": "products",
      "invalidated": false,
      "dataTags": ["catalog"]
    }
  ]
}

Get a dataset by ID

Use get dataset by id to retrieve a single dataset.

GET /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to retrieve.
`include`	array of string (query)	no	`integrations`, `issues`, `workflowUsage`, `sessionCounts`	`issues` returns unresolved dataset-repair issues. `workflowUsage` and `sessionCounts` add those details on each issue.

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Customer analytics",
  "descriptor": "Profiles, orders, and engagement",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-01-15T10:00:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "primaryEntityAlias": "customers",
  "invalidated": false,
  "dataTags": ["analytics", "pii"],
  "entityAllowList": [
    "e1b2c3d4-e5f6-7890-1234-567890abcdef",
    "e2b2c3d4-e5f6-7890-1234-567890abcdef"
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        },
        {
          "fieldId": "f2b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "NONE"
        }
      ]
    }
  ],
  "traversalRoute": [
    {
      "linkId": "l1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e2b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "customer_orders",
      "traversalMode": "GRAPH",
      "links": []
    }
  ],
  "linkAliases": [
    {
      "linkId": "l1b2c3d4-e5f6-7890-1234-567890abcdef",
      "alias": "customer_orders"
    }
  ]
}

Replace a dataset

Use update dataset to replace the mutable definition. The same rules apply as create: primaryEntityId is required and at least one of integrationId or entityGraphId must be provided. entityAllowList and dataTags are recomputed from primaryEntityId and traversalRoute on save. Include traversalRoute when relationships change.

PUT /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to replace.

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Customer analytics (revised)",
  "descriptor": "Expanded metrics and segments",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "primaryEntityAlias": "customers_v2",
  "traversalRoute": [
    {
      "linkId": "l1b2c3d4-e5f6-7890-1234-567890abcdef",
      "entityId": "e2b2c3d4-e5f6-7890-1234-567890abcdef",
      "linkAlias": "customer_orders",
      "traversalMode": "GRAPH",
      "links": []
    }
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        },
        {
          "fieldId": "f2b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ]
}

The response body matches the saved dataset (same general shape as GET).

Partially update a dataset

Use patch dataset to change label, descriptor, entityFieldSelections, or entityFieldOverrides. Any other field is rejected with 400 (see Error responses).

PATCH can change the display name, the description, which fields are selected, and pins for writable, searchable, sortable, and required-on-insert. It does not change catalog classifications or the source field schema. Use the Data Catalog for those. Changing integrationId, primaryEntityId, traversalRoute, or entityFieldPermissions requires PUT.

PATCH /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to patch.

{
  "label": "Customer analytics (renamed)",
  "descriptor": "Updated description for operators"
}

{
  "id": "ds1b2c3d-e5f6-7890-1234-567890abcdef",
  "label": "Customer analytics (renamed)",
  "descriptor": "Updated description for operators",
  "organizationId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationId": "i1b2c3d4-e5f6-7890-1234-567890abcdef",
  "entityGraphId": "eg1b2c3d-e5f6-7890-1234-567890abcdef",
  "primaryEntityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
  "integrationIds": ["i1b2c3d4-e5f6-7890-1234-567890abcdef"],
  "createdAt": "2025-01-15T10:00:00Z",
  "createdByUserId": "u1b2c3d4-e5f6-7890-1234-567890abcdef",
  "capabilities": {
    "searchOperators": {
      "STRING": ["EQUALS", "CONTAINS"]
    }
  },
  "primaryEntityAlias": "customers",
  "invalidated": false,
  "dataTags": ["analytics", "pii"],
  "entityAllowList": [
    "e1b2c3d4-e5f6-7890-1234-567890abcdef",
    "e2b2c3d4-e5f6-7890-1234-567890abcdef"
  ],
  "entityFieldPermissions": [
    {
      "entityId": "e1b2c3d4-e5f6-7890-1234-567890abcdef",
      "fieldPermissions": [
        {
          "fieldId": "f1b2c3d4-e5f6-7890-1234-567890abcdef",
          "writeMode": "FULL"
        }
      ]
    }
  ]
}

A few things to note:

Sending entityFieldSelections replaces the selection set. An empty array clears it and the dataset uses every field.
Sending entityFieldOverrides replaces the full pin set. Omit it to leave existing pins in place.
The API returns a 400 when the body includes any field other than label, descriptor, entityFieldSelections, and entityFieldOverrides.

Delete a dataset

Use delete dataset to remove a dataset. By default the row is soft-deleted (deletedAt set). Pass hard=true to remove the record from storage.

DELETE /v1/activation/dataset/{dataSetId}

name	type	required	constraints	description
`dataSetId`	string (path)	yes	UUID	Dataset to delete.
`hard`	boolean (query)	no	default `false`	When `true`, hard delete; when `false` or omitted, soft delete.

A successful call returns 200 with no body.

Error responses

Responses follow a shared error shape (code, title, detail, traceId). Exact codes depend on the operation and input; the examples below are illustrative.

PATCH returns a 400 when the body includes a field other than label, descriptor, entityFieldSelections, or entityFieldOverrides:

{
  "code": "INVALID_PATCH_DATA",
  "title": "Invalid Patch Data",
  "detail": "Unsupported fields provided in patch request. Only label, descriptor, entityFieldSelections, and entityFieldOverrides are allowed.",
  "traceId": "req_12345678-abcd-4321-9876-fedcba098765"
}

When a dataset id does not exist or you cannot access it, you may receive 404:

{
  "code": "DATASET_NOT_FOUND",
  "title": "Dataset Not Found",
  "detail": "No dataset found with the specified ID",
  "traceId": "req_12345678-abcd-4321-9876-fedcba098765"
}