Supply Chain & Retail Solutions user guide

Peak's Data Sources feature lets you ingest data from external sources into your data warehouse using connectors and feeds. Once configured, feeds run on a schedule, on demand, or via webhook.

To access Data Sources, open Manage and select Data Sources.

Peak supports four data ingestion paths:

A data feed connects to a source, copies its data, and ingests the copy into your Peak data warehouse. Feeds can run automatically using a trigger or manually on demand.

Each feed is configured through four stages:

The Feeds screen shows the status, run history, and next scheduled run for each feed. Hover over a feed to access the following actions:

Use the filter function to find feeds when you have a large number. The following filters are available:

Select a feed to open its detail view. Two tabs are available:

Shows how many rows were successfully loaded into the data warehouse, and detailed logs for each feed run. Select the Browse file icon on a log entry to open Files and view the files associated with that feed run.

If a feed run fails, error details are shown in the detailed log. To get details of individual failed records, download the STL load error files using the download icon next to the error details.

Shows basic configuration information about the feed.

Load types control how data is written to your destination table when a feed runs. You select a load type during import configuration.

Two key columns can be configured when selecting a load type:

Best suited for small data tables.

When this load type runs, the entire destination table is replaced with the data retrieved from the source.

Best suited for event-type datasets where records are added over time but not modified.

With incremental feeds, records are only inserted — existing records are never updated. At each run, only records added since the last run are fetched, based on the column value specified in the last run key.

Best suited for transactional data where records can be created or modified over time.

With upsert feeds, new records are inserted and existing records are updated. In some cases, the full dataset is fetched to ensure all updates are captured.

When creating a data feed, you select a destination where Peak ingests your data. The available destinations depend on the data warehouse configured for your Peak organization.

To see which warehouse type your organization uses, open Manage and select Data Bridge. In the Data warehouse section, check the storage type shown.

For more information about data warehouse configuration, see Data Bridge overview.

Stores data in Amazon S3. Peak uses Apache Spark to process large, unstructured CSV datasets.

Data is ingested into the Snowflake data warehouse. Snowflake is SQL-based and supports querying and analyzing data using standard SQL.

After ingestion, Peak adds these audit columns to the destination table:

Data is ingested into Amazon Redshift. Redshift is a relational database that supports SQL querying and is optimized for aggregations on large datasets. Incoming data must map exactly to the destination table schema, column by column. Any failed rows are flagged and written to a separate table.

After ingestion, Peak adds these audit columns to the destination table:

The failed row threshold defines how many rows can fail before Peak marks the feed as failed. Set a threshold that reflects your total row volume and acceptable data quality tolerance.

Behavior varies by load type:

Additional rules:

Threshold defaults and limits by destination:

Peak handles schema changes automatically when a feed run detects added or removed columns.

For CSV files, all files within a single feed run must use the same schema. Inconsistent schemas across files in the same feed run cause ingestion failure. For NDJSON files, different files can have different schemas, provided each individual file has a consistent unified schema.

When configuring a destination, you can override the inferred data type for each column. This is available for all connectors except Webhook and Braze Currents.

Supported data types:

TIMESTAMPTZ is not supported. Data in this format is ingested as STRING.

Triggers define when a data feed runs. Watchers send notifications when feed events occur.

Basic schedules run a feed at a specified time and day, or at a recurring frequency (for example, every 2 hours or every Monday and Tuesday at 12:00).

Advanced schedules use cron expressions for more precise timing.

A cron expression is a string of 6 or 7 space-separated fields:

Examples:

To validate your cron expressions before use, consider using a syntax checker such as crontab.guru.

Unlike regular APIs that require constant polling for updates, webhooks only send data when a specific event occurs — in this case, when new data is available for the feed.

For webhook triggers, Peak generates a unique webhook URL for the feed. Copy the URL into the external system that sends events. You can regenerate the URL if needed. If the external system is outside Peak, provide your tenant API key so the webhook can be authenticated. See API keys.

Run once triggers can be manual or scheduled:

Configure watchers to send notifications for feed events:

To add a watcher to a feed:

User watchers notify selected users inside Peak when a feed event occurs. Users can view notifications from the bell icon.

When configuring a user watcher, you can choose to watch all events or select a custom set.

User watchers can be configured for these events:

Webhook watchers send a notification or trigger an action in an external system or Peak feature when a feed event occurs. Examples include Slack notifications or Peak Workflows.

The webhook URL is provided by the target application. If the target is a Peak Workflow, copy the URL from the Workflow's Trigger step.

The JSON payload is optional and can include these parameters to pass feed context:

Webhook watchers can be configured for these events:

Peak uses file names to determine how files are grouped into feeds. If a file is updated and fetched regularly as part of the same feed, the file must retain the same base name with a new timestamp appended on each update.

File names must include at least one alphanumeric character.

Use one of these patterns when naming files for feeds:

Peak applies these parsing rules to extract the feed name from each file:

Peak recognizes these valid timestamp formats in file names:

If you upload data to a managed Snowflake data warehouse, Peak expects schemas for organizing your data. Schemas are sub-areas of a data warehouse used to group tables by their purpose and processing stage.

Tables do not strictly need to be placed in a particular schema, but using the recommended schemas keeps your data well organized.

The following schemas organize your data by processing stage: