# Holdout test results

`GET https://stylor.ai/api/v1/analytics/holdout`

- Authentication: private API key, sent as `Authorization: Bearer sgpt-sk-…::…`
- Rate-limit resource: `api.v1.analytics.holdout`
- Demo key: not accepted
- Web page: https://stylor.ai/guides/rest/v1/holdout-results

What the chat widget adds, measured: the visits that saw it against
the visits the organization's holdout test kept from it (set with
`holdoutPercent` in `PATCH /v1/widget/settings`). Both arms are read
the same way, from the storefront's own cart: the share of visits
that added anything (`conversionRate`), and cart value added per
visit, as the store's cart reported it. The widget arm also carries
what it did with the widget (opens, messages, adds Stylor made, adds
of products it had shown, counted by the same rules as the rest of
analytics).

## Reading it

`comparison` holds each difference (widget minus holdout), its
relative lift, a 95% confidence interval (`low`, `high`) and a
two-sided `pValue`. `verdict` calls it:

- `early`: the smaller arm has fewer than 200 visits (`visitsNeeded` says how many more).
- `split_off`: the arms did not come out at the share set, so visits were lost on one side; nothing should be read from the figures.
- `widget_better` or `widget_worse`: the conversion rate differs at 95% confidence.
- `no_difference_yet`: enough visits, and no difference at that confidence.

Visits whose arm was forced (the `stylor_arm` store-page parameter)
are left out. Figures are read from daily rollups, so today's trail by
a few minutes. A visit open across midnight counts on both days.

## Example request

```bash
curl -X GET "https://stylor.ai/api/v1/analytics/holdout" \
  -H "Authorization: Bearer $STYLOR_API_KEY"
```

## Query parameters

- `holdoutId` (string): Which test. The running one, else the one that ended last, when absent. [example `"hold_3f9a1c2b7d4e5f60"`]

## Responses

### 200

Every test, the one read and its results. `test` and `results` are `null` when the organization has never run one.

Content type `application/json`:

- `tests` (array<object>, required): Every test, the running one first.
  - `id` (string): The test.
  - `percent` (integer): The share of visits held out.
  - `startedAt` (string (date-time)): When it started.
  - `endedAt` (string (date-time), nullable): When it ended; `null` while running.
  - `running` (boolean): Whether it is the running test.
- `test` (object, nullable, required): The test read, shaped as an entry of `tests`.
- `results` (object, nullable, required): `verdict`, `visitsNeeded`, `confidence`, `arms` (`widget` and `holdout`: `visits`, `convertedVisits`, `conversionRate`, `cartAdds`, `cartValueCents`, `cartValuePerVisitCents`; the widget arm also `launcherVisits`, `openedVisits`, `messagedVisits`, `assistedCartAdds`, `stylorCartAdds`, `stylorCartValueCents`), `comparison` (`conversionRate` and `cartValuePerVisitCents`, each `treatment`, `control`, `diff`, `relative`, `low`, `high`, `pValue`, `significant`), `split` (`expected`, `observed`, `pValue`, `ok`), `days`, `startDay`, `endDay`, `timezone`.

Example:

```json
{
  "tests": [
    {
      "id": "hold_3f9a1c2b7d4e5f60",
      "percent": 10,
      "startedAt": "2026-10-01T09:00:00.000Z",
      "endedAt": null,
      "running": true
    }
  ],
  "test": {
    "id": "hold_3f9a1c2b7d4e5f60",
    "percent": 10,
    "startedAt": "2026-10-01T09:00:00.000Z",
    "endedAt": null,
    "running": true
  },
  "results": {
    "verdict": "widget_better",
    "visitsNeeded": 0,
    "confidence": 0.95,
    "arms": {
      "widget": {
        "visits": 18412,
        "convertedVisits": 1031,
        "conversionRate": 0.056,
        "cartAdds": 1290,
        "cartValueCents": 7318400,
        "cartValuePerVisitCents": 397.5,
        "launcherVisits": 18390,
        "openedVisits": 1402,
        "messagedVisits": 911,
        "assistedCartAdds": 214,
        "stylorCartAdds": 88,
        "stylorCartValueCents": 612300
      },
      "holdout": {
        "visits": 2051,
        "convertedVisits": 92,
        "conversionRate": 0.0449,
        "cartAdds": 109,
        "cartValueCents": 640200,
        "cartValuePerVisitCents": 312.1
      }
    },
    "comparison": {
      "conversionRate": {
        "treatment": 0.056,
        "control": 0.0449,
        "diff": 0.0111,
        "relative": 0.247,
        "low": 0.0016,
        "high": 0.0206,
        "pValue": 0.022,
        "significant": true
      }
    },
    "split": {
      "expected": 0.1,
      "observed": 0.1002,
      "pValue": 0.93,
      "ok": true
    },
    "days": 34,
    "startDay": "2026-10-01",
    "endDay": "2026-11-03",
    "timezone": "America/Denver"
  }
}
```

## Errors

Every error is a JSON object with an `error` string. Match on the status code; the message is written for people.

| Status | Message | When |
| --- | --- | --- |
| 404 | Holdout test not found: <holdoutId> | No test with this id belongs to the key's organization. |

### Shared authentication, quota and rate-limit errors

| Status | Message | When |
| --- | --- | --- |
| 401 | API key missing. | The `Authorization` header is absent, is not `Bearer <key>`, or the key does not have three dash-separated segments. |
| 401 | Authentication failed: public key is invalid or not recognized. | A public-key endpoint was called with a key that does not exist. |
| 401 | Authentication failed: private key provided instead of a public key. | A public-key endpoint was called with an `sgpt-sk-…` key. |
| 401 | Authentication failed: token is not recognized. | A private-key endpoint was called with a key whose lookup segment does not exist. |
| 401 | Authentication failed: private key is invalid. | A private-key endpoint was called with a key whose secret does not match. |
| 401 | Authentication failed: public key provided instead of a private key. | A private-key endpoint was called with an `sgpt-pk-…` key. |
| 401 | Authentication failed: bearer token format is invalid. | A private key was sent without its `::` lookup segment. |
| 401 | Authentication failed: API key could not be read. Regenerate the key. | The key's embedded organization data could not be decrypted or parsed. |
| 403 | This key does not have the "<scope>" scope. | A private-key endpoint was called with a key whose scopes do not include the one the endpoint needs. Nothing is counted against your quota. See Authentication. |
| 403 | This key's scopes do not cover this endpoint. | A key with scopes called a private endpoint that no scope grants yet. Give the key every scope to reach it. |
| 403 | Demo mode is not available for private-key routes. Demos are only supported with public API keys. | The demo key was sent to a private-key endpoint. |
| 403 | Demo mode is not supported on this API route. Check the documentation for available demo endpoints. | The demo key was sent to a public-key endpoint that opts out of demo mode. |
| 403 | No active subscription for this organization. | The organization that owns the key has no active subscription. |
| 403 | No quota found for organization "<organizationId>". | The organization has no quota record for the current billing period. |
| 403 | No entitlement for "<resource>". | The plan's quota template has no entry for this endpoint. |
| 403 | Access denied to "<resource>". | The plan's quota template turns this endpoint off. |
| 429 | Monthly quota exceeded for "<resource>" (<quota>/<quota> used). | The billing-period quota for this endpoint is exhausted. `Retry-After` gives the seconds until the period resets. |
| 429 | RPM limit exceeded for "<resource>" (<used>/<limit> used). | More than the allowed requests per minute were sent. `Retry-After` is 60. |
| 500 | Internal server error. | An unexpected failure on the server. The message is always this string; details are logged, never returned. Quota consumed by the request is refunded, as it is for every 5xx response. |