StylorDocs

Holdout test results

get/v1/analytics/holdout

What the chat widget adds, measured: the visits that saw it against the visits the organization's holdout test kept from it (set with holdoutPercent in PATCH /v1/widget/settings). Both arms are read the same way, from the storefront's own cart: the share of visits that added anything (conversionRate), and cart value added per visit, as the store's cart reported it. The widget arm also carries what it did with the widget (opens, messages, adds Stylor made, adds of products it had shown, counted by the same rules as the rest of analytics).

Reading it

comparison holds each difference (widget minus holdout), its relative lift, a 95% confidence interval (low, high) and a two-sided pValue. verdict calls it:

  • early: the smaller arm has fewer than 200 visits (visitsNeeded says how many more).
  • split_off: the arms did not come out at the share set, so visits were lost on one side; nothing should be read from the figures.
  • widget_better or widget_worse: the conversion rate differs at 95% confidence.
  • no_difference_yet: enough visits, and no difference at that confidence.

Visits whose arm was forced (the stylor_arm store-page parameter) are left out. Figures are read from daily rollups, so today's trail by a few minutes. A visit open across midnight counts on both days.

Private keyapi.v1.analytics.holdoutNo demo

Query parameters

1
holdoutIdstring

Which test. The running one, else the one that ended last, when absent.

Response

200Every test, the one read and its results. test and results are null when the organization has never run one.
array<object>required

Every test, the running one first.

idstring

The test.

percentinteger

The share of visits held out.

startedAtstring (date-time)

When it started.

endedAtstring (date-time)nullable

When it ended; null while running.

runningboolean

Whether it is the running test.

testobjectnullablerequired

The test read, shaped as an entry of tests.

resultsobjectnullablerequired

verdict, visitsNeeded, confidence, arms (widget and holdout: visits, convertedVisits, conversionRate, cartAdds, cartValueCents, cartValuePerVisitCents; the widget arm also launcherVisits, openedVisits, messagedVisits, assistedCartAdds, stylorCartAdds, stylorCartValueCents), comparison (conversionRate and cartValuePerVisitCents, each treatment, control, diff, relative, low, high, pValue, significant), split (expected, observed, pValue, ok), days, startDay, endDay, timezone.

Errors

20

Every error is JSON with a single error string unless noted. Match on the status code; the message is for humans.

404
Holdout test not found: <holdoutId>

No test with this id belongs to the key's organization.

curl -X GET "https://stylor.ai/api/v1/analytics/holdout" \
  -H "Authorization: Bearer $STYLOR_API_KEY"
Response · 200
{
  "tests": [
    {
      "id": "hold_3f9a1c2b7d4e5f60",
      "percent": 10,
      "startedAt": "2026-10-01T09:00:00.000Z",
      "endedAt": null,
      "running": true
    }
  ],
  "test": {
    "id": "hold_3f9a1c2b7d4e5f60",
    "percent": 10,
    "startedAt": "2026-10-01T09:00:00.000Z",
    "endedAt": null,
    "running": true
  },
  "results": {
    "verdict": "widget_better",
    "visitsNeeded": 0,
    "confidence": 0.95,
    "arms": {
      "widget": {
        "visits": 18412,
        "convertedVisits": 1031,
        "conversionRate": 0.056,
        "cartAdds": 1290,
        "cartValueCents": 7318400,
        "cartValuePerVisitCents": 397.5,
        "launcherVisits": 18390,
        "openedVisits": 1402,
        "messagedVisits": 911,
        "assistedCartAdds": 214,
        "stylorCartAdds": 88,
        "stylorCartValueCents": 612300
      },
      "holdout": {
        "visits": 2051,
        "convertedVisits": 92,
        "conversionRate": 0.0449,
        "cartAdds": 109,
        "cartValueCents": 640200,
        "cartValuePerVisitCents": 312.1
      }
    },
    "comparison": {
      "conversionRate": {
        "treatment": 0.056,
        "control": 0.0449,
        "diff": 0.0111,
        "relative": 0.247,
        "low": 0.0016,
        "high": 0.0206,
        "pValue": 0.022,
        "significant": true
      }
    },
    "split": {
      "expected": 0.1,
      "observed": 0.1002,
      "pValue": 0.93,
      "ok": true
    },
    "days": 34,
    "startDay": "2026-10-01",
    "endDay": "2026-11-03",
    "timezone": "America/Denver"
  }
}