Segmenting a photo

POST /v2/models/segment takes one photo and returns every garment and accessory in it. Each item comes with a label, a box and a mask. It works on photos of people dressed and on product shots.

Sending the photo

Send the photo as image, in one of two forms:

A public URL. Our server fetches it once. It must be a public http(s) address that answers without redirecting, and the file can be at most 12 MB.
A base64 data URL. data:image/jpeg;base64,…, image/png or image/webp, at most 8 million characters.

The model reads photos at 1024 pixels on the long side, so sending more than about 2048 pixels gains nothing. Masks always come back at the size of the photo you sent.

javascript

const response = await fetch('https://stylor.ai/api/v2/models/segment', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${STYLOR_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'stylor-segment-v1',
    image: 'https://example.com/look.jpg',
  }),
});
const result = await response.json();

python

import requests

result = requests.post(
    "https://stylor.ai/api/v2/models/segment",
    headers={"Authorization": f"Bearer {STYLOR_API_KEY}"},
    json={"model": "stylor-segment-v1", "image": "https://example.com/look.jpg"},
    timeout=60,
).json()

The response

json

{
  "model": "stylor-segment-v1",
  "width": 930,
  "height": 1395,
  "items": [
    {
      "label": "outerwear/jacket",
      "category": "outerwear",
      "type": "jacket",
      "box": [104, 290, 899, 1190],
      "area": 473186,
      "mask": "data:image/png;base64,iVBORw0KGgo..."
    }
  ],
  "ms": { "total": 612, "photo": 38, "model": 561 }
}

width and height are the photo's size after it is turned upright. A phone photo stored sideways with an EXIF rotation is turned first, and every box and mask is in the upright frame.
items is empty when the model finds nothing to segment. That is a 200, not an error.
box is [x0, y0, x1, y1] in pixels. x1 and y1 are exclusive, so the box is x1 - x0 wide.
area is the number of pixels in the mask.
mask is a 1-bit PNG the size of the photo: white where the item is, black elsewhere.
Masks never overlap. Each pixel belongs to at most one item, so a top under an open jacket gets only the pixels of it you can see.
There is no confidence score per item. The model returns only the items it is confident in.

Labels

Every label is category/type, and category and type repeat its two halves. There are 68 labels in 15 categories:

Category	Types
`top`	bodysuit, bra top, cardigan, polo, shirt, sweater, sweatshirt, t shirt, tank top, other
`bottom`	legging, pant, short, skirt, swim bottom
`full body`	dress, jumpsuit, overall, romper
`outerwear`	blazer, cape, coat, jacket, vest
`footwear`	boot, flat, lace up shoe, loafer, mule, pump, sandal, sneaker, other
`bag`	clutch, handbag, tote bag
`accessory`	belt, glove, necktie, scarf, other
`eyewear`	eyeglasses, sunglasses
`headwear`	beanie, cap, hair bow, hair clip, hair tie, hat, headband, other
`hosiery`	sock, stocking, tights
`jewellery`	bracelet, earring, necklace, ring, watch, other
`sleepwear`	nightshirt, sleeper, other
`swimwear`	swimsuit
`underwear`	bra, garter belt, panty
`other`	An item the model sees but cannot name. Its `label`, `category` and `type` are all `other`.

Both earrings are one jewellery/earring item.

Small jewellery: `zoom`

Rings, earrings and fine necklaces are a few pixels wide at the size the model reads a photo. Send zoom: true and the model looks a second time, closer, at the places where jewellery may be.

It finds more jewellery and draws it more accurately.
It takes about three times as long.
It changes nothing about clothing, shoes or bags.

Leave it off unless jewellery matters to you.

Using a mask

A mask is an ordinary PNG, so anything that reads images can use it. To cut an item out of the photo in a browser:

javascript

async function cutOut(photoUrl, item) {
  const [photo, mask] = await Promise.all([loadImage(photoUrl), loadImage(item.mask)]);
  const canvas = document.createElement('canvas');
  canvas.width = photo.naturalWidth;
  canvas.height = photo.naturalHeight;
  const ctx = canvas.getContext('2d');

  // Turn the mask's white into opacity, then keep the photo only where it is opaque.
  ctx.drawImage(mask, 0, 0);
  const pixels = ctx.getImageData(0, 0, canvas.width, canvas.height);
  for (let i = 0; i < pixels.data.length; i += 4) pixels.data[i + 3] = pixels.data[i];
  ctx.putImageData(pixels, 0, 0);
  ctx.globalCompositeOperation = 'source-in';
  ctx.drawImage(photo, 0, 0);
  return canvas;
}

const loadImage = (src) => new Promise((resolve, reject) => {
  const image = new Image();
  image.crossOrigin = 'anonymous';
  image.onload = () => resolve(image);
  image.onerror = reject;
  image.src = src;
});

In Python: