One Shape for Data from Many Sources

Integrations pile up. Payments arrive from Stripe as JSON with amounts in cents, and from an older billing system as form posts with amounts in dollars. Orders come from a storefront, a marketplace, and a CSV upload. Each source needs an adapter, the adapters live in your app's code, and when a provider changes a field, fixing it means a deploy.

With Stof, each source gets a small mapping document that turns its payload into your one shape. The shape validates itself, so a bad payload is caught at the edge. Adding or fixing a source means adding or changing a mapping, which is data.

The Shape and the Mappings

Payment is the shape the rest of your app uses, with a rule for each field. Each mapping is a function tagged #[from('source')], and normalize picks the mapping, parses the raw body in whatever format it came in, and checks the result. Press Run to normalize a Stripe event and a form post, and to catch a bad one:

payments.stofPlayground ↗

/// The one shape the rest of the app uses.
#[type]
Payment: {
  #[schema((target_val: str): bool => target_val != null && target_val.len() > 0)]
  str id: ''

  #[schema((target_val: float): bool => target_val != null && target_val > 0)]
  float amount: 0

  #[schema((target_val: str): bool => target_val != null && target_val.len() == 3)]
  str currency: ''

  str customer: ''
  str source: ''
}

/// Stripe sends JSON. Amounts are in cents and currencies are lowercase.
#[from('stripe')]
fn stripe(event: obj) -> Payment {
  const charge = event.data.object;
  new Payment {
      id: charge.id,
      amount: (charge.amount as float) / 100,
      currency: charge.currency.upper(),
      customer: charge.customer,
      source: 'stripe',
  }
}

/// The old billing system posts form data: ref=A-1009&amount=12.50&cur=usd&cust=42
#[from('legacy')]
fn legacy(form: obj) -> Payment {
  new Payment {
      id: form.ref,
      amount: form.amount,
      currency: form.cur.upper(),
      customer: 'legacy-' + form.cust,
      source: 'legacy',
  }
}

/// Turns a raw webhook body from any source into a Payment, or throws.
fn normalize(source: str, body: str, format: str) -> map {
  using raw = new {};
  parse(body, raw, format);

  for (const mapper in funcs(attributes = 'from')) {
      if (mapper.attributes().get('from') == source) {
          using payment = mapper(raw);
          if (!<Payment>.schemafy(payment)) throw(`${source} sent a payment that doesn't fit: ${body}`);
          return payment.to_map();
      }
  }
  throw(`no mapping for ${source}`);
}

#[main]
fn main() {
  pln(self.normalize('stripe', '{"type": "charge.succeeded", "data": {"object": {"id": "ch_3Pq", "amount": 2000, "currency": "usd", "customer": "cus_9s"}}}', 'json'));
  pln(self.normalize('legacy', 'ref=A-1009&amount=12.50&cur=usd&cust=42', 'urlencoded'));
  try {
      self.normalize('legacy', 'ref=A-1010&amount=0&cur=usd&cust=42', 'urlencoded');
  } catch (error) {
      pln(error);
  }
}

Output

Each mapping reads like the provider's documentation: which field is which, and what to convert. using cleans up the raw payload and the payment object after each call, so a long-running worker doesn't accumulate them.

In Your App

Keep the shape in one file and each source's mapping in its own, load them into one document, and call normalize from your webhook handlers:

import { StofDoc } from '@formata/stof';

const mappings = await StofDoc.new(await readFile('payment.stof', 'utf8'));
for (const file of await mappingFiles()) {    // stripe.stof, legacy.stof, ...
    mappings.parse(await readFile(file, 'utf8'));
}
mappings.setMaxExecutionTime(50);

async function normalize(source: string, body: string, format = 'json'): Promise<Payment> {
    return Object.fromEntries(await mappings.call('normalize', source, body, format) as Map<string, unknown>) as Payment;
}

app.post('/webhooks/stripe', express.text({ type: '*/*' }), async (req, res) => {
    verifyStripeSignature(req);                          // your app still checks who sent it
    await payments.record(await normalize('stripe', req.body));
    res.sendStatus(200);
});

app.post('/webhooks/legacy', express.text({ type: '*/*' }), async (req, res) => {
    await payments.record(await normalize('legacy', req.body, 'urlencoded'));
    res.sendStatus(200);
});

A payload that doesn't fit throws before it reaches your database, with the source and the body in the message.

Adding a Source

A new source is a new mapping file, and nothing else in your app changes. Mappings are also a good job for a model: give it the Stof skill, the Payment type, and a few sample payloads from the provider's docs, and check what it writes against those samples before you add it (see Run Rules a Model Writes). Since mappings are data, you can store them in a database and load new ones without restarting.

The formats are built in: json, yaml, toml, urlencoded, and text. See Formats.

Why This Works

• One shape, checked at the edge. Every source produces the same validated Payment, so the rest of your app trusts it.
• One small file per source. Each mapping is a few readable lines, and a change to one provider touches one file.
• Any format in. JSON, form data, YAML, or TOML, parsed by Stof, mapped the same way.
• Safe to change at runtime. Mappings can only read their payload and build the result, within a time limit.

The same pattern fits order and inventory feeds, CRM syncs, analytics events, log formats, and partner APIs: anywhere many shapes need to become one.