Skip to content
Folio

Diagrams

DataflowFigure

A pipeline: where data comes from, what happens to it, and who ends up with it.

Usage

When to reach for it

Structurally close to an architecture map, read for a different question. classification gets its own chip because a governance reviewer is looking for exactly that.

Ships from@misoto22/design/diagrams

TSX
import { DataflowFigure } from '@misoto22/design/diagrams'

Notes

A pipeline: where data comes from, what happens to it, and who ends up with it.

Structurally close to an architecture map and read for a completely different question, which is why it is a separate renderer rather than a preset. An architecture diagram is read for "what talks to what". A data-flow diagram is read for "what is IN this arrow" — and that question is why classification gets its own mono chip under the label rather than being folded into the wording. clickstream / PII touch and clickstream are two different facts, and a governance reviewer is looking for the second one.

Stages are printed as column headings on a rule across the top. They are the axis of the figure: a node's stage is its position along the pipeline, and a reader should be able to answer "how far has this got" by looking up rather than by tracing arrows.

Anatomy

DataflowFigure anatomy
ElementDescription
Figure shellrequiredThe frame around the picture. What a pipeline hands it is a flow list with the classification folded in — "clickstream — PII touch" — so the fact the diagram was opened for reaches a reader who never sees the chip.
Stage headingrequiredOne mono caption per stage on a rule across the top. It is the axis: a node’s stage is how far along the pipeline it sits, so "has this been aggregated yet" is answered by looking up rather than by tracing arrows.
Node platerequiredOne box per node, placed by stage across and row down. The row pitch leaves a clear band between two boxes in the same column, which is where a line and its wording go.
FlowA routed line with a single arrowhead, at its to end. It is the direction claim the whole figure is read for.
Classification chipclassification, printed in mono under the wording rather than inside it. "clickstream" and "clickstream / PII touch" are two different facts, and only one of them is what a governance review came for.

Best practices

Do

  • Put the governance fact in classification rather than in the label. It prints as its own chip on the line and is appended to the flow’s summary line, so it survives both a reader who is scanning the picture and one who only has the text.
  • Keep stage inside the stages you declared. A node’s x is computed from its own stage index rather than looked up, so a node at stage 5 beside four stages is drawn a full column past the last heading, under no heading at all — development prints DIAGRAM_STAGE_OUT_OF_RANGE, and the summary files it under a band called outside the declared stages rather than beside the nodes the axis does label.
  • Write a two-way exchange as two flows. Every line carries a head at its to end only, so one flow between a service and its cache says data moves one way, whatever the label claims about the round trip.

Don’t

  • via, channelX and channelY are never honoured here — a data-flow node is always placed by stage and row, so an author’s waypoints refer to a grid this renderer did not build. They are dropped and every line is routed from scratch: the specification still typechecks and the picture is not the one it drew.
  • Watch the fan-in. Lines sharing a face are spread evenly across it, and an arrowhead is 11 user units wide against a plate about 55 units tall — so four lines into one face is where the heads meet, and the fifth and sixth arrive as one thick mark. A pipeline at that density wants splitting, not more edges.

Examples

default

Two sources joining at one gate. The mono chip under each label is the classification, printed apart from the wording because a pipeline gets read for what is inside the arrow rather than for what the arrow is called — clickstream and no PII answer different questions, and only one of them is the question a governance reviewer arrived with. The stage headings are the axis, so how far something has got is answered by looking up rather than by tracing.

Event pipeline

What is in the arrow matters more than the arrow.
Event pipeline4 elements and 3 relationships. What is in the arrow matters more than the arrow.eventsWeb SDKbrowsereventsMobileiOS / AndroidPII guardConsent gatepolicy filtercuratedWarehouseanalytics tables

4 elements and 3 relationships. What is in the arrow matters more than the arrow.

  • Stage: Sources
    • Web SDK (frontend) — browser
    • Mobile (frontend) — iOS / Android
  • Stage: Gate
    • Consent gate (security) — policy filter
  • Stage: Store
    • Warehouse (database) — analytics tables
  • Web SDKConsent gate: clickstream — user events
  • MobileConsent gate: app events — device events
  • Consent gateWarehouse: accepted — no PII
Key
  • CLIENT
  • POLICY
  • DATA

one source three consumers

A fan-out, and the reason classification is a field rather than a word in the label: one table feeds three consumers and each arrow leaves carrying something different. Three flows share a single face here, so the router spreads their ports across it — stacked on the face midpoint instead, three arrowheads would read as one. The ad partner is the only consumer outside the account, which is why its flow is the one drawn as crossing a boundary.

What leaves the events table

One source, three consumers, three payloads.
What leaves the events table4 elements and 3 relationships. One source, three consumers, three payloads.PIIEvents tablerawCLOUDBI dashboardsinternalSERVICEModel trainingnightlyEXTERNALAd partnerthird party

4 elements and 3 relationships. One source, three consumers, three payloads.

  • Stage: Source
    • Events table (database) — raw
  • Stage: Consumers
    • BI dashboards (cloud) — internal
    • Model training (backend) — nightly
    • Ad partner (external) — third party
  • Events tableBI dashboards: daily rollup — aggregated
  • Events tableModel training: training set — hashed ids
  • Events tableAd partner: conversions — no PII
Key
  • DATA
  • CLOUD
  • SERVICE
  • EXTERNAL
Leaving the estate
  • Only the ad partner sits outside the account.
  • That is the arrow a reviewer reads first.

a stage the data skips

A stage is a claim about how far something has got, and that axis is what makes the bypass legible: the sampled copy leaves the SDK at stage zero and arrives at stage two without passing the redactor, still carrying its PII chip. The same three flows written out as a list of pairs would hide that, because a list has no axis to skip. The emphasised line is what the pipeline is for; the dashed one is a one per cent sample kept so somebody can debug a payload, which is exactly the kind of path an audit finds last.

Clickstream

Three stages, and one copy that skips the middle one.
Clickstream4 elements and 3 relationships. Three stages, and one copy that skips the middle one.rawWeb SDKbrowserPOLICYRedactordrops identifierscuratedLakepartitionedDATADebug bucket7-day TTL

4 elements and 3 relationships. Three stages, and one copy that skips the middle one.

  • Stage: Collect
    • Web SDK (frontend) — browser
  • Stage: Scrub
    • Redactor (security) — drops identifiers
  • Stage: Serve
    • Lake (database) — partitioned
    • Debug bucket (database) — 7-day TTL
  • Web SDKRedactor: events — PII
  • RedactorLake: hourly — no PII
  • Web SDKDebug bucket: sampled 1% — PII
Key
  • CLIENT
  • POLICY
  • DATA

what is in the arrow

Four stages, because minimisation is a stage: the reviewer's question is whether anything still labelled PII reaches the warehouse, and the answer is a line you follow rather than a paragraph you trust. Every classification is folded into the accessible summary as well as printed on the chip, so a reader who cannot see the picture gets card and email — PII rather than card and email alone. Keep the wording and the classification apart even when they feel like one phrase; the moment they merge, the chip stops being a field anything can be checked against.

Card data, minimised

Read for what crosses each stage, not for what talks to what.
Card data, minimised5 elements and 4 relationships. Read for what crosses each stage, not for what talks to what.rawCheckout formbrowserrawSupport inboxticket bodiesPOLICYTokeniserVault transitDATAWarehouseorder tablesSERVICEDashboardsfinance team

5 elements and 4 relationships. Read for what crosses each stage, not for what talks to what.

  • Stage: Collected
    • Checkout form (frontend) — browser
    • Support inbox (frontend) — ticket bodies
  • Stage: Minimised
    • Tokeniser (security) — Vault transit
  • Stage: Stored
    • Warehouse (database) — order tables
  • Stage: Served
    • Dashboards (backend) — finance team
  • Checkout formTokeniser: card and email — PII
  • Support inboxTokeniser: free text — PII, unstructured
  • TokeniserWarehouse: order rows — tokenised, no PAN
  • WarehouseDashboards: daily rollup — aggregated
Key
  • CLIENT
  • POLICY
  • DATA
  • SERVICE

Accessibility

  • A flow’s classification is folded into its summary line, so "clickstream — PII touch" reaches a reader who cannot see the chip.