Skip to content
openproduct
Search

Runbook

One page per degraded state, reached from the alert rather than found by search. Each says what a visitor sees while it is true, which route re-fires it, and what recovered looks like — so the recovery line means one thing.

How a phase is re-fired

  1. Wait one tick. Every phase runs on the next materialisation cron (every ten minutes); most stale phases are a tick that has not happened yet, and /health says when the last one did.
  2. Run the gateway's cron by hand: Cloudflare dashboard → Workers → openproduct-gateway → Triggers → the materialisation schedule → Run. That is the same code path as the schedule, with the same ledger, so /health records it.
  3. Call the route directly, from a session that holds the binding (`wrangler dev` on the gateway, or core's own `wrangler dev`): POST the route below with a request id. This bypasses the ledger for that phase, so /health will not show the run — use it to unblock, then let the next tick record.

When a promise breaks

The reader of /health judges three promises every five minutes and, when one flips, names one of these. Two of them need different people: the sweep not running is an operations problem; the sweep running and the page disagreeing is a product defect.

The search index has not been rebuiltsearch-freshness · sweep_not_run

What a visitor sees

Search answers from an old index: a listing published since the last rebuild is not found by its own name, and /health reports the searchIndex phase as stale, failed or never run.

Re-fire

/internal/openproduct/search/rebuild

Recovered

/health reports searchIndex healthy with a lastRunAt inside the last thirty minutes, and the newest published listing is found by its slug.

The index is fresh and still does not answersearch-freshness · page_disagrees

What a visitor sees

/health says searchIndex is healthy, and a listing published more than one materialisation interval ago is not in its own search results. The rebuild ran and left it out — a product defect, not a schedule.

Re-fire

Nothing to re-fire — the cause is not a schedule.

Recovered

The listing is found by its slug on a plain anonymous search; nothing to re-fire, the cause is in the index's query.

The counts on the front page are oldcount-freshness · sweep_not_run

What a visitor sees

Every count renders, and every count is older than three integrity intervals; /health reports the integrity phase as stale, failed or never run. The dash legend on /docs/checks says what a missing count means.

Re-fire

/internal/openproduct/endorsements/integrity-sweep

Recovered

/health reports integrity healthy with a lastRunAt inside the last three hours.

The front page is being rendered, not servedcache-hit · cache_cold

What a visitor sees

Nothing — the page is correct and slower. cf-cache-status on an anonymous read of / is MISS, DYNAMIC or absent more than one time in ten over the last hour. Check that Workers Caching is enabled on openproduct-web and that the document still carries s-maxage.

Re-fire

Nothing to re-fire — the cause is not a schedule.

Recovered

The ratio of HIT over the synthetic's last hour of plain reads is at or above the target.

The product could not be askedsearch-freshness · unreachable

What a visitor sees

The gateway or the site did not answer, or answered a 5xx, to an anonymous read. Every promise is unknown until it does; this line comes first and the others wait.

Re-fire

Nothing to re-fire — the cause is not a schedule.

Recovered

/health and / answer 200 to the synthetic's next cycle.

When a phase is stale

One page per sweep phase, in the order /health lists them. Each phase's row on /docs/checks links here.

The catalogueEvery ten minutes, after the upvote integrity pass and before the search index is built from it. · stale after 30 minutes

What a visitor sees

The front page and the catalogue API answer from the last catalogue computed, and say so: the moment under the heading ("as of HH:MM UTC") stops moving. A listing published since is not on the front page. /health reports catalogue stale after thirty minutes.

Re-fire

/internal/openproduct/catalogue/rebuild

Recovered

The moment on the front page moves again and /health reports catalogue healthy with a lastRunAt inside the last thirty minutes.

Search indexEvery ten minutes, when a listing has changed since the last build. · stale after 30 minutes

What a visitor sees

Search answers from the last index built: a listing published since is not found by its own name, and the search page says nothing about why. The catalogue can be current while the index is not — they are separate phases, and the index is built from the catalogue in the same tick.

Re-fire

/internal/openproduct/search/rebuild

Recovered

The newest published listing is found by its slug on an anonymous search, and /health reports searchIndex healthy.

Upvote integrityHourly. · stale after 3 hours

What a visitor sees

Every count on the site is the last discount computed: an upvote cast since is counted raw until the next pass, and a listing's clean-votes mark does not move. A dash (—) on a count is a different thing — it means never measured, and the legend on /docs/checks says which is which.

Re-fire

/internal/openproduct/endorsements/integrity-sweep

Recovered

/health reports integrity healthy with a lastRunAt inside the last three hours, and the counts move on the next front-page read.

Automatic checksEvery six hours. · stale after 18 hours

What a visitor sees

Nothing, by design: the detectors raise flags for a person to read and remove nothing, so a stale defence sweep is a reports queue quieter than the week deserves. /health reports it stale after eighteen hours.

Re-fire

/internal/openproduct/defence/sweep

Recovered

/health reports defence healthy, and the reports queue gains whatever the week held.

The day's boardEvery ten minutes; a day's numbers close shortly after midnight UTC. · stale after 30 minutes

What a visitor sees

The day's board and the leaderboard stop moving: a launch that fired today is not placed, and yesterday's numbers may not have closed. The launches strip on the front page shows the last ranking it had.

Re-fire

/internal/openproduct/launches/board-sweep

Recovered

The board's ranking moves and /health reports board healthy.

The weekly digestEvery ten minutes, doing work once a week. · stale after 30 minutes

What a visitor sees

The digest archive says nothing launched this week, and nobody who asked for the email gets one. The page is honest about what it has and silent about what it missed.

Re-fire

/internal/openproduct/digest/queue

Recovered

The week's digest page exists with its launches on it, and /health reports digest healthy.

Daily snapshotsEvery ten minutes, doing work once a day — the first tick after midnight UTC. · stale after 72 hours

What a visitor sees

A listing page's "what changed this week" compares against a snapshot up to three days older than it should be, then says there is no earlier snapshot. Not wrong, only wider; two missed days in a row is the alarm.

Re-fire

/internal/openproduct/snapshots/sweep

Recovered

/health reports snapshots healthy with a lastRunAt inside the last day.

When the deploy lags main

The deploy lags main

The rule

A re-score, a verification lane, or any measurement of "what the product does" does not run until the branch under test is the branch that is live. Measuring a deploy that lags main under-reports the fix and somebody draws the wrong conclusion about which milestone worked (WC-H).

How to read the deployed commit

  1. The deploy run: GitHub Actions on pullely/openproduct → the run for the merge commit → the job named `<component> · prod · Deploy`. Its revision line names the commit; a green job is a deploy that finished, and a job that has not run is a deploy that has not happened.
  2. The worker's versions: Cloudflare dashboard → Workers → the worker → Deployments. The active version's upload time is the deploy's moment; compare it with the merge commit's time.
  3. The live side itself: /health on the gateway carries `build` — the active Worker version's id and the moment it was uploaded (OO3, from the version_metadata binding) — and /docs/checks shows it under the heading. A version uploaded before the merge is a deploy that has not happened yet. It is a version, not a commit: the estate's composition uploads without a tag, so the SHA is read from the deploy job and the time compared here.

Recovered

The `prod · Deploy` job for the head of main is green and the version /health reports was uploaded after the merge. Then, and only then, measure.

A deploy was not verified

What a visitor sees

Possibly nothing yet — this is the line that fires BEFORE a promise breaks. /health answers with a new version, and either a phase that runs every tick has not run since that version was uploaded (the schedule did not survive the deploy: RS0's outage, caught within fifteen minutes instead of four epics), or one of the four things RS4 signed stopped being true: the signed-out upvote no longer answers 401, /signin is not openproduct's own page, the front page title lost the brand, or search stopped answering for the seeded listing.

Re-fire

/internal/openproduct/search/rebuild

Recovered

The synthetic's next cycle finds every per-tick phase with a lastRunAt after the version's upload and all four walks true, and logs deploy_verified for that version. If the phases never run: the cron did not register — `wrangler triggers deploy --env <env>` from a credentialed session, then watch /health.