1# Cascade Backpressure Implementation Plan
1# Replace KV Cascade Throttle with Cloudflare Rate Limiting Binding
2
3## Context
4
5`pg_advisory_xact_lock` consumes 99.5% of DB runtime (p50=15s, p99=203s). FileProvider sends hundreds of concurrent file writes, each triggering a separate cascade with advisory locks. They pile up on the same space, creating a convoy that saturates the DB and makes all queries slow (page loads 42s, auth 12s, "Failed query" errors).
5The KV-based cascade throttle (`cascade-throttle.ts`) has a fundamental race condition: `get` then `put` is not atomic, so under burst load all requests read `current = 0` and all get through. This makes it a no-op during FileProvider bulk sync, causing `pg_advisory_xact_lock` convoy (14s waits), worker OOM (503s), and truncated responses (dataCorrupted).
6
7Fix: cap concurrent cascades at 10 per space via KV counter. Return 429 when exceeded. FileProvider backs off via `.serverUnreachable`.
7Fix: replace with Cloudflare's built-in Rate Limiting binding — atomic, zero-latency, no race condition. Name it general-purpose so it's reusable across the API.
8
9## Step 1: Create KV namespace + bind to workers
9## Step 1: Add rate limiting binding to both workers
10
11- run `npx wrangler kv namespace create SUBLIMATED_CASCADE_THROTTLE`
12- add binding to `src/api.sublimated.com/wrangler.jsonc` in `kv_namespaces`
13- add binding to `src/git.sublimated.com/wrangler.jsonc` in `kv_namespaces`
14- add `SUBLIMATED_CASCADE_THROTTLE?: KVNamespace` to Env in `src/api.sublimated.com/src/helpers.ts` (line 43, alongside other KV bindings)
15- add same to Env in `src/git.sublimated.com/src/index.ts`
11**`src/api.sublimated.com/wrangler.jsonc`**:
12- remove `SUBLIMATED_CASCADE_THROTTLE` from `kv_namespaces`
13- add `ratelimits` section:
14```jsonc
15"ratelimits": [{
16  "name": "SUBLIMATED_RATE_LIMITER",
17  "namespace_id": "1001",
18  "simple": { "limit": 5, "period": 10 }
19}]
20```
21
17## Step 2: Create throttle module
22**`src/git.sublimated.com/wrangler.jsonc`**:
23- remove `SUBLIMATED_CASCADE_THROTTLE` from `kv_namespaces`
24- add same `ratelimits` section (same namespace_id = shared counters across workers)
25
19New file: `src/shared/services/write/cascade-throttle.ts`
26## Step 2: Update Env types
27
21- `acquireCascadeSlot(kv: KVStore, spaceId: string): Promise<ThrottleResult>` — read counter from KV key `inflight:{spaceId}`, if >= 10 return `{ acquired: false, retryAfter: 1 }`, else increment and return `{ acquired: true }`
22- `releaseCascadeSlot(kv: KVStore, spaceId: string): Promise<void>` — decrement counter (fire-and-forget safe)
23- use `KVStore` interface from `src/shared/auth/types.ts` (existing pattern: `{ get, put, delete }`)
24- TTL 60s on every KV write (safety net if worker crashes mid-cascade)
25- MAX_INFLIGHT = 10
28**`src/api.sublimated.com/src/helpers.ts`** (Env interface):
29- remove: `SUBLIMATED_CASCADE_THROTTLE?: KVNamespace`
30- add: `SUBLIMATED_RATE_LIMITER?: RateLimit`
31
27## Step 3: Add throttle guard to API write endpoints
32**`src/git.sublimated.com/src/index.ts`** (Env interface):
33- same change
34
29Each handler: acquire slot at start, release in finally. If not acquired, return 429 with `Retry-After: 1` using `raw()` helper.
35## Step 3: Replace throttle calls in all handlers
36
315 files to modify:
37Replace the acquire/try/finally/release pattern with a single `limit()` call. No try/finally needed — rate limiter is fire-and-forget, no release step.
38
331. **`src/api.sublimated.com/src/batch.ts`** — `batchUpsert()` (line 16): guard before `batchWriteItems` + `createBatchCascade` at line 57
342. **`src/api.sublimated.com/src/content.ts`** — `updateItemContent()` (line 115): guard at top of function, before CAS upload. covers both branch-save path (line 181, direct cascade) and main path (line 220, deferred cascade via updateItem)
353. **`src/api.sublimated.com/src/content-api.ts`** — folder creation cascade (line 531) and file upload paths: guard before cascade calls
364. **`src/api.sublimated.com/src/notes.ts`** — `saveNote()`: guard before `createBatchCascade` at line 169
375. **`src/git.sublimated.com/src/index.ts`** — git push handler: guard before cascade, return HTTP 429 (plain text, not pack-line) if throttled
38
39Pattern for each handler:
40```
41const kv = ctx.env.SUBLIMATED_CASCADE_THROTTLE;
42if (kv) {
43  const slot = await acquireCascadeSlot(kv, space.id);
39**Before (each handler):**
40```typescript
41const throttleKv = ctx.env.SUBLIMATED_CASCADE_THROTTLE;
42if (throttleKv) {
43  const slot = await acquireCascadeSlot(throttleKv, space.id);
44  if (!slot.acquired) {
45    return raw(JSON.stringify({ok:false, error:{type:'rate_limited', message:'Too many concurrent writes'}}), 429, {'content-type':'application/json','retry-after':'1'});
45    return raw(JSON.stringify({...}), 429, {...});
46  }
47}
48try {
49  // ... existing handler logic with cascade ...
49  // ... handler logic ...
50} finally {
51  if (kv) releaseCascadeSlot(kv, space.id).catch(() => {});
51  if (throttleKv) releaseCascadeSlot(throttleKv, space.id).catch(() => {});
52}
53```
54
55For deferred cascades (content.ts main path with `deferCascade: true`): release slot when handler returns, not when deferred cascade completes. Imprecise but safe — advisory lock still serializes at DB level.
56
57## Step 4: Client-side — map .rateLimited to .serverUnreachable
58
59File: `sdk/swift/Sublimated/Sources/SublimatedFileProvider/SublimatedFileProviderExtension.swift`
60
61In `mapError()` (line 745), add case:
62```swift
63case .rateLimited:
64    return NSFileProviderError(.serverUnreachable)
55**After:**
56```typescript
57if (ctx.env.SUBLIMATED_RATE_LIMITER) {
58  const { success } = await ctx.env.SUBLIMATED_RATE_LIMITER.limit({ key: `cascade:${space.id}` });
59  if (!success) {
60    return raw(JSON.stringify({ ok: false, error: { type: 'rate_limited', message: 'Too many concurrent writes' } }), 429, { 'content-type': 'application/json', 'retry-after': '1' });
61  }
62}
63// ... handler logic (no try/finally wrapper needed) ...
64```
65
67FileProvider already has built-in retry with backoff for `.serverUnreachable`. No manual `signalErrorResolved()` needed — system retries automatically. When the retry succeeds (counter below 10), normal sync resumes.
66Key prefix `cascade:` keeps namespace clean for future rate limiting uses (e.g. `api:{userId}`).
67
69Also add same mapping in `SublimatedEnumerator.swift` `mapError()` (line 296).
685 call sites to update:
69
71HTTPClient already detects 429 and throws `.rateLimited(retryAfter:)` — no changes needed there.
701. **`src/api.sublimated.com/src/items.ts`** — `createItem` (line 594), `updateItem` (line 1067), `deleteItem` (line 1159)
712. **`src/api.sublimated.com/src/content-api.ts`** — multipart upload handler (line 447)
723. **`src/git.sublimated.com/src/index.ts`** — git push handler (line 4001)
73
73## Step 5: Finish formatPgError fixes
74Remove imports of `acquireCascadeSlot`/`releaseCascadeSlot` from all files.
75
75Complete the remaining catch blocks in web worker page-data files (partially done):
76- `src/sublimated.com/src/lib/page-data/fetch-folder.ts` — 1 remaining (annotation stats)
77- `src/sublimated.com/src/lib/page-data/fetch-annotations.ts` — 1 catch block
78- `src/sublimated.com/src/lib/page-data/fetch-commits.ts` — 5 catch blocks (skip CAS fetch errors, fix DB-related ones)
79- `src/sublimated.com/src/lib/page-data/fetch-content.ts` — 2 catch blocks
80- `src/sublimated.com/src/lib/page-data/fetch-intents.ts` — 4 catch blocks (1 already uses formatPgError)
76## Step 4: Remove old KV throttle module
77
82## Step 6: Unit tests
78- delete `src/shared/services/write/cascade-throttle.ts`
79- delete `src/shared/test/cascade-throttle.test.ts`
80- remove re-export from `src/api.sublimated.com/src/commits.ts` if present
81
84New file: `src/shared/test/cascade-throttle.test.ts`
82## Step 5: Deploy and verify
83
86- acquire returns true when counter is 0
87- acquire returns false when counter >= 10
88- release decrements correctly
89- concurrent spaces use separate keys
90- use in-memory KVStore mock
84- `cd src/api.sublimated.com && npm run deploy`
85- `cd src/git.sublimated.com && npm run deploy`
86- check Cloudflare logs: 429s should appear under burst load, no more 503 OOM
87- run e2e tests: `cd src/e2e && bash test-api.sh`
88- FileProvider sync: files that get 429 retry via `.serverUnreachable` backoff
89
92## Step 7: E2E test
90## Files modified
91
94New file: `src/e2e/test-cascade-throttle.sh`
95
96- create anonymous user + space + file
97- fire 15 concurrent content updates via `curl &`
98- collect HTTP status codes
99- assert at least 1 returns 429 with `Retry-After: 1` header
100- assert some return 200 (those that got through)
101- wait 2s, retry a 429'd write, assert 200
102- cleanup (delete user)
103
104## Verification
105
106after deploying both API and git workers:
1071. PlanetScale insights: `pg_advisory_xact_lock` p50 should drop from 15s to <1s
1082. Cloudflare logs: timing for `resolve-caller` and `folder-cte` should be <500ms
1093. no more "Failed query" errors on unrelated pages
1104. e2e test passes: 15 concurrent writes, some get 429, retries succeed
1115. FileProvider sync still works (just slower when hitting 429 — retries automatically)
92- `src/api.sublimated.com/wrangler.jsonc` — add ratelimits, remove KV binding
93- `src/git.sublimated.com/wrangler.jsonc` — add ratelimits, remove KV binding
94- `src/api.sublimated.com/src/helpers.ts` — Env type change
95- `src/git.sublimated.com/src/index.ts` — Env type change + throttle call
96- `src/api.sublimated.com/src/items.ts` — 3 handlers simplified
97- `src/api.sublimated.com/src/content-api.ts` — 1 handler simplified
98- `src/shared/services/write/cascade-throttle.ts` — deleted
99- `src/shared/test/cascade-throttle.test.ts` — deleted