Inside the Keeper Fleet: 12 Autonomous Jobs That Run THRYX 24/7

12 min read

A token launchpad with 178 users, 1,650 tokens, and 11,400 trades needs to do a lot of housekeeping that users never see. Fees need to be swept from hooks, distributed to stakers, converted to ETH for gas sponsorship, and used to maintain the NEW/WETH pool's price peg. All of this runs as Node.js cron jobs on the THRYX server, signing transactions with the deployer key and routing them through a private mempool. If the server goes down, trading still works (the Diamond contract is permissionless), but fee collection, staker yield, and pool maintenance pause until it recovers.

The keeper topology

The dependency chain

Keepers form an implicit dependency chain through the Diamond's shared state. Hook-fee-collector sweeps fees into the Diamond, increasing its WETH balance. Fee-router-keeper reads that WETH balance and distributes it to 5 buckets. The staker share from fee-router-keeper bumps accRewardPerShareWei, which the solvency-monitor reads to compute staker debt. The pool-mm-keeper also routes 50% of LP-fee collections through the FeeRouter inline (not waiting for the fee-router-keeper's next cycle).

The dependency is unidirectional: hook-fee-collector never reads staker state, and solvency-monitor never writes fee-router state. Each keeper can run, fail, or skip independently. The system converges to the correct state over time as long as the keepers eventually run. There is no keeper that blocks another keeper from executing.

Deduplication and leader election

THRYX runs on a single Render instance, so there is no multi-server leader election problem. However, keepers still need deduplication because Node.js timers can drift, and a server restart can cause two intervals to fire in quick succession. Each keeper calls claimJobTick(name, minIntervalMs) at the start of every cycle. This function does an atomic INSERT ... ON CONFLICT UPDATE on a Turso DB row, succeeding only if the last claim was more than minIntervalMs ago. If the claim fails, the keeper skips silently. This prevents double-execution even during server restarts.

The pool-mm-keeper in detail

The pool-mm-keeper is the most complex keeper, doing three things per cycle:

1. LP fee collection: every 5th cycle (7.5 minutes), calls collectLpFeesBatch() on 4 Diamond-owned LP NFTs. Routes 50% of collected ETH immediately through FeeRouter so stakers earn from LP fees without waiting for the fee-router-keeper.
2. Drift detection: reads the pool's current tick via V4 PoolManager.extsload. Computes drift = tick - PEG_TICK (peg = 234,800). If |drift| > BAND_TICKS (1,500), an arb is needed.
3. Drift arb: if tick is above peg (NEW over-priced in the pool), calls treasuryBuyNew() with Diamond's own WETH to push tick down. If tick is below peg (NEW under-priced), calls sellNewForEthFor() to push tick up. Swap sizes are capped at 0.001 ETH or 10M NEW per cycle, with 75-100% random jitter to prevent bot pattern-matching.

The timing jitter deserves special mention. Instead of a fixed setInterval(90000), the keeper uses self-rescheduling setTimeout with a random delay in [63s, 117s] (base 90s * [0.7, 1.3]). Sandwich bots that fingerprint "a swap from 0x7a3E... hits the pool exactly every 90 seconds" now face a 54-second uncertainty window, making pre-positioning unprofitable.

The solvency-monitor recovery cascade

The solvency-monitor is the protocol's safety net. Every 60 seconds, it reads Diamond ETH, WETH, totalStakedNew, accRewardPerShareWei, and paymasterEthBalance. It computes staker debt (totalStaked * accRPS / 1e18) and surplus (Diamond WETH - staker debt + insurance fund). If surplus goes negative for 2 consecutive ticks, the monitor launches a 4-step automated recovery before emailing the admin:

1. Reconcile: calls reconcilePaymasterEthBalance() to sync stored paymaster balance with actual Diamond ETH.
2. Distribute: calls distributePending() to process any queued FeeRouter pending amounts (in case a prior distribution was blocked by the staker-debt guard).
3. Sweep: calls sweepHookFees() to pull accumulated WETH from anti-sniper hooks.
4. Emergency sell: calls sellNewForEthFor() with a small NEW amount to inject ETH directly.

One step fires per tick (60s spacing between attempts), giving on-chain state time to settle. If all four steps fail to restore solvency, the monitor sends an email to the admin with the current deficit amount, triggering manual investigation. The email has a 6-hour cooldown and only re-fires if the deficit has grown by at least 10% since the last alert.

Staker-debt guards (shared invariant)

Three keepers share the same invariant check: pool-mm-keeper, fee-router-keeper, and paymaster-auto-rescue. Before executing any operation that spends Diamond WETH or increases staker debt, they read getStakingSummary() and compute totalStaked * accRPS / 1e18. If Diamond WETH is at or below that amount, the operation is skipped. This prevents a cascade where one keeper's action makes staker claims insolvent, which was the root cause of the 2026-05-24 incident.

Kill switches and environment overrides

Every keeper has an environment variable kill switch:

Setting any of these to "false" in the Render environment variables immediately disables the keeper on the next tick. No code deploy required. The admin API at /api/admin/manage/jobs lists all registered keepers with their last tick timestamp, success/failure counts, and skip reasons.

Observability

Every keeper registers itself via registerJob(name, { intervalMs, description }). Each tick records one of three outcomes: recordSuccess(details), recordSkip(reason), or recordFailure(error). The admin endpoint GET /api/admin/manage/jobs returns the full keeper topology with per-keeper state: last tick time, consecutive failures, total ticks, and the most recent report object. This is the primary debugging surface when something goes wrong with protocol automation.

Frequently asked questions

Related Posts

• 5-Way Fee Router — The distribution logic the fee-router-keeper triggers.
• Zero-Gas Trading — The paymaster system the auto-rescue protects.
• Anti-Sniper Hooks — The hooks the fee-collector drains.
• Staking NEW $THRYX — The yield pipeline the fee-router feeds.