Hardware / process engineering
3D printing farm
Eight-printer Klipper farm across two sites, orchestrated by Jove.
- Klipper
- Moonraker
- Mainsail
- Jove
- FastAPI
- React
- Postgres
- Tailscale
- Home Assistant
Overview
A production-minded FDM pipeline spanning two buildings: two printers at the primary site, six more at a second site reached over a private mesh network. Jove is the farm orchestrator Daniel built: live printer status, a G-code library, print kits, a makespan-aware queue and planner, filament tracking, and power control through Home Assistant. The farm reliably finishes 44-hour jobs, which is the number that actually matters, because a long print is where every weakness in the stack gets exercised at once.
Problem
Hobby workflows break at scale: manual queueing, no shared view across sites, alerts when printers are deliberately powered off, and handoffs between slice → print → harvest that do not survive eight machines.
Solution
Standardize on Klipper/Moonraker/Mainsail per printer, run a Jove instance per site (FastAPI + Postgres + React UI), and connect sites over Tailscale. Home Assistant controls power; a custom health proxy gates Uptime Kuma checks on printer power state and remote host reachability.
Architecture
Each printer is a Klipper stack with Moonraker API and Mainsail/ustreamer UI. Jove talks to Moonraker for queue and live SSE status. Operators use Jove routes for farm view, dashboard timeline, G-code library, materials, kits, and makespan-aware planner. Images publish to a container registry, and the remote site runs its own monitoring stack seeded from the same scripts.
Key decisions
Build orchestration in-house (Jove) instead of spreadsheets: queue, kits, and planner in one tool. One Jove per site for local latency and independence. Power-aware monitoring over naive Moonraker pings. Proven slice profiles and batch-like-with-like before optimizing exotic automation.
Challenges
The most expensive failure was not mechanical. A seventeen-hour print died because the host paged eleven megabytes of the printer firmware process out to disk; touching one of those pages stalled it for under two seconds, which was long enough for the microcontroller timer queue to underrun and shut itself down. Three steps downstream the printer cut its own power, which is correct behaviour and also the most visible symptom, so it draws the investigation to the wrong place. The fix was pinning that one latency-critical process out of swap rather than disabling swap, which would only have converted a stall into an out-of-memory kill somewhere else. Worse, nothing alerted: a crashed print leaves the job paused rather than failed, so the completion and error alerts both stayed quiet. Beyond that: multi-site networking, environmental variance, mechanical wear across eight beds, and keeping the operator interface clear under real farm pressure.
Outcome
A coordinated eight-printer farm running software Daniel designed and operates, with alerting that now fires on the state a real crash actually leaves behind rather than only on clean completion. Evidence of full-stack and hardware systems thinking, and of reading failures back to their cause instead of their loudest symptom.