39: Sometimes a complete rewrite is the right choice

Episode 39 is the show's 40th, since numbering started at zero. Jim and Wolf ask listeners whether a video version would be worth doing, and Wolf adds "rules" to the values-and-goals framework from episode 36. In their weekly catch-up, Jim sets up his new Mac Studio M5 Max, and Wolf gets PRs merged into direnv and Quarto while recovering from wrist surgery. MAIN TOPIC: REWRITING THE CLINIC DASHBOARD Background Jim's company, Avairis, writes radiology software. His biggest client since 1985 is the largest freestanding mammography clinic in the US, which sees 500 to 600 patients a day across 6 or 7 offices in western New York. In 2006 Jim built "Dashboard" to track patient flow, so no one gets left in a waiting room, and to log hundreds of event metrics. It shows configurable "views" of boxes, each representing a room or a patient state, and lets staff drag patients between boxes. The prototype that stuck The original version was a JavaScript front end, a Perl back end (later moved to mod_perl), and Postgres. Every session polled the database every 10 seconds, with up to about 140 sessions at once. The clinic grew to 252 views. Users could write their own WHERE clauses, and the queries became bloated, some taking 18 seconds. Jim had wanted to rewrite it for about 15 years. The new system: Avairis Data Engine (AVDE) Jim started on June 1 and wrote it in TypeScript on Node.js, choosing TypeScript so the compiler would catch errors before users did. The core idea is to flip polling into push: - Triggers: lightweight Postgres triggers only send NOTIFY messages, and the Node service LISTENs for them. - Shared worker threads: each unique query, identified by a hash, gets one worker thread. Every client running a view with that query subscribes to the same worker, so 30 users means one query. - Deltas over WebSockets: a client gets the full dataset once, then only the changed rows. - Update-storm handling: a 10 ms debounce collapses duplicate notifications. If more than 25 are queued, the service drops them and does one full requery. - Reconnect grace period: workers stay alive for 60 seconds after the last client disconnects, so a reconnect doesn't trigger a full reload. All of these timings are tunable. - Leaner queries: each query selects only the fields its view needs. - Drag-and-drop actions: "on-drag" and "on-drop" actions run a small stack-based DSL that Jim wrote, parsed with Perl's Yapp and executed server-side, with a plugin system for adding functions. Because views can now be built on anything in the database, not just appointments, the clinic can track things like accounts receivable aging. Rollout and switching cost Wolf asked his standard question: what does it cost users to switch? Jim's answer was nearly nothing. A right-click converts an old view, and old and new views appear in the same list, with a column marking AVDE views. A view stays hidden from users until it's published. 85% of views convert cleanly. The other 37 all failed for the same reason: they mixed AND and OR without parentheses, which the new DSL now forbids because non-programmers can't predict what such conditions will do. In the demo, Jim converted the infamous 18-second view and it loaded in under a second, which got a real reaction from the staff. The two-hour session overwhelmed them a bit, but by the next day staff were asking good questions and building views. Real load testing with about 140 live sessions starts next week. Wolf's verdict Despite the episode title, Wolf had no fight to pick. His rule against total rewrites is really about not "stopping the train," as Netscape did when Microsoft overtook it. Jim's rewrite never stopped the train and made more progress than incremental work would have. Wolf praised the design, especially the WebSockets, which Jim had forgotten to mention. He tied it to how he sees his own job: making things faster means optimizing the whole system, including the human workflow and perceived performance, not just CPU cycles. WRAP-UP Episode 41 will feature the show's first guest, Marlon from Meta, talking about performance. Wolf described his new "answer first, then offer more detail" rule for keeping his answers short. They invited listeners to the Saturday 11 a.m. nerd lunch at Biwako in Saline Township. Wolf's next topic is cargo cults, with a nod to Feynman. Hosts: Jim McQuillan can be reached at jam@RuntimeArguments.fm Wolf can be reached at wolf@RuntimeArguments.fm Follow us on Mastodon: @RuntimeArguments@hachyderm.io If you have feedback for us, please send it to feedback@RuntimeArguments.fm Check out our webpage at http://RuntimeArguments.fm Theme music: Dawn by nuer self, from the album Digital Sky