Maintaining the Room's Hardware

Keeping a Q-SYS Core and two K-array amplifiers maintainable, by proving every change offline and making every trial reversible.

Part of Residents Platform: House Pub.

The pub’s audio runs on vendor appliances I cannot inspect: a Q-SYS Core 110f and two K-array amplifiers. They are production equipment in a building where residents sleep, there is no standing maintenance window and nobody on site, and the AV network has no internet access. This page is about how I approached keeping them maintainable, and reducing the room’s dependence on one Raspberry Pi, without risking the room’s audio. It is a story about method more than mechanism.

At a glance

  • Status: in progress. Parts are verified on the real hardware, and parts are lab-only or still design (see the table below).
  • Devices: one Q-SYS Core 110f and two K-array amplifiers, called KA18 and KA34.
  • Method: emulated labs, inputs pinned by hash, reversible trials, and redacted evidence receipts.
  • Deliberately not here: how access is maintained. That part is security-sensitive.

How it works

  1. Prove it offline. Run the real vendor software in an emulator with no network and disposable disks.
  2. Trial it reversibly. Publish a change inactive, arm it explicitly, trial it for a bounded time, and withdraw it by default if confirmation does not arrive.
  3. Record evidence. Write a redacted receipt tied to the exact inputs, and say what it does not prove.
flowchart TD
  L[Emulated lab] --> P[Publish inactive]
  P --> T[Bounded trial]
  T -->|confirmed| E[Evidence receipt]
  T -->|not confirmed| R[Automatic withdrawal]
  R --> S[Stock state restored]

Problems I had to solve

Production equipment, remote only, and no way in if it breaks

These devices are remote-only: nobody is on site to reseat a cable or press a button, and the AV network is segmented with no internet. The vendor firmware is a black box. A recovery path that only works while the device is healthy is not a recovery path.

It matters because the Raspberry Pi in the rack does too much. If it dies, the room loses song identification, TV and projector control, lighting, and remote management. The goal was to keep an owner-controlled way to maintain each device that survives reboots and firmware reinstalls, and eventually to move the Pi’s services onto equipment that is already in the room.

How do you test changes you cannot risk on the real thing?

  • Emulated copies of the real software. Disposable emulated machines with no network card run the actual Core kernel, filesystem, and vendor reset code. For the amplifiers, their ARM userland runs under an emulator with the vendor filesystem mounted read-only, and its hash is re-checked after shutdown.
  • A real test suite against a real userland. The Pi’s whole application test suite, 1,766 tests with 11 skipped, ran inside an emulator on the amplifier’s stock userland, using a separately added Python 3.9 runtime because stock Python is older than the app needs.
  • Real software with injected failures. I drove a real network-manager daemon, the same upstream version the amplifier uses, with injected failures. It exposed a bug in my own policy code that a mock would have missed: when the manager left a connection request unanswered, my code dropped its pending attempt.
  • Pinned inputs and refusals. Inputs are pinned by hash where it matters, builders refuse to write into an existing output directory, and lab code refuses to run outside the lab.

A change that can lock you out must be able to undo itself

Every change goes through separate, explicit stages: publish inactive, select, arm, then a bounded trial. On the amplifiers, if confirmation does not arrive from the newly established channel within a fixed window, access is withdrawn automatically, and the operating-system rollback is rehearsed in the lab. An attempt marker prevents a change from silently re-arming, and in lab tests a failed cleanup step did not stop the rollback.

On the Core I stopped one maintenance window before selection or arming completed. Inactive files remain, and the notes record exactly which. I chose to preserve unfamiliar state instead of cleaning it up.

Claims drift upward unless something stops them

Every write-up has a “what this does not prove” section, and results carry labels such as live-confirmed, offline-confirmed, lab-only, or unknown. I withdrew overstated claims in my own earlier documentation. Redacted receipts are bound to the exact input hashes, and the private originals are retained with a statement of which files they cover.

A failure I kept

The first same-version firmware reinstall on KA18 lost the maintenance channel. I could not confirm why, because the device’s logs did not survive the reboot. I recorded it as a failure instead of relabeling it. Then I changed the design so the channel restarts itself, and verified that with a deliberate stop test: it came back in about 35 seconds without help. After the redesign, a same-version reinstall on KA18 and a first one on KA34 each brought the channel back in about seven to nine minutes.

Silent successes

  • A check that passes on nothing. A boot-time integrity check accepted an empty input, so a missing input looked like success. I pinned the input and made the check refuse an empty one.
  • Unchanged bytes are not preserved access. In an offline rehearsal of a signed vendor update, the update left one of my own files empty, and my recovery check missed it. I tightened exactly one condition.
  • Gaps in stock userland. The amplifiers’ stock software is missing or older than some components my services need. I tested a private supplement rather than touching the vendor’s libraries.

Moving the Pi’s work onto the amplifiers

Two devices cannot simply elect a leader by heartbeat, because a network split gives you two leaders. I wrote the rejected designs down in the design spec. In a mutation test, my ownership checks caught all three deliberately unsafe changes, and four architectural counterexamples produced a double effect in simulation, so I did not adopt those designs.

The amplifiers also need their own internet uplink with ordered fallback across saved networks. From reading its source, the network manager appears not to switch networks when the upstream is dead, so I wrote a durable external policy. That has not been checked on the device. The policy passes 29 unit and 30 integration checks, and its adapter layer passes 54 unit and 39 integration checks as of the end of October. The adapter figures include the policy checks, so these are overlapping layers, not additive ones. The policy is designed never to touch Ethernet, but coexistence is untested on hardware.

Where it stands

  • Q-SYS Core.
    • Verified on hardware: a maintenance channel across ordinary reboots, in two trials.
    • Lab only: surviving a factory reset, a loader replacement, and a signed vendor firmware update.
    • Stopped: one maintenance window, before activation.
  • KA18 and KA34.
    • Verified on hardware: one successful same-version reinstall on each (KA18’s first attempt failed), and a reboot.
    • Lab only: services, ownership checks, and the Wi-Fi policy, in emulation.
    • Design: moving the Pi’s services onto the amplifiers.
  • Amplifier Wi-Fi fallback.
    • Verified on hardware: read-only inspection, which showed the radios exist and are switched off.
    • Lab only: the policy and its adapter, with a one-amp trial plan prepared.
    • Not done: any live network change.

What the amplifier trials do not show: survival of arbitrary future vendor updates, survival of a factory reset, or listening and operator-interface acceptance.

What I am leaving out, and why

I am not publishing credentials or key handling, network addresses, ports and serial numbers, the exact persistence mechanism, or vendor firmware internals and hashes. The point of this page is the method. The mechanism is security-sensitive and tied to vendor firmware.

Tools

QEMU, Linux userland tooling, Python, shell, systemd, Q-SYS Designer, and a lot of written evidence.

Glossary

  • Emulation: running a device’s real software inside a simulated machine, so a mistake cannot hurt the real device.
  • Firmware reinstall: replacing a device’s operating system image, here with the same vendor version.
  • Rollback: returning automatically to the last known good state.
  • Hash pin: fixing an input to a fingerprint, so the wrong file is refused.
  • Receipt: a redacted record of what ran, on which exact inputs, and what it showed.
  • Network manager: the program that decides which network a device uses.