Saturday, 10:42: the heart stops
Nobody heard it die. There was no smoke, no beep, no last words on a screen nobody was watching. Somewhere inside an HP EliteDesk in the corner of my lab, a small GIGABYTE SSD that had been pushing data for 45,000 hours straight (five years, no vacations, no complaints) quietly decided it had seen enough of this world, and checked out.
And here’s the obscene part: everything kept running.
That HP is my Proxmox host. The landlord, the hypervisor, the body. Its only tenant is a virtual machine called ANXEL, and ANXEL is the beating center of everything I build. Every Android app I’ve shipped or have on the bench lives there: Bukin Mic Recorder (live on Google Play), Steps GPS Route, the Magnifier, USB Camera Lab. The emulators. The adb cables reaching out to my test phones. The signing keys. And seventeen Claude sessions, each one minding its own app, like doctors in a hospital that never closes. If ANXEL goes down, the whole workshop goes dark.
The SSD that died was the host’s system disk. ANXEL’s own disk lived on a separate 2 TB SSD, safe and sound. So picture this: a man whose heart has stopped, still walking around, still talking, running on momentum and whatever blood is left in his veins. Linux keeps the programs it has already used in RAM, so the host went on living out of memory, while every new read from the dead disk came back with DID_BAD_TARGET, which is kernel for “there is nobody home”. The web console was dead. Half the commands answered “Input/output error”. And inside this walking corpse, ANXEL kept compiling, blissful and oblivious, like a passenger asleep in a car with no driver.
It took almost twelve hours before anyone noticed.
Saturday night: do not touch anything
The first rule of the situation was written in capital letters: DO NOT REBOOT. Not the host, not ANXEL. RAM was the only place the old host still existed. One restart and it would never come back, and ANXEL would go down with the ship.
So before any heroics, we built lifeboats. That night, 37 gigabytes of ANXEL’s most precious cargo (keys, secrets, source code, Claude’s memory) went across the wire to my NAS. Only then could anyone think straight.
The textbook says: plug in a monitor, boot the installer, reinstall Proxmox, rebuild everything by hand. An hour of downtime if the gods are kind, a weekend if they’re not. But my server has no monitor, no keyboard, no mouse. It lives in a corner, and I talk to it through a web page, like a sinner talking to a priest through the screen of a confessional. And the web page was dead.
Then came the crazy idea. We called it “the live route”. I call it brain surgery with the patient awake. Surgeons keep you awake during some brain operations so they can watch whether they’re cutting something important. Same here: ANXEL stays awake and working, and the new heart for its host gets built from the inside, by the guest itself.
Sunday: the donors
On Sunday I opened the case of a running server, which is the kind of sentence that makes sysadmins cross themselves. I pulled the dead GIGABYTE out and hot-plugged spare SSDs from my drawer into the open chest of a machine that could not be stopped, one at a time, while Claude read the vital signs from the inside.
Donor number two was brain-dead on arrival. It introduced itself as “Sandisk Milpitas SSD”, firmware 0.00, capacity 16 kilobytes: a controller stuck in recovery mode, the silicon version of a thousand-yard stare.
Donor number one was a SanDisk Ultra Plus born around 2013. 14,069 hours on the clock, 95% of its life still ahead of it, a clean bill of health. A pensioner with the heart of a marathon runner. We took it.
Now the real madness. The host couldn’t run its own tools anymore, because they lived on the dead disk. So Claude smuggled them in, copying programs from ANXEL straight into the host’s RAM and running them from there, like passing scalpels through a keyhole. Then, inside ANXEL, it built a complete Proxmox 9.2 system from scratch: same name, same IP address, same root password, same SSH keys, same web certificate, and ANXEL set to start on its own. The guest was building a new body for its own host.
At 20:38 the new system went down the wire onto the SanDisk. Three minutes.
And then the host went silent.
Writing those gigabytes pushed the last cached copy of the SSH login program out of the host’s memory, and there was no disk left to load it back from. The host still answered ping. It just couldn’t let anyone in. Locked-in syndrome: awake, breathing, unable to speak. We were past the point of no return, and the real operation hadn’t even started.
Sunday night: a million tests
What Claude did next, I can only describe as paranoia raised to an art form. It built a virtual twin: a copy of the new disk, booted inside a virtual machine inside ANXEL, with fake disks wearing the real serial numbers and a fake network card wearing the real MAC address. A virtual machine running a copy of its own host. I’m still dizzy.
The twin found two bugs. A leftover network file from the build would have quietly replaced the network settings at first boot: no IP, no web page, no way in. And the settings restore would have given up waiting before it even started. Both bugs sat on a disk that nobody could touch anymore.
That’s when the fear arrived properly. I typed: “are you sure we are gonna be back up again in minutes? i am afraid that it’s gonna be bad…. did you double check everything?”
Claude’s answer was more tests. And when I explained that there’s no monitor on that box, that the login page simply had to appear, smooth as silk, with ANXEL starting on its own, I asked the question that turned out to be the key: “can you maybe configure a bootstrap or some sort of startup script that will do the necessary?”
The robot surgeon
What came out of that question was a robot surgeon on a USB stick: a tiny Debian that boots, finds the SanDisk by its serial number, checks the file system, checks every installed file against its checksum, fixes the two bugs, writes the new heart into the firmware’s boot list and restarts. And if anything at all looks wrong (the wrong disk, damaged files, anything) it doesn’t guess. It switches the server off. A surgeon with the good sense to put the scalpel down.
The twin ran it, rebooted itself into Proxmox, restored my settings and reached for ANXEL. Passed.
“It’s in,” I announced proudly when the stick was back in the server. Claude asked how I had flashed it. I hadn’t. I assumed the machine would do that. The machine, it turns out, has no hands on my side of the network. One balenaEtcher later: Flash Completed, verified byte for byte.
Monday, 01:19: pulling my own plug
Here’s the detail that still gets me. Claude lives inside ANXEL. To finish the operation, the surgeon had to put himself under. I typed my password, ANXEL shut down, and the doctor walked out of the operating room, leaving me alone with a dark box, a blinking light and my imagination, which is never a safe place to be at one in the morning.
Two minutes of waiting, to let the last bytes settle. Then I held the power button until the HP died, and pressed it again.
Then ten minutes of nothing. No screen, no sound, no news. This is the part where I aged.
Later, from the logs, we could reconstruct what the robot surgeon did in the dark:
01:23:15 pve-fixer: start
01:23:16 pve-fixer: e2fsck rc=0
01:23:44 pve-fixer: dpkg --verify: 2 changed files, 0 non-config files with wrong checksums
01:23:45 pve-fixer: DONE - rebooting into Proxmox on the SanDiskZero damaged files. By 01:24 Proxmox was up on its new heart, with the same IP, the same keys, the same certificate and all my settings. ANXEL started on its own.
The login page came up. VM 100 was green. I took a deep, happy breath, the first real one in more than a day. ANXEL had been down for about six minutes.
The bureaucrat in the firmware
Every good story needs a twist, and this one came from HP’s firmware, the kind of bureaucrat that loses your file right after the surgery. After booting Proxmox exactly once, it quietly crossed it off its boot list. The next restart would have booted the USB stick, and the stick, paranoid as designed, would have switched the server off.
So the robot surgeon learned a new trick. If it ever wakes up and finds the patient already healed, it puts Proxmox back on top of the list and restarts into it. If that happens three times in half an hour, it decides something is deeply wrong and powers down instead of looping forever. Tested in the twin, installed on the stick, and the stick now lives in the server for good, like a defibrillator on the wall.
At 01:42 we restarted the server one last time, just to see. I didn’t touch it. BootCurrent: 0000. Proxmox, first try. ANXEL back in two minutes.
Coming down
I’ve been doing this long enough to know what’s normal. Moving a running virtual machine to another host? Relatively easy; people do it every day. That’s not what we did. We rebuilt the host itself, while its only guest was still running on it, and while the host was running on nothing but memory because its disk was dead. The surgeon lived inside the patient, built the new heart from in there, and passed it out through a keyhole. It’s the craziest idea in the book. We did it anyway, and the patient never really went to sleep.
Let me be honest about the credit, in the gonzo tradition: I held the screwdriver, swapped the disks, flashed the stick, typed the passwords and did most of the sweating. Claude did the impossible part. Most of it, anyway.
Lessons, for whatever they’re worth at two in the morning: back up before you get brave. Test like a paranoid, then test the tests. Never trust a firmware. And if your surgeon ever tells you he wants to operate while you’re awake, trust him, but ask how many times he practiced on your twin.