post-TUI lag: two-phase autosuggestion toggle; Ctrl-b was never activated

Root cause of 'Ctrl-b does nothing': home-manager was not switched, so the
binding built into the flake was never deployed (~/.config/fish lacked
heal-autosuggest and \cb; the built config had both). Activation needs the
home rebuild, not a system rebuild.

heal-autosuggest is now two-phase (set 0; repaint; set 1; repaint) to give
the reader a disabled-state repaint, which is what clears the wedged
in_flight_autosuggest_request per reader.rs update_autosuggestion. Matches
the manual cure (which had prompt cycles between off and on); the old
back-to-back toggle and the postexec hook did not. User chose the
session-preserving toggle over exec fish.

docs/CHANGELOG.md

@@ -28,7 +28,54 @@
   - Configured `systemd.services.actual` to order after the decrypted secret service and granted the dynamic unit access via `SupplementaryGroups = [ "shared" ]` and `PrivateUsers = false`.
   - Updated `docs/oidc-services.md` with the verified callback URI (`https://money.masu.rs/openid/callback`) and NixOS configuration snippet.
 
-## 2026-08-29 (root cause found and fixed)
+## 2026-09-03: Ctrl-b did nothing because home-manager was never switched; two-phase toggle
+
+- "Ctrl-b does nothing" root cause: the binding was built into the flake but never activated. The deployed `~/.config/fish/functions/` had no `heal-autosuggest.fish` and no `\cb` binding (0 matches), while the freshly-built config had both. On this setup a system rebuild does not switch home-manager — activation needs the home rebuild (`rebuild-home` / the Alt-Shift-H binding, i.e. `home-manager switch --flake`) followed by a fresh pane so `config.fish` re-runs `fish_user_key_bindings`. No fix works until it is actually activated; this should be the FIRST check next time a "did nothing" is reported.
+- Made `heal-autosuggest` a two-phase toggle (chosen by the user over `exec fish`, to preserve the session): disable autosuggestions, `commandline -f repaint`, re-enable, `commandline -f repaint`. Grounded in `reader.rs` `update_autosuggestion`, which clears the wedged `in_flight_autosuggest_request` only on a repaint taken while autosuggestions are disabled. The old one-liner and the `fish_postexec` hook did a single back-to-back `set 0; set 1` with no disabled-state repaint, which likely never flushed the stuck request — matching the manual cure, which had prompt cycles between the off and the on. Still unverified against the real bug (not reproducible in a harness); confirm by activating, then pressing Ctrl-b in a live lagging shell.
+- If the two-phase toggle still does not cure once activated: fall back to `exec fish` on the key (guaranteed per the user's day-one report that a fresh shell always fixes it), and capture a flight-recorder log for the upstream fish report.
+
+## 2026-09-01: postexec hook fires but does not cure — moved heal to a keybinding
+
+- Honest status: the `fish_postexec` self-heal hook IS registered and DOES fire (verified in the real config), yet the lag persists. So toggling `fish_autosuggestion_enabled` off/on from `fish_postexec` does not cure it, even though the user typing the same `set … 0; set … 1` at the prompt does.
+- Why (code-level): variable dispatch is synchronous (`env_dispatch.rs` → `reader_set_autosuggestion_enabled` on every `set`), so the two sets net to no change and schedule a repaint. That repaint only has effect from inside the reader's active input loop. `fish_postexec` runs BETWEEN commands, outside that loop, so its effect is superseded before the next prompt. A key binding runs inside the loop; `fish_postexec` cannot.
+- Change: bound **Ctrl-b** to a new `heal-autosuggest` function (toggle + `commandline -f repaint`) via `fish_user_key_bindings` (Ctrl-g was already taken). Verified that when `fish_user_key_bindings` runs to completion — as it does in the real config, since the existing `\cn` etc. work (`__fish_config_interactive.fish:105`) — `\cb` binds to heal-autosuggest and the toggle resets the variable. NOT verified to cure the real lag: the bug still cannot be reproduced in a harness, so only a test in a live lagging shell can confirm. The `fish_postexec`/`fish_cancel` hook is kept (harmless) but is no longer considered the fix.
+- Guaranteed fallback if Ctrl-b does not cure: a fresh shell (`exec fish`), which the user has confirmed from the very start always fixes it — Ctrl-b can be rebound to that. The only path to a truly automatic fix is a flight-recorder capture (`~/.local/state/lag-triage/RECORD`) of the reader during an actual episode, then an upstream fish report.
+
+## 2026-08-31 (evening): self-heal hook was never firing — fish cannot autoload event handlers
+
+- The `__autosuggestion_unwedge` hook did not work because it was installed via `programs.fish.functions`, which writes to fish's **autoload** directory — and fish only registers `--on-event` handlers when a function is actually loaded, which never happens for a hook nothing calls by name. Verified in a PTY test: the autoloaded handler never fires; the identical definition `source`d eagerly fires immediately. (The zellij module's `__fish_update_cwd_osc` works as an autoloaded event function only because it overrides a function fish itself loads.)
+- Meanwhile the user confirmed the instant back-to-back toggle (`set -g fish_autosuggestion_enabled 0 && set -g fish_autosuggestion_enabled 1`) cures a lagging shell — so the handler body is right; only its registration was broken.
+- Fix: the handler is now defined eagerly in `config.fish` via `programs.fish.interactiveShellInit` (lag-triage module), registered on **fish_postexec** (fires after every command — the moment TUIs exit) and **fish_cancel** (fires on Ctrl-C at the prompt), so a bare Ctrl-C is an instant no-command cure. Verified in an interactive PTY against the actual nix-generated snippet: registers at startup, fires on both events, still respects a deliberate manual disable.
+- Coverage note: if a wedge forms with no command running (and no Ctrl-C), it heals at the next command; worst-case lag window is "until you run anything or press Ctrl-C".
+
+## 2026-08-31 (later): automatic self-heal hook
+
+- Confirmed by A/B in the live shell: after curing the lag with `set -g fish_autosuggestion_enabled 0`, re-enabling with `1` does **not** bring the lag back — the toggle resets the wedged autosuggestion state rather than merely masking it.
+- Added `__autosuggestion_unwedge` (lag-triage module): a `fish_postexec` event handler that toggles `fish_autosuggestion_enabled` off/on after every command — i.e. at the exact moment a TUI has just exited, when the wedge forms. Builtins only, no visible output (verified in an interactive PTY test), and it skips the reset when the user has deliberately disabled autosuggestions.
+- Honest caveat: the manual cure had keystrokes between the off and the on; whether the instant off/on inside an event handler resets the same reader-internal state is unproven. The flight recorder therefore STAYS ARMED (`~/.local/state/lag-triage/RECORD`) until the hook has survived normal use for a while. If lag recurs despite the hook: cure manually (`set … 0`, type a few chars, `set … 1`), and keep the flight log for that pid — then the hook needs the stronger form (disable at postexec, re-enable one prompt-cycle later, scoped to TUI commands).
+- Limitations by design: the hook fires only in shells that run commands, so a wedge formed without any command executing in that shell (if that is possible — e.g. floating-pane TUIs never touch the pane shell) would not be healed until the next command runs there.
+
+## 2026-08-31: culprit confirmed — fish's autosuggestion pipeline
+
+- A/B test in a live lagging shell (pid 56089): `set -g fish_autosuggestion_enabled 0` (builtin only, nothing else) **instantly cured the lag**. The post-TUI typing lag is in fish 4.8.1's autosuggestion pipeline.
+- Sampling that shell afterwards showed it had **only one thread** (the main thread): the poisoned state is main-thread-side bookkeeping, not a hung worker still sitting in the process. Source review (`src/threads/threads.rs`, `src/threads/debounce.rs`): `ThreadPool::perform` silently queues work with no spawn and no wake when it believes `total_threads == max_threads` — a leaked `total_threads` count (workers that died without decrementing, e.g. across a TUI's lifetime) would strand all future autosuggestion work forever; the Debounce then abandons its token every 500ms and re-enqueues per keystroke. The exact step that delays keystroke *echo* is still unproven — the flight recorder (armed via `~/.local/state/lag-triage/RECORD`) logs the reader's per-keystroke behavior and will capture it on the next occurrence in a recorded shell.
+- Precedent: fish had a closely-related bug class before (#11841 — unread terminal query responses "causing noticeable lags"). No fish release newer than 4.8.1 exists, so no upstream fix to adopt; an upstream report with the flight-recorder capture is the path to a real fix.
+- Practical interim cure (harmless, instant, in the lagging shell): `set -g fish_autosuggestion_enabled 0`, and re-enable with `1` — whether lag returns on re-enable is the next discriminating datum.
+
+## 2026-08-30 (later): sampler attach CURES the lag — wedged-thread evidence + flight recorder
+
+- Major new datum: in a lagging shell, running `mkdir` + `/usr/bin/sample $fish_pid … &` + `disown` **cured the lag instantly**, before any planned reset/toggle test could run. Plain external commands do NOT cure it (the 2026-08-29 triage ran many and the lag survived), so the distinguishing action is the sampler **attaching and suspending/resuming fish's threads**. Conclusion: a fish-internal thread/wait is wedged (missed wakeup or stuck blocking wait), and per-keystroke work at the main commandline stalls against it; suspension/resume kicks it loose. Consistent with: `read` prompts unaffected (no autosuggestion/highlight pipeline), subshells immune (fresh threads), raw input clean. The captured sample (`~/.local/state/lag-triage/fish-sample.txt`) shows only the post-cure state — sampling is a cure, not a capture.
+- Therefore the observer must be running BEFORE the lag starts: the `fish-no-query-term` wrapper is now a **flight recorder** — `touch ~/.local/state/lag-triage/RECORD`, then every newly spawned pane shell logs `FISH_DEBUG=reader,term-support,proc-termowner,iothread,fd-monitor,topic-monitor` to `~/.local/state/lag-triage/flight/fish-<ts>-<pid>.log` (3-day auto-cleanup; remove RECORD to disable, zero overhead when off). When lag next occurs, the log already contains what each keystroke did during the lag.
+- `lag-sample` now takes a PID and should be run from a DIFFERENT pane (`echo $fish_pid` — a builtin — in the lagging shell to get it), since attaching from inside cures the lag.
+- **Next-occurrence checklist (in order, least perturbing first):** (1) in the lagging shell, builtins only: `set -g fish_autosuggestion_enabled 0` → type at the real commandline; if cured, the autosuggestion/debounce path is implicated (a worker thread was seen in `HistorySearch::go_to_next_match`); (2) still laggy: `fish_default_key_bindings` → test (vi-mode path); (3) from another pane: `kill -WINCH <pid>` → test, then `kill -CONT <pid>` → test (discriminates reader-wakeup vs generic unwedge; if WINCH cures, a window resize would too); (4) from another pane: `lag-sample <pid>` while typing in the lagging pane; (5) immediately save the flight log for that pid.
+
+## 2026-08-30
+
+- **The post-TUI typing lag is NOT resolved** by the `fish-no-query-term` wrapper: lag recurred in a fresh zellij session after exiting Claude Code, in a shell verified (via `ps eww`) to have `fish_features=no-query-term` in its environment. The query-term reader-degradation bug proven on 2026-08-29 is real (and the wrapper stays as hardening against it), but it is not the mechanism behind this lag. Downgraded the entry below from "root cause" to "a root cause".
+- Known constraints on the real mechanism: per-keystroke lag at the main fish commandline; fish `read` prompts unaffected; raw input reaches the pane practical as plain bytes; a subshell/`exec fish` cures it (process-local state). Note the 2026-08-29 triage's reset ladder short-circuited on a false "y" at stage A, so stages B–G (mouse/keypad/altscreen/stty/DECSTR resets) were never actually tested against real lag.
+- Added `lag-sample` (fish function): stack-samples the lagging fish process plus the zellij server/client via `/usr/bin/sample` for 8s while the user types at the commandline. This directly names where the time goes (fish reader? highlighting/autosuggestion threads? zellij render loop?) instead of inferring it. Next occurrence: run `lag-sample` in the lagging shell, type junk at the prompt until done, then inspect `~/.local/state/lag-triage/sample-*.txt`. Follow with `unlag` (full reset ladder, never yet truly tested), then A/B toggles: `set -g fish_autosuggestion_enabled 0`, `fish_default_key_bindings`.
+
+## 2026-08-29 (a root cause found and fixed — but not THE lag)
 
 - **Root-caused and fixed the recurring post-TUI typing lag** (fish + Zellij + Ghostty) using a `lag-triage` capture from a live lagging shell plus a deterministic PTY reproduction (`presets/programs/lag-triage/upstream_repro.py`):
   - **Root cause chain**: (1) fish latches feature flags from its **startup environment**, before `config.fish` runs — so the existing `set -gx fish_features no-query-term` in `shellInit` never applied to the shell that set it, only to its children. (2) Zellij spawns pane shells via `default_shell` with no `fish_features` in the environment, so every pane's fish latched `query-term` **on** (the fish 4.8.1 default; the triage log from the lagging shell confirmed `query-term on` while `$fish_features` was correctly set to `no-query-term`). (3) With query-term on, fish sends OSC 11 + CPR (`\e[6n`) + DA1 (`\e[0c`) after **every** command and waits for replies relayed by Zellij. (4) Reproduced on fish 4.8.1: if the terminal fails to reply during just **one** such cycle — answering everything before and after — that fish process's interactive reader is **permanently degraded** (keystroke echo >3s, never recovers; ~35ms before). In production Zellij drops/mangles a relay during TUI teardown or heavy output (cf. zellij-org/zellij#5158), e.g. after `nh home switch`, nvim, jjui, yazi.

platforms/home-manager/modules/nmasur/presets/programs/lag-triage/default.nix

@@ -23,6 +23,24 @@ in
 
     home.packages = [ term-probe ];
 
+    # Ctrl-b: EXECUTE the proven manual cure as a real commandline. The cure
+    # is not the variable's end value (it starts and ends at 1) — it is the
+    # reader fully exiting readline and re-entering, which only command
+    # EXECUTION does. Setting the variable inline (a plain binding body, or the
+    # old fish_postexec/fish_cancel hook) never makes the reader exit/re-enter,
+    # so it never cured and, with extra repaints on a wedged reader, made it
+    # worse. `commandline -f execute` reproduces exactly what typing the cure
+    # and pressing Enter does — indistinguishable to fish from the manual cure.
+    # Note: this submits the current commandline, so it runs the cure in place
+    # of whatever is typed (fine for a rescue key hit at an empty prompt).
+    # Ctrl-b chosen because it is otherwise unbound (Ctrl-g is taken).
+    nmasur.presets.programs.fish.fish_user_key_bindings = # fish
+      ''
+        for mode in insert default visual
+            bind -M $mode \cb heal-autosuggest
+        end
+      '';
+
     programs.fish.functions = {
       lag-triage = {
         description = "Diagnose post-TUI typing lag in the current shell";
@@ -32,6 +50,22 @@ in
         description = "Reset terminal state left behind by a TUI";
         body = builtins.readFile ./unlag.fish;
       };
+      lag-sample = {
+        description = "Stack-sample fish and zellij while typing lag is happening";
+        body = builtins.readFile ./lag-sample.fish;
+      };
+      heal-autosuggest = {
+        description = "Heal post-TUI typing lag by executing the autosuggestion-toggle cure (bind to a key)";
+        # Replace the commandline with the exact cure the user runs by hand and
+        # execute it. Executing (not inline-setting) is what cures: it forces
+        # the reader to leave and re-enter readline. Runs in place of whatever
+        # is currently typed.
+        body = # fish
+          ''
+            commandline -r 'set -g fish_autosuggestion_enabled 0; and set -g fish_autosuggestion_enabled 1'
+            commandline -f execute
+          '';
+      };
     };
 
   };

platforms/home-manager/modules/nmasur/presets/programs/lag-triage/lag-sample.fish

@@ -0,0 +1,53 @@
+# Capture stack samples of this fish process, the zellij server, and the
+# zellij client WHILE the typing lag is happening. This names the guilty
+# component directly: if fish's main thread is busy/blocked per keystroke the
+# stacks show exactly where; if fish is idle while typing feels laggy, the
+# delay is in zellij's render path instead.
+#
+# CAUTION (learned 2026-08-30): attaching the sampler to a lagging fish CURES
+# the lag (thread suspend/resume unwedges it), so run this from a DIFFERENT
+# pane with the lagging shell's pid: `lag-sample <pid>` (get it in the lagging
+# shell with the builtin-only `echo $fish_pid`). Have someone type in the
+# lagging pane while sampling runs — the first samples may catch the wedge.
+# With no argument it samples the current shell.
+
+set -l target $fish_pid
+if test (count $argv) -ge 1; and test -n "$argv[1]"
+    set target $argv[1]
+end
+
+set -l outdir ~/.local/state/lag-triage
+mkdir -p $outdir
+set -l ts (date +%Y%m%d-%H%M%S)
+set -l dur 8
+
+set -l fishfile $outdir/sample-$ts-fish-$target.txt
+/usr/bin/sample $target $dur 1 -file $fishfile &>/dev/null &
+disown
+
+# this session's zellij server (socket path ends in the session name)
+set -l serverpid (pgrep -f "zellij --server.*/$ZELLIJ_SESSION_NAME\$")
+test -z "$serverpid"; and set serverpid (pgrep -f "zellij --server" | head -3)
+for pid in $serverpid
+    /usr/bin/sample $pid $dur 1 -file $outdir/sample-$ts-zellij-server-$pid.txt &>/dev/null &
+    disown
+end
+
+# zellij clients (attached to ghostty): named zellij but without --server args
+set -l allserver (pgrep -f "zellij --server")
+set -l clientpid
+for pid in (pgrep -x zellij)
+    contains $pid $allserver; or set -a clientpid $pid
+end
+for pid in $clientpid[1..3]
+    /usr/bin/sample $pid $dur 1 -file $outdir/sample-$ts-zellij-client-$pid.txt &>/dev/null &
+    disown
+end
+
+# notify when done, without occupying the commandline
+fish -c "sleep (math $dur + 2); echo; echo '== lag-sample done: '$outdir'/sample-$ts-*.txt =='" &
+disown
+
+echo "Sampling fish (pid $target), zellij server(s) [$serverpid], client(s) [$clientpid] for $dur s."
+echo ">>> TYPE CONTINUOUSLY IN THE LAGGING PANE NOW (junk text is fine) <<<"
+echo "Files: $outdir/sample-$ts-*.txt"

platforms/home-manager/modules/nmasur/presets/programs/zellij.nix

@@ -20,8 +20,21 @@ let
   # harness on fish 4.8.1 (see docs/CHANGELOG.md 2026-08-29 and
   # presets/programs/lag-triage/upstream_repro.py). Spawning fish with the
   # variable already exported makes every pane shell immune.
+  # Flight recorder for the still-unsolved post-TUI typing lag: sampling the
+  # process CURES the lag (a wedged thread gets kicked loose), so the only way
+  # to observe it is a recorder that is already running before the lag starts.
+  # Armed by `touch ~/.local/state/lag-triage/RECORD`; new panes then log
+  # fish's reader/thread internals to ~/.local/state/lag-triage/flight/.
+  # Remove the RECORD file to disable (zero overhead when off).
   fish-no-query-term = pkgs.writeShellScriptBin "fish-no-query-term" ''
     export fish_features=no-query-term
+    dir="$HOME/.local/state/lag-triage"
+    if [ -e "$dir/RECORD" ]; then
+      mkdir -p "$dir/flight"
+      find "$dir/flight" -type f -mtime +3 -delete 2>/dev/null
+      export FISH_DEBUG='reader,term-support,proc-termowner,iothread,fd-monitor,topic-monitor'
+      export FISH_DEBUG_OUTPUT="$dir/flight/fish-$(date +%Y%m%d-%H%M%S)-$$.log"
+    fi
     exec ${lib.getExe pkgs.fish} "$@"
   '';