post-TUI lag: autosuggestion culprit confirmed; add self-heal hook

A/B in a live lagging shell: disabling fish_autosuggestion_enabled cures
the lag instantly, and re-enabling does NOT bring it back — the toggle
resets the wedged reader state. __autosuggestion_unwedge (fish_postexec)
now applies that reset after every command, at the moment TUIs exit.
Builtins only, invisible, respects a deliberate manual disable. Flight
recorder stays armed until the hook is proven in real use.

Also from this investigation: wedged-thread evidence (sampler attach
cures), lag-sample tool, flight recorder in the zellij fish wrapper.
This commit is contained in:
Noah Masur
2026-09-07 11:17:44 -04:00
parent db49e746b8
commit d4e56dd190
5 changed files with 124 additions and 1 deletions
+28 -1
View File
@@ -28,7 +28,34 @@
- Configured `systemd.services.actual` to order after the decrypted secret service and granted the dynamic unit access via `SupplementaryGroups = [ "shared" ]` and `PrivateUsers = false`. - Configured `systemd.services.actual` to order after the decrypted secret service and granted the dynamic unit access via `SupplementaryGroups = [ "shared" ]` and `PrivateUsers = false`.
- Updated `docs/oidc-services.md` with the verified callback URI (`https://money.masu.rs/openid/callback`) and NixOS configuration snippet. - Updated `docs/oidc-services.md` with the verified callback URI (`https://money.masu.rs/openid/callback`) and NixOS configuration snippet.
## 2026-08-29 (root cause found and fixed) ## 2026-08-31 (later): automatic self-heal hook
- Confirmed by A/B in the live shell: after curing the lag with `set -g fish_autosuggestion_enabled 0`, re-enabling with `1` does **not** bring the lag back — the toggle resets the wedged autosuggestion state rather than merely masking it.
- Added `__autosuggestion_unwedge` (lag-triage module): a `fish_postexec` event handler that toggles `fish_autosuggestion_enabled` off/on after every command — i.e. at the exact moment a TUI has just exited, when the wedge forms. Builtins only, no visible output (verified in an interactive PTY test), and it skips the reset when the user has deliberately disabled autosuggestions.
- Honest caveat: the manual cure had keystrokes between the off and the on; whether the instant off/on inside an event handler resets the same reader-internal state is unproven. The flight recorder therefore STAYS ARMED (`~/.local/state/lag-triage/RECORD`) until the hook has survived normal use for a while. If lag recurs despite the hook: cure manually (`set … 0`, type a few chars, `set … 1`), and keep the flight log for that pid — then the hook needs the stronger form (disable at postexec, re-enable one prompt-cycle later, scoped to TUI commands).
- Limitations by design: the hook fires only in shells that run commands, so a wedge formed without any command executing in that shell (if that is possible — e.g. floating-pane TUIs never touch the pane shell) would not be healed until the next command runs there.
## 2026-08-31: culprit confirmed — fish's autosuggestion pipeline
- A/B test in a live lagging shell (pid 56089): `set -g fish_autosuggestion_enabled 0` (builtin only, nothing else) **instantly cured the lag**. The post-TUI typing lag is in fish 4.8.1's autosuggestion pipeline.
- Sampling that shell afterwards showed it had **only one thread** (the main thread): the poisoned state is main-thread-side bookkeeping, not a hung worker still sitting in the process. Source review (`src/threads/threads.rs`, `src/threads/debounce.rs`): `ThreadPool::perform` silently queues work with no spawn and no wake when it believes `total_threads == max_threads` — a leaked `total_threads` count (workers that died without decrementing, e.g. across a TUI's lifetime) would strand all future autosuggestion work forever; the Debounce then abandons its token every 500ms and re-enqueues per keystroke. The exact step that delays keystroke *echo* is still unproven — the flight recorder (armed via `~/.local/state/lag-triage/RECORD`) logs the reader's per-keystroke behavior and will capture it on the next occurrence in a recorded shell.
- Precedent: fish had a closely-related bug class before (#11841 — unread terminal query responses "causing noticeable lags"). No fish release newer than 4.8.1 exists, so no upstream fix to adopt; an upstream report with the flight-recorder capture is the path to a real fix.
- Practical interim cure (harmless, instant, in the lagging shell): `set -g fish_autosuggestion_enabled 0`, and re-enable with `1` — whether lag returns on re-enable is the next discriminating datum.
## 2026-08-30 (later): sampler attach CURES the lag — wedged-thread evidence + flight recorder
- Major new datum: in a lagging shell, running `mkdir` + `/usr/bin/sample $fish_pid … &` + `disown` **cured the lag instantly**, before any planned reset/toggle test could run. Plain external commands do NOT cure it (the 2026-08-29 triage ran many and the lag survived), so the distinguishing action is the sampler **attaching and suspending/resuming fish's threads**. Conclusion: a fish-internal thread/wait is wedged (missed wakeup or stuck blocking wait), and per-keystroke work at the main commandline stalls against it; suspension/resume kicks it loose. Consistent with: `read` prompts unaffected (no autosuggestion/highlight pipeline), subshells immune (fresh threads), raw input clean. The captured sample (`~/.local/state/lag-triage/fish-sample.txt`) shows only the post-cure state — sampling is a cure, not a capture.
- Therefore the observer must be running BEFORE the lag starts: the `fish-no-query-term` wrapper is now a **flight recorder**`touch ~/.local/state/lag-triage/RECORD`, then every newly spawned pane shell logs `FISH_DEBUG=reader,term-support,proc-termowner,iothread,fd-monitor,topic-monitor` to `~/.local/state/lag-triage/flight/fish-<ts>-<pid>.log` (3-day auto-cleanup; remove RECORD to disable, zero overhead when off). When lag next occurs, the log already contains what each keystroke did during the lag.
- `lag-sample` now takes a PID and should be run from a DIFFERENT pane (`echo $fish_pid` — a builtin — in the lagging shell to get it), since attaching from inside cures the lag.
- **Next-occurrence checklist (in order, least perturbing first):** (1) in the lagging shell, builtins only: `set -g fish_autosuggestion_enabled 0` → type at the real commandline; if cured, the autosuggestion/debounce path is implicated (a worker thread was seen in `HistorySearch::go_to_next_match`); (2) still laggy: `fish_default_key_bindings` → test (vi-mode path); (3) from another pane: `kill -WINCH <pid>` → test, then `kill -CONT <pid>` → test (discriminates reader-wakeup vs generic unwedge; if WINCH cures, a window resize would too); (4) from another pane: `lag-sample <pid>` while typing in the lagging pane; (5) immediately save the flight log for that pid.
## 2026-08-30
- **The post-TUI typing lag is NOT resolved** by the `fish-no-query-term` wrapper: lag recurred in a fresh zellij session after exiting Claude Code, in a shell verified (via `ps eww`) to have `fish_features=no-query-term` in its environment. The query-term reader-degradation bug proven on 2026-08-29 is real (and the wrapper stays as hardening against it), but it is not the mechanism behind this lag. Downgraded the entry below from "root cause" to "a root cause".
- Known constraints on the real mechanism: per-keystroke lag at the main fish commandline; fish `read` prompts unaffected; raw input reaches the pane practical as plain bytes; a subshell/`exec fish` cures it (process-local state). Note the 2026-08-29 triage's reset ladder short-circuited on a false "y" at stage A, so stages BG (mouse/keypad/altscreen/stty/DECSTR resets) were never actually tested against real lag.
- Added `lag-sample` (fish function): stack-samples the lagging fish process plus the zellij server/client via `/usr/bin/sample` for 8s while the user types at the commandline. This directly names where the time goes (fish reader? highlighting/autosuggestion threads? zellij render loop?) instead of inferring it. Next occurrence: run `lag-sample` in the lagging shell, type junk at the prompt until done, then inspect `~/.local/state/lag-triage/sample-*.txt`. Follow with `unlag` (full reset ladder, never yet truly tested), then A/B toggles: `set -g fish_autosuggestion_enabled 0`, `fish_default_key_bindings`.
## 2026-08-29 (a root cause found and fixed — but not THE lag)
- **Root-caused and fixed the recurring post-TUI typing lag** (fish + Zellij + Ghostty) using a `lag-triage` capture from a live lagging shell plus a deterministic PTY reproduction (`presets/programs/lag-triage/upstream_repro.py`): - **Root-caused and fixed the recurring post-TUI typing lag** (fish + Zellij + Ghostty) using a `lag-triage` capture from a live lagging shell plus a deterministic PTY reproduction (`presets/programs/lag-triage/upstream_repro.py`):
- **Root cause chain**: (1) fish latches feature flags from its **startup environment**, before `config.fish` runs — so the existing `set -gx fish_features no-query-term` in `shellInit` never applied to the shell that set it, only to its children. (2) Zellij spawns pane shells via `default_shell` with no `fish_features` in the environment, so every pane's fish latched `query-term` **on** (the fish 4.8.1 default; the triage log from the lagging shell confirmed `query-term on` while `$fish_features` was correctly set to `no-query-term`). (3) With query-term on, fish sends OSC 11 + CPR (`\e[6n`) + DA1 (`\e[0c`) after **every** command and waits for replies relayed by Zellij. (4) Reproduced on fish 4.8.1: if the terminal fails to reply during just **one** such cycle — answering everything before and after — that fish process's interactive reader is **permanently degraded** (keystroke echo >3s, never recovers; ~35ms before). In production Zellij drops/mangles a relay during TUI teardown or heavy output (cf. zellij-org/zellij#5158), e.g. after `nh home switch`, nvim, jjui, yazi. - **Root cause chain**: (1) fish latches feature flags from its **startup environment**, before `config.fish` runs — so the existing `set -gx fish_features no-query-term` in `shellInit` never applied to the shell that set it, only to its children. (2) Zellij spawns pane shells via `default_shell` with no `fish_features` in the environment, so every pane's fish latched `query-term` **on** (the fish 4.8.1 default; the triage log from the lagging shell confirmed `query-term on` while `$fish_features` was correctly set to `no-query-term`). (3) With query-term on, fish sends OSC 11 + CPR (`\e[6n`) + DA1 (`\e[0c`) after **every** command and waits for replies relayed by Zellij. (4) Reproduced on fish 4.8.1: if the terminal fails to reply during just **one** such cycle — answering everything before and after — that fish process's interactive reader is **permanently degraded** (keystroke echo >3s, never recovers; ~35ms before). In production Zellij drops/mangles a relay during TUI teardown or heavy output (cf. zellij-org/zellij#5158), e.g. after `nh home switch`, nvim, jjui, yazi.
@@ -0,0 +1,21 @@
# Self-heal for the post-TUI typing lag (fish 4.8.1, see docs/CHANGELOG.md
# 2026-08-29..31): exiting a TUI can wedge fish's autosuggestion pipeline,
# after which every keystroke at the commandline lags until the process is
# replaced. Empirically validated cure: turn autosuggestions off and back on
# in the affected shell. This hook applies that reset after every command —
# i.e. at the exact moment a TUI has just exited — using only builtins, so it
# is effectively free and invisible.
#
# Caveats, recorded for honesty: the manual cure had keystrokes between the
# off and the on; whether an immediate off/on inside an event handler resets
# the same reader state is unproven (the flight recorder stays armed to catch
# any recurrence). If lag ever appears despite this hook, cure manually with
# set -g fish_autosuggestion_enabled 0 (type a few chars)
# set -g fish_autosuggestion_enabled 1
# and save ~/.local/state/lag-triage/flight/ logs for that shell's pid.
# Respect a deliberate user choice to keep autosuggestions off.
if test "$fish_autosuggestion_enabled" != 0
set -g fish_autosuggestion_enabled 0
set -g fish_autosuggestion_enabled 1
end
@@ -32,6 +32,15 @@ in
description = "Reset terminal state left behind by a TUI"; description = "Reset terminal state left behind by a TUI";
body = builtins.readFile ./unlag.fish; body = builtins.readFile ./unlag.fish;
}; };
lag-sample = {
description = "Stack-sample fish and zellij while typing lag is happening";
body = builtins.readFile ./lag-sample.fish;
};
__autosuggestion_unwedge = {
description = "Reset autosuggestion state after each command to prevent post-TUI typing lag";
onEvent = "fish_postexec";
body = builtins.readFile ./autosuggestion-unwedge.fish;
};
}; };
}; };
@@ -0,0 +1,53 @@
# Capture stack samples of this fish process, the zellij server, and the
# zellij client WHILE the typing lag is happening. This names the guilty
# component directly: if fish's main thread is busy/blocked per keystroke the
# stacks show exactly where; if fish is idle while typing feels laggy, the
# delay is in zellij's render path instead.
#
# CAUTION (learned 2026-08-30): attaching the sampler to a lagging fish CURES
# the lag (thread suspend/resume unwedges it), so run this from a DIFFERENT
# pane with the lagging shell's pid: `lag-sample <pid>` (get it in the lagging
# shell with the builtin-only `echo $fish_pid`). Have someone type in the
# lagging pane while sampling runs — the first samples may catch the wedge.
# With no argument it samples the current shell.
set -l target $fish_pid
if test (count $argv) -ge 1; and test -n "$argv[1]"
set target $argv[1]
end
set -l outdir ~/.local/state/lag-triage
mkdir -p $outdir
set -l ts (date +%Y%m%d-%H%M%S)
set -l dur 8
set -l fishfile $outdir/sample-$ts-fish-$target.txt
/usr/bin/sample $target $dur 1 -file $fishfile &>/dev/null &
disown
# this session's zellij server (socket path ends in the session name)
set -l serverpid (pgrep -f "zellij --server.*/$ZELLIJ_SESSION_NAME\$")
test -z "$serverpid"; and set serverpid (pgrep -f "zellij --server" | head -3)
for pid in $serverpid
/usr/bin/sample $pid $dur 1 -file $outdir/sample-$ts-zellij-server-$pid.txt &>/dev/null &
disown
end
# zellij clients (attached to ghostty): named zellij but without --server args
set -l allserver (pgrep -f "zellij --server")
set -l clientpid
for pid in (pgrep -x zellij)
contains $pid $allserver; or set -a clientpid $pid
end
for pid in $clientpid[1..3]
/usr/bin/sample $pid $dur 1 -file $outdir/sample-$ts-zellij-client-$pid.txt &>/dev/null &
disown
end
# notify when done, without occupying the commandline
fish -c "sleep (math $dur + 2); echo; echo '== lag-sample done: '$outdir'/sample-$ts-*.txt =='" &
disown
echo "Sampling fish (pid $target), zellij server(s) [$serverpid], client(s) [$clientpid] for $dur s."
echo ">>> TYPE CONTINUOUSLY IN THE LAGGING PANE NOW (junk text is fine) <<<"
echo "Files: $outdir/sample-$ts-*.txt"
@@ -20,8 +20,21 @@ let
# harness on fish 4.8.1 (see docs/CHANGELOG.md 2026-08-29 and # harness on fish 4.8.1 (see docs/CHANGELOG.md 2026-08-29 and
# presets/programs/lag-triage/upstream_repro.py). Spawning fish with the # presets/programs/lag-triage/upstream_repro.py). Spawning fish with the
# variable already exported makes every pane shell immune. # variable already exported makes every pane shell immune.
# Flight recorder for the still-unsolved post-TUI typing lag: sampling the
# process CURES the lag (a wedged thread gets kicked loose), so the only way
# to observe it is a recorder that is already running before the lag starts.
# Armed by `touch ~/.local/state/lag-triage/RECORD`; new panes then log
# fish's reader/thread internals to ~/.local/state/lag-triage/flight/.
# Remove the RECORD file to disable (zero overhead when off).
fish-no-query-term = pkgs.writeShellScriptBin "fish-no-query-term" '' fish-no-query-term = pkgs.writeShellScriptBin "fish-no-query-term" ''
export fish_features=no-query-term export fish_features=no-query-term
dir="$HOME/.local/state/lag-triage"
if [ -e "$dir/RECORD" ]; then
mkdir -p "$dir/flight"
find "$dir/flight" -type f -mtime +3 -delete 2>/dev/null
export FISH_DEBUG='reader,term-support,proc-termowner,iothread,fd-monitor,topic-monitor'
export FISH_DEBUG_OUTPUT="$dir/flight/fish-$(date +%Y%m%d-%H%M%S)-$$.log"
fi
exec ${lib.getExe pkgs.fish} "$@" exec ${lib.getExe pkgs.fish} "$@"
''; '';