diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a973365b..e42d1129 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## 2026-08-29 (root cause found and fixed) + +- **Root-caused and fixed the recurring post-TUI typing lag** (fish + Zellij + Ghostty) using a `lag-triage` capture from a live lagging shell plus a deterministic PTY reproduction (`presets/programs/lag-triage/upstream_repro.py`): + - **Root cause chain**: (1) fish latches feature flags from its **startup environment**, before `config.fish` runs — so the existing `set -gx fish_features no-query-term` in `shellInit` never applied to the shell that set it, only to its children. (2) Zellij spawns pane shells via `default_shell` with no `fish_features` in the environment, so every pane's fish latched `query-term` **on** (the fish 4.8.1 default; the triage log from the lagging shell confirmed `query-term on` while `$fish_features` was correctly set to `no-query-term`). (3) With query-term on, fish sends OSC 11 + CPR (`\e[6n`) + DA1 (`\e[0c`) after **every** command and waits for replies relayed by Zellij. (4) Reproduced on fish 4.8.1: if the terminal fails to reply during just **one** such cycle — answering everything before and after — that fish process's interactive reader is **permanently degraded** (keystroke echo >3s, never recovers; ~35ms before). In production Zellij drops/mangles a relay during TUI teardown or heavy output (cf. zellij-org/zellij#5158), e.g. after `nh home switch`, nvim, jjui, yazi. + - **Why every previous observation finally makes sense**: subshells and `exec fish` were never "resetting" anything — they *inherited* the exported `fish_features=no-query-term` from config.fish, latched query-term off at startup, and were therefore **immune**. The parent zellij-spawned shell never had the variable at startup and stayed vulnerable. Raw keystroke capture in the lagging pane showed instant plain bytes (input path fine) and no stuck terminal modes — the damage was inside the fish process, exactly as the repro shows. + - **Fix**: `zellij.nix` now spawns panes through a `fish-no-query-term` wrapper (`export fish_features=no-query-term; exec fish`), so the feature is latched off in every pane shell. Verified: interactive fish through the built wrapper with the real config reports `query-term off`; the PTY repro with `no-query-term` in the environment shows ~35ms echo through all failure phases. + - **Correction** to the earlier 2026-08-29 entry: `query-term` does **not** default to off in fish 4.8.1 — it defaults on; it only *appeared* off in non-interactive checks because the user config's `set -gx` takes effect for `fish -c` (no reader latch) but not for interactive shells. + - Upstream: fish-shell should bound the reader's wait for query replies instead of degrading permanently (repro script kept at `presets/programs/lag-triage/upstream_repro.py` for filing); Zellij's reply relaying is the trigger (zellij-org/zellij#5158). + - `lag-triage` now checks `status features` and calls out `query-term on` as the known root cause, and warns that its `read`-prompt typing tests may not exhibit main-commandline lag (which produced a false "fixed by stage A" in the first capture). + ## 2026-08-29 (later) - Added a diagnostic toolkit (`lag-triage` / `unlag` fish functions + `term-probe` binary, `presets/programs/lag-triage/`) for the still-recurring post-TUI typing lag in fish + Zellij + Ghostty, instead of another blind fix. Findings that motivated it: diff --git a/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/lag-triage.fish b/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/lag-triage.fish index 60f12ce6..63516e43 100644 --- a/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/lag-triage.fish +++ b/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/lag-triage.fish @@ -45,6 +45,23 @@ begin end >>$logfile 2>&1 _lt "Captured shell + environment snapshot." +# Proven root cause of the 2026-08 lag (see docs/CHANGELOG.md 2026-08-29): +# fish latches feature flags from its startup env before config.fish runs, so +# a shell with query-term ON sends terminal queries after every command; one +# reply zellij fails to relay permanently degrades this process's reader. +if status features | string match -qr '^query-term\s+on' + _lt "" + _lt "!! query-term is ON in this shell: fish did NOT get fish_features=" + _lt "!! no-query-term in its STARTUP environment (config.fish is too late)." + _lt "!! This is the proven root cause of the post-TUI lag — a query reply" + _lt "!! lost by zellij permanently degrades this fish process's reader." + _lt "!! Fix: spawn fish with the variable exported (zellij default_shell" + _lt "!! wrapper fish-no-query-term). Subshells are immune because they" + _lt "!! inherit the exported variable — that's why a new shell 'fixes' it." +else + _lt "query-term is off in this shell (good — the known root cause is ruled out)." +end + # ---- 3. Terminal state below the shell -------------------------------------- _lt "" _lt "Querying terminal state (takes a few seconds)..." @@ -76,6 +93,10 @@ set -l fixed none _lt "" _lt "Now applying resets one at a time. After each, type into the test prompt" _lt "to judge whether the lag is gone." +_lt "CAVEAT: fish's read prompt may NOT exhibit lag even when the main" +_lt "commandline does. If typing at these test prompts never feels laggy at" +_lt "all, answer 'u' (unsure) instead of 'y' — a 'y' here is only meaningful" +_lt "if you could feel the lag at the test prompts before the reset." if test $fixed = none _lt "" diff --git a/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/upstream_repro.py b/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/upstream_repro.py new file mode 100644 index 00000000..d8ad5b76 --- /dev/null +++ b/platforms/home-manager/modules/nmasur/presets/programs/lag-triage/upstream_repro.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Deterministic repro of permanent fish reader degradation (fish 4.8.1). + +This is evidence for an upstream fish-shell report, and the proof behind the +fish-no-query-term wrapper in presets/programs/zellij.nix. Not installed by +the nix module; run directly: python3 upstream_repro.py [queryterm|noqueryterm] + +Finding: with the query-term feature enabled (latched from the startup env, +which is fish's default), fish sends OSC 11 + CPR (\e[6n) + DA1 (\e[0c) +after every external command and waits for the replies. If the terminal +fails to reply during ONE such cycle -- even though it answered every query +before and answers every query after -- that fish process's interactive +reader is PERMANENTLY degraded: keystrokes are no longer echoed (>3s each, +never recovers). In production this happens when zellij drops/mis-relays a +reply during TUI teardown or heavy output (cf. zellij-org/zellij#5158), and +it presents as permanent typing lag cured only by replacing the process. +With fish_features=no-query-term in the startup environment, the same +sequence has zero effect (~35ms echo throughout). + +Phases: + A. terminal answers all queries -> echo ~35ms (both variants) + B. replies dropped for one command -> queryterm: echo dead, permanently + C. replies restored, another command -> queryterm: still dead +""" +import os, pty, re, select, subprocess, sys, time, fcntl, termios + +VARIANT = sys.argv[1] if len(sys.argv) > 1 else "queryterm" + +env = dict(os.environ) +env["TERM"] = "xterm-ghostty" +env["ZELLIJ"] = "0" +env["ZELLIJ_SESSION_NAME"] = "repro" +env["FISH_DEBUG"] = "term-support" +env["FISH_DEBUG_OUTPUT"] = f"/tmp/fish-lagrepro-{VARIANT}.log" +env.pop("fish_features", None) +if VARIANT == "noqueryterm": + env["fish_features"] = "no-query-term" + +master, slave = pty.openpty() +# give it a size +fcntl.ioctl(master, termios.TIOCSWINSZ, b"\x00\x28\x00\x78\x00\x00\x00\x00") +proc = subprocess.Popen( + ["fish", "-i", "--no-config"], + stdin=slave, stdout=slave, stderr=slave, env=env, + preexec_fn=lambda: (os.setsid(), fcntl.ioctl(0, termios.TIOCSCTTY, 0)), + close_fds=True, +) +os.close(slave) + +RESPOND = True +transcript = [] + +def respond(data): + """Answer terminal queries the way a well-behaved terminal would.""" + out = b"" + for m in re.finditer(rb"\x1b\[6n", data): + out += b"\x1b[40;1R" # CPR + for m in re.finditer(rb"\x1b\[0?c", data): + out += b"\x1b[?62;22c" # DA1 + for m in re.finditer(rb"\x1b\[\?u", data): + out += b"\x1b[?0u" # kitty flags + for m in re.finditer(rb"\x1b\]11;\?", data): + out += b"\x1b]11;rgb:2828/2828/2828\x1b\\" # OSC 11 + for m in re.finditer(rb"\x1b\[>0?q", data): + out += b"\x1bP>|ghostty 1.3.1\x1b\\" # XTVERSION + for m in re.finditer(rb"\x1bP\+q[0-9a-fA-F;]+\x1b\\", data): + out += b"\x1bP0+r\x1b\\" # XTGETTCAP: not found + for m in re.finditer(rb"\x1b\[\?(\d+)\$p", data): + out += b"\x1b[?%s;2$y" % m.group(1) # DECRQM: reset + return out + +def pump(timeout): + """Read fish output for `timeout` seconds, answering queries if RESPOND.""" + buf = b"" + end = time.monotonic() + timeout + while time.monotonic() < end: + r, _, _ = select.select([master], [], [], 0.03) + if r: + try: + data = os.read(master, 65536) + except OSError: + return buf + buf += data + transcript.append(data) + if RESPOND: + reply = respond(data) + if reply: + os.write(master, reply) + return buf + +def send(s): + os.write(master, s if isinstance(s, bytes) else s.encode()) + +def measure_echo(chars, settle=0.1): + """Send chars one at a time; measure time until each is echoed.""" + results = [] + for ch in chars: + pump(settle) + t0 = time.monotonic() + send(ch) + deadline = time.monotonic() + 3.0 + latency = None + buf = b"" + while time.monotonic() < deadline: + buf += pump(0.02) + if ch.encode() in buf: + latency = (time.monotonic() - t0) * 1000 + break + results.append((ch, latency)) + return results + +print(f"=== variant: {VARIANT} ===") +pump(1.2) # startup, queries answered + +send("echo warmup\r") +pump(0.8) + +print("phase A: terminal responsive, echo latency per key:") +for ch, ms in measure_echo("abcde"): + print(f" {ch}: {ms:.0f} ms" if ms else f" {ch}: NO ECHO in 3s") +send("\x15") # ctrl-u clear line +pump(0.3) + +# Run external command, then STOP answering queries (simulate lost relay) +send("sh -c true\r") +time.sleep(0.05) +RESPOND = False +pump(1.0) + +print("phase B: after external command with query replies DROPPED:") +for ch, ms in measure_echo("fghij"): + print(f" {ch}: {ms:.0f} ms" if ms else f" {ch}: NO ECHO in 3s") +send("\x15") +pump(0.3) + +# Does it persist across further commands, with responses restored? +RESPOND = True +send("sh -c true\r") +pump(1.0) +print("phase C: responses restored, after another external command:") +for ch, ms in measure_echo("klmno"): + print(f" {ch}: {ms:.0f} ms" if ms else f" {ch}: NO ECHO in 3s") + +send("\x15") +pump(0.2) +send("exit\r") +pump(0.5) +try: + proc.wait(timeout=3) +except subprocess.TimeoutExpired: + proc.kill() diff --git a/platforms/home-manager/modules/nmasur/presets/programs/zellij.nix b/platforms/home-manager/modules/nmasur/presets/programs/zellij.nix index 724cd86a..c7369017 100644 --- a/platforms/home-manager/modules/nmasur/presets/programs/zellij.nix +++ b/platforms/home-manager/modules/nmasur/presets/programs/zellij.nix @@ -9,6 +9,22 @@ let inherit (config.nmasur.settings) username; cfg = config.nmasur.presets.programs.zellij; + # fish latches feature flags from its startup ENVIRONMENT before config.fish + # runs, so the `set -gx fish_features no-query-term` in config.fish only + # protects CHILD fish processes — which is why subshells/exec fish were always + # immune to the post-TUI typing lag while zellij-spawned pane shells were not. + # With query-term latched on, fish sends OSC 11 + CPR + DA1 queries after + # every command and waits for replies; if zellij fails to relay even one + # reply (a race during TUI teardown or heavy output), that fish process's + # reader is PERMANENTLY degraded — reproduced deterministically in a PTY + # harness on fish 4.8.1 (see docs/CHANGELOG.md 2026-08-29 and + # presets/programs/lag-triage/upstream_repro.py). Spawning fish with the + # variable already exported makes every pane shell immune. + fish-no-query-term = pkgs.writeShellScriptBin "fish-no-query-term" '' + export fish_features=no-query-term + exec ${lib.getExe pkgs.fish} "$@" + ''; + zellij-switch-to-last = pkgs.writeShellScriptBin "zellij-switch-to-last" '' TARGET_SESSION=$(cat ~/.local/state/zellij-last-session) if [ -z "$TARGET_SESSION" ]; then @@ -130,7 +146,10 @@ in # Spawn fish directly instead of trusting $SHELL, which inherits the # macOS login shell. On darwin that login shell is no longer managed by # nix-darwin, so $SHELL can point at a stale /run/current-system path. - default_shell = lib.getExe pkgs.fish; + # Wrapped to export fish_features=no-query-term BEFORE fish starts — + # see the fish-no-query-term comment above for why this must happen in + # the environment rather than in config.fish. + default_shell = lib.getExe fish-no-query-term; # default_layout = "compact-top"; # Remove border pane_frames = false;