8 Commits
15 changed files with 117 additions and 707 deletions

No files matched your search

+39
View File
@@ -1,5 +1,44 @@
# Changelog
## 2026-10-17: removed post-TUI lag tooling
- With the root cause fixed (starship notifications, below), removed the `lag-triage` module: `lag-triage`, `unlag`, `lag-sample`, `term-probe`, the PTY repro script, the flight recorder filter and the Ctrl-b `heal-autosuggest` binding. Also deleted its `power-user.nix` enable and `~/.local/state/lag-triage`.
- The zellij `fish-no-query-term` wrapper now only exports `fish_features=no-query-term` before starting fish; the recorder is gone.
## 2026-10-17: post-TUI lag ROOT CAUSE — starship cmd_duration notifications
- The first timestamped episode (fish 4.9.3, fresh zellij session) showed fish handling keys in ~1ms, but every prompt took **~2s**, blocked on one child process. After Ctrl-b, prompts dropped to ~50ms.
- Cause: `starship.nix` set `cmd_duration.show_notifications = true` on darwin only (`min_time_to_notify = 30000`). On macOS, sending that notification blocks `starship prompt` for ~2s (measured: 0.03s with `--cmd-duration=100` vs 2.02s with `--cmd-duration=600000`). fish keeps `$CMD_DURATION` until another command actually executes. So after any command over 30s (i.e. every TUI session), every prompt redraw re-sent the notification and froze the shell for 2s: Enter on an empty line, vi mode switches, repaints.
- This explains every observation: macOS-only (Linux had notifications off), "post-TUI" (TUIs are the long commands), and why the toggle "cured" it. Executing *any* command resets `$CMD_DURATION`; autosuggestions were never involved. The 2026-08/09 entries blaming fish autosuggestions, query-term and thread pools were wrong.
- Fix: `show_notifications = false` everywhere. Verified: the real config with a 10-minute duration renders in 0.05s and still shows the duration.
## 2026-10-17: disable atuin tests
- Added `overlays/atuin.nix` to disable atuin 18.23.0's `checkPhase`. Its
PTY-driven e2e tests (`tests/e2e_capture.rs`, `tests/e2e_pty.rs`, notably the
bash-preexec variants) time out in the darwin build sandbox ("timed out
waiting for shell session ID", `captured: []`), which broke
`nh home switch` for `lookingglass` after the flake update. atuin isn't in
the binary cache for aarch64-darwin, so it gets built locally. Skipping
individual tests wasn't viable: cargo stops at the first failing test binary,
hiding later failures behind ~13-minute rebuilds. The 554 unit tests passed.
## 2026-10-17: fix flake check for AWS-only arrow
- `nix flake check` (run by the weekly Update Flake workflow) failed evaluating `packages.x86_64-linux.arrow-iso`: arrow now imports `amazon-image.nix` directly, which conflicts with the `iso` and `qcow-efi` generator formats (`boot.loader.timeout` 1 vs 10, and similar).
- `flake.nix` now has an `awsOnlyHosts = [ "arrow" ]` list; those hosts only get the `-aws` image. `arrow-iso` and `arrow-qcow` are no longer exported.
## 2026-10-15: post-TUI lag — re-investigation, timestamped recorder
- **Status: root cause still unproven.** Not reproducible in fresh shells: a PTY harness (with and without a real zellij 0.45.1 session, fish 4.9.3, the real config and a copy of the real history) showed no echo-latency change after hx (with nixd/marksman/rumdl running), jjui, yazi, vim, less, lazygit or Claude Code, nor with huge history autosuggestions or concurrent history writers.
- New facts:
- Timeline: fish 4.8.0 first landed in the store on 2026-07-05 and the first lag fix attempt was 2026-08-08. 4.8.0 changed history timestamps/merging (fish #10300), which is the data autosuggestions search on every keystroke.
- Today's (2026-10-15) lag was in a 36-day-old fish **4.8.1** pane. 57 of 85 running fish processes are still 4.8.1 (started before 4.9.3 was installed on 2026-09-24); there is no evidence yet either way that 4.9.3 shells lag. Long-lived shells are the main difference from every failed reproduction.
- In the shell history, the command run right before a heal was `hx` in 4 of the 5 cases with usable context (Helix, not previously on the trigger list).
- Corrections to earlier entries: the thread-pool starvation theory (2026-08-31) is not supported. In all September flight logs every enqueued work item was picked up by a worker (2090/2090). The September logs had no timestamps, so they cannot show whether fish was slow to handle keys or the delay was downstream (zellij/ghostty). In most pre-heal windows fish handled keys one at a time with a repaint after each (no input backlog), which weakly points away from fish's input loop.
- Recorder upgrade (`zellij.nix` `fish-no-query-term`, `lag-triage/flight_filter.pl`): when `~/.local/state/lag-triage/RECORD` exists, new pane shells stream `FISH_DEBUG` through a FIFO into a Perl filter that adds millisecond timestamps and redacts typed/pasted content and command lines (the September logs contained pasted secrets in clear text). Added `reader-render`, `complete`, `autoload`, `history` and `uvar-notifier` categories. `config.fish` now drops `FISH_DEBUG`/`FISH_DEBUG_OUTPUT` so subshells and `fish -c` children no longer write into the recorder's path (they previously truncated the same log file).
- `heal-autosuggest` (Ctrl-b) now first copies the last 20k lines of the pane's timestamped log to `~/.local/state/lag-triage/episodes/`, so pressing it during a lag preserves the evidence. Next step: analyze the first episode (time from each `Read byte` to the following `Writing terminal command` decides fish-side versus downstream).
## 2026-10-11
- **Configured `arrow` host for AWS EC2 deployment**:
Generated
+24 -24
View File
@@ -234,11 +234,11 @@
]
},
"locked": {
"lastModified": 1791600313,
"narHash": "sha256-/2gb2WTztITD/JU1XUCg63vDsIfJGLr0LDsjSCTTJc8=",
"lastModified": 1792098797,
"narHash": "sha256-u4DIIZgznEGzByxXrxWCXK19VcMtTBmi/M9gZ7Jk9bM=",
"owner": "nix-community",
"repo": "home-manager",
"rev": "427c74a88a93cc9f856c8bb99bf6b4a5187a6b33",
"rev": "640725978b99076e18e069f6e7d2959124584f8f",
"type": "github"
},
"original": {
@@ -277,11 +277,11 @@
"nixpkgs": "nixpkgs_5"
},
"locked": {
"lastModified": 1784936738,
"narHash": "sha256-X4w9BrFkisfzla3m1WwDJUicUbI7JRdhzq5aS7rrB0k=",
"lastModified": 1792092848,
"narHash": "sha256-L8uKQaS6wqZxXm7ULQynxnw8m5nZlqaJcbZ2+1ECXbE=",
"owner": "MichailiK",
"repo": "nix-caddy-withplugins",
"rev": "70307cfc568e5434cf42067d797b27c38ea69944",
"rev": "5181c7953264f637f7455cc95c84cf92346c8098",
"type": "github"
},
"original": {
@@ -298,11 +298,11 @@
]
},
"locked": {
"lastModified": 1791105314,
"narHash": "sha256-oTM7QFgRQ+vxBpMBMc7hEYpqBtpXAzXu2xuB/tru84U=",
"lastModified": 1791711434,
"narHash": "sha256-LpIhxnuO0tYeZ4k7BbNMxsPs2zJpB4hWXO1jdhUuE7Q=",
"owner": "nix-community",
"repo": "nix-index-database",
"rev": "e740bc4e9d8918d7819a51486d5099930c65c72e",
"rev": "9a2dd40780eeab8543aa97c59c26f11267025616",
"type": "github"
},
"original": {
@@ -401,11 +401,11 @@
},
"nixpkgs-stable": {
"locked": {
"lastModified": 1791426274,
"narHash": "sha256-MWhktfFtFE8Jl7bf9Pkp7TR5MSdYCjCswSo0Gkb4ywc=",
"lastModified": 1792030654,
"narHash": "sha256-OaQwAtPMTjQiY2sT/2/62UbSPhzP/0aDnWzMm/NCiUw=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "7c8764b7c7b09b34f632464276218ef9090eaa11",
"rev": "26f4107e47e3ad0ffd0db220469ba83b6f6d96e8",
"type": "github"
},
"original": {
@@ -465,11 +465,11 @@
},
"nixpkgs_5": {
"locked": {
"lastModified": 1781074563,
"narHash": "sha256-md8WlXOlfnIeHeOScMTTHFyf2d6iaTwPl2apR5EQ3P4=",
"lastModified": 1791956154,
"narHash": "sha256-jBYGPi4udQ9uo4KQp9m2J0546nJVQitG15gvZSaVVAs=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9ae611a455b90cf061d8f332b977e387bda8e1ca",
"rev": "55bfb4d7437bd4e09ab7cba58a43bd68eb310044",
"type": "github"
},
"original": {
@@ -481,11 +481,11 @@
},
"nixpkgs_6": {
"locked": {
"lastModified": 1791456040,
"narHash": "sha256-H83yz/do+hjOO0uujHyH8RgtIQP2v7Mw+cLwcZK/xKc=",
"lastModified": 1791956154,
"narHash": "sha256-jBYGPi4udQ9uo4KQp9m2J0546nJVQitG15gvZSaVVAs=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "e7439b6b14ad3cc35d05608ebca9bce01a25f5f8",
"rev": "55bfb4d7437bd4e09ab7cba58a43bd68eb310044",
"type": "github"
},
"original": {
@@ -519,11 +519,11 @@
]
},
"locked": {
"lastModified": 1791626603,
"narHash": "sha256-I16LCKG4Bh3WaMFHppz7gbF70smYOicr/DalSleXatU=",
"lastModified": 1792249243,
"narHash": "sha256-zDZMBwcbcj2ufGk9EGy+BnUnV3BT/NiYuNynGiQ0SqA=",
"owner": "nix-community",
"repo": "nur",
"rev": "241ca011a9fcefa68b42e8d5bd55aac59f6d9fe4",
"rev": "c21b30ed2ab024497fcd0c26abb0757a3fd1d100",
"type": "github"
},
"original": {
@@ -557,11 +557,11 @@
"nixpkgs": "nixpkgs_7"
},
"locked": {
"lastModified": 1791620365,
"narHash": "sha256-CVZXIDNw7HEwyD8X6q9Ln370Sc0ZqRlTzDRwOZ1hJMY=",
"lastModified": 1792219666,
"narHash": "sha256-RMO08tKD+hFIL5ryWi85VY4aHE5oIpdV1h4kVFctOkw=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "e90fafa71b440fabcd52dd7fb017ab2da81c781a",
"rev": "a110337465cfb76e3806b81b9db3094c3e6e770a",
"type": "github"
},
"original": {
+26 -18
View File
@@ -183,27 +183,35 @@
root = import ./hosts/x86_64-linux/swan/root.nix;
};
awsOnlyHosts = [ "arrow" ];
generators = builtins.mapAttrs (
# x86_64-linux = { arrow = ...; swan = ...; }
system: hosts:
(lib.concatMapAttrs (name: module: {
"${name}-aws" = lib.generateImage {
inherit system module;
format = "amazon";
specialArgs = { inherit hostnames; };
};
"${name}-iso" = lib.generateImage {
inherit system module;
format = "iso";
specialArgs = { inherit hostnames; };
};
"${name}-qcow" = lib.generateImage {
inherit system module;
format = "qcow-efi";
specialArgs = { inherit hostnames; };
# extraModules = [ "${nixpkgs}/nixos/modules/virtualisation/oci-image.nix" ];
};
}) hosts)
(lib.concatMapAttrs (
name: module:
{
"${name}-aws" = lib.generateImage {
inherit system module;
format = "amazon";
specialArgs = { inherit hostnames; };
};
}
# Hosts that import amazon-image.nix directly conflict with other formats
// lib.optionalAttrs (!builtins.elem name awsOnlyHosts) {
"${name}-iso" = lib.generateImage {
inherit system module;
format = "iso";
specialArgs = { inherit hostnames; };
};
"${name}-qcow" = lib.generateImage {
inherit system module;
format = "qcow-efi";
specialArgs = { inherit hostnames; };
# extraModules = [ "${nixpkgs}/nixos/modules/virtualisation/oci-image.nix" ];
};
}
) hosts)
) lib.linuxHosts # x86_64-linux = { arrow = ...; swan = ...; }
;
+10
View File
@@ -0,0 +1,10 @@
_inputs: _final: prev: {
# atuin 18.23.0's PTY-driven e2e tests (e2e_capture, e2e_pty) time out in the
# darwin build sandbox: shell hooks never report a session ID or history
# entry. Unit tests pass, but cargo aborts on the first failing e2e binary.
atuin = prev.atuin.overrideAttrs (_old: {
doCheck = false;
});
}
@@ -33,7 +33,7 @@ in
# size = 11.0;
};
services.polybar.config."bar/main".font-0 = "Hack Nerd Font:size=10;2";
programs.rofi.font = "Hack Nerd Font 14";
programs.rofi.settings.font = "Hack Nerd Font 14";
programs.alacritty.settings.font.normal.family = "Maple Mono NF";
programs.kitty.font.name = "Maple Mono NF";
nmasur.presets.programs.wezterm.font = "Maple Mono NF";
@@ -28,6 +28,7 @@ in
cask "topnotch" # Darkens the menu bar to complete black
cask "ghostty" # Terminal application (not buildable on Nix on macOS)
cask "discord" # nixpkgs bundle breaks its notarization seal on macOS 26+
cask "vorssaint" # Customizations and automations for macOS
# Mac App Store apps (requires 'mas' CLI, optional)
# mas "Tailscale", id: 1475387142
@@ -1,72 +0,0 @@
{
config,
pkgs,
lib,
...
}:
let
cfg = config.nmasur.presets.programs.lag-triage;
term-probe = pkgs.writeScriptBin "term-probe" ''
#!${lib.getExe pkgs.python3}
${builtins.readFile ./term_probe.py}
'';
in
{
options.nmasur.presets.programs.lag-triage.enable =
lib.mkEnableOption "Terminal input-lag triage tools";
config = lib.mkIf cfg.enable {
home.packages = [ term-probe ];
# Ctrl-b: EXECUTE the proven manual cure as a real commandline. The cure
# is not the variable's end value (it starts and ends at 1) — it is the
# reader fully exiting readline and re-entering, which only command
# EXECUTION does. Setting the variable inline (a plain binding body, or the
# old fish_postexec/fish_cancel hook) never makes the reader exit/re-enter,
# so it never cured and, with extra repaints on a wedged reader, made it
# worse. `commandline -f execute` reproduces exactly what typing the cure
# and pressing Enter does — indistinguishable to fish from the manual cure.
# Note: this submits the current commandline, so it runs the cure in place
# of whatever is typed (fine for a rescue key hit at an empty prompt).
# Ctrl-b chosen because it is otherwise unbound (Ctrl-g is taken).
nmasur.presets.programs.fish.fish_user_key_bindings = # fish
''
for mode in insert default visual
bind -M $mode \cb heal-autosuggest
end
'';
programs.fish.functions = {
lag-triage = {
description = "Diagnose post-TUI typing lag in the current shell";
body = builtins.readFile ./lag-triage.fish;
};
unlag = {
description = "Reset terminal state left behind by a TUI";
body = builtins.readFile ./unlag.fish;
};
lag-sample = {
description = "Stack-sample fish and zellij while typing lag is happening";
body = builtins.readFile ./lag-sample.fish;
};
heal-autosuggest = {
description = "Heal post-TUI typing lag by executing the autosuggestion-toggle cure (bind to a key)";
# Replace the commandline with the exact cure the user runs by hand and
# execute it. Executing (not inline-setting) is what cures: it forces
# the reader to leave and re-enter readline. Runs in place of whatever
# is currently typed.
body = # fish
''
commandline -r 'set -g fish_autosuggestion_enabled 0; and set -g fish_autosuggestion_enabled 1'
commandline -f execute
'';
};
};
};
}
@@ -1,53 +0,0 @@
# Capture stack samples of this fish process, the zellij server, and the
# zellij client WHILE the typing lag is happening. This names the guilty
# component directly: if fish's main thread is busy/blocked per keystroke the
# stacks show exactly where; if fish is idle while typing feels laggy, the
# delay is in zellij's render path instead.
#
# CAUTION (learned 2026-08-30): attaching the sampler to a lagging fish CURES
# the lag (thread suspend/resume unwedges it), so run this from a DIFFERENT
# pane with the lagging shell's pid: `lag-sample <pid>` (get it in the lagging
# shell with the builtin-only `echo $fish_pid`). Have someone type in the
# lagging pane while sampling runs — the first samples may catch the wedge.
# With no argument it samples the current shell.
set -l target $fish_pid
if test (count $argv) -ge 1; and test -n "$argv[1]"
set target $argv[1]
end
set -l outdir ~/.local/state/lag-triage
mkdir -p $outdir
set -l ts (date +%Y%m%d-%H%M%S)
set -l dur 8
set -l fishfile $outdir/sample-$ts-fish-$target.txt
/usr/bin/sample $target $dur 1 -file $fishfile &>/dev/null &
disown
# this session's zellij server (socket path ends in the session name)
set -l serverpid (pgrep -f "zellij --server.*/$ZELLIJ_SESSION_NAME\$")
test -z "$serverpid"; and set serverpid (pgrep -f "zellij --server" | head -3)
for pid in $serverpid
/usr/bin/sample $pid $dur 1 -file $outdir/sample-$ts-zellij-server-$pid.txt &>/dev/null &
disown
end
# zellij clients (attached to ghostty): named zellij but without --server args
set -l allserver (pgrep -f "zellij --server")
set -l clientpid
for pid in (pgrep -x zellij)
contains $pid $allserver; or set -a clientpid $pid
end
for pid in $clientpid[1..3]
/usr/bin/sample $pid $dur 1 -file $outdir/sample-$ts-zellij-client-$pid.txt &>/dev/null &
disown
end
# notify when done, without occupying the commandline
fish -c "sleep (math $dur + 2); echo; echo '== lag-sample done: '$outdir'/sample-$ts-*.txt =='" &
disown
echo "Sampling fish (pid $target), zellij server(s) [$serverpid], client(s) [$clientpid] for $dur s."
echo ">>> TYPE CONTINUOUSLY IN THE LAGGING PANE NOW (junk text is fine) <<<"
echo "Files: $outdir/sample-$ts-*.txt"
@@ -1,182 +0,0 @@
# Guided diagnosis for the post-TUI typing-lag problem (Ghostty + Zellij + fish).
# Run this IN THE LAGGING SHELL the moment you notice the lag, BEFORE starting
# a new shell. It captures evidence, then applies targeted resets one at a time
# so the stage that cures the lag identifies the layer holding stuck state.
# Everything is logged for filing an upstream issue.
set -l logdir ~/.local/state/lag-triage
mkdir -p $logdir
set -l logfile $logdir/(date +%Y%m%d-%H%M%S).log
function _lt --inherit-variable logfile
echo $argv | tee -a $logfile
end
function _lt_ask --inherit-variable logfile
# usage: _lt_ask VARNAME prompt... -> sets global $VARNAME (default: skip)
set -l __name $argv[1]
read -g -P "$argv[2..] " $__name
or set -g $__name skip
test -z "$$__name"; and set -g $__name skip
echo "ANSWER $__name: $$__name" >>$logfile
end
_lt "== lag-triage "(date)" =="
_lt "Log: $logfile"
_lt "Answer y / n, or press Enter to skip a question."
_lt ""
# ---- 1. Context -------------------------------------------------------------
_lt_ask ans_tui "Which TUI did you just exit (nvim/jjui/yazi/other)?"
_lt_ask ans_launch "Launched via (f)loating-pane keybind or (c)ommand typed in this shell?"
# ---- 2. Snapshot ------------------------------------------------------------
begin
echo "-- snapshot --"
fish --version
echo "fish pid: $fish_pid, started: "(ps -o lstart= -p $fish_pid 2>/dev/null)
zellij --version 2>/dev/null
echo "escape delay: '$fish_escape_delay_ms' sequence delay: '$fish_sequence_key_delay_ms'"
env | grep -iE '^(TERM|ZELLIJ|GHOSTTY|COLORTERM)' | sort
echo "-- status features --"
status features
echo "-- stty -a --"
stty -a
end >>$logfile 2>&1
_lt "Captured shell + environment snapshot."
# Proven root cause of the 2026-08 lag (see docs/CHANGELOG.md 2026-08-29):
# fish latches feature flags from its startup env before config.fish runs, so
# a shell with query-term ON sends terminal queries after every command; one
# reply zellij fails to relay permanently degrades this process's reader.
if status features | string match -qr '^query-term\s+on'
_lt ""
_lt "!! query-term is ON in this shell: fish did NOT get fish_features="
_lt "!! no-query-term in its STARTUP environment (config.fish is too late)."
_lt "!! This is the proven root cause of the post-TUI lag — a query reply"
_lt "!! lost by zellij permanently degrades this fish process's reader."
_lt "!! Fix: spawn fish with the variable exported (zellij default_shell"
_lt "!! wrapper fish-no-query-term). Subshells are immune because they"
_lt "!! inherit the exported variable — that's why a new shell 'fixes' it."
else
_lt "query-term is off in this shell (good — the known root cause is ruled out)."
end
# ---- 3. Terminal state below the shell --------------------------------------
_lt ""
_lt "Querying terminal state (takes a few seconds)..."
term-probe report 2>&1 | tee -a $logfile
_lt ""
_lt " ^ Things to look for: kitty flags with a reply > 0, modifyOtherKeys > 1,"
_lt " any mouse/alternate-screen mode SET while at a shell prompt, or a slow"
_lt " DA1 round-trip (> 100 ms means the input path itself is delayed)."
# ---- 4. Raw keystroke capture (bypasses fish entirely) ----------------------
_lt ""
_lt "Raw input capture: type ~10 characters at a steady pace, including one"
_lt "ESC press and one arrow key. This shows the exact bytes this pane delivers"
_lt "and their timing, with fish's input handling out of the picture."
term-probe keylog 2>&1 | tee -a $logfile
_lt_ask ans_keylog_instant "Did each keypress appear INSTANTLY in the capture? (y/n)"
_lt_ask ans_keylog_plain "Were plain letters single plain bytes like b'a' (not escape sequences)? (y/n)"
# ---- 5. Scope ---------------------------------------------------------------
_lt ""
_lt_ask ans_scope_pane "Optional: open a NEW zellij pane/tab and type — laggy there too? (y/n)"
_lt_ask ans_scope_window "Optional: type in a separate Ghostty window (outside this zellij session) — laggy? (y/n)"
# ---- 6. Staged resets -------------------------------------------------------
# Each stage resets one category of state a TUI could have left behind.
# The first stage that cures the lag names the culprit.
set -l fixed none
_lt ""
_lt "Now applying resets one at a time. After each, type into the test prompt"
_lt "to judge whether the lag is gone."
_lt "CAVEAT: fish's read prompt may NOT exhibit lag even when the main"
_lt "commandline does. If typing at these test prompts never feels laggy at"
_lt "all, answer 'u' (unsure) instead of 'y' — a 'y' here is only meaningful"
_lt "if you could feel the lag at the test prompts before the reset."
if test $fixed = none
_lt ""
_lt "Stage A - kitty keyboard protocol: pop stack + clear all flags"
printf '\e[<9u\e[=0;1u'
_lt_ask ans_stage_a " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_a" = y; and set fixed "A (kitty keyboard state)"
end
if test $fixed = none
_lt "Stage B - modifyOtherKeys off"
printf '\e[>4;0m'
_lt_ask ans_stage_b " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_b" = y; and set fixed "B (modifyOtherKeys)"
end
if test $fixed = none
_lt "Stage C - normal keypad + normal cursor keys"
printf '\e>\e[?1l'
_lt_ask ans_stage_c " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_c" = y; and set fixed "C (application keypad/cursor mode)"
end
if test $fixed = none
_lt "Stage D - disable mouse, focus reporting, synchronized output"
printf '\e[?1000l\e[?1001l\e[?1002l\e[?1003l\e[?1005l\e[?1006l\e[?1015l\e[?1016l\e[?1004l\e[?2026l'
_lt_ask ans_stage_d " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_d" = y; and set fixed "D (mouse/focus/sync modes)"
end
if test $fixed = none
_lt "Stage E - leave alternate screen"
printf '\e[?1049l'
_lt_ask ans_stage_e " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_e" = y; and set fixed "E (alternate screen)"
end
if test $fixed = none
_lt "Stage F - stty sane (line-discipline reset)"
stty sane
_lt_ask ans_stage_f " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_f" = y; and set fixed "F (termios/line discipline)"
end
if test $fixed = none
_lt "Stage G - DECSTR soft terminal reset"
printf '\e[!p'
_lt_ask ans_stage_g " Test typing here, then Enter — lag gone? (y/n)"
test "$ans_stage_g" = y; and set fixed "G (DECSTR-resettable mode)"
end
# ---- 7. Verdict --------------------------------------------------------------
_lt ""
_lt "== Verdict =="
if test $fixed != none
_lt "Lag cleared by stage $fixed."
_lt "That state was stuck BELOW fish — in the Zellij pane or relayed to"
_lt "Ghostty — and the TUI you exited ($ans_tui) failed to restore it, or"
_lt "Zellij failed to restore it when the pane closed."
_lt "Re-probing terminal state after the fix for comparison:"
term-probe report 2>&1 | tee -a $logfile
_lt ""
_lt "-> File this log against zellij (or ghostty, if a separate window also"
_lt " lagged). The before/after probe diff pinpoints the exact stuck mode."
else if test "$ans_keylog_instant" = y; and test "$ans_keylog_plain" = y
_lt "Raw input reaches this pane instantly as plain bytes, and no terminal"
_lt "state reset helps: the lag lives INSIDE this fish process (reader state)."
_lt "Confirm now: run 'exec fish' — if that cures it, it is fish-internal."
_lt ""
_lt "-> To catch it in the act, run your next long-lived shell as:"
_lt " FISH_DEBUG='reader,term-support' FISH_DEBUG_OUTPUT=$logdir/fish-debug.log fish"
_lt " then re-run lag-triage when it recurs and file both logs to fish-shell."
else
_lt "Keystrokes were delayed or arrived as escape sequences BEFORE fish saw"
_lt "them: the problem is in Zellij (client stdin parser / server) or Ghostty."
_lt " new pane also laggy: $ans_scope_pane (y -> session-wide, not this pane)"
_lt " separate window laggy: $ans_scope_window (y -> Ghostty itself)"
_lt "-> File this log against zellij; include the keylog byte capture."
end
_lt ""
_lt "Full log: $logfile"
functions -e _lt _lt_ask
@@ -1,165 +0,0 @@
"""Probe the terminal state of the current pane, below the shell.
Modes:
report - query kitty-keyboard flags, modifyOtherKeys, and DEC private
modes directly on /dev/tty, reporting each reply (or lack of
one) and its round-trip latency. Answers may come from Zellij
(pane state) or be relayed from Ghostty (window state).
keylog - raw-mode keystroke capture: prints the exact bytes and
inter-key latency for every keypress, bypassing the shell's
input machinery entirely. Press q to finish.
Used by the `lag-triage` fish function to pin down which layer
(fish / zellij / ghostty) is holding stuck state when typing lags
after a TUI exits.
"""
import os
import re
import select
import sys
import termios
import time
import tty
# DECRQM reply values
DECRQM_VALUES = {
"0": "not recognized",
"1": "SET",
"2": "reset",
"3": "permanently set",
"4": "permanently reset",
}
DEC_MODES = [
(1, "application cursor keys (DECCKM)"),
(25, "cursor visible"),
(1000, "mouse click reporting"),
(1002, "mouse drag reporting"),
(1003, "mouse all-motion reporting"),
(1004, "focus reporting"),
(1006, "SGR mouse encoding"),
(1049, "alternate screen"),
(2004, "bracketed paste"),
(2026, "synchronized output"),
(2031, "color theme reporting"),
]
QUERIES = [
("kitty keyboard flags (\\e[?u)", b"\x1b[?u", rb"\x1b\[\?(\d+)u", None),
("modifyOtherKeys (XTQMODKEYS)", b"\x1b[?4m", rb"\x1b\[>4;(\d+)m", None),
("background color (OSC 11)", b"\x1b]11;?\x1b\\", rb"\x1b\]11;([^\x07\x1b]+)", None),
] + [
(
f"DEC mode {num} — {desc}",
b"\x1b[?%d$p" % num,
rb"\x1b\[\?%d;(\d+)\$y" % num,
DECRQM_VALUES,
)
for num, desc in DEC_MODES
]
def read_for(fd, seconds):
buf = b""
end = time.monotonic() + seconds
while True:
remaining = end - time.monotonic()
if remaining <= 0:
break
r, _, _ = select.select([fd], [], [], remaining)
if not r:
break
buf += os.read(fd, 4096)
return buf
def report(fd):
lines = []
raw_dump = b""
for label, query, pattern, value_names in QUERIES:
raw_dump += read_for(fd, 0.02) # drain stragglers
start = time.monotonic()
os.write(fd, query)
buf = b""
match = None
deadline = time.monotonic() + 0.35
while time.monotonic() < deadline:
buf += read_for(fd, 0.05)
match = re.search(pattern, buf)
if match:
break
raw_dump += buf
if match:
latency = (time.monotonic() - start) * 1000
value = match.group(1).decode("ascii", "replace")
if value_names:
value = f"{value} ({value_names.get(value, '?')})"
lines.append(f" {label:45s} = {value:24s} [{latency:6.1f} ms]")
else:
lines.append(f" {label:45s} = (no reply)")
# DA1 as a fence: every terminal answers it, so its round-trip time
# measures the whole input path (ghostty -> zellij -> pane -> here).
start = time.monotonic()
os.write(fd, b"\x1b[c")
buf = b""
match = None
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
buf += read_for(fd, 0.05)
match = re.search(rb"\x1b\[\?([0-9;]*)c", buf)
if match:
break
raw_dump += buf
if match:
latency = (time.monotonic() - start) * 1000
lines.append(
f" {'device attributes (DA1) round-trip':45s} = "
f"{match.group(1).decode():24s} [{latency:6.1f} ms]"
)
else:
lines.append(f" {'device attributes (DA1) round-trip':45s} = (NO REPLY in 2s!)")
lines.append(f" raw bytes received: {raw_dump!r}")
return lines
def keylog(fd):
sys.stdout.write("keylog: capturing raw bytes from the tty. Press q to finish.\r\n")
sys.stdout.flush()
last = time.monotonic()
while True:
select.select([fd], [], [], None)
data = os.read(fd, 4096)
now = time.monotonic()
delta_ms = (now - last) * 1000
last = now
sys.stdout.write(f" +{delta_ms:8.1f} ms {data!r} hex={data.hex(' ')}\r\n")
sys.stdout.flush()
if data in (b"q", b"\x03", b"\x04"):
break
def main():
mode = sys.argv[1] if len(sys.argv) > 1 else "report"
fd = os.open("/dev/tty", os.O_RDWR)
old = termios.tcgetattr(fd)
lines = None
try:
tty.setraw(fd)
if mode == "report":
lines = report(fd)
elif mode == "keylog":
keylog(fd)
else:
raise SystemExit(f"unknown mode: {mode}")
finally:
termios.tcsetattr(fd, termios.TCSADRAIN, old)
os.close(fd)
if lines:
print("terminal state as seen from this pane:")
print("\n".join(lines))
if __name__ == "__main__":
main()
@@ -1,15 +0,0 @@
# One-shot reset of terminal state a TUI may have left behind (kitty keyboard
# flags, modifyOtherKeys, application keypad/cursor, mouse/focus reporting,
# alternate screen, termios). fish re-enables the modes it wants at the next
# prompt, so this is safe to run any time.
#
# Diagnostic value: if this cures the lag, the stuck state was below fish
# (run lag-triage next time to find which mode). If only `exec fish` cures
# it, the lag is inside the fish process itself.
printf '\e[<9u\e[=0;1u'
printf '\e[>4;0m'
printf '\e>\e[?1l'
printf '\e[?1000l\e[?1001l\e[?1002l\e[?1003l\e[?1005l\e[?1006l\e[?1015l\e[?1016l\e[?1004l\e[?2026l'
printf '\e[?1049l'
stty sane
echo "terminal state reset — if typing still lags, run lag-triage (before exec fish!)"
@@ -1,151 +0,0 @@
#!/usr/bin/env python3
"""Deterministic repro of permanent fish reader degradation (fish 4.8.1).
This is evidence for an upstream fish-shell report, and the proof behind the
fish-no-query-term wrapper in presets/programs/zellij.nix. Not installed by
the nix module; run directly: python3 upstream_repro.py [queryterm|noqueryterm]
Finding: with the query-term feature enabled (latched from the startup env,
which is fish's default), fish sends OSC 11 + CPR (\e[6n) + DA1 (\e[0c)
after every external command and waits for the replies. If the terminal
fails to reply during ONE such cycle -- even though it answered every query
before and answers every query after -- that fish process's interactive
reader is PERMANENTLY degraded: keystrokes are no longer echoed (>3s each,
never recovers). In production this happens when zellij drops/mis-relays a
reply during TUI teardown or heavy output (cf. zellij-org/zellij#5158), and
it presents as permanent typing lag cured only by replacing the process.
With fish_features=no-query-term in the startup environment, the same
sequence has zero effect (~35ms echo throughout).
Phases:
A. terminal answers all queries -> echo ~35ms (both variants)
B. replies dropped for one command -> queryterm: echo dead, permanently
C. replies restored, another command -> queryterm: still dead
"""
import os, pty, re, select, subprocess, sys, time, fcntl, termios
VARIANT = sys.argv[1] if len(sys.argv) > 1 else "queryterm"
env = dict(os.environ)
env["TERM"] = "xterm-ghostty"
env["ZELLIJ"] = "0"
env["ZELLIJ_SESSION_NAME"] = "repro"
env["FISH_DEBUG"] = "term-support"
env["FISH_DEBUG_OUTPUT"] = f"/tmp/fish-lagrepro-{VARIANT}.log"
env.pop("fish_features", None)
if VARIANT == "noqueryterm":
env["fish_features"] = "no-query-term"
master, slave = pty.openpty()
# give it a size
fcntl.ioctl(master, termios.TIOCSWINSZ, b"\x00\x28\x00\x78\x00\x00\x00\x00")
proc = subprocess.Popen(
["fish", "-i", "--no-config"],
stdin=slave, stdout=slave, stderr=slave, env=env,
preexec_fn=lambda: (os.setsid(), fcntl.ioctl(0, termios.TIOCSCTTY, 0)),
close_fds=True,
)
os.close(slave)
RESPOND = True
transcript = []
def respond(data):
"""Answer terminal queries the way a well-behaved terminal would."""
out = b""
for m in re.finditer(rb"\x1b\[6n", data):
out += b"\x1b[40;1R" # CPR
for m in re.finditer(rb"\x1b\[0?c", data):
out += b"\x1b[?62;22c" # DA1
for m in re.finditer(rb"\x1b\[\?u", data):
out += b"\x1b[?0u" # kitty flags
for m in re.finditer(rb"\x1b\]11;\?", data):
out += b"\x1b]11;rgb:2828/2828/2828\x1b\\" # OSC 11
for m in re.finditer(rb"\x1b\[>0?q", data):
out += b"\x1bP>|ghostty 1.3.1\x1b\\" # XTVERSION
for m in re.finditer(rb"\x1bP\+q[0-9a-fA-F;]+\x1b\\", data):
out += b"\x1bP0+r\x1b\\" # XTGETTCAP: not found
for m in re.finditer(rb"\x1b\[\?(\d+)\$p", data):
out += b"\x1b[?%s;2$y" % m.group(1) # DECRQM: reset
return out
def pump(timeout):
"""Read fish output for `timeout` seconds, answering queries if RESPOND."""
buf = b""
end = time.monotonic() + timeout
while time.monotonic() < end:
r, _, _ = select.select([master], [], [], 0.03)
if r:
try:
data = os.read(master, 65536)
except OSError:
return buf
buf += data
transcript.append(data)
if RESPOND:
reply = respond(data)
if reply:
os.write(master, reply)
return buf
def send(s):
os.write(master, s if isinstance(s, bytes) else s.encode())
def measure_echo(chars, settle=0.1):
"""Send chars one at a time; measure time until each is echoed."""
results = []
for ch in chars:
pump(settle)
t0 = time.monotonic()
send(ch)
deadline = time.monotonic() + 3.0
latency = None
buf = b""
while time.monotonic() < deadline:
buf += pump(0.02)
if ch.encode() in buf:
latency = (time.monotonic() - t0) * 1000
break
results.append((ch, latency))
return results
print(f"=== variant: {VARIANT} ===")
pump(1.2) # startup, queries answered
send("echo warmup\r")
pump(0.8)
print("phase A: terminal responsive, echo latency per key:")
for ch, ms in measure_echo("abcde"):
print(f" {ch}: {ms:.0f} ms" if ms else f" {ch}: NO ECHO in 3s")
send("\x15") # ctrl-u clear line
pump(0.3)
# Run external command, then STOP answering queries (simulate lost relay)
send("sh -c true\r")
time.sleep(0.05)
RESPOND = False
pump(1.0)
print("phase B: after external command with query replies DROPPED:")
for ch, ms in measure_echo("fghij"):
print(f" {ch}: {ms:.0f} ms" if ms else f" {ch}: NO ECHO in 3s")
send("\x15")
pump(0.3)
# Does it persist across further commands, with responses restored?
RESPOND = True
send("sh -c true\r")
pump(1.0)
print("phase C: responses restored, after another external command:")
for ch, ms in measure_echo("klmno"):
print(f" {ch}: {ms:.0f} ms" if ms else f" {ch}: NO ECHO in 3s")
send("\x15")
pump(0.2)
send("exit\r")
pump(0.5)
try:
proc.wait(timeout=3)
except subprocess.TimeoutExpired:
proc.kill()
@@ -42,8 +42,13 @@ in
};
cmd_duration = {
min_time = 5001;
show_notifications = if pkgs.stdenv.hostPlatform.isLinux then false else true;
min_time_to_notify = 30000;
# Must stay off. On macOS the notification blocks the prompt for ~2s,
# and fish's $CMD_DURATION keeps its value until the next command
# actually runs. So after any long command (every TUI session),
# every prompt redraw -- Enter on an empty line, vi mode switches,
# repaints -- re-sent the notification and froze the shell for 2s.
# That was the "post-TUI typing lag" (see docs/CHANGELOG.md).
show_notifications = false;
format = "[$duration]($style) ";
};
custom = {
@@ -10,31 +10,13 @@ let
cfg = config.nmasur.presets.programs.zellij;
# fish latches feature flags from its startup ENVIRONMENT before config.fish
# runs, so the `set -gx fish_features no-query-term` in config.fish only
# protects CHILD fish processes — which is why subshells/exec fish were always
# immune to the post-TUI typing lag while zellij-spawned pane shells were not.
# With query-term latched on, fish sends OSC 11 + CPR + DA1 queries after
# every command and waits for replies; if zellij fails to relay even one
# reply (a race during TUI teardown or heavy output), that fish process's
# reader is PERMANENTLY degraded — reproduced deterministically in a PTY
# harness on fish 4.8.1 (see docs/CHANGELOG.md 2026-08-29 and
# presets/programs/lag-triage/upstream_repro.py). Spawning fish with the
# variable already exported makes every pane shell immune.
# Flight recorder for the still-unsolved post-TUI typing lag: sampling the
# process CURES the lag (a wedged thread gets kicked loose), so the only way
# to observe it is a recorder that is already running before the lag starts.
# Armed by `touch ~/.local/state/lag-triage/RECORD`; new panes then log
# fish's reader/thread internals to ~/.local/state/lag-triage/flight/.
# Remove the RECORD file to disable (zero overhead when off).
# runs, so `set -gx fish_features no-query-term` in config.fish only affects
# child fish processes. Spawning pane shells with the variable already
# exported keeps fish from sending terminal queries (OSC 11, CPR, DA1) that
# zellij may fail to relay; on fish 4.8.1 a single missed reply was shown to
# degrade that shell's reader for good.
fish-no-query-term = pkgs.writeShellScriptBin "fish-no-query-term" ''
export fish_features=no-query-term
dir="$HOME/.local/state/lag-triage"
if [ -e "$dir/RECORD" ]; then
mkdir -p "$dir/flight"
find "$dir/flight" -type f -mtime +3 -delete 2>/dev/null
export FISH_DEBUG='reader,term-support,proc-termowner,iothread,fd-monitor,topic-monitor'
export FISH_DEBUG_OUTPUT="$dir/flight/fish-$(date +%Y%m%d-%H%M%S)-$$.log"
fi
exec ${lib.getExe pkgs.fish} "$@"
'';
@@ -178,6 +160,10 @@ in
show_startup_tips = false;
# For nested sessions, automatically descend into the session without
# fullscreening it.
nested_session_handling = "descend";
keybinds = {
locked = lib.mkIf (pkgs.stdenv.hostPlatform.isLinux && !config.programs.ghostty.enable) {
# For servers that I SSH into, use c-a-G to unlock
@@ -63,7 +63,6 @@ in
git.enable = lib.mkDefault true;
helix.enable = lib.mkDefault true;
jujutsu.enable = lib.mkDefault true;
lag-triage.enable = lib.mkDefault true;
lazygit.enable = lib.mkDefault true;
# neovim.enable = lib.mkDefault true;
nix-index.enable = lib.mkDefault true;