[1.6.1] Chart output corrupts after a few minutes on macOS?

[quinncomendant]

I recently upgraded ttyplot on macOS from 1.5.2 to 1.6.1 (via brew upgrade), and the new version displays some misplaced character artifacts in the chart after a few minutes of running. I use this command:

ping -i .1 8.8.8.8 | sed -u 's/^.*time=//g; s/ ms//g; s/^.*timeout.*$//g' | ttyplot -t "ping to 8.8.8.8" -u ms

I've run the first part of the command simutaneously in a separate window to view the input data to ensure it is not causing ttyplot to become corrupted, and it does not: each line contains just one numeric value.

The chart starts out displaying correctly:

But after a few minutes, once the variation in values changes enough for ttyplot to rescale the chart, it can look like this:

Since this was not happening with ttyplot version 1.5.2, it must be something with the new version (which has a new plotting algorithm?).

[hartwork]

Hi @quinncomendant, thanks for the report! I do not have access to macOS outside of CI personally, running your command for 5 minutes (with multiple re-scales) on Linux does not show anything off for me. Is there a chance that you could:

• try the same with version 1.6.1 on Linux?
• try the same with version 1.6.1 on other macOS hardware?
• try the same with version 1.6.1 compiled from source on macOS rather than using Homebrew binaries?
• fabricate a reproducer using something like { echo "1 2 3"; sleep 1; echo "4 5 6"; } | ttyplot so that we can get rid of the influence of ping and get stable results?
• investigate the influence of runtime more, e.g. how much minimum and/or maximum runtime it takes for you to see the effect?

[quinncomendant]

Hi @hartwork! Thanks for the quick reply with suggestions. Yeah, if your internet is stable, ping may not reveal the issue.

In order to reproduce this, you need to generate values that:

• are in a quantity great enough to saturate the x axis (so the screen is full horizontally),
• and contain a range of values such that the scale of the chart will need to redraw occasionally.

Here's a test case you can use to reproduce the issue:

1. Set your terminal window to about 90 columns by 20 rows.
2. Run this:

for i in 13.669 13.669 13.669 17.235 17.235 17.235 17.235 17.235 17.235 17.235 10.831 10.831 10.831 10.831 10.831 10.831 93.902 93.902 93.902 93.902 93.902 93.902 93.902 93.902 2.984 2.984 2.984 2.984 2.984 2.984 2.984 6.528 6.528 6.528 6.528 6.528 6.528 6.528 6.528 5.143 5.143 5.143 5.143 5.143 5.143 5.143 5.143 8.741 8.741 8.741 8.741 8.741 8.741 8.741 8.741 2.310 2.310 2.310 2.310 2.310 2.310 2.310 10.943 10.943 10.943 10.943 10.943 10.943 10.943 10.943 18.117 18.117 18.117 18.117 18.117 18.117 18.117 18.117 11.670 11.670 11.670 11.670 11.670 11.670 11.670 11.670 5.296 5.296 5.296 5.296 5.296 5.296 5.296 5.296 8.840 8.840 8.840 8.840 8.840 8.840 8.840 8.840 7.446 7.446 7.446 7.446 7.446 7.446 7.446 7.446 7.446 6.027 6.027 6.027 6.027 6.027 6.027 6.027 6.027 4.598 4.598 4.598 4.598 4.598 4.598 4.598 4.598 4.598 8.156 8.156 8.156 8.156 8.156 8.156 8.156 8.156 8.156 6.724 6.724 6.724 6.724 6.724 6.724 6.724 6.724 5.336 5.336 5.336 5.336 5.336 5.336 5.336 5.336 5.336 8.920 8.920 8.920 8.920 8.920 8.920 8.920 8.920 2.485 2.485 2.485 2.485 2.485 2.485 2.485 2.485 2.485 6.056 6.056 6.056 6.056 6.056 6.056 6.056 6.056 3.284 3.284 3.284 3.284 3.284 3.284 3.284 3.284 6.840 6.840 6.840 6.840 6.840 6.840 6.840 6.840 6.840 5.382 5.382 5.382 5.382 5.382 5.382 5.382 5.382 3.986 3.986 3.986 3.986 3.986 3.986 3.986 3.986 3.986 2.608 2.608 2.608 2.608 2.608 2.608 2.608 2.608 1.134 1.134 1.134 1.134 1.134 1.134 1.134 1.134 1.134 4.777 4.777 4.777 4.777 4.777 4.777 4.777 4.777 3.337 3.337 3.337 3.337 3.337 3.337 3.337 3.337 3.337 6.924 6.924 6.924 6.924 6.924 6.924 6.924 6.924 50.511 50.511 50.511 50.511 50.511 50.511 50.511 50.511 50.511 4.095 4.095 4.095 4.095 4.095 4.095 4.095 4.095 2.677 2.677 2.677 2.677 2.677 2.677 2.677 2.677 2.677 1.278 1.278 1.278 1.278 1.278 1.278 1.278 1.278 3.443 3.443 3.443 3.443 3.443 3.443 3.443 3.443 3.443 7.014 7.014 7.014 7.014 7.014 7.014 7.014 7.014 5.616 5.616 5.616 5.616 5.616 5.616 5.616 5.616 5.616 7.731 7.731 7.731 7.731 7.731 7.731 7.731 7.731 7.731 6.371 6.371 6.371 6.371 6.371 6.371 6.371 6.371 4.914 4.914 4.914 4.914 4.914 4.914 4.914 4.914 4.914 3.492 3.492 3.492 3.492 3.492 3.492 3.492 3.492 2.086 2.086 2.086 2.086 2.086 2.086 2.086 2.086 2.086 0.660 0.660 0.660 0.660 0.660 0.660 0.660 0.660 4.234 4.234 4.234 4.234 4.234 4.234 4.234 4.234 4.234 2.853 2.853 2.853 2.853 2.853 2.853 2.853 2.853 2.853 1.382 1.382 1.382 1.382 1.382 1.382 1.382 1.382 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 8.607 8.607 8.607 8.607 8.607 8.607 8.607 8.607 2.206 2.206 2.206 2.206 2.206 2.206 2.206 2.206 2.206 5.747 5.747 5.747 5.747 5.747 5.747 5.747 5.747 7.980 7.980 7.980 7.980 7.980 7.980 7.980 7.980 6.509 6.509 6.509 6.509 6.509 6.509 6.509 6.509 6.509 5.088 5.088 5.088 5.088 5.088 5.088 5.088 5.088 5.088 3.682 3.682 3.682 3.682 3.682 3.682 3.682 3.682 2.264 2.264 2.264 2.264 2.264 2.264 2.264 2.264 2.264 5.868 5.868 5.868 5.868 5.868 5.868 5.868 5.868 4.424 4.424 4.424 4.424 4.424 4.424 4.424 4.424 4.424 7.988 7.988 7.988 7.988 7.988 7.988 7.988 7.988 6.607 6.607 6.607 6.607 6.607 6.607 6.607 6.607 6.607 5.181 5.181 5.181 5.181 5.181 5.181 5.181 5.181 8.746 8.746 8.746 8.746 8.746 8.746 8.746 8.746 8.746; do echo $i; sleep 0.1; done | ttyplot

This prints a value to the screen every 1/10 seconds. After about 12 seconds, the screen will look like this:

[hartwork]

Hi @quinncomendant, thanks for the update! Neither after 12 seconds nor anywhere between start and finish anything like that appears for me with 1.6.1 on (Gentoo) Linux, my terminal was exactly 90x20 in Terminator. Could you try with Linux or from source or on different (macOS) hardware?

[hartwork]

@quinncomendant update: no corruptions like that with 90x20 in Terminator on Debian bookworm with ttyplot 1.6.1 built from source, either.

[edgar-bonet]

Hi @quinncomendant, and thanks for the report. I couldn't reproduce the issue either. I am using:

• ttyplot 1.6.1 built from sources
• Gnome Terminal @ 90×20
• Ubuntu 22.04

[quinncomendant]

I did more experiments. The issue only occurs on macOS, when using iTerm version 3.5.0beta18, running ttyplot v1.6.1.

• The problem occurs on a clean install of macOS + iTerm 3.5.0beta18 + ttyplot 1.6.1.
• No problem when using the stable version of iTerm (3.4.23).
• No problem when using the standard macOS Terminal.app.
• No problem when running ttyplot 1.6.1 on an Ubuntu server connected from macOS via SSH using iTerm.
• No problem when using ttyplot 1.5.2 with iTerm.
• I've ruled out my choice of font (Iosevka) causing the issue; the problem continues even when using the default Menlo.

I know terminal emulation can be difficult to troubleshoot because of all the variables on multiple layers. I would be happy to just blame iTerm because it's a huge ball of wax. However, the fact that ttyplot v1.5.2 works, and v1.6.1 does not, in identical environments, is evidence that something changed that made ttyplot less resilient to working well in obscure environments, namely, mine.

Here's a screen recording showing v1.5.2 and v1.6.1 running side-by-side.

[hartwork]

@quinncomendant thanks a lot for investigating further and sharing your findings and the recording! My personal key takeway from what you shared is that the combination of local operation + iTerm2 + ttyplot 1.6.1 is the problem. (I'm assuming here that you already are running the latest version 3.4.23 of iTerm2 so that we're not dealing with already fixed bugs in iTerm, either something novel or outside of iTerm2?) To me it sounds like the local NCurses layer could be involved (because with SSH part of NCurses is no longer local). There were two bigger pull requests between 1.5.1 and 1.6.0 that changed things in the NCurses area:

• Get signal handling into the land of defined behavior (fixes #62) + fix terminal resize issues + fix stuck clock display + fix mis-render for empty input #98
• Refactor the handling of input data (+ add minimal support for negative input values) #115

What I am guessing right now is the symptom would go away adding (or re-adding) "full-screen" redrawing somewhere. I'm ignoring the question here whether ttyplot or iTerm would be at fault then then but that's at least an area where things changed and where we could experiment to see if it makes the symptoms go away. The pull requests are not small and not trivial but maybe you'd still see something that stands out to you as a candidate for the source of that problem. Could you have a quick look at the two and see if something catches your intention for a potential cause or for something with potential for experimentation?

[edgar-bonet]

@quinncomendant: Some random ideas...

• What is $TERM when running a local shell on iTerm?
• what is $TERM when running an ssh session on iTerm?
• are the relevant macOS and Ubuntu terminfo files different?
• if so, what happens if you use the Ubuntu terminfo file on your mac?

[quinncomendant]

I tested this again on an older MacBook which I recently reinstalled the OS, so it has a virgin terminal environment (never used, never customized). I installed iTerm and ttyplot for the first time. The issue occurs there!

Also, I discovered it only occurs with the beta version of iTerm (3.5.0beta18), and not the stable release (3.4.23).

The beta version of iTerm is far ahead of the stable release, so I've been using it everywhere. I think the iTerm team even recommends it. Searching the changelog of the beta releases I can see many dozens of fixes related to drawing text and screen management, so it's no surprise that there is a difference.

@hartwork, I checked those two pull requests and briefly scanned the commits, but I'm not familiar enough with ttyplot's design to know which might be more the cause. If I had to guess on a hunch, it seems like c150198 has the most changes related to screen redraw, so we could start there? If there any way I can get a copy of the source code before these pull requests and merge them myself? (I'm not very familiar with GitHub.)

@edgar-bonet, I can do the random ideas you suggested, but I think it may not be useful since I already discovered ttyplot 1.6.1 does work fine locally on macOS when using the default Terminal.app or the stable version of iTerm, it only occurs with the beta version of iTerm. It's not environmental differences that cause the issue.

[edgar-bonet]

@quinncomendant:

If there any way I can get a copy of the source code before these pull requests and merge them myself?

The Web UI way:

• browse https://github.com/tenox7/ttyplot/tree/162fefe1ba7ca439128e7d4aadb07033e3f9a12a
• click on the green “Code” button near the top right of the page
• click on “Download ZIP”

The git way:

git clone https://github.com/tenox7/ttyplot.git
cd ttyplot
git checkout c150198^  # the “^” suffix means “parent of”

Note that git can be very helpful for bisecting the history and get the exact commit that broke something:

git bisect start
git bisect good 1.5.2
git bisect bad 1.6.1

At this point git checks out an intermediate revision and asks you to test it. You reply by typing either git bisect good or git bisect bad. Repeat. After O(log(n)) steps, git tells you the ID of the commit that caused the breakage.

It's not environmental differences that cause the issue.

That's not certain. Ttyplot relies on ncurses, and ncurses uses the TERM environment variable and the terminfo database to know what escape sequences it has to give to the terminal to get a specific result. Terminal.app uses a different terminfo entry from iTerm. When you ssh to Ubuntu, ttyplot uses the terminfo database installed on the Ubuntu server, instead of the one on your Mac. If the iTerm entries in these databases differ, that could lead to different behaviors.

[hartwork]

@edgar-bonet suggesting git bisect is a great idea, didn't think of in this context yet, but should have.

@quinncomendant one thing I'm wondering is if 1.5.2 works for you and 1.6.1 how did you install each and where do they get ther (N)Curses from — from Homebrew or from stock macOS? Both of these versions? I see potential for more insights and variance there, maybe it can be pinned down further. To re-summarize, so far we have that it needs all of:

• macOS (not Linux)
• iTerm2 3.5.0beta18 (not stable 3.4.23)
• ttyplot 1.6.1 (not 1.5.2)
• local (not through SSH)

So (1) from source and/or homebrew and (2) the precise source and version of (N)Curses could give additional insights.

[quinncomendant]

@edgar-bonet Thanks for the tip to use git bisect, that's super cool! The process of bisecting and testing went smoothly until a commit that failed to compile. I was unable to git bisect skip because it was the last commit that needed testing. Here's the output at the end of the git bisect process:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
c73270ccfb74cf2dddfe9526f68645630b43f9f5
61c2cf5ccc4a951d5b6bf4f9caf7ee5f75c64d3e
We cannot bisect more!

I manually checkout’ed a series of commits in this range and tested them one-by-one, with the following results (in order from newest commits to older):

• 61c2cf5 compiles but runs with corrupt output 🐞
• c73270c fails to compile (screenshot with errors) 💣
• 07ce2f9 compiles and runs successfully ✅

This is confirms what git bisect already said, that the bug was introduced in c73270c or 61c2cf5.

It's not environmental differences that cause the issue.

That's not certain. Ttyplot relies on ncurses, and ncurses uses the TERM environment variable […]

Throughout my testing, $TERM has always been xterm-256color everywhere. I still don't understand how my environment could be different between two ttyplot versions, but maybe I missed your point.

@hartwork As far as I can tell there is only one ncurses installed: the one included with the macOS developer tools. According to otool -L, all the versions I've tested use the same ncurses library, including the one installed by Homebrew (which just uses make during its install process).

❯ ./ttyplot -v
ttyplot 1.5.2

❯ otool -L ./ttyplot
./ttyplot:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

❯ /opt/homebrew/bin/ttyplot -v
ttyplot 1.6.1

❯ otool -L /opt/homebrew/bin/ttyplot
/opt/homebrew/bin/ttyplot:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

[hartwork]

Hi @quinncomendant, thanks for investigating more and sharing your findings with us!
I believe what you found means the answer may be be somewhere in c73270c, which notably switches from ncurses to ncursesw. I'm surprised that ttyplot links to /usr/lib/libncurses.5.4.dylib without any "w" in it on your system. Could you share the output of pkg-config --cflags ncursesw ; pkg-config --libs ncursesw?

If we're looking at some flavor of memory corruption, maybe compiling with AddressSanitizer and UndefinedBehaviorSanitizer and running again could lead to insights:

env {C,LD}FLAGS=-fsanitize=address,undefined make clean all
[..data piping here..] | MallocNanoZone=0 ./ttyplot [..arguments here..]

If it now crashes at runtime, that would be a first clue.

@edgar-bonet any ideas what in c73270c it could possible be (if anything)? I remember because of the global static we were sure that all memory is zero-initialized. Anything else it could possible be?