git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Prematurely-closed stdin using async NodeJS to smudge large file on Mac

From
SHStephen Hicks <stephenhicks@gmail.com>
Date
Dec 15, 2022, 03:59 UTC
Message-ID
<CAKNkOnNAvV0tTkQXyRXjTiiUqPpu2cJSg5emrODu3AoGm79v+A@mail.gmail.com>

I recently observed some odd behavior when setting up a repo on a new computer. I isolated it to some weird behavior where git seems to be closing stdin prematurely.

Repro steps: ``` # Make a new repo mkdir smudge-test cd smudge-test git init

# Grab the first directory on the path (NOTE: we'll write to $BIN/smudge) BIN=${PATH%%:*}

# Set up the smudge filter
echo 'file filter=smudge' > .gitattributes
cat >> .git/config <<EOF
[filter "smudge"]
  clean = smudge
  smudge = smudge
  required
EOF
# Write the smudge script into $BIN/smudge
cat > $BIN/smudge <<EOF
#!/usr/bin/env node
const fs = require('fs');
fs.promises.readFile('/dev/stdin').then(src => {
  console.error('Read ' + src.length + ' bytes');
  process.exit(1);
});
EOF
chmod +x $BIN/smudge

# Make a 2mb file to smudge head -c 2000000 /dev/random > file

# Add it to the repo git add . ```

What I see is that every time I run `git add .`, it quotes me a different number of bytes read. If I truncate the file down to (say) 1mb, it works consistently every time (note: the add still fails because smudge exits 1, but the bytes read is consistent, which is what I'm looking for). The problems seem to start somewhere around 1.2mb or 1.3mb for me.

Changing the NodeJS script to use `fs.readFileSync` seems to fix it, so it appears to be something peculiar to how NodeJS handles the main task exiting before all of its async work is done (FWIW this is my current workaround). Piping the file to smudge directly (rather than via git) works consistently as well, so it's apparently an interaction with how git is handling the pipe. I also put together a quick shell script as a wrapper. If the shell script `cat`s stdin to a temp file first and redirects the temp file to NodeJS, it works consistently. If it just redirects /dev/stdin directly to NodeJS, it's inconsistent.

I'm running git 2.39.0 freshly-installed via MacPorts on macOS Monterey 12.6.1, Darwin Kernel 21.6.0, Node v18.12.1.

Message 1 of 1 in “Prematurely-closed stdin using async NodeJS to smudge large file on Mac”
  1. Stephen HicksDec 15, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.