git/list[1] front-page[2] threads[3] people[4] search[5] about
 

fatal: Out of memory, getdelim failed under NFS mounts

From
Yaroslav Halchenko <yoh@onerussian.com>
Date
Aug 9, 2017, 17:39 UTC
Message-ID
<20170809173928.h2ylvg5tp2p5inem@hopa.kiewit.dartmouth.edu>
Dear Git gurus,

More context (may be different issue(s)) could be found at http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/ but currently I am consistently reproducing it while running git (1:2.11.0-3 debian stretch build) within debian stretch singularity environment [1].

External system is Centos 6.9, and git 1.7.1 (and installed in modules 2.0.4) do not show similar buggy behavior.

NFS mounted partitions are bind mounted inside the sinularity space and when I try to do some git operations, I get that error inconsistently , e.g.

	yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin master
	fatal: Out of memory, getdelim failed
	error: git://github.com/datalad/datalad did not send all necessary objects
	yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin master
	fatal: Out of memory, getdelim failed
	error: git://github.com/datalad/datalad did not send all necessary objects
	yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin master
	From git://github.com/datalad/datalad
	 * branch              master     -> FETCH_HEAD
	fatal: Out of memory, getdelim failed

and some times it succeeds. So it smells that some race condition somewhere...?

any recommendations on how to pin point the "offender"? ;) Here is the trailer of one of the straced calls:

... [pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24 [pid 19713] stat(".git", {st_mode=S_IFDIR|0755, st_size=322, ...}) = 0 [pid 19713] lstat(".git/HEAD", {st_mode=S_IFREG|0644, st_size=41, ...}) = 0 [pid 19713] open(".git/HEAD", O_RDONLY) = 3 [pid 19713] read(3, "39f80454d31cfb691b006302b1f29dee"..., 255) = 41 [pid 19713] read(3, "", 214) = 0 [pid 19713] close(3) = 0 [pid 19713] lstat(".git/commondir", 0x7ffc1a571190) = -1 ENOENT (No such file or directory) [pid 19713] access(".git/objects", X_OK) = 0 [pid 19713] access(".git/refs", X_OK) = 0 [pid 19713] lstat(".git/commondir", 0x7ffc1a571120) = -1 ENOENT (No such file or directory) [pid 19713] open(".git/config", O_RDONLY) = 3 [pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0 [pid 19713] mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ba7894e7000 [pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257 [pid 19713] read(3, "", 524288) = 0 [pid 19713] close(3) = 0 [pid 19713] munmap(0x2ba7894e7000, 528384) = 0 [pid 19713] stat(".", {st_mode=S_IFDIR|0755, st_size=907, ...}) = 0 [pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24 [pid 19713] chdir(".") = 0 [pid 19713] getcwd("/ihome/yhalchen/datalad", 130) = 24 [pid 19713] lstat("/ihome/yhalchen/datalad", {st_mode=S_IFDIR|0755, st_size=907, ...}) = 0 [pid 19713] chdir("/ihome/yhalchen/datalad") = 0 [pid 19713] stat(".git", {st_mode=S_IFDIR|0755, st_size=322, ...}) = 0 [pid 19713] lstat(".git/commondir", 0x7ffc1a571140) = -1 ENOENT (No such file or directory) [pid 19713] access("/etc/gitconfig", R_OK) = -1 ENOENT (No such file or directory) [pid 19713] access("/ihome/yhalchen/.config/git/config", R_OK) = -1 ENOENT (No such file or directory) [pid 19713] access("/ihome/yhalchen/.gitconfig", R_OK) = -1 ENOENT (No such file or directory) [pid 19713] access(".git/config", R_OK) = 0 [pid 19713] open(".git/config", O_RDONLY) = 3 [pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0 [pid 19713] brk(0x2ba78afea000) = 0x2ba78afea000 [pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257 [pid 19713] read(3, "", 524288) = 0 [pid 19713] close(3) = 0 [pid 19713] access("/etc/gitconfig", R_OK) = -1 ENOENT (No such file or directory) [pid 19713] access("/ihome/yhalchen/.config/git/config", R_OK) = -1 ENOENT (No such file or directory) [pid 19713] access("/ihome/yhalchen/.gitconfig", R_OK) = -1 ENOENT (No such file or directory) [pid 19713] access(".git/config", R_OK) = 0 [pid 19713] open(".git/config", O_RDONLY) = 3 [pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0 [pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257 [pid 19713] read(3, "", 524288) = 0 [pid 19713] close(3) = 0 [pid 19713] open(".git/objects/pack", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 [pid 19713] fstat(3, {st_mode=S_IFDIR|0755, st_size=270, ...}) = 0 [pid 19713] getdents(3, /* 6 entries */, 524288) = 336 [pid 19713] access(".git/objects/pack/pack-fd9e70075570d8ec41f12605852f54f1cb9771a8.keep", F_OK) = -1 ENOENT (No such file or directory) [pid 19713] stat(".git/objects/pack/pack-fd9e70075570d8ec41f12605852f54f1cb9771a8.pack", {st_mode=S_IFREG|0444, st_size=10019975, ...}) = 0 [pid 19713] access(".git/objects/pack/pack-f1fc124e3aa1619d65a6ba56219f84871a762775.keep", F_OK) = -1 ENOENT (No such file or directory) [pid 19713] stat(".git/objects/pack/pack-f1fc124e3aa1619d65a6ba56219f84871a762775.pack", {st_mode=S_IFREG|0444, st_size=610330, ...}) = 0 [pid 19713] getdents(3, /* 0 entries */, 524288) = 0 [pid 19713] close(3) = 0 [pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24 [pid 19713] open(".git/objects/info/alternates", O_RDONLY|O_NOATIME|O_CLOEXEC) = -1 ENOENT (No such file or directory) [pid 19713] open(".git/objects/17", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) [pid 19713] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 17), ...}) = 0 [pid 19713] close(1) = 0 [pid 19713] exit_group(0) = ? [pid 19713] +++ exited with 0 +++ [pid 19710] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 19713 [pid 19710] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19713, si_uid=14076, si_status=0, si_utime=0, si_stime=0} --- [pid 19710] exit_group(1) = ? [pid 19710] +++ exited with 1 +++ <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 19710 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19710, si_uid=14076, si_status=1, si_utime=0, si_stime=2} --- exit_group(1) = ? +++ exited with 1 +++

PS please CC me in replies! Thanks in advance

[1] http://datasets.datalad.org/singularity/neurodebian-v2.1.img.tgz it is a tarball to extract/run with , I promise I did no evil in there ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Next: René Scharfe
Message 1 of 21 in “fatal: Out of memory, getdelim failed under NFS mounts”
  1. Yaroslav HalchenkoAug 9, 2017
  2. René ScharfeAug 10, 2017
  3. Yaroslav HalchenkoAug 10, 2017
  4. René ScharfeAug 10, 2017
  5. Junio C HamanoAug 10, 2017
  6. Yaroslav HalchenkoAug 10, 2017
  7. René ScharfeAug 10, 2017
  8. Jeff KingAug 10, 2017
  9. Yaroslav HalchenkoAug 10, 2017
  10. strbuf: clear errno before calling getdelim(3)René Scharfe, Aug 10, 2017
  11. Jeff KingAug 10, 2017
  12. Yaroslav HalchenkoAug 10, 2017
  13. Junio C HamanoAug 10, 2017
  14. Simon RuderichAug 11, 2017
  15. René ScharfeAug 11, 2017
  16. Simon RuderichAug 12, 2017
  17. René ScharfeAug 12, 2017
  18. René ScharfeAug 12, 2017
  19. Jeff KingAug 13, 2017
  20. René ScharfeAug 13, 2017
  21. Jeff KingAug 13, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.