{"thread":{"id":"50842","subject":"test suite: why does git add large_file create a pack, rather than an object?","startedAt":"2019-03-30T14:13:19Z","lastAt":"2019-04-02T10:36:13Z","messageCount":4,"participants":["Philip Oakley","Junio C Hamano","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"372798","messageId":"f0c838f0-2f75-2b05-1aeb-3db4743ce89a@iee.org","threadId":"50842","inReplyTo":null,"subject":"test suite: why does git add large_file create a pack, rather than an object?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-03-30T14:13:15Z","receivedAt":"2019-03-30T14:13:19Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"I'm looking at the Git-for-Windows(GfW) >4Gb  large file problem \nfollowing Torsten and Thomas's work (and others). [1,2,3, etc.]\n\nI've added a few more changes to my branch [2] to get the zlib to \nproperly count past its 32bit limit when wrapped in the git_inflate \n(etc) wrapper code. [4]\n\nAt the moment I'm using an extended _test_ case that starts by adding a \n~5.1Gb file and then using verify-pack, which aborts with an error.\n\n         dd if=/dev/zero of=file bs=1M count=5100 &&\n         git config core.compression 0 &&\n         git config core.looseCompression 0 &&\n         git add file &&\n         git verify-pack -s .git/objects/pack/*.pack &&\n         git fsck --verbose --strict --full &&\n         ...\n\nIf however I simple execute the commands from the GfW bash, the added \nfile is stored as a blob object, rather than a pack.\n\nI'm at a loss to understand the reason for the change in behaviour \n[store file as pack, vs store as object] between running the code as a \ntest script and at the terminal. What am I missing?\n\nI have 'good' output from the test script on the WSL (and have \nidentified the packs' specific byte differences), but my gdb experience \nis limited so executing that test while within the test script meant I \ncouldn't start debugging there. Hence the direct execution from the \nterminal that raised the issue.\n\n-- \n\nPhilip\n[1] <20190131203842.633ztr4yckn7kl2d@tb-raspi4>\n[2] https://github.com/gitgitgadget/git/pull/115#issuecomment-4753008375\n[3] https://github.com/git-for-windows/git/issues/1848\n[4] https://github.com/PhilipOakley/git/tree/size_t2\n\n"},{"id":"372876","messageId":"xmqqftr2uhmg.fsf@gitster-ct.c.googlers.com","threadId":"50842","inReplyTo":"f0c838f0-2f75-2b05-1aeb-3db4743ce89a@iee.org","subject":"Re: test suite: why does git add large_file create a pack, rather than an object?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-01T10:47:35Z","receivedAt":"2019-04-01T10:47:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Philip Oakley <philipoakley@iee.org> writes:\n\n> At the moment I'm using an extended _test_ case that starts by adding\n> a ~5.1Gb file and then using verify-pack, which aborts with an error.\n>\n>         dd if=/dev/zero of=file bs=1M count=5100 &&\n>         git config core.compression 0 &&\n>         git config core.looseCompression 0 &&\n>         git add file &&\n>         git verify-pack -s .git/objects/pack/*.pack &&\n>         git fsck --verbose --strict --full &&\n>         ...\n>\n> If however I simple execute the commands from the GfW bash, the added\n> file is stored as a blob object, rather than a pack.\n>\n> I'm at a loss to understand the reason for the change in behaviour\n> [store file as pack, vs store as object] between running the code as a\n> test script and at the terminal. What am I missing?\n\nTo which test are you adding the above piece?  Perhaps one of those\nthat configures core.bigfilethreashold?\n"},{"id":"372893","messageId":"8850d755-07ce-d8d2-6e5c-88393fce34de@iee.org","threadId":"50842","inReplyTo":"xmqqftr2uhmg.fsf@gitster-ct.c.googlers.com","subject":"Re: test suite: why does git add large_file create a pack, rather than an object?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-04-01T15:09:51Z","receivedAt":"2019-04-01T15:09:55Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"hi Junio,\nOn 01/04/2019 11:47, Junio C Hamano wrote:\n> Philip Oakley <philipoakley@iee.org> writes:\n>\n>> At the moment I'm using an extended _test_ case that starts by adding\n>> a ~5.1Gb file and then using verify-pack, which aborts with an error.\n>>\n>>          dd if=/dev/zero of=file bs=1M count=5100 &&\n>>          git config core.compression 0 &&\n>>          git config core.looseCompression 0 &&\n>>          git add file &&\n>>          git verify-pack -s .git/objects/pack/*.pack &&\n>>          git fsck --verbose --strict --full &&\n>>          ...\n>>\n>> If however I simple execute the commands from the GfW bash, the added\n>> file is stored as a blob object, rather than a pack.\n>>\n>> I'm at a loss to understand the reason for the change in behaviour\n>> [store file as pack, vs store as object] between running the code as a\n>> test script and at the terminal. What am I missing?\n> To which test are you adding the above piece?  Perhaps one of those\n> that configures core.bigfilethreashold?\nThe test script (t-large-files-on-winows.sh: [1] below) was specific to \nthis debugging.\n\nI didn't set core.bigfilethreshold - Is that done (or unset) by the test \nsetup at all?\n\nIt does prompt me to check that all the bigfilethreshold checks are \nactually size_t, rather than a simple 'long'/uInt which would only be \n32bits on Windows and potentially a downcast comparison, resulting in \nmistaken bigfile actions because of the modulo 2^32 action.\n\nSo when I run the test script [1] on Windows I get my error from \nverify-pack, and the trash directory contains a single pack file.\nI tried doing the commands singly on a fresh repo, but that time found \nthat the add/verify produced a blob object (rather than a pack with one \nobject), so it got me wondering if I was testing like for like.\n\nWhen I tried using gdb at the add stage, with a break point, I got a \nback trace [2], and when run to completion it had the loose object, so I \nwas confused. (my fixup code is at [3])\n\nDoes the test_expect_success clear all the local/user config variable to \nsandbox the tests?\n\nPhilip\n\n\n[1] \nhttps://github.com/PhilipOakley/git/blob/size_t2/t/t-large-files-on-windows.sh \n\n\n    #!/bin/sh\n\n    test_description='test large file handling on windows'\n    . ./test-lib.sh\n\n    test_expect_success SIZE_T_IS_64BIT 'blah blubb' '\n\n    \ttest-tool zlib-compile-flags >zlibFlags.txt &&\n    \tdd if=/dev/zero of=file bs=1M count=5100 &&\n    \tgit config core.compression 0 &&\n    \tgit config core.looseCompression 0 &&\n    \tgdb git  &&\n    \tgit verify-pack -s .git/objects/pack/*.pack &&\n    \tgit fsck --verbose --strict --full &&\n    \tgit commit -m msg file &&\n    \tgit verify-pack -s .git/objects/pack/*.pack &&\n    \tgit log --stat &&\n    \tgit fsck --verbose --strict --full &&\n    \tgit repack -a -f &&\n    \tgit verify-pack -s .git/objects/pack/*.pack &&\n    \tgit verify-pack -v .git/objects/pack/*.pack &&\n    \tgit gc\n    '\n\n    test_done\n\n\n[2] Thread 1 hit Breakpoint 1, git_deflate (strm=0x138dac0, flush=0) at \nzlib.c:235\n235                     zlib_pre_call(strm);\n(gdb) bt\n#0  git_deflate (strm=0x138dac0, flush=0) at zlib.c:235\n#1  0x00000000005f4d11 in write_loose_object (oid=0x7ff4fda90070,\n     hdr=0x138f560 \"blob 5347737600\", hdrlen=16, buf=0x7fff0000,\n     len=5347737600, mtime=0) at sha1-file.c:1707\n#2  0x00000000005f50db in write_object_file (buf=0x7fff0000, len=5347737600,\n     type=0x6e3e9c <__ac_HASH_UPPER+20> \"blob\", oid=0x7ff4fda90070)\n     at sha1-file.c:1779\n#3  0x00000000005f5696 in index_mem (istate=0x78a3c0 <the_index>,\n     oid=0x7ff4fda90070, buf=0x7fff0000, size=5347737600, type=OBJ_BLOB,\n     path=0x13b6c2c \"file\", flags=1) at sha1-file.c:1901\n#4  0x00000000005f5aca in index_core (istate=0x78a3c0 <the_index>,\n     oid=0x7ff4fda90070, fd=4, size=5347737600, type=OBJ_BLOB,\n     path=0x13b6c2c \"file\", flags=1) at sha1-file.c:1975\n#5  0x00000000005f5c5d in index_fd (istate=0x78a3c0 <the_index>,\n     oid=0x7ff4fda90070, fd=4, st=0x138f860, type=OBJ_BLOB,\n     path=0x13b6c2c \"file\", flags=1) at sha1-file.c:2019\n#6  0x00000000005f5d9a in index_path (istate=0x78a3c0 <the_index>,\n     oid=0x7ff4fda90070, path=0x13b6c2c \"file\", st=0x138f860, flags=1)\n     at sha1-file.c:2040\n#7  0x000000000059e818 in add_to_index (istate=0x78a3c0 <the_index>,\n     path=0x13b6c2c \"file\", st=0x138f860, flags=0) at read-cache.c:763\n#8  0x000000000059ea04 in add_file_to_index (istate=0x78a3c0 <the_index>,\n     path=0x13b6c2c \"file\", flags=0) at read-cache.c:796\n#9  0x0000000000404b73 in add_files (dir=0x138f9c0, flags=0)\n     at builtin/add.c:378\n#10 0x0000000000405286 in cmd_add (argc=1, argv=0x1390658, prefix=0x0)\n     at builtin/add.c:534\n#11 0x0000000000402e86 in run_builtin (p=0x68a040 <commands>, argc=2,\n     argv=0x1390650) at git.c:422\n#12 0x000000000040326a in handle_builtin (argc=2, argv=0x1390650) at \ngit.c:654\n#13 0x00000000004034ac in run_argv (argcp=0x138fd90, argv=0x138fd38)\n     at git.c:708\n#14 0x00000000004038ca in cmd_main (argc=2, argv=0x1390650) at git.c:830\n#15 0x00000000004c2c5f in main (argc=3, argv=0x1390648) at common-main.c:45\n(gdb)\n\n[3] https://github.com/PhilipOakley/git/tree/size_t2\n\n"},{"id":"372973","messageId":"CACsJy8Bs6uso4oL6sSz9xC5gv-xcxkNc2KRcxX=eCT5KcMjSSQ@mail.gmail.com","threadId":"50842","inReplyTo":"8850d755-07ce-d8d2-6e5c-88393fce34de@iee.org","subject":"Re: test suite: why does git add large_file create a pack, rather than an object?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-04-02T10:35:44Z","receivedAt":"2019-04-02T10:36:13Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 1, 2019 at 10:10 PM Philip Oakley <philipoakley@iee.org> wrote:\n>\n> hi Junio,\n> On 01/04/2019 11:47, Junio C Hamano wrote:\n> > Philip Oakley <philipoakley@iee.org> writes:\n> >\n> >> At the moment I'm using an extended _test_ case that starts by adding\n> >> a ~5.1Gb file and then using verify-pack, which aborts with an error.\n> >>\n> >>          dd if=/dev/zero of=file bs=1M count=5100 &&\n> >>          git config core.compression 0 &&\n> >>          git config core.looseCompression 0 &&\n> >>          git add file &&\n> >>          git verify-pack -s .git/objects/pack/*.pack &&\n> >>          git fsck --verbose --strict --full &&\n> >>          ...\n> >>\n> >> If however I simple execute the commands from the GfW bash, the added\n> >> file is stored as a blob object, rather than a pack.\n> >>\n> >> I'm at a loss to understand the reason for the change in behaviour\n> >> [store file as pack, vs store as object] between running the code as a\n> >> test script and at the terminal. What am I missing?\n> > To which test are you adding the above piece?  Perhaps one of those\n> > that configures core.bigfilethreashold?\n> The test script (t-large-files-on-winows.sh: [1] below) was specific to\n> this debugging.\n>\n> I didn't set core.bigfilethreshold - Is that done (or unset) by the test\n> setup at all?\n>\n> It does prompt me to check that all the bigfilethreshold checks are\n> actually size_t, rather than a simple 'long'/uInt which would only be\n> 32bits on Windows and potentially a downcast comparison, resulting in\n> mistaken bigfile actions because of the modulo 2^32 action.\n>\n> So when I run the test script [1] on Windows I get my error from\n> verify-pack, and the trash directory contains a single pack file.\n> I tried doing the commands singly on a fresh repo, but that time found\n> that the add/verify produced a blob object (rather than a pack with one\n> object), so it got me wondering if I was testing like for like.\n>\n> When I tried using gdb at the add stage, with a break point, I got a\n> back trace [2], and when run to completion it had the loose object, so I\n> was confused. (my fixup code is at [3])\n\nStreaming a blob directly to a pack is done by index_stream(). I\nsuggest you force a crashwhen that function is called (from your test\nscript) then examine with gdb for more info. You should be able to see\nwhat's its caller (in case it's not index_fd), then perhaps you could\nadd a bunch of printtfs to show all the conditions that lead (or not\nlead) to that function?\n\nThere are some would_convert_ calls in index_fd(). Maybe some other\nconfig keys are affecting this.\n\nPS. I also don't know what index_stream_convert_blob() does. Not sure\nif it's really streaming to blob or streaming from somewhee to a\nconverter. You might want to check that too.\n-- \nDuy\n"}]}