{"thread":{"id":"53359","subject":"Re: [PATCH 06/15] run-job: auto-size or use custom pack-files batch","startedAt":"2020-04-30T16:48:38Z","lastAt":"2020-04-30T20:13:55Z","messageCount":2,"participants":["Son Luong Ngoc","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":15},"messages":[{"id":"396670","messageId":"CAL3xRKcsa_P6X5Y+c2LWoftfjqEw9eheikrxfwXU=y6KuFHjtQ@mail.gmail.com","threadId":"53359","inReplyTo":null,"subject":"Re: [PATCH 06/15] run-job: auto-size or use custom pack-files batch","fromName":"Son Luong Ngoc","fromEmail":"sluongng@gmail.com","sentAt":"2020-04-30T16:48:24Z","receivedAt":"2020-04-30T16:48:38Z","isPatch":true,"sender":{"key":"sluongng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/26684313?v=4"},"body":"Hi Derrick,\n\nI have been reviewing these jobs' mechanics closely and have some questions:\n\n> The dynamic default size is computed with this idea in mind for\n> a client repository that was cloned from a very large remote: there\n> is likely one \"big\" pack-file that was created at clone time. Thus,\n> do not try repacking it as it is likely packed efficiently by the\n> server. Instead, try packing the other pack-files into a single\n> pack-file.\n>\n> The size is then computed as follows:\n>\n> batch size = total size - max pack size\n\nCould you please elaborate why is this the best value?\nIn practice I have been testing this out with the following\n\n> % cat debug.sh\n> #!/bin/bash\n>\n> temp=$(du -cb .git/objects/pack/*.pack)\n>\n> total_size=$(echo \"$temp\" | grep total | awk '{print $1}')\n> echo total_size\n> echo $total_size\n>\n> biggest_pack=$(echo \"$temp\" | sort -n | tail -2 | head -1 | awk '{print $1}')\n> echo biggest pack\n> echo $biggest_pack\n>\n> batch_size=$(expr $total_size - $biggest_pack)\n> echo batch size\n> echo $batch_size\n\nIf you were to run\n\n> git multi-pack-index repack --batch-size=$(./debug.sh | tail -1)\n\nthen nothing would be repack.\n\nInstead, I have had a lot more success with the following\n\n> # Get the 2nd biggest pack size (in bytes) + 1\n> $(du -b .git/objects/pack/*pack | sort -n | tail -2 | head -1 | awk '{print $1}') + 1\n\nI think you also used this approach in t5319 when you used the 3rd\nbiggest pack size\n\n> test_expect_success 'repack creates a new pack' '\n> (\n> cd dup &&\n> ls .git/objects/pack/*idx >idx-list &&\n> test_line_count = 5 idx-list &&\n> THIRD_SMALLEST_SIZE=$(test-tool path-utils file-size .git/objects/pack/*pack | sort -n | head -n 3 | tail -n 1) &&\n> BATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) &&\n> git multi-pack-index repack --batch-size=$BATCH_SIZE &&\n> ls .git/objects/pack/*idx >idx-list &&\n> test_line_count = 6 idx-list &&\n> test-tool read-midx .git/objects | grep idx >midx-list &&\n> test_line_count = 6 midx-list\n> )\n> '\n\nLooking forward to a re-roll of this RFC.\n\nCheers,\nSon Luong.\n"},{"id":"396692","messageId":"f7a193f8-8d5f-3f71-45b1-e117742df8f9@gmail.com","threadId":"53359","inReplyTo":"CAL3xRKcsa_P6X5Y+c2LWoftfjqEw9eheikrxfwXU=y6KuFHjtQ@mail.gmail.com","subject":"Re: [PATCH 06/15] run-job: auto-size or use custom pack-files batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-04-30T20:13:48Z","receivedAt":"2020-04-30T20:13:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/30/2020 12:48 PM, Son Luong Ngoc wrote:\n> Hi Derrick,\n> \n> I have been reviewing these jobs' mechanics closely and have some questions:\n> \n>> The dynamic default size is computed with this idea in mind for\n>> a client repository that was cloned from a very large remote: there\n>> is likely one \"big\" pack-file that was created at clone time. Thus,\n>> do not try repacking it as it is likely packed efficiently by the\n>> server. Instead, try packing the other pack-files into a single\n>> pack-file.\n>>\n>> The size is then computed as follows:\n>>\n>> batch size = total size - max pack size\n> \n> Could you please elaborate why is this the best value?\n\nThe intention was to repack everything _except_ the biggest pack,\nbut clearly that doesn't always work. There is some logic to \"guess\"\nthe size of the resulting pack that doesn't always reach the total\nbatch size, so nothing happens. More investigation is required here.\n\n> In practice I have been testing this out with the following\n> \n>> % cat debug.sh\n>> #!/bin/bash\n>>\n>> temp=$(du -cb .git/objects/pack/*.pack)\n>>\n>> total_size=$(echo \"$temp\" | grep total | awk '{print $1}')\n>> echo total_size\n>> echo $total_size\n>>\n>> biggest_pack=$(echo \"$temp\" | sort -n | tail -2 | head -1 | awk '{print $1}')\n>> echo biggest pack\n>> echo $biggest_pack\n>>\n>> batch_size=$(expr $total_size - $biggest_pack)\n>> echo batch size\n>> echo $batch_size\n> \n> If you were to run\n> \n>> git multi-pack-index repack --batch-size=$(./debug.sh | tail -1)\n> \n> then nothing would be repack.> \n> Instead, I have had a lot more success with the following\n> \n>> # Get the 2nd biggest pack size (in bytes) + 1\n>> $(du -b .git/objects/pack/*pack | sort -n | tail -2 | head -1 | awk '{print $1}') + 1\n> \n> I think you also used this approach in t5319 when you used the 3rd\n> biggest pack size\n\nThe \"second biggest pack\" is an interesting approach. At first glance it\nseems like we will stabilize with one big pack and many similarly-sized\npacks. However, even a small deviation in size is inevitable and will\ncause two or more packs to combine and create a \"new second biggest\"\npack.\n\n> Looking forward to a re-roll of this RFC.\n\nI do plan to submit a new version of the RFC, but it will look quite\ndifferent based on the feedback so far. I'm still digesting that\nfeedback and will take another attempt at it after I wrap up some other\nitems that have my attention currently.\n\nThanks!\n-Stolee\n\n\n"}]}