git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/8] pack-objects: produce a stable pack when --skip is given

From
Duy Nguyen <pclouds@gmail.com>
Date
Feb 5, 2016, 23:25 UTC
Message-ID
<CACsJy8CkQPX4zqW39gpODSkU+habM7TGTJ85MU-S0UuNPk9SCw@mail.gmail.com>
In-Reply-To
<xmqq1t8rdmky.fsf@gitster.mtv.corp.google.com>
On Sat, Feb 6, 2016 at 1:43 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 25 quoted lines
> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
>
>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> index 417c830..c58a9cb 100644
>> --- a/builtin/pack-objects.c
>> +++ b/builtin/pack-objects.c
>> @@ -2709,6 +2709,11 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
>>                       if (get_oid_hex(skip_hash_hex, &skip_hash))
>>                               die(_("%s is not SHA-1"), skip_hash_hex);
>>               }
>> +
>> +             /*
>> +              * Parallel delta search can't produce stable packs.
>> +              */
>> +             delta_search_threads = 1;
>>       }
>>
>>       argv_array_push(&rp, "pack-objects");
>
> A multi-threaded packing is _one_ source of regenerating the same
> pack for the same set of objects, but we shouldn't be tying our
> hands by promising it will forever be the _only_ source of it by
> doing things like this.  We may want to dynamically tweak the
> packing behaviour depending on the load of the minute and such for
> example.

You noticed that tying the behavior only happens when the user asks for it, right? I don't expect people to do resumable fetch/clone by default. There are tradeoffs to make and they decide it, we offer options. So, it does not really tie our hands in the normal case.

Show 11 quoted lines
> I think a more sensible approach for "resuming" is to attack cloning
> first.  Take a reasonable baseline snapshot periodically (depending
> on the activity level of the project, the interval may span between
> 12 hours to 2 weeks and you would want to make it configurable) to
> create a bundle, teach "clone" to check the bundle first and perform
> a resumable and bulk transfer for the stable part of the history
> (e.g. up to the latest tag or a branch that does not rewind, the set
> of refs to use as the stable anchors you would want to make
> configurable), and then fill the gap between that baseline snapshot
> and up-to-date state by doing another round of "git fetch" and then
> you'd have solved the most serious problem already.

_most_. On a troubled connection, fetch can fail as well, especially when the repo is not uptodate. What about shallow clone? I don't think you want to prepare snapshots for all depths (or some "popular" depths). Cloning full then cutting it shallow locally might work, but we're wasting bandwidth and disk space.

> This is an indication that the approach this series takes is taking
> us in a wrong direction.

The way I see it, the two approaches complement each other. Snapshots, like pack bitmaps, helps tremendously, but it has corner cases that can be covered by this.

-- 
Duy
Previous: Junio C HamanoNext: Junio C Hamano
Message 6 of 22 in “Resumable clone revisited, proof of concept”
  1. 0/8 Resumable clone revisited, proof of conceptNguyễn Thái Ngọc Duy, Feb 5, 2016
  2. 1/8 pack-objects: add --skip and --skip-hashNguyễn Thái Ngọc Duy, Feb 5, 2016
  3. Johannes SchindelinFeb 5, 2016
  4. 2/8 pack-objects: produce a stable pack when --skip is givenNguyễn Thái Ngọc Duy, Feb 5, 2016
  5. Junio C HamanoFeb 5, 2016
  6. Duy NguyenFeb 5, 2016
  7. Junio C HamanoFeb 6, 2016
  8. Duy NguyenFeb 8, 2016
  9. 3/8 index-pack: add --append-pack=<path>Nguyễn Thái Ngọc Duy, Feb 5, 2016
  10. 4/8 upload-pack: new capability to pass --skip* to pack-objectsNguyễn Thái Ngọc Duy, Feb 5, 2016
  11. Johannes SchindelinFeb 5, 2016
  12. 5/8 fetch-pack.c: send "skip" line to pack-objectsNguyễn Thái Ngọc Duy, Feb 5, 2016
  13. Eric SunshineFeb 7, 2016
  14. 6/8 fetch: add --resume-pack=<path>Nguyễn Thái Ngọc Duy, Feb 5, 2016
  15. 7/8 index-pack: --append-pack implies --strictNguyễn Thái Ngọc Duy, Feb 5, 2016
  16. Eric SunshineFeb 7, 2016
  17. 8/8 one ugly test to verify basic functionalityNguyễn Thái Ngọc Duy, Feb 5, 2016
  18. Elia PintoFeb 5, 2016
  19. Duy NguyenFeb 5, 2016
  20. Elia PintoFeb 5, 2016
  21. Johannes SchindelinFeb 5, 2016
  22. Elia PintoFeb 5, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.