Re: [kernel.org users] Re: auto-packing on kernel.org? please?

Nick Hengeveld <nickh@reactrix.com> writes:

Notice I did not say "we do not have yet but we will need" -- I just said "we do not have yet".

The assumption, which is the property the suggested packing strategy has, is that older objects that are needed to complete the history leading to the current tip are packed in those n-month/n-week packs, so if we do not have them we would likely be needing them, although we might not have walked that far back in history yet.

The previous "packing strategy" picture was certainly too simplified. Obviously we would not want to repack everything every week for different periods all the way back -- we would want to leave old huge pack untouched to help server side (and mirroring), so instead of having a single "pack optimization boundary", we would probably need some staggering as well for archived material.

This is a revised example.

1yr ----- 9mo -------- 6mo ---------- 3mo ------------------ 1mo ------------ 2wk -------- 1wk ----

We keep track of "the current heads and tags" for each week. Every week, we can do something like this:

 - rotate the record, and create a new one:
   mv .save/wk11 .save/wk12
   mv .save/wk10 .save/wk11
   mv .save/wk9 .save/wk10
   ...
   mv .save/wk0 .save/wk1
   find .git/refs -type f -print | xargs cat >.save/wk0
 
 - prepare a pack to allow a single pack fetch to bring a
   repository that had everything reachable from wk$N refs
   up-to-date to the current, for selected recent weeks (say N=1,
   2, 4, 12):

   for N in 1 2 4 12
   do
       name=$(git-rev-list --objects \
                 $(sed -e 's/^/^/' .save/wk$N) \
                 $(cat .save/wk0) |
              git-pack-object pack-) &&
       mv pack-$name.* .git/objects/pack/.
   done

   remove the pack files that we created this way last week from
   the repository (if the repository did not have any activity
   during the last week we would have created the same set of
   packs.  make sure we do not remove them).

 - except that, we keep the longest period (i.e. N=12 in this
   example) one every N weeks (that's how 1yr, 9mo, 6mo packs in
   the picture are kept).

This way, really old stuff (say, older than 3mo) will stay intact and will not be repacked, so people reasonably up-to-date (within 12 weeks in the example) need to fetch only one pack (and unpacked objects since the last pack optimization), but people without the ancient history need to go further back.