git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: fetching packs and storing them as packs

From
Nicolas Pitre <nico@cam.org>
Date
Oct 27, 2006, 17:23 UTC
Message-ID
<Pine.LNX.4.64.0610271252390.11384@xanadu.home>
In-Reply-To
<7viri6i6uu.fsf@assigned-by-dhcp.cox.net>
On Thu, 26 Oct 2006, Junio C Hamano wrote:
Show 39 quoted lines
> I'd almost say "heavy repository-wide operations like 'repack -a
> -d' and 'prune' should operate under a single repository lock",
> but historically we've avoided locks and instead tried to do
> things optimistically and used compare-and-swap to detect
> conflicts, so maybe that avenue might be worth pursuing.
> 
> How about (I'm thinking aloud and I'm sure there will be
> holes -- I won't think about prune for now)...
> 
> * "repack -a -d":
> 
>  (1) initially run show-ref (or "ls-remote .") and store the
>      result in .git/$ref_pack_lock_file;
> 
>  (2) enumerate existing packs;
> 
>  (3) do the usual "rev-list --all | pack-objects" thing; this
>      may end up including more objects than what are reachable
>      from the result of (1) if somebody else updates refs in the
>      meantime;
> 
>  (4) enumerate existing packs; if there is difference from (2)
>      other than what (3) created, that means somebody else added
>      a pack in the meantime; stop and do not do the "-d" part;
> 
>  (5) run "ls-remote ." again and compare it with what it got in
>      (1); if different, somebody else updated a ref in the
>      meantime; stop and do not do the "-d" part;
> 
>  (6) do the "-d" part as usual by removing packs we saw in (2)
>      but do not remove the pack we created in (3);
> 
>  (7) remove .git/$ref_pack_lock_file.
> 
> * "fetch --thin" and "index-pack --stdin":
> 
>  (1) check the .git/$ref_pack_lock_file, and refuse to operate
>     if there is such (this is not strictly needed for
>     correctness but only to give an early exit);

I don't think this is a good idea. A fetch should always work irrespective of any repack taking place. The fetch really should have priority over a repack since it is directly related to the user experience. The repack can fail or produce suboptimal results if a race occurs, but the fetch must not fail for such a reason.

>  (2) create a new pack under a temporary name, and when
>      complete, make the pack/index pair .pack and .idx;

Actually this is what already happens if you don't specify a name to git-index-pack --stdin.

>  (3) update the refs.

So the actual race is the really small interval between the time the new pack+index are moved to .git/objects/pack/ and the moment the refs are updated. In practice this is probably less than a second. All that is needed here is to somehow go back to (2) if that interval occurs between (2) and (3).

Previous: Junio C HamanoNext: Nicolas Pitre
Message 32 of 51 in “fetching packs and storing them as packs”
  1. Nicolas PitreOct 26, 2006
  2. Eran TromerOct 26, 2006
  3. Linus TorvaldsOct 27, 2006
  4. Junio C HamanoOct 27, 2006
  5. Shawn PearceOct 28, 2006
  6. Junio C HamanoOct 28, 2006
  7. Linus TorvaldsOct 28, 2006
  8. Junio C HamanoOct 28, 2006
  9. Shawn PearceOct 28, 2006
  10. Shawn PearceOct 28, 2006
  11. Junio C HamanoOct 28, 2006
  12. Shawn PearceOct 29, 2006
  13. Junio C HamanoOct 29, 2006
  14. Shawn PearceOct 29, 2006
  15. Junio C HamanoOct 29, 2006
  16. Shawn PearceOct 29, 2006
  17. Linus TorvaldsOct 28, 2006
  18. Junio C HamanoOct 28, 2006
  19. Eran TromerOct 28, 2006
  20. Shawn PearceOct 29, 2006
  21. Jakub NarebskiOct 29, 2006
  22. Shawn PearceOct 29, 2006
  23. send-pack --keep: do not explode into loose objects on the receiving end.Junio C Hamano, Oct 29, 2006
  24. Shawn PearceOct 29, 2006
  25. Junio C HamanoOct 29, 2006
  26. Nicolas PitreOct 30, 2006
  27. Eran TromerOct 26, 2006
  28. Nicolas PitreOct 27, 2006
  29. Shawn PearceOct 27, 2006
  30. SeanOct 27, 2006
  31. Junio C HamanoOct 27, 2006
  32. Nicolas PitreOct 27, 2006
  33. Nicolas PitreOct 27, 2006
  34. Eran TromerOct 27, 2006
  35. Shawn PearceOct 27, 2006
  36. SeanOct 27, 2006
  37. Jakub NarebskiOct 27, 2006
  38. SeanOct 27, 2006
  39. Eran TromerOct 27, 2006
  40. Shawn PearceOct 27, 2006
  41. Alex RiesenOct 27, 2006
  42. Shawn PearceOct 27, 2006
  43. Alex RiesenOct 27, 2006
  44. Shawn PearceOct 27, 2006
  45. Nicolas PitreOct 27, 2006
  46. Petr BaudisOct 27, 2006
  47. J. Bruce FieldsOct 27, 2006
  48. Petr BaudisOct 27, 2006
  49. J. Bruce FieldsOct 27, 2006
  50. J. Bruce FieldsOct 27, 2006
  51. Junio C HamanoOct 27, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.