git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [hacky PATCH 0/2] speeding up trivial for-each-ref invocations

From
ZheNing Hu <adlternative@gmail.com>
Date
Sep 9, 2021, 13:20 UTC
Message-ID
<CAOLTT8TSNtVqZrdYpRpUp-JfvHUpDAMEo5k4SkiLJUjMmefcLA@mail.gmail.com>
In-Reply-To
<YTehP58NfcnDSWEc@coredump.intra.peff.net>
Jeff King <peff@peff.net> 于2021年9月8日周三 上午1:28写道:
Show 52 quoted lines
>
> On Mon, Sep 06, 2021 at 09:30:45PM +0800, ZheNing Hu wrote:
>
> > Jeff King <peff@peff.net> 于2021年9月5日周日 下午8:49写道:
> > >
> > > On Sun, Sep 05, 2021 at 04:19:53PM +0800, ZheNing Hu wrote:
> > >
> > > > > In this version there are 2 patches, tested against 'git for-each-ref
> > > > > --format="%(objectname) %(refname)"' on a fully packed repo with 500k
> > > > > refs:
> > > > >
> > > >
> > > > Regarding this 500k refs, is there any way I can reproduce it?
> > >
> > > Try this in a clone of linux.git (or any other repo):
> > >
> > >   git rev-list HEAD |
> > >   head -500000 |
> > >   perl -lne 'print "create refs/foo/$. $_"' |
> > >   git update-ref --stdin
> > >
> > >   git pack-refs --all --prune
> > >
> >
> > Sorry, It seems that the above command is difficult to complete on my
> > machine (it took more than ten minutes). It may be stuck on git update-ref.
> > So I tried to reproduce it in a repo which containing 76K refs:
>
> Mine didn't take nearly that wrong, but it does depend on filesystem and
> disk performance. It's going to create 500k lock files in refs/foo. :)
>
> You can cheat a bit like this:
>
>   {
>     # grab existing packed refs; don't worry about peel lines or the
>     # header comment, we're producing a lowest-common denominator
>     # version of the file
>     grep '^[0-9a-f]' packed-refs
>
>     # now make our new fake refs
>     git rev-list HEAD |
>     head -500000 |
>     perl -lne 'print "$_ refs/foo/$."'
>   } >packed-refs.tmp
>   mv packed-refs.tmp packed-refs
>
>   # and now ask Git to repack to get everything sorted, etc
>   git pack-refs --all --prune
>
> It sounds like you were able to come up with a smaller version to play
> with anyway, but I enjoy coming up with such hacks. :)
>
Thanks, this method really works. :-)
> -Peff

-- ZheNing Hu

Previous: Jeff KingNext: Patrick Steinhardt
Message 25 of 26 in “speeding up trivial for-each-ref invocations”
  1. 0/2 speeding up trivial for-each-ref invocationsJeff King, Sep 4, 2021
  2. 1/2 ref-filter: hacky "streaming" modeJeff King, Sep 4, 2021
  3. ZheNing HuSep 5, 2021
  4. Jeff KingSep 5, 2021
  5. ZheNing HuSep 7, 2021
  6. Jeff KingSep 7, 2021
  7. ZheNing HuSep 9, 2021
  8. Jeff KingSep 10, 2021
  9. ZheNing HuSep 15, 2021
  10. ZheNing HuSep 15, 2021
  11. Jeff KingSep 16, 2021
  12. ZheNing HuSep 20, 2021
  13. Jeff KingSep 16, 2021
  14. Jeff KingSep 5, 2021
  15. ZheNing HuSep 7, 2021
  16. 2/2 ref-filter: implement "quick" formatsJeff King, Sep 4, 2021
  17. ZheNing HuSep 5, 2021
  18. Jeff KingSep 5, 2021
  19. ZheNing HuSep 6, 2021
  20. Junio C HamanoSep 7, 2021
  21. ZheNing HuSep 5, 2021
  22. Jeff KingSep 5, 2021
  23. ZheNing HuSep 6, 2021
  24. Jeff KingSep 7, 2021
  25. ZheNing HuSep 9, 2021
  26. Patrick SteinhardtSep 6, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.