git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [Summit topic] Crazy (and not so crazy) ideas

From
Son Luong Ngoc <sluongng@gmail.com>
Date
Oct 21, 2021, 12:30 UTC
Message-ID
<CAL3xRKck=cbdo=7gqv0q=MLjn+5J1ES6fbzHP1mEj90pLdUdAA@mail.gmail.com>
In-Reply-To
<nycvar.QRO.7.76.6.2110211144490.56@tvgsbejvaqbjf.bet>
Hi,

On Thu, Oct 21, 2021 at 1:56 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 8 quoted lines
>
> This session was led by Elijah Newren. Supporting cast: Johannes "Dscho"
> Schindelin, Jonathan Tan, Jonathan "jrnieder" Nieder, brian m. carlson,
> Jeff "Peff" King, Ævar Arnfjörð Bjarmason, Emily Shaffer, CB Bailey,
> Taylor Blau, and Philip Oakley.
>
> Notes:
>
...
Show 6 quoted lines
>
> * Biggest idea: there are a lot of people who version control things via
>   tarballs or .zip files per version. This prevents history from
>   compressing well. Some people check in those compressed files into Git
>   for purposes of history.
>
...
Show 22 quoted lines
>
>    * Old suggestion of a “blob-tree” type that allows storing a single
>      index entry that corresponds to multiple trees and blobs in the
>      background, possibly.
>
>    * One long-term dream (inspired by Avery Pennarun’s “bup” tool) is to
>      store large binary files in a tree-structured way that can store
>      common regions as deltas, improve random access, parallelized
>      hashing. Involves a consistent way to split the file into stable
>      pieces, like --rsyncable uses (based on a rolling hash being zero).
>
>    * Peff: you can do that at the object model layer or at the storage
>      layer. The latter is less invasive.
>
>    * jrnieder: The benefits of blobtree are greater at the object model
>      layer --- e.g. not having to transmit chunks over the wire that you
>      already have. I think the main obstacle has been that the benefits
>      haven’t been enough to be worth the complexity. If that changes, we
>      can imagine bundling it with some other object format changes, e.g.
>      putting blob sizes in tree objects, and rolling it out as a new
>      object-format.
>

I think this was implemented as 'Blob Ref' in Yandex's vcs named Arc. I was suggesting this to Gitlab folks earlier (1) as a possible solution to large file storage.

Very glad to hear that it was brought up during the summit.

Cheers, Son Luong.

(1): https://gitlab.com/gitlab-org/git/-/issues/93
Previous: Johannes SchindelinNext: Eric Wong
Message 3 of 58 in “Notes from the Git Contributors' Summit 2021, virtual, Oct 19/20”
  1. Johannes SchindelinOct 21, 2021
  2. [Summit topic] Crazy (and not so crazy) ideasJohannes Schindelin, Oct 21, 2021
  3. Son Luong NgocOct 21, 2021
  4. scripting speedups [was: [Summit topic] Crazy (and not so crazy) ideas]Eric Wong, Oct 26, 2021
  5. Ævar Arnfjörð BjarmasonOct 30, 2021
  6. test suite speedups via some not-so-crazy ideas (was: scripting speedups[...])Ævar Arnfjörð Bjarmason, Nov 3, 2021
  7. Junio C HamanoNov 3, 2021
  8. Johannes SchindelinNov 2, 2021
  9. [Summit topic] SHA-256 UpdatesJohannes Schindelin, Oct 21, 2021
  10. [Summit topic] Server-side merge/rebase: needs and wants?Johannes Schindelin, Oct 21, 2021
  11. Bagas SanjayaOct 22, 2021
  12. Johannes SchindelinOct 22, 2021
  13. Ævar Arnfjörð BjarmasonOct 23, 2021
  14. Taylor BlauNov 8, 2021
  15. Ævar Arnfjörð BjarmasonNov 9, 2021
  16. Christian CouderNov 30, 2021
  17. [Summit topic] Submodules and how to make them worth usingJohannes Schindelin, Oct 21, 2021
  18. [Summit topic] Sparse checkout behavior and plansJohannes Schindelin, Oct 21, 2021
  19. [Summit topic] The state of getting a reftable backend working in git.gitJohannes Schindelin, Oct 21, 2021
  20. Han-Wen NienhuysOct 25, 2021
  21. Ævar Arnfjörð BjarmasonOct 25, 2021
  22. Han-Wen NienhuysOct 26, 2021
  23. Philip OakleyOct 28, 2021
  24. Philip OakleyOct 26, 2021
  25. [Summit topic] Documentation (translations, FAQ updates, new user-focused, general improvements, etc.)Johannes Schindelin, Oct 21, 2021
  26. Jean-Noël AvilaOct 22, 2021
  27. Ævar Arnfjörð BjarmasonOct 22, 2021
  28. Jean-Noël AvilaOct 27, 2021
  29. Jeff KingOct 27, 2021
  30. [Summit topic] Increasing diversity & inclusion (transition to `main`, etc)Johannes Schindelin, Oct 21, 2021
  31. Son Luong NgocOct 21, 2021
  32. vale check, was Re: [Summit topic] Increasing diversity & inclusion (transition to `main`, etc)Johannes Schindelin, Oct 22, 2021
  33. Johannes SchindelinOct 22, 2021
  34. [Summit topic] Improving Git UXJohannes Schindelin, Oct 21, 2021
  35. changing the experimental 'git switch' (was: [Summit topic] Improving Git UX)Ævar Arnfjörð Bjarmason, Oct 21, 2021
  36. Junio C HamanoOct 21, 2021
  37. Bagas SanjayaOct 22, 2021
  38. martinOct 22, 2021
  39. Ævar Arnfjörð BjarmasonOct 22, 2021
  40. Sergey OrganovOct 22, 2021
  41. martinOct 22, 2021
  42. Sergey OrganovOct 23, 2021
  43. MartinOct 24, 2021
  44. Junio C HamanoOct 24, 2021
  45. Ævar Arnfjörð BjarmasonOct 25, 2021
  46. Junio C HamanoOct 25, 2021
  47. Sergey OrganovOct 25, 2021
  48. Ævar Arnfjörð BjarmasonOct 25, 2021
  49. Sergey OrganovOct 27, 2021
  50. [Summit topic] Improving reviewer quality of life (patchwork, subsystem lists?, etc)Johannes Schindelin, Oct 21, 2021
  51. Konstantin RyabitsevOct 21, 2021
  52. Ævar Arnfjörð BjarmasonOct 22, 2021
  53. Missing notes, was Re: Notes from the Git Contributors' Summit 2021, virtual, Oct 19/20Johannes Schindelin, Oct 22, 2021
  54. Johannes SchindelinOct 22, 2021
  55. Johannes SchindelinOct 22, 2021
  56. Johannes SchindelinOct 22, 2021
  57. Let's have public Git chalk talks, was Re: Notes from the Git Contributors' Summit 2021, virtual, Oct 19/20Johannes Schindelin, Oct 22, 2021
  58. Ævar Arnfjörð BjarmasonOct 25, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.