git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC] Submodules in GIT

From
JWJosef Weidendorfer <josef.weidendorfer@gmx.de>
Date
Dec 15, 2006, 21:42 UTC
Message-ID
<200612152242.50472.Josef.Weidendorfer@gmx.de>
In-Reply-To
<e7bda7770612150943j71a7362bmb509cea3b7756003@mail.gmail.com>
On Friday 15 December 2006 18:43, Torgil Svensson wrote:
Show 13 quoted lines
> On 12/15/06, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> > However, there is nothing wrong with it. Yet, you perhaps want
> > the 2 Lib submodules not to go out of sync. This easily
> > can be done with symlinking the Lib checkouts. As they are submodules,
> > everything should work fine.
> 
> This is interesting. In my notation:
> 
> /path/to/link/name -> <commit>/path/to/subtree
> 
> means that there is a link named "name" in the tree object for
> "path/to/link". The link points to a "link object" specifying a
> subtree or blob of the tree that is pointed to in a submodule commit.
Ah, now I understand. I somehow missed this notation.
Show 8 quoted lines
> This is not currently implemented but has at least the following
> advantages:
> 
> 1. You can access files in a submodule without fetching the whole
> submodule (which may be very large). (App1 is only interested in
> lib1.h, the rest is toally irrelevant)
> 2. Superproject can access referenced (linked) files in it's own
> folder-structure without being forced a structure by the subproject.
That all sounds fine, but how do you create such symlinks in practice?
Do you want to introduce special porcelain commands to create them?
Especially, what is the SCM user supposed to do to change the link
target, ie. from
 <commit>/path/to/subtree
to 
 <commit>/path2/to2/subtree2
?
Should this do a re-checkout at the other point?

By linking a file from a submodule, such a link seems to force that this file has to be at a fixed position in the submodule. Otherwise, some magic has to happen when the file is moved in the submodule, possibly leading to a dangling link, eg. if the whole subdirectory specified in the link is removed.

IMHO this is getting way to complex. Much simpler is to include the full submodule at some path in the supermodule, and create normal symlinks from the supermodule into the submodule.

If you only want to check out part of a submodule, this should be done with path-limiting checkouts, which should be a feature totally independent from submodules.

And if you want to limit the number of objects transferred in cloning
of a subproject, it is better to further split this subproject into
multiple subprojects itself.
 
> If you do a symlink instead, doesn't you loose versioning information?

Of course, you need the submodule fully checked out somewhere in the supermodule, and the link goes into the submodule directory. The versioning is given by the supermodule/submodule link.

> What happens with the symlinks if someone clones the superproject?

As already said: the link has to go into a submodule directory, which will be checked out automatically with the clone of the supermodule.

Show 14 quoted lines
> > Perhaps an option you want to have is to force a checkout
> > of AppBuild to make these symlinking itself when it detects
> > identical submodules links.
> >
> > Hmmm... the only problem with a symlink is that it can go wrong
> > when moved. Unfortunately, I do not have a good solution for
> > this. We can not make UNIX symlinks smart in any way.
> > Hardlinking directories would be a solution, but that is not
> > possible.
> >
> 
> Wouldn't specifying the submodule path in the link object fit in well
> here? Then each "link object" can represent a checked out tree from
> the subproject in the superproject directory-structure.

The problem is not the representation in the git repository, but the checked out module/submodule, where you need to use normal UNIX file semantics. To move submodules around, the user should be able to just use the normal UNIX "mv" commands, and git should be able to detect move actions after the fact. The simple thing here is that currently, git does not have this problem as it tracks content, and does not even try to detect any moves at commit time. This is different with submodules, as there, you want to be able to track moves of any submodule root directories.

This now becomes a problem if you use symlinks to "unify" multiple checkouts of the same submodule at multiple places in the supermodule, and move the symlink around, as it easily can get dangling this way. Thus, you would not have a way to see what submodule this link was talking about.

And for this thing, I do not see how your link object could help.

So it is better to use a simple submodule concept, and for this corner cases, we perhaps could expect the user to fix e.g. a dangling symlink to a previous submodule checkout himself, using a meaningful error message.

> > BTW, build project commits probably should not depend on any
> > history of other build commits.
> 
> Why? Can you give an example here.

If you have a source commit chain A => B => C => D, you want to make any build commits totally independent: you first only are interested in a build commit for source versions A and D, and later find out that a build commit for B and C would be nice, too. If you force build commits into some history order, this order now would be A => D => B => C, which makes no sense.

Build commit independence can easily be achieved by making every commit parentless, without further history. You still have the link to the source version via the submodule link in the tree. But to not loose any such build commits, they have to appear as tags or refs (unless integrated in another superproject build commit).

Show 16 quoted lines
> > > Link: /headers/lib1.h -> <lib1-commit3>/src/lib1.h
> > > Link: /bin/lib1.so -> <build1-commit>/i386/lib1/lib1.so
> > > Link: /bin/app1 -> <build1-commit>/i386/app1/app1
> > >
> > >
> > > <lib1-commit1>, <lib1-commit2> and <lib1-commit3> should be the same,
> > > dictated by the app1 project.
> >
> > I do not see any problem here. Symlinks are stored in the git repository.
> > As the AppBuild commit depends on App and LibBuild submodule commits, the
> > symlinks always should be correct.
> 
> The main reason for these "links" are for versioning purposes: the
> uniqe SHA1 of the "link" representing a tree/blob in a version of the
> submodule should be "included" in the supermodules commit. Symlinks
> won't give that at all.

The version coupling will be there if the whole submodule is available at some path in the supermodule checkout, as said above.

Previous: Torgil SvenssonNext: Torgil Svensson
Message 102 of 160 in “Re: [RFC] Submodules in GIT”
  1. Andy ParkinsNov 28, 2006
  2. Jakub NarebskiNov 28, 2006
  3. Andy ParkinsNov 28, 2006
  4. Shawn PearceNov 28, 2006
  5. Andy ParkinsNov 28, 2006
  6. Shawn PearceNov 28, 2006
  7. Jon LoeligerNov 28, 2006
  8. Martin WaitzNov 29, 2006
  9. sfNov 30, 2006
  10. Steven GrimmNov 28, 2006
  11. Shawn PearceNov 28, 2006
  12. Martin WaitzNov 29, 2006
  13. Andy ParkinsNov 29, 2006
  14. Andreas EricssonNov 30, 2006
  15. Andy ParkinsNov 30, 2006
  16. Martin WaitzNov 30, 2006
  17. Andreas EricssonNov 30, 2006
  18. sfDec 1, 2006
  19. Martin WaitzDec 1, 2006
  20. sfDec 1, 2006
  21. Martin WaitzDec 1, 2006
  22. Stephan FederDec 1, 2006
  23. Martin WaitzDec 1, 2006
  24. Stephan FederDec 1, 2006
  25. Martin WaitzDec 1, 2006
  26. Uwe Kleine-KoenigDec 5, 2006
  27. Andreas EricssonDec 5, 2006
  28. Jakub NarebskiDec 5, 2006
  29. Uwe Kleine-KoenigDec 5, 2006
  30. Andreas EricssonDec 5, 2006
  31. Sven VerdoolaegeDec 5, 2006
  32. Andy ParkinsDec 1, 2006
  33. Martin WaitzDec 1, 2006
  34. sfDec 1, 2006
  35. Martin WaitzDec 1, 2006
  36. sfDec 1, 2006
  37. Martin WaitzDec 1, 2006
  38. Andreas EricssonDec 1, 2006
  39. Martin WaitzDec 1, 2006
  40. Andreas EricssonDec 1, 2006
  41. Martin WaitzDec 1, 2006
  42. Andreas EricssonDec 1, 2006
  43. Linus TorvaldsDec 1, 2006
  44. sfDec 1, 2006
  45. Andreas EricssonDec 1, 2006
  46. Linus TorvaldsDec 1, 2006
  47. Martin WaitzDec 1, 2006
  48. Alan ChandlerDec 1, 2006
  49. Josef WeidendorferDec 1, 2006
  50. Martin WaitzDec 1, 2006
  51. Josef WeidendorferDec 1, 2006
  52. Martin WaitzDec 1, 2006
  53. Josef WeidendorferDec 1, 2006
  54. Martin WaitzDec 2, 2006
  55. Josef WeidendorferDec 3, 2006
  56. Martin WaitzDec 3, 2006
  57. Linus TorvaldsDec 1, 2006
  58. sfDec 1, 2006
  59. Josef WeidendorferDec 1, 2006
  60. Linus TorvaldsDec 1, 2006
  61. Josef WeidendorferDec 1, 2006
  62. Linus TorvaldsDec 2, 2006
  63. Andy ParkinsDec 2, 2006
  64. Josef WeidendorferDec 2, 2006
  65. Linus TorvaldsDec 2, 2006
  66. Martin WaitzDec 2, 2006
  67. Linus TorvaldsDec 2, 2006
  68. Martin WaitzDec 2, 2006
  69. Josef WeidendorferDec 3, 2006
  70. Martin WaitzDec 2, 2006
  71. Linus TorvaldsDec 2, 2006
  72. Martin WaitzDec 2, 2006
  73. Linus TorvaldsDec 2, 2006
  74. Linus TorvaldsDec 2, 2006
  75. Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT]Josef Weidendorfer, Dec 3, 2006
  76. Linus TorvaldsDec 3, 2006
  77. Shawn PearceDec 3, 2006
  78. Josef WeidendorferDec 3, 2006
  79. Jakub NarebskiDec 3, 2006
  80. Josef WeidendorferDec 3, 2006
  81. Martin WaitzDec 3, 2006
  82. sfDec 1, 2006
  83. Torgil SvenssonDec 2, 2006
  84. Linus TorvaldsDec 2, 2006
  85. Torgil SvenssonDec 3, 2006
  86. Linus TorvaldsDec 3, 2006
  87. Torgil SvenssonDec 4, 2006
  88. Linus TorvaldsDec 4, 2006
  89. Torgil SvenssonDec 4, 2006
  90. Andreas EricssonDec 5, 2006
  91. Jakub NarebskiDec 5, 2006
  92. Andreas EricssonDec 5, 2006
  93. Jakub NarebskiDec 5, 2006
  94. Andy ParkinsDec 3, 2006
  95. Daniel BarkalowDec 5, 2006
  96. sfDec 5, 2006
  97. R. Steve McKownDec 9, 2006
  98. Torgil SvenssonDec 10, 2006
  99. Torgil SvenssonDec 14, 2006
  100. Josef WeidendorferDec 14, 2006
  101. Torgil SvenssonDec 15, 2006
  102. Josef WeidendorferDec 15, 2006
  103. Torgil SvenssonDec 15, 2006
  104. Torgil SvenssonDec 16, 2006
  105. Torgil SvenssonDec 16, 2006
  106. Jakub NarebskiDec 16, 2006
  107. Torgil SvenssonDec 16, 2006
  108. Jakub NarebskiDec 16, 2006
  109. Junio C HamanoDec 16, 2006
  110. Torgil SvenssonDec 16, 2006
  111. Torgil SvenssonDec 16, 2006
  112. Jakub NarebskiDec 16, 2006
  113. Torgil SvenssonDec 17, 2006
  114. Linus TorvaldsDec 16, 2006
  115. Linus TorvaldsDec 16, 2006
  116. Torgil SvenssonDec 16, 2006
  117. Martin WaitzDec 2, 2006
  118. Josef WeidendorferDec 1, 2006
  119. Martin WaitzDec 1, 2006
  120. Linus TorvaldsDec 1, 2006
  121. Josef WeidendorferDec 2, 2006
  122. Linus TorvaldsDec 2, 2006
  123. Andy ParkinsDec 2, 2006
  124. Michael K. EdwardsDec 4, 2006
  125. Sam VilainDec 5, 2006
  126. Sven VerdoolaegeDec 3, 2006
  127. Linus TorvaldsDec 3, 2006
  128. Jakub NarebskiDec 3, 2006
  129. Josef WeidendorferDec 4, 2006
  130. sfDec 1, 2006
  131. Jon LoeligerDec 8, 2006
  132. Sven VerdoolaegeDec 8, 2006
  133. Andreas EricssonDec 12, 2006
  134. Martin WaitzDec 1, 2006
  135. Martin WaitzDec 1, 2006
  136. Andreas EricssonDec 1, 2006
  137. Martin WaitzDec 1, 2006
  138. Stephan FederDec 1, 2006
  139. Martin WaitzDec 1, 2006
  140. Stephan FederDec 1, 2006
  141. Martin WaitzDec 1, 2006
  142. Stephan FederDec 1, 2006
  143. Martin WaitzDec 1, 2006
  144. sfDec 1, 2006
  145. Martin WaitzDec 2, 2006
  146. Andy ParkinsDec 1, 2006
  147. Martin WaitzDec 1, 2006
  148. Andy ParkinsDec 1, 2006
  149. Martin WaitzDec 1, 2006
  150. Andy ParkinsDec 1, 2006
  151. Martin WaitzDec 1, 2006
  152. Andy ParkinsDec 2, 2006
  153. Josef WeidendorferDec 2, 2006
  154. Martin WaitzDec 2, 2006
  155. Josef WeidendorferDec 3, 2006
  156. Martin WaitzDec 2, 2006
  157. Jakub NarebskiDec 2, 2006
  158. Jakub NarebskiDec 2, 2006
  159. Jakub NarebskiDec 2, 2006
  160. Andy ParkinsDec 3, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.