# [RFC] Code reorgnization 14 messages from 2016-03-17 to 2016-03-18. Participants: Duy Nguyen, Johannes Schindelin, Junio C Hamano, Thomas Adam, Stefan Beller, Pranit Bauva, John Keeping, Jeff King. Thread: https://gitlist.dev/t/41726 ## Duy Nguyen, 2016-03-17 11:11 Subject: [RFC] Code reorgnization Message-ID: <20160317111136.GA21745@lanh> URL: https://gitlist.dev/e/20160317111136.GA21745%40lanh ``` Git's top directory is crowded and I think it's agreed that moving test-* to t/helper is a good move. I just wanted to check if we could take this opportunity (after v2.8.0) to move some other files too. I propose the following new subdirs lib --- This contains files that are about data structures or algorithms. Very general purpose. This directory includes argv-array.[ch] base85.c column.[ch] delta.h diff-delta.c hashmap.[ch] hex.c khash.h kwset.[ch] levenshtein.[ch] mergesort.[ch] patch-delta.c prio-queue.[ch] sha1-array.[ch] sha1-lookup.[ch] strbuf.[ch] string-list.[ch] url.[ch] urlmatch.[ch] utf8.[ch] varint.[ch] versioncmp.c wildmatch.[ch] odb --- The grouping of object database files is to easily make connections between them. Unlike, for example, diff-related files which either start with "diff" or has that word in the file name to make connections. alloc.c blob.[ch] bulk-checkin.[ch] commit-slab.h commit.[ch] object.[ch] pack.h pack-revindex.[ch] replace_object.c sha1_file.c streaming.[ch] tag.[ch] tree.[ch] index ----- For the same reason of odb subdir. This directory contains cache-tree.[ch] name-hash.c preload-index.c read-cache.c split-index.[ch] unpack-trees.[ch] sys (or maybe util or support) ------------------------------ These are still general purpose but is usually system-related. They are still far away from git's core logic. I want to separate them to make it easier to spot "important" files at top dir. abspath.c color.[ch] copy.c csum-file.[ch] ctype.c date.c editor.c exec_cmd.[ch] gettext.[ch] gettext.h gpg-interface.[ch] ident.c lockfile.[ch] mailinfo.[ch] mailmap.[ch] pager.c parse-options-cb.c parse-options.[ch] pathspec.[ch] pkt-line.[ch] progress.[ch] prompt.[ch] quote.[ch] run-command.[ch] sideband.[ch] sigchain.[ch] symlinks.c tar.h tempfile.[ch] thread-utils.[ch] trace.[ch] unix-socket.[ch] usage.c userdiff.[ch] wrapper.c write_or_die.c zlib.c Good? Bad? Ugly? -- Duy ``` ## Johannes Schindelin, 2016-03-17 13:32 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.20.1603171431000.4690%40virtualbox In-Reply-To: <20160317111136.GA21745@lanh> ``` Hi Duy, On Thu, 17 Mar 2016, Duy Nguyen wrote: > Git's top directory is crowded and I think it's agreed that moving > test-* to t/helper is a good move. I just wanted to check if we could > take this opportunity (after v2.8.0) to move some other files too. I > propose the following new subdirs > > lib > --- > This contains files that are about data structures or algorithms. Very > general purpose. This directory includes > > argv-array.[ch] base85.c column.[ch] delta.h diff-delta.c hashmap.[ch] > hex.c khash.h kwset.[ch] levenshtein.[ch] mergesort.[ch] patch-delta.c > prio-queue.[ch] sha1-array.[ch] sha1-lookup.[ch] strbuf.[ch] > string-list.[ch] url.[ch] urlmatch.[ch] utf8.[ch] varint.[ch] > versioncmp.c wildmatch.[ch] The name "lib" makes it sound as if this contains the source code of libgit.a. Maybe "generic" or "common" or "util" would be better (my favorite would be "util"). > odb > --- > The grouping of object database files is to easily make connections > between them. Unlike, for example, diff-related files which either > start with "diff" or has that word in the file name to make > connections. > > alloc.c blob.[ch] bulk-checkin.[ch] commit-slab.h commit.[ch] > object.[ch] pack.h pack-revindex.[ch] replace_object.c sha1_file.c > streaming.[ch] tag.[ch] tree.[ch] > > index > ----- > For the same reason of odb subdir. This directory contains > > cache-tree.[ch] name-hash.c preload-index.c read-cache.c > split-index.[ch] unpack-trees.[ch] > > sys (or maybe util or support) > ------------------------------ > These are still general purpose but is usually system-related. They > are still far away from git's core logic. I want to separate them to > make it easier to spot "important" files at top dir. > > abspath.c color.[ch] copy.c csum-file.[ch] ctype.c date.c editor.c > exec_cmd.[ch] gettext.[ch] gettext.h gpg-interface.[ch] ident.c > lockfile.[ch] mailinfo.[ch] mailmap.[ch] pager.c parse-options-cb.c > parse-options.[ch] pathspec.[ch] pkt-line.[ch] progress.[ch] > prompt.[ch] quote.[ch] run-command.[ch] sideband.[ch] sigchain.[ch] > symlinks.c tar.h tempfile.[ch] thread-utils.[ch] trace.[ch] > unix-socket.[ch] usage.c userdiff.[ch] wrapper.c write_or_die.c zlib.c > > Good? Bad? Ugly? Disruptive. Probably a change for 3.0? Ciao, Dscho ``` ## Duy Nguyen, 2016-03-17 13:35 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/CACsJy8BFdk6jtRqSKE0ThtcZi39hHC_QbDx6oka0qWk7eiD4wQ%40mail.gmail.com In-Reply-To: ``` On Thu, Mar 17, 2016 at 8:32 PM, Johannes Schindelin wrote: >> Good? Bad? Ugly? > > Disruptive. Probably a change for 3.0? We tested it with the builtin rename a long time ago, so it's probably not bad. By the principle of "dogfooding", we should try it soon and make sure it's not disruptive, or prepare ourselves for such a change (I think git-am can't track renames, for example, without us giving it a clue somehow) -- Duy ``` ## Junio C Hamano, 2016-03-17 16:21 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/xmqqbn6d13f5.fsf%40gitster.mtv.corp.google.com In-Reply-To: <20160317111136.GA21745@lanh> ``` Duy Nguyen writes: > Good? Bad? Ugly? Too fine-grained to induce confusion for things that have to work as a bridge between two categories (e.g. odb & index). In short, bad and ugly. I am OK with a looser classification e.g. (1) things that can be used without Git at all like strbuf, string-list vs (2) things that are Git but shared across different subcommands like read-cache, sha1_file, vs (3) command implementations (e.g. builtin/ and also standalone). ``` ## Thomas Adam, 2016-03-17 17:00 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/CA%2B39Oz6jNwcyCQFiakh%3DEch6p8UYRW9pn95e6cTGXf8nFcwwWQ%40mail.gmail.com In-Reply-To: <20160317111136.GA21745@lanh> ``` On 17 March 2016 at 11:11, Duy Nguyen wrote: > Git's top directory is crowded and I think it's agreed that moving > test-* to t/helper is a good move. I just wanted to check if we could > take this opportunity (after v2.8.0) to move some other files too. I > propose the following new subdirs I wonder whether previous discussions on this still count? See: http://marc.info/?l=git&m=129650572621523&w=1 -- Thomas Adam ``` ## Junio C Hamano, 2016-03-17 17:48 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/xmqq7fh10zcp.fsf%40gitster.mtv.corp.google.com In-Reply-To: ``` Thomas Adam writes: > On 17 March 2016 at 11:11, Duy Nguyen wrote: >> Git's top directory is crowded and I think it's agreed that moving >> test-* to t/helper is a good move. I just wanted to check if we could >> take this opportunity (after v2.8.0) to move some other files too. I >> propose the following new subdirs > > I wonder whether previous discussions on this still count? See: > > http://marc.info/?l=git&m=129650572621523&w=1 If you refer to ancient discussion, especially to a large thread like that one, please spend a bit more time to summarize it. It is between one person spends a bit more time, and all others independently go there and read. The essense of the proposal [1] back then was to move all the source file to src/, rename t/ to testsuite. And I think [2] is a pretty good summary of the common feeling back then that explains why the proposal died out: Moving everything into src/ and calling it "organized" doesn't actually accomplish much other than perhaps making the README file more visible to newbs; things are _still_ a mess, just a mess with four more letters... This round is slightly more organized, so many points the old thread raised would not apply, I suspect. [References] *1* http://thread.gmane.org/gmane.comp.version-control.git/165720/focus=165748 *2* http://thread.gmane.org/gmane.comp.version-control.git/165720/focus=166019 ``` ## Stefan Beller, 2016-03-17 18:37 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/CAGZ79kbcwFcPSJ9xwE6xi4gQ871m3brtfAut2TChGNzL-foxdQ%40mail.gmail.com In-Reply-To: <20160317111136.GA21745@lanh> ``` On Thu, Mar 17, 2016 at 4:11 AM, Duy Nguyen wrote: > Good? Bad? Ugly? For now I would just go with 3 directories: non-git/ (or util, helpers, or anything that could be ripped out and be useful e.g. strbufs, argv-array run-command, lockfile git/ (maybe called lib? All stuff that is pure Git and is used for libgit builtin/ (as we have it today + all that stuff that doesn't go into git/ very well?) Thanks, Stefan ``` ## Junio C Hamano, 2016-03-17 19:10 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/xmqqy49gzzrf.fsf%40gitster.mtv.corp.google.com In-Reply-To: ``` Stefan Beller writes: > For now I would just go with 3 directories: > > non-git/ (or util, helpers, or anything that could be ripped out and be useful > e.g. strbufs, argv-array run-command, lockfile > git/ (maybe called lib? All stuff that is pure Git and is used for libgit > > builtin/ (as we have it today + all that stuff that doesn't go into > git/ very well?) It is unclear where you want to have standalone programs in the above. I'd say lib/ and src/ for the first two, where lib/ is for things that could be lifted without any Git dependencies and src/ for everything else. Aren't there some folks who link directly with our codebase (I am thinking about cgit, but hjemli.net/git/cgit does not seem to be responding anymore)? ``` ## Pranit Bauva, 2016-03-17 21:03 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/CAFZEwPMBr4oK9tM7rMsBUDhON-0%2BjymGT%3Dt1sdkztbh3eNyt5g%40mail.gmail.com In-Reply-To: ``` On Fri, Mar 18, 2016 at 12:40 AM, Junio C Hamano wrote: > Stefan Beller writes: > >> For now I would just go with 3 directories: >> >> non-git/ (or util, helpers, or anything that could be ripped out and be useful >> e.g. strbufs, argv-array run-command, lockfile >> git/ (maybe called lib? All stuff that is pure Git and is used for libgit >> >> builtin/ (as we have it today + all that stuff that doesn't go into >> git/ very well?) > > It is unclear where you want to have standalone programs in the > above. I'd say lib/ and src/ for the first two, where lib/ is for > things that could be lifted without any Git dependencies and src/ > for everything else. > > Aren't there some folks who link directly with our codebase (I am > thinking about cgit, but hjemli.net/git/cgit does not seem to be > responding anymore)? They now have a new mailing list.[1] I guess they forward all the mails to the new one[2]. The new one seems active. [3] [1] : cgit@lists.zx2c4.com [2] : https://lists.zx2c4.com/pipermail/cgit/2013-May/001380.html [3] : http://news.gmane.org/gmane.comp.version-control.cgit ``` ## John Keeping, 2016-03-17 21:43 Subject: Re: [RFC] Code reorgnization Message-ID: <20160317214355.GA32317@serenity.lan> URL: https://gitlist.dev/e/20160317214355.GA32317%40serenity.lan In-Reply-To: ``` On Thu, Mar 17, 2016 at 12:10:44PM -0700, Junio C Hamano wrote: > Stefan Beller writes: > > > For now I would just go with 3 directories: > > > > non-git/ (or util, helpers, or anything that could be ripped out and be useful > > e.g. strbufs, argv-array run-command, lockfile > > git/ (maybe called lib? All stuff that is pure Git and is used for libgit > > > > builtin/ (as we have it today + all that stuff that doesn't go into > > git/ very well?) > > It is unclear where you want to have standalone programs in the > above. I'd say lib/ and src/ for the first two, where lib/ is for > things that could be lifted without any Git dependencies and src/ > for everything else. > > Aren't there some folks who link directly with our codebase (I am > thinking about cgit, but hjemli.net/git/cgit does not seem to be > responding anymore)? CGit lives at https://git.zx2c4.com/cgit/ these days. The organisation of the git code shouldn't make a difference since CGit just links with libgit.a, even if it does CGit pulls in git.git as a submodule so it can just fix any problems in the same commit that updates the submodule reference. ``` ## Junio C Hamano, 2016-03-17 21:49 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/xmqqh9g4zsf8.fsf%40gitster.mtv.corp.google.com In-Reply-To: <20160317214355.GA32317@serenity.lan> ``` John Keeping writes: > The organisation of the git code shouldn't make a difference since CGit > just links with libgit.a, even if it does CGit pulls in git.git as a > submodule so it can just fix any problems in the same commit that > updates the submodule reference. I was mostly worried about where Duy and Stefan want to place *.h ``` ## Duy Nguyen, 2016-03-18 00:28 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/CACsJy8Bt211aSusEZ0xTDS96ZQgCjrMi-sDmZXkz_jCuUnkcTA%40mail.gmail.com In-Reply-To: ``` On Fri, Mar 18, 2016 at 4:49 AM, Junio C Hamano wrote: > John Keeping writes: > >> The organisation of the git code shouldn't make a difference since CGit >> just links with libgit.a, even if it does CGit pulls in git.git as a >> submodule so it can just fix any problems in the same commit that >> updates the submodule reference. > > I was mostly worried about where Duy and Stefan want to place *.h *.h stay with their *.c. CFLAGS has two more -Isrc and -Ilib. I don't expect any #include line changes. Maybe we can start moving stuff to "lib" soon. Many of them rarely receive changes these days. The creation of "src" could be more disruptive and can wait until $(topdir) is once again unbearable. -- Duy ``` ## Jeff King, 2016-03-18 05:24 Subject: Re: [RFC] Code reorgnization Message-ID: <20160318052447.GD22327@sigill.intra.peff.net> URL: https://gitlist.dev/e/20160318052447.GD22327%40sigill.intra.peff.net In-Reply-To: <20160317111136.GA21745@lanh> ``` On Thu, Mar 17, 2016 at 06:11:36PM +0700, Duy Nguyen wrote: > Git's top directory is crowded and I think it's agreed that moving > test-* to t/helper is a good move. I just wanted to check if we could > take this opportunity (after v2.8.0) to move some other files too. I > propose the following new subdirs I guess I don't really see the "crowded" problem, but perhaps that is because I am more or less familiar with where things are in git's code base. I suppose if you were looking for a "utility" function, you might look in "util" and therefore have a smaller set of files to check. But I think we also run into the opposite problem: I am looking for some particular function, but I can't find it, because I am looking in "util" and it is in some other directory. And when files move around, it makes history harder to follow (maybe that is because git sucks and we need to make it better, but certainly I run into mild annoyances with the builtin/ rename when digging in history). And you have a similar problem when creating new files. Which slot do they go in? What if they could feasibly go into two slots? So there can be friction either way. In practice I find I just use ctags to jump to the functions I am interested in, and I don't care that much about filenames. The reorganization that _would_ be more interesting to me is not files in directories, but rather functions in files. I wish everything were designed more as modules with a pair of matching ".c" and ".h" files, with a public interface defined in the ".h", and messier, private stuff in the ".c". But we have some real dumping grounds: 1. cache.h has the declarations for at least a dozen different modules; besides being hard to navigate, it causes more frequent recompilation than necessary. 2. a few of the .c files could probably be split (e.g., dir.c is where all of the pathspec code lives, even though that is used for much more than filesystem access these days). Splitting those up would _also_ introduce friction (and actually worse than whole-file renames, because finding code movement between files is an even harder / more expensive problem). But I feel like it would buy a lot more in terms of code clarity, and in reducing the scope of code which has access to private, static interfaces. -Peff ``` ## Duy Nguyen, 2016-03-18 05:59 Subject: Re: [RFC] Code reorgnization Message-ID: URL: https://gitlist.dev/e/CACsJy8AmrdVeB95RUbVPamM7DMxFKRJ9REd0SN_oq_4HEb6E9g%40mail.gmail.com In-Reply-To: <20160318052447.GD22327@sigill.intra.peff.net> ``` On Fri, Mar 18, 2016 at 12:24 PM, Jeff King wrote: > On Thu, Mar 17, 2016 at 06:11:36PM +0700, Duy Nguyen wrote: > >> Git's top directory is crowded and I think it's agreed that moving >> test-* to t/helper is a good move. I just wanted to check if we could >> take this opportunity (after v2.8.0) to move some other files too. I >> propose the following new subdirs > > I guess I don't really see the "crowded" problem, but perhaps that is > because I am more or less familiar with where things are in git's code > base. I suppose if you were looking for a "utility" function, you might > look in "util" and therefore have a smaller set of files to check. > > But I think we also run into the opposite problem: I am looking for some > particular function, but I can't find it, because I am looking in "util" > and it is in some other directory. And when files move around, it makes > history harder to follow (maybe that is because git sucks and we need to > make it better, but certainly I run into mild annoyances with the > builtin/ rename when digging in history). Yeah, for finding a particular function, I just "git grep" (or rgrep from emacs) if I fail to locate it after the first guess. We have this problem nowadays anyway. Besides builtin, we also have ewah, refs and some more subdirs. > And you have a similar problem when creating new files. Which slot do > they go in? What if they could feasibly go into two slots? Everything goes to topdir (or later on "src") by default and only goes to "lib" when it's _obvious_ that it's disconnected from git (i'm talking about the "lib/src" layout). > So there can be friction either way. In practice I find I just use ctags > to jump to the functions I am interested in, and I don't care that much > about filenames. > > The reorganization that _would_ be more interesting to me is not files > in directories, but rather functions in files. I wish everything were > designed more as modules with a pair of matching ".c" and ".h" files, > with a public interface defined in the ".h", and messier, private stuff > in the ".c". But we have some real dumping grounds: > > 1. cache.h has the declarations for at least a dozen different > modules; besides being hard to navigate, it causes more frequent > recompilation than necessary. > > 2. a few of the .c files could probably be split (e.g., dir.c is where > all of the pathspec code lives, even though that is used for much > more than filesystem access these days). Heh.. that's what I wanted to do (or at least discuss) after files are moved :) > Splitting those up would _also_ introduce friction (and actually worse > than whole-file renames, because finding code movement between files is > an even harder / more expensive problem). .. and this is why I did not raise it in the first mail. > But I feel like it would buy a > lot more in terms of code clarity, and in reducing the scope of code > which has access to private, static interfaces. -- Duy ```