What's cooking in git.git (Jan 2011, #06; Sun, 30)

Here are the topics that have been cooking. Commits prefixed with '-' are only in 'pu' while commits prefixed with '+' are in 'next'.

1.7.4 is out. I'd like to stop and calm the tree down for a few days so that we can catch any brown-paper-bag bugs before moving things forward, and then open the floodgate for the next cycle, which I am inclined to designate as "We would have done these differently if we were creating git from scratch with the experience we have and wisdom we have gained" cycle, allowing minor backward incompatibilities, somewhat like 1.6.0 but not so drastic. The result would probably be called 1.8.0--the details in a separate message.

-------------------------------------------------- [New Topics]

* jc/fsck-fixes (2011-01-26) 2 commits
 - fsck: do not give up too early in fsck_dir()
 - fsck: drop unused parameter from traverse_one_object()

-------------------------------------------------- [Stalled]

* nd/index-doc (2010-09-06) 1 commit
 - doc: technical details about the index file format

Half-written but it is a good start. I may need to give some help in describing more recent index extensions.

* cb/ignored-paths-are-precious (2010-08-21) 1 commit
 - checkout/merge: optionally fail operation when ignored files need to be overwritten

This needs tests; also we know of longstanding bugs in related area that needs to be addressed---they do not have to be part of this series but their reproduction recipe would belong to the test script for this topic.

It would hurt users to make the new feature on by default, especially the ones with subdirectories that come and go.

* jk/tag-contains (2010-07-05) 4 commits
 - Why is "git tag --contains" so slow?
 - default core.clockskew variable to one day
 - limit "contains" traversals based on commit timestamp
 - tag: speed up --contains calculation

The idea of the bottom one is probably Ok, except that the use of object flags needs to be rethought, or at least the helper needs to be moved to builtin/tag.c to make it clear that it should not be used outside the current usage context.

* jc/rename-degrade-cc-to-c (2011-01-06) 3 commits
 - diffcore-rename: fall back to -C when -C -C busts the rename limit
 - diffcore-rename: record filepair for rename src
 - diffcore-rename: refactor "too many candidates" logic

IIRC, this was a weather-baloon "if you wanted to, this may be how you would do it" without test updates. People who care need to help moving things forward.

* jc/rerere-remaining (2011-01-06) 1 commit
 - rerere "remaining"

Just a handful of weatherballoon patches without proper tests, in response to feature/minor fix requests.

* ab/p4 (2011-01-11) 1 commit
 - git-p4: correct indenting and formatting

Lacks sign-off but is trivial.

* tr/maint-branch-no-track-head (2010-12-14) 1 commit
 - branch: do not attempt to track HEAD implicitly

Probably needs a re-roll to exclude either (1) any ref outside the hierarchies for branches (i.e. refs/{heads,remotes}/), or (2) only refs outside refs/ hierarchies (e.g. HEAD, ORIG_HEAD, ...). The latter feels safer and saner.

* hv/mingw-fs-funnies (2010-12-14) 5 commits
 - mingw_rmdir: set errno=ENOTEMPTY when appropriate
 - mingw: add fallback for rmdir in case directory is in use
 - mingw: make failures to unlink or move raise a question
 - mingw: work around irregular failures of unlink on windows
 - mingw: move unlink wrapper to mingw.c

Will be rerolled (Heiko, 2010-12-23)

* mz/rebase (2010-12-28) 31 commits
 - rebase -i: remove unnecessary state rebase-root
 - rebase -i: don't read unused variable preserve_merges
 - git-rebase--am: remove unnecessary --3way option
 - rebase -m: don't print exit code 2 when merge fails
 - rebase -m: remember allow_rerere_autoupdate option
 - rebase: remember strategy and strategy options
 - rebase: remember verbose option
 - rebase: extract code for writing basic state
 - rebase: factor out sub command handling
 - rebase: make -v a tiny bit more verbose
 - rebase -i: align variable names
 - rebase: show consistent conflict resolution hint
 - rebase: extract am code to new source file
 - rebase: extract merge code to new source file
 - rebase: remove $branch as synonym for $orig_head
 - rebase -i: support --stat
 - rebase: factor out call to pre-rebase hook
 - rebase: factor out clean work tree check
 - rebase: factor out reference parsing
 - rebase: reorder validation steps
 - rebase -i: remove now unnecessary directory checks
 - rebase: factor out command line option processing
 - rebase: align variable content
 - rebase: align variable names
 - rebase: stricter check of standalone sub command
 - rebase: act on command line outside parsing loop
 - rebase: improve detection of rebase in progress
 - rebase: remove unused rebase state 'prev_head'
 - rebase: read state outside loop
 - rebase: refactor reading of state
 - rebase: clearer names for directory variables

Will be rerolled (Martin, Jan 28).

-------------------------------------------------- [Cooking]

* jn/unpack-lstat-failure-report (2011-01-12) 2 commits
  (merged to 'next' on 2011-01-24 at 1245180)
 + unpack-trees: handle lstat failure for existing file
 + unpack-trees: handle lstat failure for existing directory

* rr/fi-import-marks-if-exists (2011-01-15) 1 commit
 - fast-import: Introduce --import-marks-if-exists

Looked reasonable.

* tr/diff-words-test (2011-01-18) 4 commits
 - t4034 (diff --word-diff): add a minimum Perl drier test vector
 - t4034 (diff --word-diff): style suggestions
 - userdiff: simplify word-diff safeguard
 - t4034: bulk verify builtin word regex sanity

I thought this was Ok; further comments, anybody?

* ef/alias-via-run-command (2011-01-07) 1 commit
  (merged to 'next' on 2011-01-06 at 1fbd4a0)
 + alias: use run_command api to execute aliases

* uk/checkout-ambiguous-ref (2011-01-11) 1 commit
  (merged to 'next' on 2011-01-11 at 2aa30de)
 + checkout: fix bug with ambiguous refs

* cb/setup (2010-12-27) 1 commit
  (merged to 'next' on 2011-01-05 at 790b288)
 + setup: translate symlinks in filename when using absolute paths

* ae/better-template-failure-report (2010-12-18) 1 commit
  (merged to 'next' on 2011-01-05 at d3f9142)
 + Improve error messages when temporary file creation fails

* jc/unpack-trees (2010-12-22) 2 commits
 - unpack_trees(): skip trees that are the same in all input
 - unpack-trees.c: cosmetic fix

* jn/cherry-pick-strategy-option (2010-12-10) 1 commit
  (merged to 'next' on 2011-01-05 at 3ccc590)
 + cherry-pick/revert: add support for -X/--strategy-option

* nd/struct-pathspec (2010-12-15) 21 commits
  (merged to 'next' on 2011-01-24 at 08f1774)
 + t7810: overlapping pathspecs and depth limit
 + grep: drop pathspec_matches() in favor of tree_entry_interesting()
 + grep: use writable strbuf from caller for grep_tree()
 + grep: use match_pathspec_depth() for cache/worktree grepping
 + grep: convert to use struct pathspec
 + Convert ce_path_match() to use match_pathspec_depth()
 + Convert ce_path_match() to use struct pathspec
 + struct rev_info: convert prune_data to struct pathspec
 + pathspec: add match_pathspec_depth()
 + tree_entry_interesting(): optimize wildcard matching when base is matched
 + tree_entry_interesting(): support wildcard matching
 + tree_entry_interesting(): fix depth limit with overlapping pathspecs
 + tree_entry_interesting(): support depth limit
 + tree_entry_interesting(): refactor into separate smaller functions
 + diff-tree: convert base+baselen to writable strbuf
 + glossary: define pathspec
 + Move tree_entry_interesting() to tree-walk.c and export it
 + tree_entry_interesting(): remove dependency on struct diff_options
 + Convert struct diff_options to use struct pathspec
 + diff-no-index: use diff_tree_setup_paths()
 + Add struct pathspec
 (this branch is used by en/object-list-with-pathspec.)

* en/object-list-with-pathspec (2010-09-20) 2 commits
  (merged to 'next' on 2011-01-24 at 134f65c)
 + Add testcases showing how pathspecs are handled with rev-list --objects
 + Make rev-list --objects work together with pathspecs
 (this branch uses nd/struct-pathspec.)

I've been toying with the above two topics and am reasonably happy. I suspect that there could be (and probably need to be) further consolidation of the two remaining pathspec API, but this seems to be already usable.

* tr/merge-unborn-clobber (2010-08-22) 1 commit
 - Exhibit merge bug that clobbers index&WT

* ab/i18n (2010-10-07) 161 commits
 - po/de.po: complete German translation
 - po/sv.po: add Swedish translation
 - gettextize: git-bisect bisect_next_check "You need to" message
 - gettextize: git-bisect [Y/n] messages
 - gettextize: git-bisect bisect_replay + $1 messages
 - gettextize: git-bisect bisect_reset + $1 messages
 - gettextize: git-bisect bisect_run + $@ messages
 - gettextize: git-bisect die + eval_gettext messages
 - gettextize: git-bisect die + gettext messages
 - gettextize: git-bisect echo + eval_gettext message
 - gettextize: git-bisect echo + gettext messages
 - gettextize: git-bisect gettext + echo message
 - gettextize: git-bisect add git-sh-i18n
 - gettextize: git-stash drop_stash say/die messages
 - gettextize: git-stash "unknown option" message
 - gettextize: git-stash die + eval_gettext $1 messages
 - gettextize: git-stash die + eval_gettext $* messages
 - gettextize: git-stash die + eval_gettext messages
 - gettextize: git-stash die + gettext messages
 - gettextize: git-stash say + gettext messages
 - gettextize: git-stash echo + gettext message
 - gettextize: git-stash add git-sh-i18n
 - gettextize: git-submodule "blob" and "submodule" messages
 - gettextize: git-submodule "path not initialized" message
 - gettextize: git-submodule "[...] path is ignored" message
 - gettextize: git-submodule "Entering [...]" message
 - gettextize: git-submodule $errmsg messages
 - gettextize: git-submodule "Submodule change[...]" messages
 - gettextize: git-submodule "cached cannot be used" message
 - gettextize: git-submodule $update_module say + die messages
 - gettextize: git-submodule die + eval_gettext messages
 - gettextize: git-submodule say + eval_gettext messages
 - gettextize: git-submodule echo + eval_gettext messages
 - gettextize: git-submodule add git-sh-i18n
 - gettextize: git-pull "rebase against" / "merge with" messages
 - gettextize: git-pull "[...] not currently on a branch" message
 - gettextize: git-pull "You asked to pull" message
 - gettextize: git-pull split up "no candidate" message
 - gettextize: git-pull eval_gettext + warning message
 - gettextize: git-pull eval_gettext + die message
 - gettextize: git-pull die messages
 - gettextize: git-pull add git-sh-i18n
 - gettext docs: add "Testing marked strings" section to po/README
 - gettext docs: the Git::I18N Perl interface
 - gettext docs: the git-sh-i18n.sh Shell interface
 - gettext docs: the gettext.h C interface
 - gettext docs: add "Marking strings for translation" section in po/README
 - gettext docs: add a "Testing your changes" section to po/README
 - po/pl.po: add Polish translation
 - po/hi.po: add Hindi Translation
 - po/en_GB.po: add British English translation
 - po/de.po: add German translation
 - Makefile: only add gettext tests on XGETTEXT_INCLUDE_TESTS=YesPlease
 - gettext docs: add po/README file documenting Git's gettext
 - gettextize: git-am printf(1) message to eval_gettext
 - gettextize: git-am core say messages
 - gettextize: git-am "Apply?" message
 - gettextize: git-am clean_abort messages
 - gettextize: git-am cannot_fallback messages
 - gettextize: git-am die messages
 - gettextize: git-am eval_gettext messages
 - gettextize: git-am multi-line getttext $msg; echo
 - gettextize: git-am one-line gettext $msg; echo
 - gettextize: git-am add git-sh-i18n
 - gettext tests: add GETTEXT_POISON tests for shell scripts
 - gettext tests: add GETTEXT_POISON support for shell scripts
 - Makefile: MSGFMT="msgfmt --check" under GNU_GETTEXT
 - Makefile: add GNU_GETTEXT, set when we expect GNU gettext
 - gettextize: git-shortlog basic messages
 - gettextize: git-revert split up "could not revert/apply" message
 - gettextize: git-revert literal "me" messages
 - gettextize: git-revert "Your local changes" message
 - gettextize: git-revert basic messages
 - gettextize: git-notes "Refusing to %s notes in %s" message
 - gettextize: git-notes GIT_NOTES_REWRITE_MODE error message
 - gettextize: git-notes basic commands
 - gettextize: git-gc "Auto packing the repository" message
 - gettextize: git-gc basic messages
 - gettextize: git-describe basic messages
 - gettextize: git-clean clean.requireForce messages
 - gettextize: git-clean basic messages
 - gettextize: git-bundle basic messages
 - gettextize: git-archive basic messages
 - gettextize: git-status "renamed: " message
 - gettextize: git-status "Initial commit" message
 - gettextize: git-status "Changes to be committed" message
 - gettextize: git-status shortstatus messages
 - gettextize: git-status "nothing to commit" messages
 - gettextize: git-status basic messages
 - gettextize: git-push "prevent you from losing" message
 - gettextize: git-push basic messages
 - gettextize: git-tag tag_template message
 - gettextize: git-tag basic messages
 - gettextize: git-reset "Unstaged changes after reset" message
 - gettextize: git-reset reset_type_names messages
 - gettextize: git-reset basic messages
 - gettextize: git-rm basic messages
 - gettextize: git-mv "bad" messages
 - gettextize: git-mv basic messages
 - gettextize: git-merge "Wonderful" message
 - gettextize: git-merge "You have not concluded your merge" messages
 - gettextize: git-merge "Updating %s..%s" message
 - gettextize: git-merge basic messages
 - gettextize: git-log "--OPT does not make sense" messages
 - gettextize: git-log basic messages
 - gettextize: git-grep "--open-files-in-pager" message
 - gettextize: git-grep basic messages
 - gettextize: git-fetch split up "(non-fast-forward)" message
 - gettextize: git-fetch update_local_ref messages
 - gettextize: git-fetch formatting messages
 - gettextize: git-fetch basic messages
 - gettextize: git-diff basic messages
 - gettextize: git-commit advice messages
 - gettextize: git-commit "enter the commit message" message
 - gettextize: git-commit print_summary messages
 - gettextize: git-commit formatting messages
 - gettextize: git-commit "middle of a merge" message
 - gettextize: git-commit basic messages
 - gettextize: git-checkout "Switched to a .. branch" message
 - gettextize: git-checkout "HEAD is now at" message
 - gettextize: git-checkout describe_detached_head messages
 - gettextize: git-checkout: our/their version message
 - gettextize: git-checkout basic messages
 - gettextize: git-branch "(no branch)" message
 - gettextize: git-branch "git branch -v" messages
 - gettextize: git-branch "Deleted branch [...]" message
 - gettextize: git-branch "remote branch '%s' not found" message
 - gettextize: git-branch basic messages
 - gettextize: git-add refresh_index message
 - gettextize: git-add "remove '%s'" message
 - gettextize: git-add "pathspec [...] did not match" message
 - gettextize: git-add "Use -f if you really want" message
 - gettextize: git-add "no files added" message
 - gettextize: git-add basic messages
 - gettextize: git-clone "Cloning into" message
 - gettextize: git-clone basic messages
 - gettext tests: test message re-encoding under C
 - po/is.po: add Icelandic translation
 - gettext tests: mark a test message as not needing translation
 - gettext tests: test re-encoding with a UTF-8 msgid under Shell
 - gettext tests: test message re-encoding under Shell
 - gettext tests: add detection for is_IS.ISO-8859-1 locale
 - gettext tests: test if $VERSION exists before using it
 - gettextize: git-init "Initialized [...] repository" message
 - gettextize: git-init basic messages
 - gettext tests: skip lib-gettext.sh tests under GETTEXT_POISON
 - gettext tests: add GETTEXT_POISON=YesPlease Makefile parameter
 - gettext.c: use libcharset.h instead of langinfo.h when available
 - gettext.c: work around us not using setlocale(LC_CTYPE, "")
 - builtin.h: Include gettext.h
 - Makefile: use variables and shorter lines for xgettext
 - Makefile: tell xgettext(1) that our source is in UTF-8
 - Makefile: provide a --msgid-bugs-address to xgettext(1)
 - Makefile: A variable for options used by xgettext(1) calls
 - gettext tests: locate i18n lib&data correctly under --valgrind
 - gettext: setlocale(LC_CTYPE, "") breaks Git's C function assumptions
 - gettext tests: rename test to work around GNU gettext bug
 - gettext: add infrastructure for translating Git with gettext
 - builtin: use builtin.h for all builtin commands
 - tests: use test_cmp instead of piping to diff(1)
 - t7004-tag.sh: re-arrange git tag comment for clarity

It is getting ridiculously painful to keep re-resolving the conflicts with other topics in flight, even with the help with rerere.

Needs a bit more minor work to get the basic code structure right.

Heya,

On Mon, Jan 31, 2011 at 06:53, Junio C Hamano <gitster@pobox.com> wrote:

Nice, is this based on the topics that are currently cooking, or on people having indicated an intent to submit such patches?

Now that we're past 1.7.4., perhaps it's time to resurrect a dead thread. From a past "What's cooking":

On Tue, Dec 14, 2010 at 03:20, Junio C Hamano <gitster@pobox.com> wrote:

Ævar, you didn't respond to that message. Junio, do I understand correctly that if this problem is addressed the topic is ready to be merged to next?

Heya,

On Mon, Jan 31, 2011 at 16:08, Sverre Rabbelier <srabbelier@gmail.com> wrote:

It's been a week, no response from either Junio or Ævar ? :(

Sverre Rabbelier <srabbelier@gmail.com> writes:

Necessary? yes. Sufficient? I dunno, but it is a good start, I think.

Now the 1.7.4 release is out, I'd like people to help thinking about the next cycle(s).

As a discussion-starter, here are my random wishes. Even though this does not attempt to be exhaustive, keeping the number of goals manageably small may help us focus.

 * The i18n effort Ævar Arnfjörð Bjarmason started two cycles ago has
   stalled. If enough people feel i18n's Porcelain UI is worth having, I
   think we would need a brief calming period in the entire tree in order
   for us to get the minimum support (definition of _() macro that is
   empty is a good start) and _() mark-up of existing strings in, and then
   ask everybody to rebase their ongoing work on top of it.

 * There was a discussion on documentation updates to reduce "here we tell
   you only the basics; see elsewhere for details", and consolidating the
   description of configuration options in one place.

 * Nguyễn has been scratching my longstanding itch by attempting to unify
   two pathspec semantics (the ones based on tree-diff matches only
   leading path while others know globs) to reduce inconsistencies. I
   would really want to see this polished and in the main release.

 * Elijah's fix to "rev-list --objects", together with the updated
   pathspec semantics will allow us to cleanly implement narrow cloning
   (possibly deprecating and replacing the narrow checkout in the
   future).  I am hoping that we can lay groundwork on this inside one
   cycle and the initial end-to-end implementation in another.

 * Shawn Pearce says that the diff implementation JGit uses (histogram
   diff) performs way better than the xdiff implementation we use by
   default. It would be great if somebody can spend time taking a look at
   it and possibly port it back to C-git.

Over the time we have discussed minor glitches and inconsistencies that we all (or at least most of us anyway) agreed we would have done differently if we were writing Git from scratch, yet we cannot retroactively introduce differences not to harm existing users. We may also want to revisit these discussions during this round--if there are reasonable number of them that we can agree the benefit of tweaked semantics/behaviour outweighs the risk of breaking and having to update ancient scripts that exploited obscure corner case behaviour of Git, we would want warn the users loudly, bite the bullet and break them so that we can move forward. We would however need to roll such potentially disruptive changes into a big single cycle, like we did in 1.6.0.

I'll follow-up this message with a couple of example proposals. Please send your own, imitating the format of the message, as a reply to this message. Do not forget to retitle your message when you do so (iow, I don't want to see "Re: Planning for 1.7.5 and 1.8.0").

Thanks.

Proposal:

Currently "git merge" without saying what to merge is a no-op, and succeeds silently. As many downstream developers (that are by definition more numerous than people who do not have "upstream") run merge from the configured upstream of the current branch most of the time, it would make it more convenient for them to change the default to merge from the upstream of the current branch. Running "git merge" without argument, when there is no upstream defined for the current branch, will be an error.

Risks:

Existing scripts may prepare what to merge in an array (e.g. in Bourne, accumulating them in "$@" by repeatedly doing 'set "$@" "$newbranch"') and call 'git merge "$@"', relying on the current behaviour that zero argument means no-op. Such scripts will be broken by this change. Driving "git merge" with xargs without --no-run-if-empty (not POSIX), feeding the branches to merge in an Octopus, will be broken the same way.

Migration plan:

Add merge.defaultUpstream configuration variable, which defaults to false when unconfigured. Change "git merge" so that when this configuration is set and the command is run without the commit to merge to use the configured upstream of the current branch (or error out if there isn't one). Merge this change in the next 1.7.x series.

One release before 1.8.0, issue a warning when "git merge" is run without the commit to merge and this configuration variable is not explicitly set either way, and notify the user of upcoming incompatibility.

In 1.8.0, flip the default for merge.defaultUpstream to true.

On Mon, Jan 31, 2011 at 09:06:33AM -0800, Junio C Hamano wrote:

I am not sure these things are not already broken. "git merge" without arguments right now is not a no-op. It is an error that spews usage to stderr and exits 129. Yes, there can be scripts which stupidly do not bother to see if the merge succeeded, but I'm not sure how much we should care about such poorly written junk.

One nit: upon reading the name of the variable, I assumed it would be "the default upstream to merge". Perhaps "merge.defaultToUpstream" is a more descriptive name?

Don't we already issue a giant warning when "git merge" is run without a commit, namely:

  usage: git merge [options] <remote>...
  ... etc ...

? If people are already not paying attention to that (either because they are throwing away stderr and exit code indiscriminately, or because the no-arguments case is a simply an obscure corner that their script doesn't usually exercise), why would they pay attention to a new warning?

Other than that, I think the proposal and migration plan are fine.

-Peff

Jeff King <peff@peff.net> writes:

Heh, thanks.

It's not my proposal though ;-). It is just an illustration of how I see an item from people's wishlist should look like to make it easier to discuss them.

On Mon, Jan 31, 2011 at 10:17 PM, Junio C Hamano <gitster@pobox.com> wrote:

Ok, I share the same opinion as Jeff; we should not care about scripts doing something so wrong. But if the migration path is the only way to get the change in, I guess I can do that.

Ok, here is an update.

I wonder how we should keep track of the "proposal" series, though. Sending the full-text replacement like I am doing here feels eh not gittish.

Perhaps I should start a new directory in todo branch (say, 1.8.0), accept patches from people? I'd grudgingly admit that using Wiki on k.org may be less burdensome (I hate editing inside the browser myself), but I'd want to keep the mailing list the center of discussion and am afraid that forcing people to go to Wiki would fragment the discussion.

I dunno.

--

Proposal:

Currently "git merge" without saying what to merge fails loudly. As many downstream developers (that are by definition more numerous than people who do not have "upstream") run merge from the configured upstream of the current branch most of the time, it would make it more convenient for them to change the default to merge from the upstream of the current branch. Running "git merge" without argument, when there is no upstream defined for the current branch, will be an error.

Risks:

Existing scripts may prepare what to merge in an array (e.g. in Bourne, accumulating them in "$@" by repeatedly doing 'set "$@" "$newbranch"') and call 'git merge "$@"', relying on the current behaviour that zero argument will flag an error condition. Such scripts will be broken by this change.

Driving "git merge" with xargs without --no-run-if-empty (not POSIX), feeding the branches to merge in an Octopus, will be broken the same way.

Migration plan:

Add merge.defaultToUpstream configuration variable, which defaults to false when unconfigured. Change "git merge" so that when this variable is set and the command is run without the commit to merge to use the configured upstream of the current branch (or error out if there isn't one). Merge this change in the next 1.7.x series.

Update the error message issued when when "git merge" is run without the commit to merge and this configuration variable is not explicitly set either way to notify the user of upcoming incompatibility.

In 1.8.0, flip the default for merge.defaultUpstream to true.

Helped-by: Jeff King <peff@peff.net>

On Mon, Jan 31, 2011 at 12:50:30PM -0800, Junio C Hamano wrote:

I really wish we had a git-backed wiki. I also hate using the browser for such things (though browser extensions to edit textareas in a Real Editor at least make it tolerable, it still ends up clunky).

GitHub's wiki gets this right. I'm not saying we should host our wiki there (well, it _would_ make setting it up pretty damn easy). But their wiki system (gollum) is open-source, albeit in ruby. And surely there are other git-backed alternatives (it's been a while since I've looked).

Well, I don't know this is still just an example, but your update looks fine to me. :)

-Peff

On 31 January 2011 22:55, Jeff King <peff@peff.net> wrote:

You mean like ikiwiki (were it used on the main sites)?

http://ikiwiki.info

-- Thomas Adam

Hey,

On Mon, Jan 31, 2011 at 2:55 PM, Jeff King <peff@peff.net> wrote:

If you want to use the wiki on the git/git repo on GitHub that is being mirrored from the canonical repository, I've added Junio and peff to the account. If you want to use that wiki, anyone with a github account can edit wiki pages on the site or clone and edit it locally and push changes up. You can also turn off site edits so people have to send Junio a patch instead. It's up to you guys, but the access is there now if you want.

Scott

On Tue, Feb 01, 2011 at 10:34:24AM -0800, Scott Chacon wrote:

Out of curiosity, I scraped the kernel wiki and put it into gollum. The results are here, if people want to see what it looks like:

  https://github.com/peff/foo/wiki

It's extremely quick and dirty, which is why I didn't put it in the actual git/git/wiki spot you made. Some of the formatting is off (note that I didn't do any conversion; it understands mediawiki natively), and some of the content is probably missing (my scraper was extremely naive). For a real import I would try to get the actual wiki db from kernel.org and import the entire history.

I dunno what the next step would be. I would really prefer a git-backed wiki, but moving to something like this is a pretty big step in workflow for people who use the current wiki. I would be curious to hear general opinions (on the idea of moving, assuming we did a conversion that actually looked good). I'll change the subject and see if we get any comments.

-Peff

On Tue, Feb 1, 2011 at 3:11 PM, Jeff King <peff@peff.net> wrote:

A git-backed git wiki would be great. As a related matter, the hosting infrastructure for https://git.wiki.kernel.org/index.php/Main_Page seems overloaded. About half the time I try to access it, it's either down completely or very slow to respond. If the wiki were git-backed I could get to it even if the central site were down. :-)

That said, didn't the wiki just migrate to Mediawiki recently?

j.

On 02/01/2011 02:36 PM, Jay Soffian wrote:

The wiki will almost universally have a "central site" no matter what the backend. Personally I see little advantage to having a git backed wiki myself.

Speaking to the slowness, it's caused by at least 2 different kernel related bugs on the two boxes that run the wikis that I haven't had enough time to track down to nail to specific developers to fix. I have 20u of equipment sitting in my apartment that is heading to Portland in the next two weeks to eliminate the bits I'm pretty sure are the root cause of the problems.

Trust me when I say it's not only been a thorn in my side, and something I've been rather angry at several people about, but it's something that has kept me up at night trying to get fixed.

It did.

- John 'Warthog9' Hawley

  On 01/02/11 22:48, J.H. wrote:

with git based wiki, you can clone the whole wiki on your local machine, and read/edit/commit on it locally using standard editor tool (i.e. $EDITOR). and the history/revision/diff is completely built-in.

On Wed, Feb 2, 2011 at 11:55 AM, Vincent Hanquez <tab@snarc.org> wrote:

But there's no git based wiki (or any other wiki) that has even a fraction of the features that MediaWiki has, or IMO a markup format similarly sane.

Felipe Contreras <felipe.contreras@gmail.com> writes:

Gollum (the engine used by git-based wikis on GitHub) offers support for MediaWiki format. I don't know how it is with editing and presenting history (end e.g. reverting spam), especially on older or text browsers.

On 02/02/2011 01:55 AM, Vincent Hanquez wrote:

That would be fine for things like source code or documentation, but you end up with a single person who would need to merge / push things to a central location, a-la git.wiki.kernel.org. You are now taking something, that is already editable by anyone, and making it only editable by a single person.

You also have a scalability problem. Git is *VERY* memory and i/o intensive. While you basically have a cache of data that is static (the basic pages you are viewing) things like the history, edits, etc can be quite expensive to generate.

Think about a site, we'll use git.wiki.kernel.org, where it's not running on a single machine, but a cluster of machines (how many web infrastructures, including git.wiki.kernel.org run) and now you have a problem of an edit happens and commits on node3, a different conflicting edit happens on node9 and when those try to merge - you get conflicts.

Let me be clear here, I think the idea is interesting, but I think in trying to replace a full wiki it's a horrible idea, particularly since you are pushing a lot of manual work - somewhere, and trying to use git as a nosql database without some sort of locking system.

Just my $0.02 though.

- John 'Warthog9' Hawley

On Wed, Feb 02, 2011 at 06:24:23PM -0800, J.H. wrote:

I don't think it makes sense to use the same workflow for the wiki as git.git itself uses. The point of having a wiki is to keep the barrier to editing extremely low; the point of source code control is to keep the quality of contributions high.

But that doesn't mean they can't be accessed by the same tool.

Forget about a git-backed wiki for a moment, and imagine a regular old Mediawiki. What are the operations you can perform? You can look at the current or any past version of a page, you can do diffs between versions of pages, and you can create a new version of a page. All through some CGI forms.

So what stops us from replacing the CGI interface with a git one (or adding it alongside)? Given the ability to retrieve current and past versions of all pages, I could surely build a git repository of the whole wiki (and update it incrementally as new edits were made). And pushing a set of commits is just a sequential series of page edits, no?

And I think that would be enough for the purposes of this discussion. Most of us don't really care if git is the ultimate storage mechanism. I could even build the git interface as a purely client thing on top of the CGI interface; the problem is that scraping the wiki pages for new versions over the net would be horribly inefficient.

But the point is that accessing the wiki via git is not about changing the wiki workflow. It's about providing a richer set of tools for doing those page views, diffs, and edits.

Getting back to git as the actual backend:

Sure. But is that any worse than running gitweb, which you already do? Or a site like GitHub, which basically is just running git constantly on a bunch of repos? Or how much worse is it than running regular wiki software? I mean, Foswiki is backed by RCS, for god's sake. Surely git is more efficient than that. ;)

If it sounds like I'm handwaving away scalability problems, I am. I'd be curious to see some performance numbers for gollum or ikiwiki versus more traditional wiki formats.

Don't you have the same problem with a regular wiki? Or with stock git repos, for that matter? You need database consistency.

-Peff

Heya,

On Thu, Feb 3, 2011 at 18:45, Jeff King <peff@peff.net> wrote:

Note that MediaWiki has a stable API that you could use instead :).

On Thu, Feb 03, 2011 at 08:06:34PM +0100, Sverre Rabbelier wrote:

The initial import is still pretty painful over the network, but at least I know I am getting accurate data from the API now. And I can share the git repository, and just use the RecentChanges API to get updates.

The result is at:

  https://github.com/peff/wikitest/wiki

You can see an example of the full history:

  https://github.com/peff/wikitest/wiki/_history

or a page history:

  https://github.com/peff/wikitest/wiki/GitFaq/_history

Or download the repository for yourself:

  git://github.com/peff/wikitest.wiki.git

The whole git wiki is about a 4M git repository (and at least some of that is spam that's deleted but still in the history).

This is has been a cute exercise, but I'm not sure mirroring it in a gollum wiki really makes sense. It's cool that gollum understands MediaWiki enough to actually render the pages at all, but there are obviously lots of corner cases that it just doesn't get (formatting issues, missing images, some redirect naming issues).

Still, it's useful as a local repo doing diffs. I just wrote the initial importer (which I'm happy to share if somebody wants to see some ugly code), but to act as a useful client, it still needs:

  1. Incremental updating from the RecentChanges API.

  2. Push support (or more likely something similar to "git svn
     dcommit") to push edits back.

-Peff

On Thu, Feb 3, 2011 at 7:45 PM, Jeff King <peff@peff.net> wrote:

Howe about these?

1) Support for discussion; since changes can be controversial.

2) Support for article move; so everything is kept organized.

3) Support for "who is linking here". Also helps reorganization.

4) Support for categories. Ditto.

5) Support for watchlist, e-mail notifications. So that you are
up-to-date with the changes.

6) Support for contribution backtracking. So that it's easy to know who's who.

7) Personal wiki pages (with discussion). So you can put information
about yourself, and general notes.

On Thu, Feb 03, 2011 at 10:34:38PM +0200, Felipe Contreras wrote:

I've never really used a wiki in any in-depth way, so be gentle if my utter cluelessness about these features shows through.

Doesn't this just happen on special Talk: pages? Couldn't these just be pages with special names?

Isn't that even simpler in a git-backed wiki? You just move the files. Obviously you would want to update links, too, and presumably the wiki software helps with that. But that is outside the scope of data storage. In a git-backed wiki, you would get one atomic commit with the move and link updating.

Again, is that a fundamental storage issue? It seems like you could implement that easily on top of basic storage with a grep (and some caching if you are going to let people do it a lot via the web).

I have no idea how categories work. Special page naming and/or directories?

Post-receive hook?

git log? git blame? :)

Specially named pages for people?

Obviously I'm just filling in these features off the top of my head. MediaWiki is a mature system, and I doubt that either ikiwiki or gollum has nearly the same featureset. But that was never my point. In the bit you quoted, my point was that a git-interface to a wiki was useful and feasible. And I stand by that.

Even with just the basic functionality of fetch/diff/push, that still makes it a useful interface into an existing wiki for a large number of users who just want to do simple stuff (or power users who happen to be doing simple stuff at the moment).

I also happen to think you could put all of those features into a git-backed wiki, and build the web features on _top_ of git access. But I'm not volunteering to work on that.

-Peff

On Fri, Feb 4, 2011 at 8:16 AM, Jeff King <peff@peff.net> wrote:

Could be.

Yeah.

I doubt that. That's where you need an SQL database, to make it fast.

Each page has a tag [[Category::Tips]], and then the the Category::Tips page gets a new link automatically. Again, SQL helps.

Yeah, but you need to store the watchers, and their preferences. Again, SQL.

Sure, 'git log' would do it... Very inefficiently.

Right.

It might be possible to implement some functionality of a full blown wiki such as MediaWiki on a git backed wiki, but my point is that it's not there _now_, and more likely would never be.

Exactly, and nobody is volunteering for that. MediaWiki is the best, it has all the features, and it's already there.

Cheers.

Jeff King wrote:

Ikiwiki builds static pages, this tends to mean it doesn't have performance, because there is little more to perform than httpd < file > network :) So I've routinely had ikiwiki sites slashdotted, and not noticed.

Ikiwiki is not enormously fast in the rare cases when it does have to run as a CGI, but little of that has to do with git. About the worst case is that saving a page edit leads to a git commit -- if git decides to gc the repository right then, it could make the save stall for a while. There are easy ways to avoid that. (ie, git gc in cron job) In general, though ikiwiki as a CGI is fast enough to not be annoying -- although it won't scale to a site the size of wikipedia.

Since I'm lazy, ikiwiki does not include a history or diff viewer. It just points off to gitweb or a similar tool. As you say, gitweb can be fast enough, and really most wiki users do read their current content, or maybe make an edit; digging in the history is comparatively rare. And once users realize the wiki is in git, they can use gitk to dig in the history without using any server resources. :)

The feature I like best with using git for a wiki (besides ease of branching) is that it can actually make a legitimate use of the woefully underused git push over git:// feature. Ikiwiki can be configured to check such pushes, running via the pre-receive hook. This allows it to limit the changes that can be pushed anonymously to changes that could be made using the web interface. So if you've chosen to lock some pages, or virus filter attachments, or whatever, in the web side of the wiki, that all applies to the anonymous git pushes too. Details at <http://ikiwiki.info/tips/untrusted_git_push/>

On Fri, 4 Feb 2011, Joey Hess wrote:

I think you mean it doesn't have performance _problems_ ;-)

David Lang

Jeff King schrieb:

ikiwiki, for example, with Hosting on <http://branchable.com/>?

Adds the option merge.defaultupstream to add support for merging from the upstream branch by default. The upstream branch is found using branch.[name].upstream. ---

This is just testing code; it works but I'm not yet sure if it breaks other things. I tried to code it so it doesn't.

Note: First time using git send-email; I hope I set it up correctly with the
In-reply-to and Cc's and such.

 builtin/merge.c |   41 +++++++++++++++++++++++++++++++++++------
 1 files changed, 35 insertions(+), 6 deletions(-)

Proposal:

Traditionally, Git had two distinct semantics for pathspecs. Anything based on tree-diff (i.e. "log" family of commands when limiting the history by paths or "diff" family of commands limiting the output) used "leading paths match" without globbing support. All others (e.g. "grep", "ls-files") supported globbing. This resulted in subtly inconsistent behaviour when one part of the program collected paths from the index and the working tree while another part of the program used differences between the index and the HEAD, e.g. "git add". Unify "pathspec" semantics to make all of them learn the globbing.

Risks:

If coded poorly, performance bugs can be introduced to the tree-diff codepath, making it inefficient.

Some projects may track a file whose name is asterisk (e.g. "foo/*") and output from "git log 'foo/*'" would look different. Before the change, only commits that touch that exact path would be shown, but after the change, any commit that touch a path underneath "foo/" directory will be shown. This is a backward incompatible change.

Migration plan:

We could conditionally enable globbing support when implementing unified pathspec API, default to the traditional and inconsisntent behaviour during the 1.7.x series, and flip the default to accept globs everwhere in the 1.8.0 release. Practically, however, nobody sane would track paths that have shell metacharacters in them, so we may not need to do the usual "introduce as an opt-in, warn about incompatibility, and flip the default" migration.

On Tue, Feb 1, 2011 at 12:07 AM, Junio C Hamano <gitster@pobox.com> wrote:

Can we support quoting wildcards? I can imagine a file name such as '***DO NOT DO IT***'. People who wish to match exactly that file would have hard time ahead without a way to tell git those stars are literal.

A prefix/special leading symbol or cmdline option to indicate the given pathspec is literal is fine too (e.g "!literal:***hey***" or --literal "***hey***"). In fact I can extend that to support negative pathspecs.

To me the source tree looks highly untidy to me.

We do have subdirectories for documentation, tests, contributions, etc. But a sizeable part of the tree is just a big splat of source files dumped right in the root of the tree.

So I'd suggest doing the following:

1) Create a src/ directory and move *.c, *.h, *.sh, *.perl, *.py and 
   the builtin directory from the root directory to it.

2) Create a build/ directory, or bin/ if prefered, to hold the result of 
   the build.

3) Consider dropping the ppc/ directory.  Unless someone really cares 
   deeply, it would be nice to simply always use the block-sha1 code and 
   move it straight into src/.

4) Consider moving some more directories into src/ such as xdiff/.
   I'd leave compat/ outside src/ to make it more explicit that this is 
   not about Git proper.

5) Rename t/ to testsuite/ so this doesn't look like some garbage 
   leftover.

6) And fix up all the Makefiles to cope with the above movements.

What do you think?

Nicolas

Nicolas Pitre <nico@fluxnic.net> writes:

Knee-jerk reaction: not very motivated to make the top-level directory just a skeleton that holds various directories with a handful of administrative files like Makefile, README, etc. Under your proposal, the bulk of the current content at the top would simply move to another single directory anyway, so I don't immediately see much point of such a move, other than adding merge burden on me and rebase burden on others, that is.

But that is just a knee-jerk reaction, just to fill the "Risks:" section you didn't fill. Your missing "Migration Plans" section might outline a clever approach to lessen the interim hurt while merging in-flight topics.

Junio C Hamano <gitster@pobox.com> writes:

There would be at least one obvious benefit: currently, we have this

git$ ls | wc -l 623

(that's after a build)

It's a bit hard to find the interesting bits (README, Documentation/, contrib/ for example) in the output of "ls".

That can be seen as a test of how good Git is at bulk rename management ;-).

All that said, I cannot really say whether the benefit is higher than the cost.

On Mon, 31 Jan 2011, Matthieu Moy wrote:

There is a huge value in inflicting on ourselves such a test case for the tool we produce. That helps avoiding the ivory tower syndrome.

Nicolas

On Mon, 31 Jan 2011, Junio C Hamano wrote:

I really think that the top directory is not the proper place for source files to live, especially considering how big a project Git is now. The top directory should be like a table of content and not the content itself.

But if you the maintainer doesn't see a long-term value in this to be greater than the one-time burden, then I'm afraid there's nothing I can do to help it.

Well, there is no such plan. Given that 1.8 is meant to be an inflexion point for users, it could as well be for developers the best time to clean up this mess too.

Nicolas

On Mon, Jan 31, 2011 at 03:28:37PM -0500, Nicolas Pitre wrote:

Wouldn't this just be the same giant splat of source files, but in a different tree? I don't really see the advantage, and it seems like an extra annoyance. Besides being just one more directory to go up and down, it does make history browsing more annoying. As much as I love git's "don't record renames" philosophy, our handling of renames on the viewing side is often annoying. I already get annoyed sometimes following stuff across the s!builtin-!builtin/! change. This would be like that but more so.

Or maybe it is a good thing for that reason, as we will eat our own rename dogfood. :)

Ugh, more typing. :P

-Peff

On Mon, 31 Jan 2011, Jeff King wrote:

Like I said to Junio, if you don't see the advantage, there's nothing I can do for you. To me this is simple good source code hygiene.

So... we do suck at something? So why not take this opportunity to shake yourself out of this easy comfort and improve Git as a result on both front? :-)

Exactly!  And maybe we'll make Git even more useful in the process.

Come on!  You sound like an old fart now!  ;-)

Nicolas

Nicolas Pitre <nico@fluxnic.net> writes:

This part I _could_ actually buy; even though I do not think moving files without much reason is a good project hygine, it does happen in real life, and we would want to keep things smooth for real people.

I am not sure about this "t/" vs "testsuite/".

If we make the top-level directory lean enough, we probably can tab complete after typing just "cd t" to go to testsuite/ or tests/ or whatever you come up with, so "more typing" is not a huge issue to me personally.

I however think the directory name "t/" is not our invention but what we took from somebody else (perhaps Perl?), and I suspect some people expect to find tests under there since we have had them there for a long time.

I think a good code hygiene is important, and some suggestions here are relevant:

1) I also think there should be a directory for the source code
(namely, src/), and the top level should act as a table of contents.
As a newcomer myself who's trying to grasp Git, I can say an organized
project makes people more inclined join. Therefore, directories should
be named as clearly as possible: see item 2;

2) For item 1, t/ should be renamed to testsuite/. As Junio said, if
we get a more organized project, people could just "cd t" and tab to
autocomplete, or "cd tes" at the worst scenario, which is not such a
big hassle. About people expecting the testsuite to be inside t/, once
they type "cd t" and get an error, most people would look for an
alternative and eventually find the correct folder, or tabbing would
just suggest the name.

3) The top level should hold files that point people towards where
they want to go, helpful files like README and even some
Documentation/ files could get a promotion.

3) As 1.8.0 can be an inflexion for users, so could be for the
developers as well.

On Mon, 31 Jan 2011, Junio C Hamano wrote:

If those people are not able to figure out that "testsuite" means where tests are, especially within a lean top directory, then we might question the reliability of the tests they might contribute.

Nicolas

On Mon, Jan 31, 2011 at 04:28:49PM -0500, Nicolas Pitre wrote:

Yes, we do suck at rename following. The problem is that it is partially an implementation issue, and partially a fundamental issue. Obviously --follow sucks pretty hard right now, and that could be fixed. Namely it follows only a single file, and it interacts very badly with history simplification.

But even with those things fixed, there will still be annoyances.

It will still be _slower_ to turn it on all the time, for one[1]. And that's due to fundamental design decisions of the git data structure. And I'm not knocking those decisions; I think they made the right tradeoff. But that doesn't mean we don't pay the cost for that tradeoff.

And no matter what your model, renames can be annoying. On-going topics will have a painful rebase or merge. And people looking at history will have to deal with the code-base having different names at different points. Yeah, you can say it's all just "content", but the filenames we put things in are actually useful.

So I don't think it's wrong to say "renames are a pain, and so should not be done lightly". I do think it's wrong to say "renames can't be done"; I just the cost needs to be considered.

-Peff

[1] I'd be interested to see how much we can get around that slowness using a notes-cache.

On Mon, 31 Jan 2011, Jeff King wrote:

This is no excuse not to do proper source tree reorganization.

Telling people not to move files around because Git sucks at tracking them is also the wrong answer.

I agree. However sitting on our back and resisting a cleanup just because our very tool does poorly in that scenario is just like putting our heads in the sand and pretend that the problem doesn't exist. Better do what most people without internal knowledge of Git would do and just clean up the tree, and then benefit from this extraordinary opportunity of having this environment right at home where Git developers have a much greater incentive to work on this issue and improve things.

Of course. But such is life. Many projects out there are just like that, and facing this situation ourselves will just help us figure out ways to make Git even more useful to more people.

I disagree. This is like saying: "renames are not well supported, so let's avoid them while using Git." People used to say that of merges with CVS. Are we going to follow suit de facto? Imagine the Git detractors taking our source tree mess to exemplify this Git flaw since "Git developers themselves are unwilling to move files around because Git sucks at it".

Instead, why not saying: "Rename tracking is not as optimal as it could be, so let's work it out." ?

Nicolas

On Mon, Jan 31, 2011 at 07:29:54PM -0500, Nicolas Pitre wrote:

I think this is the crux of our disagreement. I don't agree that your proposal is any way more "proper" than what is there now. Leaving the rename issue aside (i.e., if we were starting a new project), I would still be slightly against a src/ directory. I find them annoying.

But I don't care _that_ much, and I would rather not waste either of our time debating it more. I would much rather you spend your time on pack v4. :)

For the record, part of my argument was that renaming is annoying to some degree in _all_ systems, not just git.

I did also say that. :)

-Peff

On Mon, 31 Jan 2011, Jeff King wrote:

Let's agree to disagree then. What I see in the root of the Git source tree is a huge clutter of source files, binary files, scripts, and subdirectories all mixed together. If you know by hart where things are because you've been hacking on them for the last 5 years then of course you might not see the point. But since I didn't work much on Git lately, things are not as obvious to me as they used to be. Looking back at it now with some distance, this tree looks like a mess and it is really annoying to work with.

I wish... I wish. But I have a plan which might involve taking some vacation from $day_job in the Caribbeans with $wife and no kids, where $wife is going to do scuba diving with her club mates while I'll be alone with a laptop and no net connection and therefore nothing else to do for a week. I've been craving for such free time for quite a while now.

Nicolas

Nicolas Pitre wrote:

But judging by that assessment, shouldn't we strive to make it *easier* to find things?

In particular a prospective git hacker would not care whether something is a source file or a script (you seem to imply the opposite). He would instead expect to find git-foo implemented in something named of that sort, so we could probably help him by mapping

  git-foo.sh      ->   git-foo.sh
  builtin/bar.c   ->   git-bar.c
  baz.c           ->   lib/baz.c
  baz.o           ->   build/baz.o (or whatever, just elsewhere)
  baz.gcov        ->   build/baz.gcov (ditto)

(I'm no huge fan of src/ either, but this should be orthogonal.)

Thomas Rast wrote:

I agree. This sets off my "time to resist change" alarms much less than "git mv *.c *.sh src/", for what it's worth.

Maybe something like this to start?

-- 8< --
Subject: Makefile: basic support for separate build dir

 - python and perl machinery haven't been tweaked yet
 - requires good VPATH support
 - relies on COMPUTE_HEADER_DIRECTORIES to make the object file
   directories
 - does not support paths with spaces

Usage:

	mkdir output
	cd output
	echo COMPUTE_HEADER_DIRECTORIES=1 >config.mak
	make -f ../Makefile GIT_SRC=$(pwd)/../ -j2

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Makefile            |   41 ++++++++++++++++++++++++++---------------
 generate-cmdlist.sh |    4 ++--
 perl/Makefile       |    2 +-
 3 files changed, 29 insertions(+), 18 deletions(-)

Jonathan Nieder wrote:

Erm, COMPUTE_HEADER_DEPENDENCIES. Anyway, if someone wants to make setting GIT_SRC take care of making the directories automatically, I wouldn't mind. ;-)

On Tue, 1 Feb 2011, Thomas Rast wrote:

I'm not proposing to go that far, especially given the current resistance to any changes. IMHO anything that unclutters the top directory is good.

Nicolas

On Tue, Feb 1, 2011 at 7:42 PM, Thomas Rast <trast@student.ethz.ch> wrote:

A hacker is expected to RTFM first, IMO. Put up a document describing how things are organized in git and we're good. git-grep will take care from there.

Jeff King <peff@peff.net> writes:

I am with you, both counts on this topic. I don't necessarily agree that having sources at the top-level is bad, I don't want to see Nico wasting his time arguing, and I do see some value in the proposal that gives us an opportunity for dogfooding (but we already have done so with builtin/ and it was not all that annoying---I think the timing was rather good and the tree was semi-quiescent).

Ehh, that makes it not "both" but "two and half" counts ;-).

As long as the new directories are named sanely (one of the things I detest is abbreviated uppercase silliness like "Src", "Lib"), I am fine with the proposal. Also I have a mild preference to keep build-products next to the source (i.e. no separate "obj" directory).

On Tue, Feb 1, 2011 at 12:12 AM, Jeff King <peff@peff.net> wrote:

Does it really have to be? I mean, for whole-file rename-detection, if we say that we automatically enable rename-detection (by default) as we reach the first commit that doesn't have a given tree-entry, then it would only kick in as we're about to terminate the log-output. And since we're usually reading through a pager, it should means that it takes a little bit more time before the user knows he's at the end of the log. It shouldn't really affect the throughput of the data before the point that becomes an annoyance, right?

At least that's the part that I find the most annoying with the current rename detection; having to enable the flag as I reach the end of the history for a file, often having to search through a lot of commits again.

On Tue, Feb 01, 2011 at 01:35:16AM +0100, Erik Faye-Lund wrote:

That's not quite true. You may be following not just a single file, but some arbitrary pathspec. Any time there is a file creation event that matches that pathspec, you will want to rename-detect any possible sources for that created file, and add them to the list of interesting paths.

But yeah, we don't have to do rename detection for every single case. So the slowness may turn out to be not that bad. When I had arbitrary-pathspec following this summer I thought I did some numbers, but I don't remember the results and I can't find them on the list.

-Peff

Heya,

On Tue, Feb 1, 2011 at 00:12, Jeff King <peff@peff.net> wrote:

Do you mean something like a refs/notes/renames namespace in which we stick notes on commits indicating that a rename indicated at that commit, with an option of the user, after-the-fact, adding this information manually?

On Tue, Feb 01, 2011 at 02:00:05AM +0100, Sverre Rabbelier wrote:

Yes, without the "option of the user..." bit. Basically just cache the list of renames for a given commit against its parent (which should be immutable[1]), under the assumption that it is cheaper to look up the note than it is to calculate the renames.

But I would make it purely a cache, not some place for users to stick their own generated rename information (if people really want to do that, I would much rather see it go into the commit itself as a pseudo-header).

-Peff

[1] It's technically not immutable if you limit the pathspec, or if you have non-standard rename options. But you could define some "canonical" rename set, like all of the pairs found doing rename detection with -M90 when considering the whole set of removed files as sources and added files as destinations. That would cover the common case of people running "git log", and then more specialized detection would not use the cache.

On Mon, Jan 31, 2011 at 6:12 PM, Jeff King <peff@peff.net> wrote:

I have been dealing with this quite a bit lately as Chromium has been doing mass renaming. It's no small project and sometimes those merges are big:

[diff]
	renames = copies
	renameLimit = 2000

:-)

What I can say is, yes, it's annoying, but also: git does quite a decent job of it. I've found myself having to do this occasionally, but not too often:

  git diff ...MERGE_HEAD -- /path/to/old/name | patch /path/to/new/name

(Typically when a header is renamed, a stub is left in place at the old name which just includes the new name, and the local changes don't come across to the new name 100% cleanly.)

Anyway, I would welcome git.git getting a little taste of that. :-) :-) :-)

j.

On 01/31/2011 10:28 PM, Nicolas Pitre wrote:

Personally, I kinda like the capital D in Documentation for tab completion reasons. Keeping frequently used files and directories with short unique prefixes makes perfect sense from a typing point of view. Using longer mnemonic names makes perfect sense from a regex search/replace point of view.

I'm kinda with Junio on the ./*.[ch] -> src/*.[ch] move though, but perhaps that's just because I hate autoconf projects which generate a ton of cruft in the root before one can even start building it.

It would probably help matters along if buildproducts ended up in their own directory though. That way .gitignore won't have so many extra commits when new source files are added, and 'make clean' gets easier to maintain.

So to sum up what I'm for;

t/ -> Test/ (or Testsuite, but some more mnemonic name anyways with a short unique prefix for tab completion). This would also exercise our rename machinery quite a bit, altohugh not to the point where people get annoyed if it gets sort-of-broken.

buildproducts to Build/ (capital B to avoid completing against builtin*). This also has the benefit that %.o: %.c rules makes tab completion work better when object files are already built, and "git add git-foo.*" doesn't throw the "git-foo.o is ignored" error and forces one to re-type it as "git-foo.[ch]"

The ppc stuff I don't really care about and it wouldn't be hard to resurrect if we remove it and people complain. Everything will also still work if we do, although with possibly a slight decrease in performance.

Bulk of source-files stay as ./*.[ch]. I see absolutely no benefit to moving them, but two potential drawbacks. One is that it'll cause more typing for those of us who use console-based editors and run 'make' manually (yes, we do exist). The afore- mentioned merge+rebase hell is also a considerable drawback, even though that's primarily an issue for maint releases.

Risks: Well... dunno about that really. Mucking up the build and
test systems, I suppose, but it's easy enough to test.

Cons: Rebasing and merging stuff to the Makefile and test-stuff
will suck a bit, but as has been pointed out that's not only a
bad thing.

Pros: Less typing all-round. Simpler "make clean" rules. Fewer
tacked-on .gitignore patches for new commands. We (well, Junio)
get to experience first-hand the problems of directory renames
across releases but on a smaller scale than moving *everything*
around in one go.

Nicolas Pitre <nico@fluxnic.net> writes:

Nope. 't/' is the standard name for directory with "normal" tests, at least in Perl / CPAN land, where TAP comes from ('xt/' is for extra tests)

On Mon, 31 Jan 2011, Jakub Narebski wrote:

So what? It is not because Perl has set this horrible precedent that we have to perpetuate it. I personally never saw t used as a directory name for tests before Git, and I'm not that young anymore unfortunately.

Nicolas

The MySQL project (and its clones) also uses the t/ convention.

On Tue, 1 Feb 2011, Alex Budovski wrote:

OK, that makes for another one.

Now what about those hundred counter-example projects _not_ using "t" but something more descriptive?

Nicolas

On Tue, 1 Feb 2011, Nicolas Pitre wrote:

CPAN is around 21000 distributions and 88000 modules. Most of those use 't/' for tests (default CPAN client to download and install modules runs those tests before installing module).

You mean those counter-example projects that use and include comprehensive tests, isn't it?

Well, GCC uses 'testsuite/'.  Mozilla uses 'testing/'.

Nicolas Pitre <nico@fluxnic.net> writes:

I am fine with "tests/", by the way.

On 01/02/11 14:15, Junio C Hamano wrote:

FWIW, CCAN and Rubygems both use plain "test".

"t/" is a long standing convention for many projects, dating back to at least 1987 with Perl 1.0. As far as I know it was unusual at the time for being a very test-driven development and inventing the TAP (Test Anything Protocol). It is something of a mark of respect to the heritage of TDD to have it there, but that doesn't mean that "test" might be better for the project now.

I'd also think about removing the "t" from the front of all the test script names, those are a PITA :-). And possibly allowing the scripts to be broken into subdirectories.

Sam

Two largest files in git repository are gitk and gitweb, see the output of the following command

  $ git ls-tree --abbrev -r -t -l v1.7.4 | sort -k4,4 -n | tail

I can't say much about splitting gitk, besides the fact that git-gui which was of comparable size IIRC got split into smaller files; I guess similar thing could be done for gitk.

There was an attempt (ultimately failed) on splitting gitweb during Google Summer of Code 2010. At least providing infrastructure for multiple gitweb modules is very much required for adding code-intensive new features, like gitweb output caching.

On the other hand this might make gitweb harder to install...

Next in size is compat/nedmalloc/malloc.c.h (I don't know if it can be reduced in size), and git-svn. I think we could separate core functionality into Git::Svn module or something, and make git-svn smaller (perhaps reusing some code in/from svn remote helper).

What do you think?

Jakub Narebski <jnareb@gmail.com> writes:

What does this have to do anything with 1.8.0? Isn't this all internal implementation that can be brought in without affecting end users?

Dnia wtorek 1. lutego 2011 23:15, Junio C Hamano napisał:

In the case of gitk we have "prior art" i.e. git-gui, which got split. In the case of gitweb I am not sure if there having multiple files to install wouldn't be inconvenient for users (though with "install-gitweb" target...)

Anyway, I have posted this in subthread of

  [1.8.0] reorganize the mess that the source tree has become

because it is also code reorganization.

On Mon, 31 Jan 2011, Nicolas Pitre wrote:

I don't care much about the other items on the list, but I do agree with this one. The biggest reason I like this is because it makes it easier to tab complete. In all the cases so far that I have tab completed "git-rebase--i" to open it in an editor or to run some git command on it, I have wanted "git-rebase--interactive.sh"; I have never wanted the build result.

It is also nice to have one less file to edit (.gitignore) when you add a new source file, but that is of course much less important.

Are there any arguments against this part of Nicolas's proposal?

Btw, this is not really related to 1.8.0, is it? It seems to me like it could be done at any time...

/Martin

Proposal:

Running "git fetch origin master" only updates FETCH_HEAD, not origin/master, which turns out to be quite confusing for newcomers especially after running 'git pull origin master'.

Since the remote branches in some sense reflect the "last known state" of the remote, it would make sense to also update them to whatever a two-argument fetch got.

Risks:

Scripts might rely on the current behaviour. The most likely case I can think of would be to go along the lines of

  git fetch origin master
  git rev-list origin/master...FETCH_HEAD | do_something

to avoid relying on reflogs to get the same result. Seems a bit arcane to me though. Such usage would see the updated state, i.e., process an empty range.

Migration plan:

Add a fetch.updateRemoteNamespace (or so) configuration variable that defaults to false. When enabled, it turns on the auto-updating behaviour.

In 1.8.0, flip the default.

Thomas Rast <trast@student.ethz.ch> writes:

+1

We just got the case on this list of a newbie running "git pull origin master" to update his local branch, and "git status" still complaining that the local branch was ahead of the remote by many commits.

Updating the remote-tracking in this case would avoid such confusion.

Thomas Rast <trast@student.ethz.ch> writes:

The overall goal is a good one, I think, but it is not a migration plan without a period where we issue a loud, in-your-face, warning to force users to choose, is it? I suspect you just didn't write it because it is so obvious, but I am just making sure it is written down somewhere, so that whoever ends up implementing this will not forget.

Thanks.

On Mon, Jan 31, 2011 at 4:44 PM, Thomas Rast <trast@student.ethz.ch> wrote:

+1 I really wanted to write this one my self;)

I would disagree with the migration plan though. IMHO there is no need to introduce the variable. If it will start update both FETCH_HEAD and the remote-tracking branches since 1.8 it will not break any code, because it is added functionality... In this case FETCH_HEAD will reflect the latest fetch and it doesn't even need to be removed later on.

just my 2 cents...

Thanks, Eugene

Eugene Sajine <euguess@gmail.com> writes:

Then you didn't understand the risks section, did you? Thomas clearly illustrated with an example where the script _expects_ origin/master to stay the same after "git fetch origin master".

On Mon, Jan 31, 2011 at 6:06 PM, Junio C Hamano <gitster@pobox.com> wrote:

I did understand what Thomas illustrated. I'm just thinking that the range origin/master...FETCH_HEAD seems to be useful but in fact is pretty useless, because you cannot guarantee the state of the origin/master _before_ the fetch and therefore you cannot rely on the results of range selections involving it. By "guaranteeing the state" i mean that because of the current implementation origin/master doesn't always mean/reflect the same thing: it might be some _old_ and outdated push or it might be some new state of the remote branch which are IMHO completely different semantics.

That's exactly why it seems to me that it is important to always update remote-tracking branches upon any related network operation. So remote-tracking branch always represents the _same_ thing - the latest state of the remote branch that you interacted with.

Thanks, Eugene

Eugene Sajine <euguess@gmail.com> writes:

With the current system, origin/master is what I _used to_ have before running this fetch, iow, what I have been looking at as a reference material while I did my own work so far. The current "when fetching refs explicitly, do not touch tracking branches" behaviour guarantees that.

The range above shows what I already knew about the work the other people did, and what is new on both sides, to help me decide what I want to do next with the fetched result. At least that is the workflow the "when fetching refs explicitly, do not touch tracking branches" behaviour allows us to support (I am not recommending that the workflow in particular, by the way). The next action after seeing what they added is sane may be to run "git pull" (no arguments), which would fast-forward my view of the other side to whatever I reviewed this round.

On Mon, Jan 31, 2011 at 10:44:09PM +0100, Thomas Rast wrote:

I like this one. FYI, I posted a patch in Aug 2009, and there was some discussion about the impacts (it spills over into the sibling subthreads):

  http://article.gmane.org/gmane.comp.version-control.git/127215

I'm actually still carrying the patch in my repository (and using it every day!). I guess I should get on polishing it up...

-Peff

On Mon, Jan 31, 2011 at 4:44 PM, Thomas Rast <trast@student.ethz.ch> wrote:

Would it make sense to group the pre-1.8 compatibility switches together in some way, if there will be several of them? Maybe

[compat]
   fetchUpdateRemoteNamespace = false
   ...

(Thinking out loud.)

j.

On Tue, Feb 1, 2011 at 2:04 PM, Jay Soffian <jaysoffian@gmail.com> wrote:

It is. I was thinking of it as a group of "short"-lived configs to help maintain backward compatibility for some time (not for ever) until users are forced to migrate.

Jay Soffian <jaysoffian@gmail.com> writes:

I don't think so.

If these configuration variables are expected to be removed in some future, such a layout might make sense, but what is proposed is to default them to a different behaviour from today at 1.8.0 boundary, and we are not going to remove the ability to invoke the current behaviour with these variables. It would make it a lot easier to find and understand if the variables are grouped by functionality like all the other regular variables, as these new ones are after all regular ones.

Just thinking aloud, too.

On 01/31/2011 04:44 PM, Thomas Rast wrote:

If this is proposing to break:

	get-fetch ${REPO} ${SRC_REF}:${DST_REF}

then I am against this since that form _is_ used and *is* plumbing.

A Large Angry SCM wrote:

You're mixing up the two proposals.  This one is to teach

  git fetch repo foo

to update refs/remotes/repo/foo with the new value (maybe we should also have it update in the foo:bar case, but I haven't thought that through).

The other one is to forbid 'git pull repo foo:bar' and would not change git-fetch at all.

On 02/01/2011 05:39 PM, Thomas Rast wrote:

I'm not concerned about the pull proposal (I haven't really thought about it, yet) but I am concerned that your proposal may break (as in change the behavior of) the case I identified above.

Proposal:

git-pull inherits the full fetchspec invocation syntax from git-fetch, so that you can do e.g.

  git pull origin master:master

usually shooting yourself in the foot in the process.  See e.g.

  http://thread.gmane.org/gmane.comp.version-control.git/130819/focus=130879 [item 1]

Prohibit this invocation, i.e., disallow any second argument to git-pull that contains ':'.

History:

I submitted a patch ages ago:

  http://article.gmane.org/gmane.comp.version-control.git/130822

Sean seemed to be the only one in favour of the old behaviour, but I was too lazy to push it past him. (There were some issues with the test as well.)

I honestly still don't see a valid use-case for the full syntax, let alone having seen one in the wild; in cases where you are inclined to do

  git pull origin foo:bar

you probably don't need 'bar' to begin with if you can use 'origin/foo' instead, assuming the previous proposal on remote-branch updating is also included.

Risks:

It's an incompatibility, so scripts may break.

Migration plan:

None; disallow it at the 1.8.0 boundary.

If someone makes a good case why his hands are trained that way, we might introduce a config variable instead.

Thomas Rast <trast@student.ethz.ch> writes:

As I summarized in $gmane/135813 I don't think there was any objection against forbidding "git pull" with refspec with colon. There indeed was an interesting tangent topic that was about valid use cases of "git fetch" with such refspec, but I think this is orthogonal to that issue.

On Mon, Jan 31, 2011 at 10:55:59PM +0100, Thomas Rast wrote:

Hmm... I have always thought about "git pull repo refspec" as "git fetch repo refspec && git merge FETCH_HEAD" and "git fetch" refuses to fetch into the current branch of a non-bare repository, so I expected "git merge" to fail in this case too, but it succeeded though with some warning that fetch updated the current branch head. I think it is inconsistent and should be fixed, and that will fix the mentioned confusion as well.

As to disallowing ':' in refspec completely, I am not so sure... Not that I think it is very useful, but also I don't see how it can hurt someone provided that the target branch cannot be the current branch.

Dmitry

Dmitry Potapov wrote:

IRC experience shows that people, while on some topic branch, run

  git pull origin master:master

expecting it to "pull master into master" (or even worse with three different branch names). So no, the current branch safeguard does not prevent the fundamental mistake.

On Tue, Feb 01, 2011 at 04:14:57PM +0100, Thomas Rast wrote:

I am not sure what you mean by three different branches names. You referred to item 1, and I agree it is confusing, but it can be prevented by the current branch safeguard.

and the current semantic of "git pull" is very clear:

"git pull repo refspec" = "git fetch repo refspec && git merge FETCH_HEAD"

IMHO, the full confusion was caused by incorrect information on github, which was corrected a long time ago. Have you heard about any new users who are confused by git-pull?

And if we really want to disallow ':' in git pull refspec then the documentation should be corrected too. For instance, if there are options to git fetch that make no sense if you cannot specify lbranch. Also description of refspec should be corrected in git pull man page.

Dmitry

On Tue, Feb 1, 2011 at 12:05 AM, Junio C Hamano <gitster@pobox.com> wrote:

Another random wish, which does not come with a proposal. How about tag namespace (ie. tags from a remote stay in remote namespace)?

On Tue, 1 Feb 2011, Nguyen Thai Ngoc Duy wrote:

Please make this into a proper proposal. this would be indeed a huge improvement.

Nicolas

On 11-01-31 10:20 PM, Nguyen Thai Ngoc Duy wrote:

I had just started writing up such a proposal yesterday. What I have so far is pretty preliminary:

Proposal:

Change tag refspecs to distinguish between remote and local tags. An unadorned tag "foo" could point to different commits in different repositories. A remote could move/edit it's "foo" tag and have that update smoothly propagated to clones.

I believe this was last brought up in November while discussing the refs base for notes:

http://thread.gmane.org/gmane.comp.version-control.git/160503/focus=160655

Risks:

I think the main risk lies in breaking plain <tagname> refs, as they would become "origin/<tagname>" refs instead. But I think that can be mitagated against (see below).

The other risk folks might raise, though I disagree, is breaking the immutability assumption for tags. I'm willing to debate this, though (see the above-linked thread).

Another "risk" is that this change might be too much of an earthquake. It may be something more suitable to a major release, like 2.0.

Migration plan:

Add a "tags.relative" (name TBD) configuration variable which defaults to false. When tags.relative is true, "git fetch" puts any received tags under (location TBD) refs/remotes/<remote>/tags/. In 1.8.0 we flip tags.realtive's default value.

To help mitigate the risk of breaking plain "<tagname>" refs, "git rev-parse" can look for plain names (i.e. ones without a /) in the remote tags location.

		M.

On Tue, Feb 1, 2011 at 9:54 PM, Marc Branchaud <marcnarc@xiplink.com> wrote:

Thanks. I wrote another proposal (should we have more or less standard "Git Enhancement Proposal" process like Python's PEP or Scheme's SRFI to keep track of these over time?) but it's good to see others look at it too.

One problem with the proposed ref layout is that it breaks current layout (remotes/<remote>/head -> remotes/<remote>/heads/head). Some sort of migration support is needed. On the other hand, new layout is cleaner than my proposal (remote-tags/<remote>/tag).

Hmm I thought "a-ref" would check "refs/tags/a-ref" and "refs/remotes/*/a-ref". But you are right. Maybe "tags.relative" can take three values instead of boolean (names TBD):

 - deprecated (current behavior)
 - migrating (fetch tags to refs/remotes, but tag lookup will look in
refs/tags as well as refs/remotes/*/tags)
 - migrated (fetch tags to refs/remotes, look up in order)

We may slowly turn default value step by step until it becomes "migrated".

The following four commands have been marked as deprecated in command-list.txt for quite some time[*]:

   command          deprecated since replaced by
   ---------------- ---------------- ---------------------
   git-lost-found   2007-11-08       git fsck --lost-found
   git-peek-remote  2007-11-24       git ls-remote
   git-repo-config  2008-01-17       git config
   git-tar-tree     2007-11-08       git archive

Let's just remove them. There's a risk that someone is still using the old commands, of course, but they have been deprecated long enough now.

[*] data source:
    git blame -C command-list.txt |
    sed -n 's/.*\(....-..-..\).*\(git-[^ ]*\).*deprecated.*/\2\t\1/p'

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

Sounds fine.

Hi,

René Scharfe wrote:

Some quick thoughts based on a code search:

 - removing lost-found seems safe
 - removing peek-remote seems safe
 - repo-config should probably warn before it is removed
 - removing tar-tree will probably break "make dist" for old
   projects.  I still am tempted to say removing it should be
   okay.

It can stay in contrib/examples for inspiration.

No one seems to be using it (github.com/gitpan/App-GitHub-FindRepository.git uses it as a fallback when ls-remote is not present).

giggle[1] still uses it --- see libgiggle-git/giggle-git-config-read.c and giggle-git-config-write.c.

Likewise darcs2git[2] and the stgit testsuite.

webkit's VCSUtils.pm only uses repo-config as a fallback when git config is not present.

Already prints a deprecation notice. WWW::PkgFind from CPAN uses it but doesn't seem to be maintained.

pilgrim[3] uses tar-tree in its "make dist" target. I wouldn't be surprised if some other projects use it in a similar way.

Jonathan

[1] git://git.gnome.org/giggle.git [2] git://repo.or.cz/darcs2git.git [3] git://git.fedorahosted.org/pilgrim.git

Am 02.02.2011 01:57, schrieb Jonathan Nieder:

Sure.

How did you search for current usage?  How comprehensive are the results?

Well, the release notes for 1.5.4 promised that the "next feature release will remove it". Perhaps notifying the developers of the projects you discovered is enough?

That said, the benefit for final removal of this command, which is effectively just an alias, is the smallest of the four.

Possibly, and this shows that deprecation warnings don't fully solve the problem of educating users to switch to the replacements.

I think it's relatively safe to remove the command anyway because the users in this case are developers and packagers, i.e. the ones who put the command in the Makefile in the first place. They should be able to cope easily.

René

René Scharfe wrote:

By searching for "peek-remote" at http://www.google.com/codesearch

I tried to check out all hits that weren't just enumerating git commands, but that doesn't rule out unindexed or non-public use. No interesting hits.

But that's pretty weird, given that there was no deprecation notice, nor anything else to encourage a transition.

*checks history*

Ah, peek-remote and ls-remote seem to have been introduced at the same time. ls-remote could use all git-supported protocols, while peek-remote could only use git protocol. So very few people had reason to use peek-remote, anyway.

[...]

[...]

The list is probably not exhaustive. On the bright side, repo-config tends to be run in user-visible contexts, so I think a deprecation notice could be effective.

After adding a deprecation notice and filing some bugs, I think we can forget about it and wait another year. ;-)

[...]

I agree. The remaining users look like holdouts that will be hard to get at by other means.

Jonathan Nieder <jrnieder@gmail.com> writes:

Yes, peek-remote was a backend for ls-remote, just like http-fetch was one of the backends for fetch.

Am 10.02.2011 21:56, schrieb Jonathan Nieder:

-- >8 --
Subject: [PATCH] repo-config: add deprecation warning

repo-config was deprecated in 5c66d0d4 on 2008-01-17. Warn the remaining users that it has been replaced by config and is going to be removed eventually.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
 builtin.h        |    3 ++-
 builtin/config.c |    6 ++++++
 git.c            |    2 +-
 3 files changed, 9 insertions(+), 2 deletions(-)

René Scharfe wrote:

Looks good to me, for what it's worth.  Thanks for taking care of it.

Thanks; applied.

Proposal:

Handle the submodule options "diff.ignoreSubmodules" and "submodule.<name>.ignore" consistently in diff plumbing.

I see two basic ways to change the behavior of the plumbing diff commands:

a) Let them use the "diff.ignoreSubmodules" configuration too.

b) Don't let them use the "submodule.<name>.ignore" entries either.
   But if we go that way, we might have to revert the default of
   recursing into populated submodules too, as it might cause
   unexpected behavior when all configuration options introduced
   to control that recursion are just ignored unless explicitly
   told otherwise.

History:

When the submodule recursion for the diff commands was introduced, all diff commands - including plumbing - learned to recurse into submodules by default. This was done to mark submodules with uncommitted changes as dirty so no user could accidentally forget to commit his changes there before pushing in the superproject.

Some time after that "--ignore-submodules" learned some values to achieve more control over what conditions mark a submodule dirty. Then the "submodule.<name>.ignore" option was added to .git/config and the .gitmodules file to to be able to specify these values for one or more submodules. In a later commit the "diff.ignoreSubmodules" option was added, but the plumbing diff commands weren't taught to use that config option.

Risks:

a) Those scripts which depend on the plumbing commands to ignore
   the setting from "diff.ignoreSubmodules" will break.

b) All scripts written or changed since 1.7.0 which depend on the
   current behavior to recurse into submodules and use the
   "submodule.<name>.ignore" entries will be broken.

Me thinks the risks are much smaller when doing a), as people who learned to use the recursive behavior since 1.7.0 will see that changed under their feet when we do b) and I expect much more code to rely on the recursion than on the "diff.ignoreSubmodules" setting. And doing a) would fix a real life problem too, see [1].

Migration plan:

a) During the 1.7.x series a new "noconfig" value is added for the
   "--ignore-submodules" command line option for people who don't
   want user configuration to interfere with the recursion, e.g. in
   scripts (turning off the recursion is already implemented, just
   use the "--ignore-submodules" option). And then starting with
   1.8.0 "diff.ignoreSubmodules" will be used by diff plumbing.

b) During the 1.7.x series a new value for "--ignore-submodules"
   called "porcelain" is added which enables recursion and also
   tells diff plumbing use all configuration settings. Then all
   relevant call sites (like git gui, gitk and PS1 from completion
   and others) are changed to use this option. Changing the default
   behavior to ignore "submodule.<name>.ignore" and to not recurse
   anymore will be done in the 1.8.0 release.

Personally I'm in favor of solution a), but lets hear what other people say.

[1] http://thread.gmane.org/gmane.comp.version-control.git/164166/focus=164172

See "Tracking empty directories" subthread on git mailing list, starting with

  http://thread.gmane.org/gmane.comp.version-control.git/165655/focus=165831

The idea is for git to be able to track empty directories without the .keepme (or empty .gitignore) file trick. This was one of the most requested features is native support for tracking empty directories, with 35.2% support (from those who answered question) in "Git User's Survey 2010":

  https://git.wiki.kernel.org/index.php/GitSurvey2010#17._Which_of_the_following_features_would_you_like_to_see_implemented_in_git.3F

There were some concerns about *backwards compatibility* of this feature, see for example this email (which I would summarize here).

  http://thread.gmane.org/gmane.comp.version-control.git/165655/focus=165847

That's why it is in 1.8.0 proposals.

The problem with backward compatibility is twofold. First and more important is while git supports empty tree object (it has it hardcoded for some time, as it is necessary e.g. for initial diff, or merging unrelated branches without common ancestor), and there is no problem with entry for empty tree in a tree object

  040000 tree 22d5826c087c4b9dcc72e2131c2cfb061403f7eb	empty

there is (supposedly) problem when checking out such tree (see email referenced above) with an old git.

Second is that tracking empty directories would require extension to the git index (storing trees in index, like we store submodules)... but that is purely local matter.

There is also issue with pre-change git automatically removing empty directories, but that is probably matter for pre-commit hook to check, or pre-receive hook in publishing repository.

On Wed, Feb 2, 2011 at 6:56 AM, Jakub Narebski <jnareb@gmail.com> wrote:

Instead of using an empty tree, construct a tree containing a single sentinel file whose contents are a suitable warning not to delete/edit said file using pre-1.8.0 git. Meanwhile git-1.8.0 never writes the file to the filesystem. Too ugly?

j.

On Feb 2, 2011, at 3:23 PM, Jay Soffian <jaysoffian@gmail.com> wrote:

I don't like where this is going. Users are not always right. Touch .gitignore and be done with it. This is a big change with negligible benefits. I don't understand why this is needed.

Dnia czwartek 3. lutego 2011 00:33, David Aguilar napisał:

Too ugly.

Two issues: first is interaction with other SCM which keep empty files (or keep empty files when requested). Second is skeleton of directory structure automatically generated by some git-unaware tool.

I've never felt the need but more than 50% of Git User's Survey 2010 responders did.

On Wednesday, February 02, 2011 16:33:09 David Aguilar wrote:

I am not usually bothered too much that git doesn't story directories, and when I need to do it, I can do the .gitignore trick just like anyone else.

However here are a few reasons that I miss this feature sometimes:

  1) Why WOULDN'T you want to track empty directories? We track empty files: 
isn't that just as pointless? What if git couldn't track empty files and 
automatically removed files when they became empty? Well, I could live with 
that just as well, with some silly workarounds every once in a while (e.g. 
echo empty > some_file).

  2) One of git's best strengths is that it's so easy to interact with other 
SCM software, primarily because git's features are a SUPERSET of other SCMs. 
However, almost every other SCM can track empty directories, except git, 
which makes it much harder to use as universal tool, where I can trust that 
everything will round-trip as well as possible. Also, other SCMs don't want 
".gitignore" files cluttering their repository any more than we want their 
SCM tool's random control files in our repositories.

  3) Forget for a moment the cuddly git idosynchracies that through use we 
have come to know and "love". From a design perspective, does putting and 
tracking, a file called IGNORE in a directory you want to KEEP make sense? 
It's one thing to use "touch .gitignore" in an empty directory to keep it 
*as a workaround*, based on implementation details (i.e. any file in the 
directory will do, but we probably will have a .gitignore anyway eventually, 
so might as well as use that) but it's a strange *design*. =)

  4) On many projects I work on with a huge number of people, the workflow 
is to create a very, very intricate directory hierarchy skeleton, so that 
it's clear to everyone where everything goes and how it is organized, even 
before any work is started. In this case with git, it's annoying to do this 
because there are worthless .gitignore files cluttering up everything, which 
makes it harder to find where there are *actual* ignore rules being applied.

  5) Git not tracking empty directories and the (perceived?) arrogant 
reaction from git experts ("no big deal, just touch .gitignore" -- I've said 
this to people too, since it's the canonical answer, but I always feel a 
little chagrined after hyping up everything git can do) sometimes is just 
one more thing that makes git harder to sell to others, especially when they 
are already in love with Subversion or whatever.

Most of all, think of it this way: maybe git doesn't need to track empty directories in order to be awesome, but is there some reason that tracking empty directories would make it less awesome?

Anyway, just some thoughts.

Wesley J. Landaker wrote:

See http://thread.gmane.org/gmane.comp.version-control.git/52875

Not really. For example, many other SCMs can store per-file comments, arbitrary revision properties, a detailed provenance of a file, and detailed permissions for each directory entry.

What might make git nice as an interoperability tool is that it tracks the _relevant_ information for the history of a software project. Example of what is not relevant information and why that matters:

 http://thread.gmane.org/gmane.comp.version-control.git/53494

All that said, I do want support for explicitly[1] tracking empty directories, mostly for the sake of the ability to clone an svn repo with empty directories in a simple way.

The aforementioned "share a project skeleton" use case is just a nice bonus.

Hope that helps, Jonathan

[1] Part of the value of the "explicitly" is to make it explicit that early adopters are asking for trouble. :) FWIW I imagine a transition like this:

 1. Teach "git read-tree" and "git checkout-index" to honor empty
    directories (but not "git update-index" or "git write-tree").

 2. Teach "git write-tree" to accept empty directories.

 3. Teach "git update-index" to accept empty directories if a
    configuration item indicates so.  That configuration would
    default to false.

 4. (Maybe) add porcelain support for tracking of empty directories.
    Also teach "git diff-tree" and "git apply" about empty
    directories.

 5. Change the default to true.

An orthogonal question is how the empty directories would be stored. I do not like the idea of a ".empty-directory" file, since what happens when you try to import a repository with a genuine ".empty-directory" file?

Based on a quick test, currently read-tree _ignores_ empty tree entries. Would it be okay to say that anyone who turns on the switch from step (3) has declared he is willing to write tree objects that git fsck versions before v1.5.5-rc0~63 (fsck.c: fix bogus "empty tree" check, 2008-03-04) will reject and current git might mishandle?

"Wesley J. Landaker" <wjl@icecavern.net> writes:

[...]

Just adding my 2 cents: my first clash with Git's non-management of empty directories was a combination of both. A colleague created an SVN project with several empty directories, along the lines of "Here it is. Now, put your stuff in there".

git-svn didn't import these empty directories (I think I actually could have worked around this with "git svn mkdirs"). Adding .gitignore files would have been a really dirty workaround since I didn't want to put Git stuff in the SVN repo.

I don't think my colleague did anything wrong, I did want to use Git, and that was still frustrating to see such a simple scenario not managed by my favorite tool.

So, yes, I can clearly leave without empty directory support, but that would be a nice addition to Git.

On 02/02/2011 03:33 PM, David Aguilar wrote:

I worked for a huge company that was converting a large project to Git. They had a wiki about the conversion. There was a section titled "Git Gotchas" that had one entry: Git Doesn't Track Empty Directories.

This has come up as something to be worked around in each of the conversions to Git I've been part of.

And unlike other things Git doesn't track, such as permissions or modification times, I can't argue that Git's behavior here is superior to what they were expecting. I'd love to see this feature in Git if only to make the issue go away. And I don't think it's a slippery slope, that once this is in then they'll be clamoring for resource forks and ACLs. Developers know some things aren't handled by SCMs, they just don't think that includes directories.

--Pete

Jakub Narebski:

Proposal:

- Implement the possibility to checkout/read/handle empty directories as soon 
as possible, even in the next 1.7.x release.
- Don't implement the possibility to create/commit empty directories yet.
- Implement the possibility to checkin empty directories next year, but allow 
it only, if the user knows that this breaks backwards compatibility of the 
repo. (Generate warning and require --commit-empty-directories option)

Sorry if this should be crap. I'm not a GIT dev.

Best regards,

Thomas Koch, http://www.koch.ro

Heya,

On Sat, Feb 5, 2011 at 19:31, Thomas Koch <thomas@koch.ro> wrote:

I think regardless of whatever else we do, this makes sense. I think it's been suggested by Junio in a neighboring thread as well.

I don't know about this part though.

I would say that if we are going to require an option, it should be core.emptyDirectory , which would default to false. It would be much more annoying than having to supply an option every time. I also disagree with warning for this.

Sverre Rabbelier <srabbelier@gmail.com> writes:

Huh?

Heya,

On Sun, Feb 6, 2011 at 21:41, Junio C Hamano <gitster@pobox.com> wrote:

To teach the plumbing to handle empty directories first, and then in a later release add porcelain?

On Sun, Feb 6, 2011 at 1:31 AM, Thomas Koch <thomas@koch.ro> wrote:

(Elaborate the "handle" part from your first item)

- Teach diff machinery about empty trees. At least diff-tree should
show empty tree addition/removal. diff-index and diff-files can learn
later when index supports empty trees.

- Teach merge empty trees.

Proposal:

1. Change 'git stash <not-a-stash-command>' to give a usage message
   instead of using <not-a-stash-command> as the stash message.

2. Change 'git stash -p <args>' to treat the <args> as filename
   arguments similar to add -p.  Possibly add a -m option that lets
   you specify a message anyway, if desired.

Rationale:

The first one has long been a pet peeve of others, too. It makes the stash command prone to typo accidents and such.

The second one was my own fault, and breaks the symmetry with the rest of the -p family.

Risks:

Users trained to either usage will obviously have to retrain their fingers. Scripts may also have to be changed, but if anything they probably use a 'git stash; do_something; git stash pop' pattern which would not be affected. (I also suspect most scripts using this pattern forget to check whether there is anything to stash, and thus are broken if there isn't.)

Migration plan:

In 1.7.5, give a loud warning for both syntaxes.

In 1.8.0, switch them as described.

On Wed, Feb 2, 2011 at 09:23, Thomas Rast <trast@student.ethz.ch> wrote:

Oh please, yes, please do this. We should have done this long, long ago. Its easy enough to train your fingers or fix your scripts to say `git stash save list` rather than `git stash lsit` once stash errors out and gives you a usage message once.

Just fix it. I can't imagine anyone cares enough that `git stash blah` stops working without first saying save.

Shawn Pearce <spearce@spearce.org> writes:

Err, hasn't this been fixed long ago already?

$ git stash not-a-stash-command
Usage: git stash list [<options>]
   or: git stash show [<stash>]
   or: git stash drop [-q|--quiet] [<stash>]
   or: git stash ( pop | apply ) [--index] [-q|--quiet] [<stash>]
   or: git stash branch <branchname> [<stash>]
   or: git stash [save [--patch] [-k|--[no-]keep-index] [-q|--quiet] [<message>]]
   or: git stash clear
$ git stash save --no-such-option
error: unknown option for 'stash save': --no-such-option
       To provide a message, use git stash save -- '--no-such-option'
Usage: [...]

Only this could be seen as a problem:

$ git stash save this-is-not-a-stash-name Saved working directory and index state On master: this-is-not-a-stash-name

in particular, it is wrt:

Thomas Rast <trast@student.ethz.ch> writes:

I'm not a user of "stash with messages" myself, so I can't say how annoying the migration would be, but -m "message" sounds good to me.

Matthieu Moy wrote:

Oh, you're actually right. I have totally missed this, and should obviously have tested first.

Still, I think the change to 'git stash -p' is also worthwhile.

On Wed, Feb 2, 2011 at 11:51 AM, Thomas Rast <trast@student.ethz.ch> wrote:

There's still something annoyingly different about git-stash. Most git commands that take sub-commands simply list useful information when you don't provide a sub-command:

git branch --> lists local branches git tag --> lists tags git remote --> lists remotes git stash --> creates a new stash

It would be much more predictable if git-stash without a sub-command just listed the current stashes. But, at this point it's probably too late to change that behavior.

Proposal:

Stop copying the "submodule.<name>.update" entries into .git/config on "git submodule init". The current behavior makes it impossible for upstream to change defaults later, as this value can only be altered through user intervention when it resides in .git/config. This is a good thing when he chose to copy it there, but it doesn't seem to make much sense doing it by default. When for example upstream wants to provide a branch for hacking on an otherwise ignored submodule, it cannot do that without the users intervention right now.

This change also makes sense in preparation of recursive submodule checkout, as that will be controlled by this option. Also notice that the proposed behavior is already used for the "ignore" and "fetchRecurseSubmodules" entries.

History:

I can't tell why that copy is done as I wasn't following the list closely at the time this was implemented. Maybe someone else can shed some light on that?

Risks:

Users could be surprised that upstream can change the value of the "update" configuration when the submodule was initialized with a git version >= 1.8.0.

Migration plan:

In 1.8.0 stop copying the "submodule.<name>.update" entries into .git/config and add an option ("--update"?) to reactivate the old behavior of "git submodule init" if the user wants that. Announce both in the release notes of 1.8.0.

Jens Lehmann <Jens.Lehmann@web.de> writes:

Doesn't it just come from the usual "upstream can give a sane default as recommendation to users who may not bother to set up .git/config, and the user can tweak that if that doesn't suit his/her needs" convention?

I have a feeling that the correct fix (not limited to "update" but all the submodule related configuration that share the same "give default, allow tweak" philosophy) is to:

 (1) record submodule.<name>.update at initialization time, to allow the
     upstream a chance to give a sensible default, as we do now;

 (2) in addition to that, record the fact that the value came as upstream
     default.  You could do so in multiple ways:

     a) record the commit that gave the suggested default to .git/config,
     perhaps submodule.<name>.defaultedFrom (notice that this is
     independent from "update", and covers all such configuration
     variables with a single value); or

     b) record the value the upstream gave to .git/config in a separate
     variable, perhaps submodule.<name>.updateSuggested; or

     c) some other clever way you can think of, as long as it lets us do
     the next step.

 (3) when updating from the upstream results in a change in .gitmodules
     file that changes the previously suggested default the user
     considered, tell that to the user and have him/her choose.  If you
     took (a) in the previous step, you can use "git diff" to determine if
     the suggested default has changed; if you took (b) in the previous
     step, you can compare submodule.<name>.update in .gitmodules with
     submodule.<name>.updateSuggested to do so; if you did (c), you are on
     your own ;-).  After the user updates (or chooses to keep the current
     setting), record the current suggested default just like you did at
     the init time in step (2).

One thing to be careful is in (3) you should not bother users who chose to ignore the upstream default (i.e. has submodule.<name>.update set differently from what is suggested by the .gitmodules at the time of initialization). The reason (3) updates the "current suggested default" is exactly for that purpose---the user has seen what the last suggested default was, and decided to either go with it or have his/her own setting.

Am 23.02.2011 21:28, schrieb Junio C Hamano:

Yup, but when *copying* it locally upstream won't be able to change that default ever again. Wouldn't it suffice to copy that *only* if the user really wants to tweak it?

And now I read that again I notice that I forgot to add a very important sentence, sorry about that:

"Take the setting from .gitmodules if submodule.<name>.update is not configured by the user in .git/config."

I agree that all should behave the same way for consistency reasons.

My proposal uses the values in .gitmodules for those proposed by upstream and those in .git/config as those tweaked by the user. Isn't that simpler than 1) and 2) while giving us the same functionality? And additionally it is not setting the upstream default in stone (which is rather arbitrary as it just happened to be present in our .gitmodules when we did the "git submodule init" and might be different at another point in the history)?

Consider the following situation: The .git directories of the submodules reside in the .git directory of the superproject so their work trees can be safely deleted and we have means to let upstream configure which submodules should be populated on clone. (Hopefully this is the future ;-)

What happens when we fetch a commit which records a new submodule marked to be populated on clone in the .gitmodules of that commit? I assume we would want to fetch the bare submodule into a subdirectory of the .git directory of the superproject so that we have it present so we can populate it when the superproject's commit is checked out later, no?

Should we ask the user if he wants to fetch the bare submodule into the .git directory of the superproject or to ignore the clone setting while fetching? Should we ask him again on checkout time if he wants to use the upstream setting of checking out the submodule? Or should we just let it happen and when the user decides that he never wanted this submodule he says so in his .git/config while everybody who is happy with the default just moves on? (And of course we would honor any configurations the user did beforehand, like e.g. ignoring all upstream clone defaults)

I have the impression that using submodules won't work smoothly unless we honor the defaults set by upstream until told otherwise by the user. If we copy the settings into .git/config right away we take away much of the flexibility that I believe is needed here, and I can't really see the upsides of that. But the downside is that the setting is copied at an random point in time and therefore is rather arbitrary.

Another example is the repo where submodules normally use the on-demand fetch mode (so everybody only gets those submodule commits he really needs) while having topic branches where certain submodules are always fetched in full to be able to record new commits of those in the superproject. Depending on what branch you were when the copy into .git/config occurred you would be stuck with either setting, which I think is suboptimal.

What am I missing?

Jens Lehmann <Jens.Lehmann@web.de> writes:

Yeah, that changes everything.

It actually is even more flexible, in that it allows the user to say "I don't care either way; I'll follow along whatever the version that I happen to have checked out says"; I am however tempted to think that this particular flexibility is not necessarily a good thing to have.

If the setting changes the behaviour of the command so drastically, would the "I'll follow along" mode be really sensible? A setting in .gitmodules might be different between 'maint' and 'master' from the upstream and depending on which branch is checked out and you are working on, you may get different behaviour (e.g. 'checkout' vs 'merge' vs 'rebase' depending on the value-of-the-day in submodule.<name>.update). Depending on the nature of the setting, it may or may not be.

The "record once and then help the users to adjust but make aware of the new duggestion" actually comes from a very old discussion circa 1.5.2 days:

  Cf. http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621

Imagine the upstream URL to fetch submodule from has changed over time. You don't want to interact with an ancient and now defunct URL only because you happen to have a checkout of an old version by reading from its .gitmodules file. The (2) and (3) in the previous message when taken to their logical conclusion will be the "seen values" that is hinted in the above message from May 2007. Let the user record what s/he wants under the condition that s/he has seen these choices, and do not bother her/im again when choices do not change, but do ask permission and/or confirmation when a new choice appears. git-submodule script hasn't learned to do that yet, unfortunately.

In any case, URL is a good example of variables that would want to stay around (while giving the user helping hand to update it when choice changes). "update" would be a good example of a variable that may want to be per branch (e.g. 'maint' might encourage "checkout" while 'master' might encourage "rebase"). So most likely we would need to support both modes of operation.

Yes, this is very much needed and that is one of the reasons we introduced the support for textual ".git" file.

Marked by who?  The supplier of the superproject?

If the user consents (e.g. "git clone --recursive"), yes.

If we don't know what the user wants yet, yes. Note that explicit command line options "git clone --recursive" and $HOME/.gitconfig counts as the user letting us know what s/he wants.

Yes if the available choice and/or suggestion by the upstream has changed, otherwise no.

Am 24.02.2011 01:34, schrieb Junio C Hamano:

Yeah, I totally agree that the URL has to be copied to .git/config and must be updated only on the users request. I'm not sure the "giving the user a helping hand" approach will work so well as I suspect switching back and forth between branches might spam the user with helping hands every time, but maybe I'm missing something here.

Looks like we are in the same boat here. Only if either the always-recurse or on-demand mode for fetch is enabled - either by command line or by configuration - fetch should do that automatically. Otherwise it would do nothing (me not being so sure about the feasibility of asking the user).