Google Summer of Code 2011

Anyone want to mentor this year?

On Thu, Mar 03, 2011 at 10:08:25AM -0800, Shawn O. Pearce wrote:

I'd be happy to. I can also act as org admin this year if you want. I bowed out last year due to impending baby, but have no such excuse this year. :)

Should we also start a call for project suggestions? I haven't been paying attention to the GSoC timeline.

-Peff

On Thu, Mar 3, 2011 at 10:59, Jeff King <peff@peff.net> wrote:

I would appreciate that, I'm too busy this year. :-(

Org application deadline is March 11: 23:00 UTC

On Thu, Mar 03, 2011 at 11:04:51AM -0800, Shawn O. Pearce wrote:

OK, there is now:

  https://git.wiki.kernel.org/index.php/SoC2011Application

which is mostly just an adapted version of last year's application.

I'll give people a week or so to make changes, and then do the application with Google probably next Wednesday or Thursday.

Looks like accepted projects are announced on the 18th, and then we should be talking to students about it. So we should probably have a relatively complete list of ideas by the 18th. I set up a bare-bones idea page here if people want to start adding ideas:

  https://git.wiki.kernel.org/index.php/SoC2011Ideas

-Peff

Jeff King <peff@peff.net> writes:

Quote:

 "In 2010, we had three projects: native svn support, libgit2, and a
  line level history browser.

  All three projects were successful. [...]"

What about 'integrated web client for git', aka. "Splitting gitweb and developing write functionalities" project wit Pavan Kumar Sankara (pkumar), which failed midterm evaluations?

See https://git.wiki.kernel.org/index.php/SoC2010Projects#Splitting_gitweb_and_developing_write_functionalities_.28Integrated_web_client_for_git.29

On Thu, Mar 03, 2011 at 01:25:16PM -0800, Jakub Narebski wrote:

Thanks, I had totally forgotten about that project and didn't find it mentioned on the GSoC site (I guess because it failed). I couldn't find any discussion of its failure, though. Is there more information somewhere?

-Peff

On Thu, Mar 03, 2011 at 03:33:23PM -0500, Jeff King wrote:

We need to list a backup admin. Can I volunteer you for that (or are there are volunteers)?

-Peff

On Wed, Mar 9, 2011 at 08:39, Jeff King <peff@peff.net> wrote:

Sure, I can be the backup admin.

On Thu, Mar 03, 2011 at 03:33:23PM -0500, Jeff King wrote:

Our application is officially in. However, we can still edit it until the deadline on Friday. If you have changes, feel free to make them on the wiki page but make sure to let me know, as I have to migrate them to the official application. I'll also probably do a once-over on Friday to check for any updates. If only the wiki was kept in git. :)

Also, the application links to our ideas page. Please add ideas! It will give the GSoC people a sense of what we are thinking of for projects, and students will probably start looking at them after the list of accepted organizations is published (which is next Friday, the 18th).

-Peff

On Wed, Mar 9, 2011 at 09:49, Jeff King <peff@peff.net> wrote:

The ideas page is very important. Git needs a good set of ideas on its ideas page in order to be accepted. Google has stated this many times in the past, its a key part of the decision making process (not the only part, but an important part nonetheless).

We should have a good ideas page by the application deadline, so that when Google goes to review applications, they can at least make a fair assessment of our ideas list.

On Wed, Mar 09, 2011 at 09:52:17AM -0800, Shawn O. Pearce wrote:

Then let's repeat it with a more eye-catching subject, and cc all of the people who have said they would mentor so far.

For those just joining us:

The Google Summer of Code application deadline is this Friday (March 11) at 23:00 UTC. They will be looking at our ideas page at:

    https://git.wiki.kernel.org/index.php/SoC2011Ideas

If you have any ideas, please add them to the page! Whether you are available to mentor the project, or simply think it would make a good project and want to inspire others to mentor it, it's appropriate to go there.

-Peff

Hi,

Jeff King wrote:

Thanks for a pointer. Some ideas still at the "throw them against the wall and see if they stick" stage: please feel free to add to the page if you think you can find subsets with the right scope. Later today I can look more carefully.

1. Cross-compilable msysgit: ideally, allowing someone to run
   "make msysgit-installer" on Linux from an msysgit-source.git repo
   to get an installer.  Nice subsets to start with might be msys.dll
   (I think there has been some work on that already) and the basic
   msys utilities.

2. Merge libgit2 into git.git: polish the existing code in libgit2 so
   that standard git can use it.  Presumably this would happen by
   porting one function at a time, not by making git link to libgit2
   immediately.  People familiar with libgit2 might be able to pick
   out particularly interesting subsets to start with (diff
   generation?).

3. Remote helpers: bidi git remote-svn (or one-way remote-hg, or
   remote-cvs, or ...).

4. filter-branch killer: using fast-import's new features to implement
   common filter-branch operations (--subdirectory-filter,
   --prune-empty, obliterating certain files) faster.

5. rev-list: Coping with timeskew.

6. shallow clone: Push support.

7. fixing workdirs (and alternates), along the lines explained in the
   thread <http://thread.gmane.org/gmane.comp.version-control.git/150559>.
   I would be very much interested in this.

7. packfilev4.

On Wed, Mar 09, 2011 at 06:10:17PM -0600, Jonathan Nieder wrote:

That's definitely bigger than a SoC project at this point. However, one smaller project might be to start implementing a subset of git commands (probably plumbing) on top of libgit2, and then see how well the result passes git.git tests.

Yeah, I like that. Or maybe somebody can pick up git-sequencer, which in theory could be used to filter-branch, too (or maybe sequencer can go on top of fast-import? :) ).

I don't think this is big enough for a SoC project. It's on my todo list for the near-term, though.

I suspect that is too complex for a SoC project. But you never know.

-Peff

On Thu, Mar 10, 2011 at 08:30, Jeff King <peff@peff.net> wrote:

It is. Nico is also probably working on it.

My big concern with pack v4 right now is the on-disk format of commits/trees is different from the current wire protocol. If a pack v2 only client connected to a server using pack v4, the server would have to spend a lot of time to inflate/deflate about 60% of the objects in the repository (commits and trees). That is a lot of CPU time. So even if a SoC-er finished the work, it probably wouldn't be merged because of this penalty.

On 10.03.2011 18:31, Shawn Pearce wrote:

I'm very interested in this. Can anyone please point me towards further information on the packfile format and proposed/in-progress changes?

Jonathan Nieder wrote:

Some stuff off the top of my head, please apply a similar filter:

* Word-based merge helper

  The existing merge algorithms are all tailored to line-based formats
  such as code.  Writing, e.g., LaTeX or even asciidoc requires
  sticking to a strict word-wrapped format.  Worse even, re-wrapping
  leads to headaches if people work on the same areas a lot, much like
  the effects of code reindents.

  Something similar was already proposed in 2010, IIRC by Dscho:

    https://git.wiki.kernel.org/index.php/SoC2010Ideas#Merge_helper_for_LaTeX_files

  One possible angle of attack given --word-diff=porcelain would be:

  1. Fix --word-diff to properly represent both sides of the diff at
     least optionally.  (It was observed in a list post some months
     ago that it doesn't even represent *one* side faithfully.)

  2. Use --word-diff=porcelain as input to some to-be-written merge
     algorithm.

* Clean up add -p

  git-add--interactive.perl became a bit of a mess.  Partly due to my
  own efforts with {checkout,stash,...} it has bolted-on interfaces to
  other commands.  There are some UI issues that simply fall out of
  its design, e.g., you cannot go back from one file to another,
  Ctrl-C stops applying to the current file but does not discard
  earlier files, etc.  And that's not saying anything about 'add -i'
  which I don't really know.

  This would probably not be a very fun project, but it could add a
  little edge of usability to the tool and it's probably one of the
  few pure-Perl ideas we can offer.

I would phrase this as follows:

* fast-import-filter

  Implement a utility that can execute certain transformations to
  fast-import streams, such as:

  - delete or rename entire files/directories
  - split out subtrees
  - ...

  Ideally, this should be a thin command line interface around a set
  of primitives (such as a Perl package) that allow a competent
  scripter to go beyond the CLI.  (There have been threads on this.)

  Later on, make an interface that automatically does

    git fast-export | fast-import-filter | git fast-import

  with suitable options.  Nice because it should be largely orthogonal
  to everything except the fast-import stream format.

In addition, Jeff wrote:

I'd be interested to hear how that goes, because I think the tools are fundamentally different. The rebase and thus sequencer family is delta-based, and the fast-import and filter-branch families are tree-based. Feel free to prove me wrong of course.

On Thu, Mar 10, 2011 at 6:15 PM, Thomas Rast <trast@student.ethz.ch> wrote:

Not a complete/right solution (and not even extensively tested), but I did a "trivial" merge program for LaTeX files that:

1.- Rewraps all files as the first one, using dwdiff 2.- Perform a normal line based merge

[wlmerge] #!/bin/bash tmp1=$(mktemp -t wlmerge.XXXXXXX) tmp2=$(mktemp -t wlmerge.XXXXXXX) trap "rm -f $tmp1 $tmp2" 0 1 2 3 15 [ ! -e "$1" ] && exit 1 file1=$1 && shift [ ! -e "$1" ] && exit 1 orig=$1 && shift [ ! -e "$1" ] && exit 1 file2=$1 && shift dwdiff -2 -w "" -x "" "$orig" "$file1" > $tmp1 dwdiff -2 -w "" -x "" "$file2" "$file1" > $tmp2 merge "$file1" $tmp1 $tmp2

I even have a "wldiff", but nowadays I use --color-words exclusively!

HTH, Santi

On Thu, Mar 10, 2011 at 06:15:23PM +0100, Thomas Rast wrote:

One more wishlist item for this. I use "add -p" for almost all of my adds these days, because I like the final review check. So after a conflicted merge, I find myself doing "add -p" to stage my resolution. The current behavior is that it shows the --cc diff and exits. It would be cool to handle staging the resolution, which would involve converting the combined diff into something that can be applied.

Hmm, yeah, that is a fundamental difference. It might be possible to overcome it, but I admit I haven't really given it any thought yet at this point.

-Peff

Jeff King <peff@peff.net> writes:

I think the end-result would be a nice feature. I suspect that it would not involve conversion from --cc, but more like using the difference between the HEAD and the working tree, generated as if there is no multi-stage index.

On Thu, Mar 10, 2011 at 11:21:47AM -0800, Junio C Hamano wrote:

The trouble is that I would like to see the combined diff, then say "OK" and have it apply the result to the index. But because we work on a per-hunk basis, you need to match the combined diff hunks to the regular diff hunks, taking into account that hunks could be split. Which maybe is straightforward, but I haven't convinced myself yet that there are no corner cases where they don't line up.

I guess one could argue that I should be using mergetool, and making a mergetool helper that is more like "add -p" (show hunks, say "choose left, choose right, choose none"). But I generally fix up the conflict using an editor and just want "add -p" as a final check.

I'll probably look into it more eventually, but it's not a high priority at this point.

-Peff

Jeff King <peff@peff.net> writes:

Forgetting for now the implementation, I _think_ what you would want is for "git add -p" to notice that you are resolving conflicts, and do not bother you about cleanly merged parts (I take it is a given that you would always want to add them to the index without even inspecting when running "add -p"), and make the per-hunk selection loop ask only about the parts that originally had conflicts.

But it is rather hard to arrange. Neither "--cc" nor "-c" during the merge is about showing conflicts but is about showing the result with respect to the two originals. If you resolved conflicts in your editor already, there are no lines with "<<</>>>" markers that are different from either original to cause the "conflicted parts" to appear in their output. If your conflict resolution ended up in taking what one side did, "--cc" will hide it as a non-event. So at least you would be using "-c" to implement this.

Also, you have to remember that "add -p" is about adding the state you deem Ok incrementally to the index, that once you add a path to the index the higher stages for the path will collapse to stage #0, and that "-c" and "--cc" make their comparison based on what you have in higher stages.

I would imagine that a workable implementation might look like this:

 0. The solution introduces a new index extension, "PRSF" (partial
    resolution so far)".  This is a mapping from pathname to a blob object
    name.

 1. "git add -p" notices that you are in the middle of conflict
    resolution, by noticing that you have higher stages to the path.

 2. Look up the path from the PRSF extension.  If there is no entry for
    the path, recreate the content-level 3-way merge using the content of
    three stages (i.e. the state immediately after the mergy operation
    that caused the conflicts before you touched the corresponding file in
    your working tree), and register this image (with the full glory of
    "<<</>>>" conflict markers) to the extension.  If there already is an
    entry in the extension for the path, skip this step.

 3. The patch the user will see in the interactive hunk selection is the
    difference between the working tree and the PRSF image, not your
    regular index (nor the HEAD version). This allows you to see how you
    resolved the conflict incrementally so far.

 4. Choosing a hunk to "apply" would not affect the index entry for the
    path, as it would collapse its higher stages.  It instead updates the
    blob registered in the PRSF extension using the hunk you are applying.

If you are running "git add -p" incrementally (i.e. edit a little, review with diff, then "add -p" the part you are sure about, and repeat the whole thing), the next invocation of "git add -p" would notice that you still have unresolved conflicts in PRSF. It will skip the step 2 above and the step 3 will let you review the remaining difference between the PRSF image (which you updated in step 4 during the last round) and what you have in the working tree. The step 4 will update the PRSF image for the next invocation of "git add -p". And you continue until you are done.

At the end (we would probably need a good way to detect the user might want to declare "end" automatically for a good user experience), use the PRSF image to collapse the higher stages for the path down to stage 0.

On Thu, Mar 10, 2011 at 12:54:29PM -0800, Junio C Hamano wrote:

Yes, I don't want to see cleanly merged parts. And "--cc" already does what I want by not showing them. But of course they still need to be added to the index.

So my thinking was more along the lines of:

  1. Get "git diff HEAD file" and store its hunks.

  2. Get "git diff --cc file" and stores its hunks.

  3. For each hunk in (1), if it does not have an analagous hunk in (2),
     mark it for staging without asking the user.

  4. For the remaining hunks in (1), show the user the analagous --cc
     hunk from (2), and mark the hunk from (1) for staging if requested.

  5. Create the final patch from the marked hunks, apply it to
     HEAD:file, and put that in the index.

There are two issues (and I think you know this, but it took me reasoning out why this wouldn't work to quite understand what you were saying in your email, so I'll elaborate here for the benefit of other readers):

  a. Step (3) glosses over the definition of "analagous hunk". In simple
     cases the hunk headers match up (i.e., they start at the same
     offsets). But the --cc diff is actually a diff against the merge
     base, and the diff in step (1) is against HEAD. So actually we
     would want step (1) and (5) to deal not with the file in HEAD, but
     the file in the merge-base.

     Or we can use "-c" in the first place, which gives us interesting
     and uninteresting parts, and then suppress the uninteresting ones.

  b. You can't stage part of a resolution. This is the "adding a path
     collapses stages to #0" that you mentioned. So either you need an
     index extension, or you need to make it all-or-nothing.

That description makes sense to me, but is way overkill for my workflow. I think at its simplest, what I would like is to be shown the entire --cc diff, have it say "do you want this and all of the cleanly merged bits added?" and then either stage the whole thing or not.

Which really I could do with:

  for i in `git diff-files --name-only --diff-filter=U`; do
    git diff --cc $i
    echo 'OK?'
    read r
    test "$r" = y && git add $i
  done

It would just be a little nicer to have it integrated into the "git add -p" loop.

Even though it is obviously a less featureful solution than what you proposed, I am tempted to implement it anyway because the current behavior is so bad (it shows the diff but doesn't consider it a selectable hunk at all, so it just skips it, exiting immediately if there are no other files).

And then if somebody wants to spend time allowing partial resolution selection, it would be a nice improvement on top of that.

-Peff

Jeff King <peff@peff.net> writes:

Are you sure about that?

What "--cc" would show largely depends on what you have in your working tree. If your two branches fixed the same bug with very different approaches, you may resolve the conflict favouring what one side did while discarding everything the other side did. The file in your work tree might be the same as "git checkout --ours $that_path" after a mergy operation. "diff --cc" won't show anything to you in such a case.

Actually, the largest problem I see in the above approach is _when_ you perform the first two steps, not the details of the later steps.

The user has complete freedom outside our control between the time the mergy operation leaves conflicts in the index and the time "add -p" is invoked (except obviously the user cannot do any "add" to register the resolution for the whole file to the index, as there won't be any per-hunk resolution recording left to do by "add -p").

And as I repeatedly said, grabbing "--cc file" must be done before the user starts mucking with the file in the working tree for the approach to be any useful.

Obviously I don't think so. I'd rather consider what it does an absolute minimum (not the part that it stores in an index extension, but the part that the reference it uses to compare against the working tree file is a original, mechanical merge result). By the way, I just relized that an index extension similar to PRSF might be a way to properly give index log similar to the reflog you wanted to have the other day---every time you "git add", you record the previous state of the index entry there, so that it can be replayed in reverse.

As you already know, I disagree the usefulness of this approach (see the "Are you sure about that?" discussion), hence I doubt the usefulness of "have it integrated into the "git add -p" loop".

On Thu, Mar 10, 2011 at 02:58:07PM -0800, Junio C Hamano wrote:

Well, no. But my experience has been that its output is useful.

Actually, I think diff --cc is doing what I want there. If I take one side's content completely, then it is not interesting to me anymore. What I am really concerned about is doing a tricky content-level merge in my editor and then screwing up the result. Or _trying_ to take one side of the merge, and then screwing it up. So the "diff --cc" output after I have mucked is useful for both those cases.

I'm not sure I agree. The output after I have mucked is useful to me, and that is what this use-case is based on. So I think we are talking about related but slightly different use cases.

OK, let me put it this way. I am not volunteering to work on the approach you outlined. If you are, great. But if not, then what should be done? The current behavior to show the diff and then exit is quite confusing. At the very least, we should say something to the user about what happened (or even suppress the diff and just say "these paths are unmerged, and we can't handle them in add -p").

-Peff

Jeff King wrote:

Good idea, thanks.

I put this (with your idea) and the word-merge thing on the wiki. I'll let Jonathan decide what to do about the fast-import stream filtering.

Jonathan Nieder <jrnieder@gmail.com> writes:

How it would be different from existing reposurgeon tool by ESR (cross-VC thanks to using fast-import format), or git_fast_filter by Elijah Newren (more of a library than a ready tool)?

Jakub Narebski wrote:

Git integration? I haven't looked at these tools for long enough to say how much they fit the ticket and how fast they are, so for me it's hard to say how much work it would be.

Jakub Narebski wrote:

Good question.

. reposurgeon loads the whole history into a convenient format,
  manipulates it, then dumps it as a separate step.  So it is even more
  flexible than filter-branch, but with a large history and especially
  on a small machine it can be slower.

. git_fast_filter is what Thomas mentioned --- a filter to go between
  fast-export and fast-import.

Both involve unpacking all trees. Which is not a huge expense, mind you, but there is room to go faster.

What I was suggesting is instead a tool that relies on fast-import to do the heavy lifting. So, for example, when you ask to delete path/to/remove.txt from all commits, it would write:

 commit refs/heads/master
 mark :172
 author A U Thor <author@example.com> ...
 committer ...
 data ...
 from ...
 merge ...
 M 040000 [old tree name for that commit] ""
 D path/to/remove.txt

 commit refs/heads/mater
 mark :173
 ...

The idea is inspired by http://thread.gmane.org/gmane.comp.version-control.git/158375 I am interested in it because I imagine something like this could be useful for splitting out branches from a naive import of an svn repo (and similar surgeries).

Hi Jonathan,

Jonathan Nieder writes:

I wrote a small introduction and created an entry for git-remote-svn
in the wiki [1]. Two minor concerns:
1. The student will have to work very closely with us, and pick up a
   lot of scattered WIP patches. I hope he doesn't get confused
   between what's merged and what's in-progress.
2. The project isn't as de-coupled as we'd ideally like. The student
   has to learn about the new fast-import features, and the
   information contained in an SVN replay dumpstream before diving in.

Other than these, I think most of the pending work is in the mapper.

This is an interesting project that I'd also thought about: it might get tangled up along with the Sequencer project though. Should we put up this project on the wiki nevertheless?

p.s- Sorry about the late reply; I wasn't in station.

[1] SoC2011Ideas#Welcome and SoC2011Ideas#Remote_helper_for_Subversion

-- Ram

On Thu, Mar 10, 2011 at 4:58 AM, Jeff King <peff@peff.net> wrote:

Jeff, how is the "support multiple transport (torrent, bundle, http..) cloning" going? The one that you were working on from the "initial clone from bundle" thread? Is it ok to make it a SoC project (presumably with at least bundle clone support)?

On Thu, Mar 10, 2011 at 07:19:36AM +0700, Nguyen Thai Ngoc Duy wrote:

I pushed it back in favor of other topics, but I will get to it eventually. If you want to make a SoC proposal, go for it. Though I think it might not be big enough by itself, and might need to be folded into a larger project.

-Peff

Comments on the "Better big-file support" section:

"While git can handle arbitrary-sized binary content [...]"

This is very much not true. Git tries at many places to load the complete file into memory and usually fails with "out of memory" if it can't. With the 32bit msysGit client this places the upper file size limit, from purely empirical observation, at 600-700 MByte. When a file is to large git fails late, adding and committing works (as long as there are no filters or other complications), but you can forget about pushing, rebasing or otherwise manipulating that commit. Even worse yet, commits consisting of smaller files but with a combined size over the limit will also cause out-of-memories.

Thus a main focus should be the memory problem, e.g. by using stream-like file handling everywhere, since not working at all is orders of magnitude worse than working slowly :)

"In some cases, this may be as simple as having a "large file" codepath that avoids pulling whole files into memory (e.g., during "git add")."

Ironically git add is one of the few things that work with large files, as mentioned above. Presumably the stream-oriented zlib enforced/encouraged a steam-like handling here :) Slow as hell though and of course it is usually not sensible to compress a 1.5 GByte file.

I'm very willing to work on this topic. Though I'm not a student and as a git code newbie I also don't have the skills for mentoring yet.

On Thu, Mar 10, 2011 at 10:40:01PM +0100, Alexander Miseler wrote:

I think we are picking nits here. What I meant was two things:

  1. The fundamental design of git does not prevent storing
     arbitrary-sized binary data.

  2. How big a piece of data the current implementation can handle
     sanely is up to how much hardware you throw at it. On my 64-bit
     machine with 8G of RAM running Linux, I can easily work with 2
     gigabyte files. Some operations are slow, of course, but it works.

     I'm willing to accept that 32-bit msysgit has more trouble with
     a case like that.

But I think we are probably in agreement with what needs to be done to make things better. Specifically, I am thinking of:

  1. Streaming blobs wherever possible (e.g., add, filters, textconv).

  2. Converting the diff code not to have in-memory files is probably
     going to be quite difficult. But most of these files don't have
     interesting diffs _anyway_. They're usually binary, and we don't
     generate binary diffs by default. So what we need to focus on is
     avoiding loading them when we can. Things like:

       a. Using caching textconv, and when we do run the textconv
          filter, streaming the blob to the filter.

       b. Avoid loading the whole file to check whether it is binary. We
          can already avoid this by marking it binary with
          gitattributes, but there is no reason git can't just load the
          first 4K or so to check for binary-ness, and get this
          optimization automatically. We can also consider caching
          binary-ness for large files so we don't have to look at them
          at all after the first time.

       c. Handle rename detection better. It may be a matter of saying
          "this file is too big for detection". But we may also be able
          to stream it through the spantree-hashing, and then possibly
          also cache the resultant data.

  3. The above deal with memory problems. There is also a storage
     problem. If I have a 100G repo, right now I use at least 100G in
     the .git directory and 100G in the working tree. That's a problem
     for repos of that size. If I have a storage server on the LAN and
     want to accept the latency hit, it would be nice to keep the
     commits local and the giant blobs on the server. Especially coupled
     with the optimizations in (2), we can possibly avoid even having to
     touch those blobs at all in many cases, so the latency wouldn't be
     a big deal.

  4. A related storage problem is that we put big files in packs, and
     the I/O on rewriting the pack becomes a problem. They would do
     better loose or in their own packs.

And I'm sure there are more variations on those things. Part of the project would be identifying the problem areas.

That generally should work OK. The diff and packing code tries to keep memory usage reasonable, which generally equates to two times the largest file. If you have a test case that shows problems, there may very well be a bug.

Agreed. I think they are sort of the same problem. Whether it works slowly or not at all is simply a matter of how much memory you have. ;)

I just tried "git add" on a 2G file of random bytes. It took about a minute or so to calculate the sha1 and compress it, but the memory usage did jump to 2G. So we could obviously do better on the memory, and there is almost certainly no point in zlib compressing something that big. In my case, it was obviously just random junk. But most files of that size are already going to have some kind of lossy compression, so we are just wasting CPU. You can always set core.compression, but I really just want it off for certain files.

It's on my agenda, too. We'll see if a student steps up for the GSoC project. But don't let that stop you if you want to take a look at it; I'm sure there is plenty of work to go around. :)

-Peff

On 10.03.2011 23:18, Jeff King wrote:

agreed

agreed

I will look into it.

Certainly ^_^

I failed to create an artificial test case for this (thus probably misunderstanding the exact circumstances leading to this). I will investigate it when I get a real life test case again.

On 09.03.2011 22:58, Jeff King wrote:

Added two new ideas: Port Git to Android Resumable clone

I'm new to the Git code and therefore don't necessarily have an idea what I'm talking about. Please feel free to change, correct, mutilate and massacre this ideas. This are just two things that I would like to see, but don't want to work on myself since I have BIGGER fishes to fry ^_^

On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:

This is very very hard. Not so much to implement but to design the way it can be done without assuming things (like object sort orders) that aren't stable.

-Ilari

On Fri, Mar 11, 2011 at 7:52 PM, Ilari Liusvaara <ilari.liusvaara@elisanet.fi> wrote:

Yes it's hard. I have some experimental thing that nearly works [1], although whether it is an acceptable approach is to be seen. If anyone's interested, I'll post it some time

A simpler way to restartable clone is to facilitate bundles (Nicolas' idea). Some glue is needed to teach git-fetch/git-daemon to use the bundles, and git-push to automatically create bundles periodically (or a new command that can be run from cron). I think this way fit in GSoC scope better.

[1] The idea of my work above was mentioned elsewhere, history is cut down by path. Each file/dir's history a very long chain of deltas. We can stream deltas (in parallel if needed) over the wire, resuming where the chain stops last time.

There are many problems. One is that a deep chain can make git run out of stack, so chains have to be broken down before storing (not done). Another one is that not many deltas can be reused, so it will consume more power than normal clone. But once you clone this way, the cloned repo have lots of delta suitable for another clone (but probably not for anything else).

On 11.03.2011 14:48, Nguyen Thai Ngoc Duy wrote:

This may all be aiming to short. IMHO the best solution would be some generic way for the client to specify exactly what 
it wants to get and to get just that. This would lay the groundwork for:
- lazy clones
- sparse clones
- resumable cloning
- resumable fetching
and probably quite a few other nifty tricks.

I guess that would be far beyond the scope of a SoC project though.

On Fri, Mar 11, 2011 at 9:10 PM, Alexander Miseler <alexander@miseler.de> wrote:

But how do you define "what it wants to get"? Pros and cons are different and depend on that defintion.

Git's way of saying now is "I have these commits, give me these branches (cutting at certain depth)". A near future extension would be "I have these commits _restricted to these paths only_, give me...". If it's broken half way, start again.

The bundle way is basically Git's way except that the remote side says "I've got this bundle, fetch it here, then you can ask us again for the rest in a normal way".

mirrorsync (aka gittorrent) says "I have this commit, give me objects so that I can have this commit", which leaves the (probably) huge initial commit out of view.

My way is "give me a chain of deltas starting from this SHA-1". A big blob can be considered as a history of smaller pieces.

The last way of saying is just "give me this object", which would be a big waste of bandwidth.

Your move :)

On 12/03/11 03:27, Nguyen Thai Ngoc Duy wrote:

Just FWIW, I recently produced some diagrams and demonstrated the resumable clone algorithm; speed is currently very limited by the implementation at 15 minutes to slice up git.git; but I'm playing with libgit2 to see if I can make a more optimal version.

See http://vilain.net/comp/git/gittorrent/commit_reel.html

Sam

Heh, I'm still pretty hampered by having no clue ^_^

I originally missed the fact that it is completely irrelevant where in the chain the reference point is and that you can go easily from any arbitrary point in the chain to any other arbitrary point in the same chain. This makes it a lot more flexible than I thought. But to support more than just resumable cloning You would still need a more powerful way to negotiate "I have this; I want that" between client and server.

My point was mostly that lazy clone, sparse clone and resumable clone are closely related and should ideally be solved in one go. You currently seem to aim at only one of them, which is a shame. For lazy clone the client would start with all blobs referenced by HEAD (and possibly other branches) and then it would walk backwards through the delta chains if/as far/whenever it wants. Though it would also need to detect when it needs to start reading new delta chains as it walks backwards through history. Sparse clone would use the same mechanism and just restrict it to certain paths.

On Wed, Mar 9, 2011 at 22:58, Jeff King <peff@peff.net> wrote:

Here's a simple one:

Currently we have a global UI color setting, but pretty much only git-status/log/etc use it.

Change git-bisect, git-am etc. to also support color output. It would be a easy but big UI win.

Great idea! Sent from my BlackBerry device on the Rogers Wireless Network

-----Original Message-----
From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Sender: git-owner@vger.kernel.org
Date: Fri, 11 Mar 2011 13:43:39 
To: Jeff King<peff@peff.net>
Cc: Shawn Pearce<spearce@spearce.org>; Ramkumar Ramachandra<artagnon@gmail.com>; Jonathan Nieder<jrnieder@gmail.com>; Jens Lehmann<Jens.Lehmann@web.de>; Christian Couder<chriscool@tuxfamily.org>; Thomas Rast<trast@student.ethz.ch>; git<git@vger.kernel.org>
Subject: Re: Summer of Code project ideas due this Friday

On Wed, Mar 9, 2011 at 22:58, Jeff King <peff@peff.net> wrote:

Here's a simple one:

Currently we have a global UI color setting, but pretty much only git-status/log/etc use it.

Change git-bisect, git-am etc. to also support color output. It would be a easy but big UI win. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Jeff King <peff@peff.net> writes:

A few project ideas I am not sure if they are feasible for GSoC:

* merging-in support for caching in gitweb, and benchmarking/profiling
  gitweb in high load situation

  J.H., you would probably want gitweb [output] caching to be merged-in
  sooner that the end of Google Summer of Code 2011, isn't it?

* embedding graphical diff and graphical merge tool in git-gui, e.g. as
  "git gui diff".  I think that we can use xxdiff; the license is 
  compatibile.

  Pat and Shawn, is it something worth doing?  Does it look like a good
  project for GSoC2011, or is it too small of a project for this?  Would
  we be able to find mentor for this idea?

* splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui

  Pat and Paul, do you think it is right scope, or is it too large project
  to put as an GSoC idea?

Hi,

On Thu, Mar 17, 2011 at 04:40:44PM -0700, Jakub Narebski wrote:

I would be interested in mentoring those projects. I do not know whether we can reach consensus how to split up things. The last time we discussed a possible approach[1] to share code between gitk and git gui we did not reach one.

Cheers Heiko

[1] http://thread.gmane.org/gmane.comp.version-control.git/166663/focus=166695

On 03/17/2011 04:40 PM, Jakub Narebski wrote:

I think the work needed to get gitweb-caching actually merged upstream is going to be a slight bit messier than I'm willing to put a student through, and couple that with the fact I'm going to be herding 2 or 3 students for kernel.org as already, I won't have enough time to try and mentor a student through that quagmire.

Should a student be interested in the graphing stuff for gitweb, I'm happy to make commentary and review things as they come, but that's at best a co-mentor from me.

Jakub Narebski <jnareb@gmail.com> writes:

There is also tkdiff for stealing from. I'm not sure about the worth - there are lots of free merge tools around. But if someone wants to do that then fine.

It shouldn't be too large. You are likely looking at doing Git.pm as a Tcl package (without looking in much detail). Testing that gitk and git-gui didn't get broken will probably be tedious. So adding some test suite to each could help. The Tcl test package is quite capable of being used to test Tk apps provided some tests get written.

On Fri, 25 Mar 2011, Pat Thoyts wrote:

I meant TkDiff, not xxdiff here; I'm sorry for the mistake.

TkDiff has the advantage that is already written in Tcl/Tk, so we can borrow code (like e.g. "refining" diff), not only algorithms.

I thought that would be mainly re-using library part of git-gui in gitk; tests are decidely a good idea, though I don't know if there are ready tools to do tests for programs written in Tcl/Tk , and for testing UI.

Hi,

Shawn Pearce writes:

Although I'm probably not experienced enough to independently mentor a project, I'd be more than happy to co-mentor, or step in if a mentor is suddenly unavailable. I should have a lot of time to give in summer.

-- Ram

Hi,

Ramkumar Ramachandra wrote:

I'd also be willing to co-mentor. Time might be a constraint, but I can find time.

Jonathan

Heya,

On Thu, Mar 3, 2011 at 23:08, Jonathan Nieder <jrnieder@gmail.com> wrote:

Perhaps we can mentor a student to work on git-remote-svn with the three of us? :)

Hi,

Sverre Rabbelier writes:

Sure, but I was hoping for an exciting new project idea- what do you think?

-- Ram

Heya,

On Tue, Mar 8, 2011 at 13:33, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

I don't know, having a working bidi git-remote-svn is pretty exciting in itself :)

Am 03.03.2011 19:08, schrieb Shawn Pearce:

I can offer to either be a co-mentor (being a fill in for a mentor who is not available the whole time, like I did last year for Thomas) or to mentor a student full time when (s)he is working on submodules.

Hi,

On Thursday 03 March 2011 19:08:25 Shawn Pearce wrote:

I can co-mentor this year too.

Thanks, Christian.

Hi Shawn,

On 04/03/11 07:08, Shawn Pearce wrote:

This year I'm likely to be far too busy to mentor anyone or review applications; however I expect to find time to deliver on my earlier promise to revisit GitTorrent with a series of articles and how-to's on the comp.git section of my blog. I've made at least administrative progress towards this goal; by setting up ikiwiki and arranging a discrete comp.git section, so that readers can follow without getting my other articles.

The introductory article is at http://vilain.net/comp/git/gittorrent/past_synthesis.html and I hope to be able to keep producing one article a week. I would love following, input, criticism etc from anyone who has been involved in the GitTorrent project or its many spin-offs over the years.

Cheers and hope GSoC 2011 works out well for git! Sam

Hi,

On Thu, Mar 03, 2011 at 10:08:25AM -0800, Shawn Pearce wrote:

I can offer to mentor a project either about submodule improvements or on gui stuff (git-gui or gitk). Since I am still not a native speaker of the full git code base co-mentoring would be an option for other projects.

Cheers Heiko

On Mon, Mar 07, 2011 at 08:40:47PM +0100, Heiko Voigt wrote:

That's sounds very interesting.

I'm using git with submodules a lot (daily in 5-10 different smaller projects with 1-5 developers in each) and suffers from lacking submodule support, specially in the gui-tools.

Being a student this is a part of git I would be very interested in helping to improve.

Hi,

On Mon, Mar 07, 2011 at 09:50:23PM +0100, Fredrik Gustafsson wrote:

Sounds promising. When I was talking about gui or submodules I did not think about gui *and* submodules. Do you have some concrete areas you would like to improve in mind?

Cheers Heiko

On Wed, Mar 09, 2011 at 10:52:56PM +0100, Heiko Voigt wrote:

Many of the ideas I had was already mentioned in https://github.com/jlehmann/git-submod-enhancements/wiki/

The ones I feel is most important for me is:
* gitk: Add popup menu for submodules to see the detailed history of
changes
* Check before a push in the superproject that all submodules HEADs are
pushed
* Showing that a submodule has a HEAD not on any branch in “git status”
* Move the submodules git directories into the superproject’s .git so that
submodules can be created and deleted

With this said, I'm not familiar with the git codebase yet and does not have any concrete solutions. For example, I don't understand why a submodule doesn't get a default branch.

Hi,

On Thu, Mar 10, 2011 at 12:16:04AM +0100, Fredrik Gustafsson wrote:

Sounds like a good roadmap. From my viewpoint this includes enough work for a summer. There are also enough small steps so even if only part of this would be finished it would contain enough improvement for git. Since this is the first time I am offering to mentor I am not that familar how much work can be done during summer. What do others think?

(added Jens to the CC)

The familiarity with the code should not be a problem since the idea of the Summer of Code is to allow more people to get involved into a project.

Regarding the default branch: The short answer is that git does not record any branchname but a specific revision using its SHA1. This is on purpose to make sure that you get exactly the code you had when you created the commit. But a default branch has been requested many times to make working easier. AFAIR it the consensus was that creating a branch recursively by default would not be a good idea. But doing that on explicit user request could be implemented as an option to ease users workflow. So something like this could be discussed.

Cheers Heiko

Hi Shawn

Shawn Pearce wrote:

I can mentor (or co-mentor) again.