threads / discuss / 46358

"groups of files" in Git?

Subject: "groups of files" in Git?

## tl;dr

26 messages between Jul 11, 2017 and Jul 13, 2017.

replies: 25people: 8as markdown or json

Nikolay Shustov· Jul 11, 2017, 15:45 UTC · lore

Hi, I have been recently struggling with migrating my development workflow from Perforce to Git, all because of the following thing:

I have to work on several features in the same code tree parallel, in the same Perforce workspace. The major reason why I cannot work on one feature then on another is just because I have to make sure that the changes in the related areas of the product play together well.

With Perforce, I can have multiple changelists opened, that group the changed files as needed.

With Git I cannot seem to finding the possibility to figure out how to achieve the same result. And the problem is that putting change sets on different Git branches (or workdirs, or whatever Git offers that makes the changes to be NOT in the same source tree) is not a viable option from me as I would have to re-build code as I re-integrate the changes between the branches (or whatever changes separation Git feature is used). Build takes time and resources and considering that I have to do it on multiple platforms (I do cross-platform development) it really denominates the option of not having multiple changes in the same code tree.

Am I ignorant about some Git feature/way of using Git that would help? Is it worth considering adding to Git a feature like "group of files" that would offer some virtutal grouping of the locally changed files in the checked-out branch?

Thanks in advance,
- Nikolay
Stefan Beller· Jul 11, 2017, 17:18 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:

Show 8 quoted lines
> Hi,
> I have been recently struggling with migrating my development workflow
> from Perforce to Git, all because of the following thing:
>
> I have to work on several features in the same code tree parallel, in
> the same Perforce workspace. The major reason why I cannot work on one
> feature then on another is just because I have to make sure that the
> changes in the related areas of the product play together well.

So in that case the features are not independent, but related to each other? In that case you want to have these things in the same working tree as well as in the same branch.

Take a look at git.git itself, for example:
    git clone git://github.com/git/git
    git log --oneline --graph

You will see a lot of "Merge X into master/maint" commits, but then you may want to dive into each feature by:

    git log --oneline e83e71c5e1

for example and then you'll see lots of commits (that were developed in the same branch), but that are closely related. However they are different enough to be in different commits. (different features, as I understand)

Show 10 quoted lines
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
>
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me as I would have to re-build code as I re-integrate the
> changes between the branches (or whatever changes separation Git
> feature is used).

you would merge the branches and then run the tests/integration. Yes that seems cumbersome.

Show 9 quoted lines
> Build takes time and resources and considering that I have to do it on
> multiple platforms (I do cross-platform development) it really
> denominates the option of not having multiple changes in the same code
> tree.
>
> Am I ignorant about some Git feature/way of using Git that would help?
> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

The way of Git is that a commit (snapshot) by definition describes a set of files (The set of all files in the project). So If you need two features there at the same time, you probably want it in the same commit.

If they are different enough such that you could have them independently, but really want to test them together, your testing may need to become more elaborate (test a merge of all feature branches) I would think.

>
> Thanks in advance,
> - Nikolay
Nikolay Shustov· Jul 11, 2017, 17:47 UTC · re: Stefan Beller · lore

Re: "groups of files" in Git?

> The way of Git is that a commit (snapshot) by definition describes a
> set of files (The set of all files in the project). So If you need two features
> there at the same time, you probably want it in the same commit.

Thank you, but if I wanted these two features to be in the same commit, I would have no reasons to see them as two distinctive groups. I mean, groups of uncommitted files.

The general problem of not having multiple features in the same code tree is the cost of doing multiple builds and integration testing runs. Now I imagine there could be workaround of having two features developed at different branches and then merging them into 3rd branch for building/testing; however this introduces overhead of maintaining at lest two code trees: one for "dirty changes" where I do the code changes that are not guaranteed to be even build-able and another "build/test" code tree. Plus merging the changes from one to another. A bit too much, IMHO.

On Tue, Jul 11, 2017 at 1:18 PM, Stefan Beller <sbeller@google.com> wrote:
Show 65 quoted lines
> On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
> <nikolay.shustov@gmail.com> wrote:
>> Hi,
>> I have been recently struggling with migrating my development workflow
>> from Perforce to Git, all because of the following thing:
>>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>
> So in that case the features are not independent, but related to each other?
> In that case you want to have these things in the same working tree as
> well as in the same branch.
>
> Take a look at git.git itself, for example:
>
>     git clone git://github.com/git/git
>     git log --oneline --graph
>
> You will see a lot of "Merge X into master/maint" commits, but then
> you may want to dive into each feature by:
>
>     git log --oneline e83e71c5e1
>
> for example and then you'll see lots of commits (that were developed
> in the same branch), but that are closely related. However they are
> different enough to be in different commits. (different features, as
> I understand)
>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>
> you would merge the branches and then run the tests/integration. Yes that
> seems cumbersome.
>
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> The way of Git is that a commit (snapshot) by definition describes a
> set of files (The set of all files in the project). So If you need two features
> there at the same time, you probably want it in the same commit.
>
> If they are different enough such that you could have them independently,
> but really want to test them together, your testing may need to become
> more elaborate (test a merge of all feature branches) I would think.
>
>>
>> Thanks in advance,
>> - Nikolay
Junio C Hamano· Jul 13, 2017, 18:22 UTC · re: Stefan Beller · lore

Re: "groups of files" in Git?

Stefan Beller <sbeller@google.com> writes:
Show 12 quoted lines
> On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>
> you would merge the branches and then run the tests/integration. Yes that
> seems cumbersome.

Sometimes the need to make trial merge for testing cannot be avoided and having branches for separate topics is the only sensible approach, at least in the Git world.

Imagine your project has two components that are interrelated, say, the server and the client, that have to work well with each other. In addition, you want to make sure your updated server works well with existing clients, and vice versa.

One way that naturally maps this scenario to the development workflow is to have a server-update topic and a client-update topic branches, and separate changes to update each side with their own commits:

             s---s---S    server-update topic
            /
    ---o---o----o----M    mainline
            \
             c---c---C    client-update topic

And during the development of these *-update topics, you try three merges:

 (1) Merge S to the mainline M and test the whole thing, to make sure
     that existing client will still be able to talk with the
     updated server.
 (2) Merge C to the mainline M and test the whole thing, to make
     sure that updated clients will still be able to talk with the
     existing server.
 (3) Merge both S and C to the mainline M and test the whole thing,
     to make sure the updated ones talk to each other.

If there is no significant development going on on the mainline in the meantime, (1) and (2) can be done by trying out S and C alone without making a trial merge with M. The same for (3)---it can be just a trial merge between S and C without updates that happened on the mainline.

I'd love to hear from folks in Perforce or other world how they address this scenario with their system.

Nikolay Shustov· Jul 13, 2017, 19:47 UTC · re: Junio C Hamano · lore

Re: "groups of files" in Git?

For me the roadblock for multiple iterations through merging of the different parts (S, C, then C+S) is the time that will be spent on rebuilding the mainline. That's why I would like to have C+S in the same source tree then run tests for S, tests for C (if they can be run standalone) and C+S tests, then tests for whatever other pieces may be affected. (As I mentioned, there are more layers than client + server in my situation, e.g. client + transport + server).

I am not really try to ignite the holy war between Perforce and Git (and why would one???), but if you are interested in the answer on how you'd do your scenario in Perforce, it would be: "use shelved changelists". In Perforce, you could "shelve" the changelist, similar to "stash" in Git, but the difference is that the Perforce shelved changes are accessible across clients. I.e. the other developer can "unshelve" these pending changes to its sandbox (to the same or the different branch) so that sandbox would get the pending changes as well. That would be like the developer made these changes himself. Whatever automated/manual process is involved, it is typical to run "a trial build/tests" on shelved changelist (i.e. uncommitted yet files) to verify the quality of changes. Git achieves the same through the ease of manipulation with branches and I like the way it does it much more.

My question was about how to robustly handle "multiple pending commits" which in Perforce are represented by concept of pending changelists.

On Thu, Jul 13, 2017 at 2:22 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 57 quoted lines
> Stefan Beller <sbeller@google.com> writes:
>
>> On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
>>
>>> With Git I cannot seem to finding the possibility to figure out how to
>>> achieve the same result. And the problem is that putting change sets
>>> on different Git branches (or workdirs, or whatever Git offers that
>>> makes the changes to be NOT in the same source tree) is not a viable
>>> option from me as I would have to re-build code as I re-integrate the
>>> changes between the branches (or whatever changes separation Git
>>> feature is used).
>>
>> you would merge the branches and then run the tests/integration. Yes that
>> seems cumbersome.
>
> Sometimes the need to make trial merge for testing cannot be avoided
> and having branches for separate topics is the only sensible
> approach, at least in the Git world.
>
> Imagine your project has two components that are interrelated, say,
> the server and the client, that have to work well with each other.
> In addition, you want to make sure your updated server works well
> with existing clients, and vice versa.
>
> One way that naturally maps this scenario to the development
> workflow is to have a server-update topic and a client-update topic
> branches, and separate changes to update each side with their own
> commits:
>
>              s---s---S    server-update topic
>             /
>     ---o---o----o----M    mainline
>             \
>              c---c---C    client-update topic
>
> And during the development of these *-update topics, you try three
> merges:
>
>  (1) Merge S to the mainline M and test the whole thing, to make sure
>      that existing client will still be able to talk with the
>      updated server.
>
>  (2) Merge C to the mainline M and test the whole thing, to make
>      sure that updated clients will still be able to talk with the
>      existing server.
>
>  (3) Merge both S and C to the mainline M and test the whole thing,
>      to make sure the updated ones talk to each other.
>
> If there is no significant development going on on the mainline in
> the meantime, (1) and (2) can be done by trying out S and C alone
> without making a trial merge with M.  The same for (3)---it can be
> just a trial merge between S and C without updates that happened on
> the mainline.
>
> I'd love to hear from folks in Perforce or other world how they
> address this scenario with their system.
Junio C Hamano· Jul 13, 2017, 21:20 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Nikolay Shustov <nikolay.shustov@gmail.com> writes:
> I am not really try to ignite the holy war between Perforce and Git
> (and why would one???), but if you are interested in the answer on how
> you'd do your scenario in Perforce, it would be: "use shelved
> changelists".

Oh, that was not my intention, either. My interest was to see if there is a good solution that we could steal from other world.

Show 11 quoted lines
> In Perforce, you could "shelve" the changelist, similar to "stash" in
> Git, but the difference is that the Perforce shelved changes are
> accessible across clients. I.e. the other developer can "unshelve"
> these pending changes to its sandbox (to the same or the different
> branch) so that sandbox would get the pending changes as well. That
> would be like the developer made these changes himself. Whatever
> automated/manual process is involved, it is typical to run "a trial
> build/tests" on shelved changelist (i.e. uncommitted yet files) to
> verify the quality of changes.
> Git achieves the same through the ease of manipulation with branches
> and I like the way it does it much more.

Thanks. Shelving and letting others unshelve is like keeping the changes in separate branches and privately share them among developers, so they sound pretty much equivalent features to me.

> My question was about how to robustly handle "multiple pending
> commits" which in Perforce are represented by concept of pending
> changelists.

And in Git, they are represented by concept of commits that are not yet pushed out to the public repository to become the final history carved in stone.

Igor Djordjevic· Jul 13, 2017, 22:39 UTC · re: Junio C Hamano · lore

Re: "groups of files" in Git?

On 13/07/2017 23:20, Junio C Hamano wrote:
Show 8 quoted lines
> Nikolay Shustov <nikolay.shustov@gmail.com> writes:
>> My question was about how to robustly handle "multiple pending
>> commits" which in Perforce are represented by concept of pending
>> changelists.
> 
> And in Git, they are represented by concept of commits that are not
> yet pushed out to the public repository to become the final history
> carved in stone.

If I may, I don`t think "multiple pending commits" is the issue here (as that is indeed what a private branch is), but more something like "multiple branches pending/live merge branch", or something.

To illustrate, let`s say this is our starting position:
(1)      o---o---o (featureA)
        /         \
    ---o---o---o---M (master, HEAD)
        \         /
         o---o---o (featureB)

We`re currently on commit "M", being a merge commit between our "master" and two feature branches.

Now, what seems lacking, while still possible through a series of steps, is an easy (single step) way to modify current state and commit the change to the _feature branch_, while still being on the "master" branch, still having everything merged in.

So after I make a "featureA" related change while on "M", to be able 
to issue a single command, for example:
 
    $ git commit --branch=featureA
... or:
 
    $ git commit -b featureA 
 
..., where "featureA" would need to be one of the parents of the 
current commit we are at (commit "M", in our case), and get a 
situation like this:
(2)      o---o---o---A (featureA)
        /             \
    ---o---o---o-------M' (master, HEAD)
        \             /
         o---o---o---/ (featureB)

Here, "A" is a new commit/change I`ve just made (while still being on the "master" branch), and it is automatically commited to related "featureA" branch, with merge commit "M" now recreated into "M'" to hold the new "featureA" commit "A" as well.

I guess it would be a kind of alias to doing:
    $ git checkout featureA
    $ git add ...
    $ git commit
    $ git checkout master
    $ git reset --hard HEAD^
    $ git merge featureA featureB

... or something, where last merge step would need to remember previous merge commit "M" parent branches and merge them again to produce an updated "M'" merge commit.

In the same manner, it should be possible to drop a commit from the feature branch in a single step, for example returning to the state as shown in (1), or even "port" it from one branch to the other, like this (without a need for it to be the last commit, even):

(3)      o---o---o---\ (featureA)
        /             \
    ---o---o---o-------M' (master, HEAD)
        \             /
         o---o---A'--o (featureB)

Something like "rebase on steroids", lol, keeping the HEAD where it is, and its merge commit beneath updated.

This indeed seems similar to Mercurial`s patch "queues", except being 
much better as everything is still version controlled at all times, 
no additional tools needed to version control the patches (unless 
that`s already been addressed in Mercurial as well, dunno).
 
And it still seems to be following Git`s "multiple commits per 
feature, single feature per branch" spirit, just allowing for 
easier/faster branch integration testing.

p.s. Even if my short sample might be flawed in one way or the other, it should show the essence of the functionality we`re discussing here, I think.

Regards, Buga

Igor Djordjevic· Jul 13, 2017, 23:32 UTC · re: Igor Djordjevic · lore

Re: "groups of files" in Git?

Just a small update/fixup:
On 14/07/2017 00:39, Igor Djordjevic wrote:
Show 9 quoted lines
> I guess it would be a kind of alias to doing:
> 
>     $ git checkout featureA
>     $ git add ...
>     $ git commit
>     $ git checkout master
>     $ git reset --hard HEAD^
>     $ git merge featureA featureB
> 
This should, in fact, be:    
    
    $ git checkout featureA
    $ git commit
    $ git checkout master
    $ git reset --hard HEAD^
    $ git merge <HEAD@{1} parents>
    
(removed "git add" step, as that is needed for proposed single step 
solution as well, as a usual step preceding the commit; also replaced 
concrete branch names in the last step with a more generic 
description, better communicating real intent)
Show 10 quoted lines
> In the same manner, it should be possible to drop a commit from the 
> feature branch in a single step, for example returning to the state 
> as shown in (1), or even "port" it from one branch to the other, like
> this (without a need for it to be the last commit, even):
> 
> (3)      o---o---o---\ (featureA)
>         /             \
>     ---o---o---o-------M' (master, HEAD)
>         \             /
>          o---o---A'--o (featureB)
Here, the diagram should look like this:
(3)      o---o---o---\ (featureA)
        /             \
    ---o---o---o-------M'' (master, HEAD)
        \             /
         o---o---A''-o (featureB)

(replaced leftover M' from the previous diagram with M'' to show it`s yet another (updated) merge commit, different from both M and M' in terms of SHA1, yet the contents would probably, but not necessarily, be the same for all three; same for leftover A', replaced with A'')

Regards, Buga

Igor Djordjevic· Jul 13, 2017, 23:40 UTC · re: Igor Djordjevic · lore

Re: "groups of files" in Git?

Eh, yet another one, sorry:
On 14/07/2017 01:32, Igor Djordjevic wrote:
Show 6 quoted lines
>     
>     $ git checkout featureA
>     $ git commit
>     $ git checkout master
>     $ git reset --hard HEAD^
>     $ git merge <HEAD@{1} parents>

The last line should stress <HEAD@{1} *parent branches*>, as we`re not merging exact parent commits the previous merge commit was made of, but updated tips of the branches the previous merge commit was made of... or something along those lines :)

Randall S. Becker· Jul 11, 2017, 17:27 UTC · re: Nikolay Shustov · lore

RE: "groups of files" in Git?

-----Original Message----- On July 11, 2017 11:45 AM Nikolay Shustov wrote:

>I have been recently struggling with migrating my development workflow from Perforce to Git, all because of the following thing:
>I have to work on several features in the same code tree parallel, in the same Perforce workspace. The major reason why I cannot
>work on one feature then on another is just because I have to make sure that the changes in the related areas of the product play together well.
>With Perforce, I can have multiple changelists opened, that group the changed files as needed.
Show 5 quoted lines
>With Git I cannot seem to finding the possibility to figure out how to achieve the same result. And the problem is
>that putting change sets on different Git branches (or workdirs, or whatever Git offers that makes the changes to
>be NOT in the same source tree) is not a viable option from me as I would have to re-build code as I re-integrate
>the changes between the branches (or whatever changes separation Git feature is used).
>Build takes time and resources and considering that I have to do it on multiple platforms (I do cross-platform development) it really denominates the option of not having multiple changes in the same code tree.
Change sets are core git functionality. When you commit, you commit a group of changes across multiple files, not single file at a time, like most legacy SCM systems. Individual features are managed typically managed using topic branches that can be switched (using checkout) rapidly, which in your case will cause a localized rebuild based on what files were swapped.
If you need something slightly different than topic branches, do multiple clones off a base integration branch. This will give you multiple working source trees. Switch each clone to its own branch and work with them locally. If you need to move changes from one branch to another, commit, push on one branch, and pull merge to the other branch.
You could also use stash to accomplish similar things, but I wouldn't.

Cheers, Randall

Junio C Hamano· Jul 11, 2017, 17:27 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Nikolay Shustov <nikolay.shustov@gmail.com> writes:
Show 13 quoted lines
> I have to work on several features in the same code tree parallel, in
> the same Perforce workspace. The major reason why I cannot work on one
> feature then on another is just because I have to make sure that the
> changes in the related areas of the product play together well.
>
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
>
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me ...

Naturally. If these separate changes need to work together, it is way too inconvenient if these changes do not appear in a single unified working tree to be built and tested.

> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

Let's step back and let me make sure if I understand you correctly. You want to work on a system with two distinct areas (say, the frontend and the backend), that have to work together, but you want to make two commits, one for each area. You make changes for both areas in your working tree, build and test them to make sure the whole thing works well together, and at the end, you make two commits.

In your real project, you may be doing more than two areas and more than two commits, but is the above a good degenerative case that shows the basic idea? If not, then please disregard all of the following.

You can make partial commits in Git. In the simplest case, you may have two separate files backend.py and frontend.py, you make edits to both files and then make two commits:

	$ git commit backend.py
	$ git commit frontend.py

Changes to some files may contain both changes for the backend and for the frontend that does not allow you to separate commits at file boundary, and Git even lets you handle such a case. If you have the third file in addition to the above two, called common.py, you could instead

	$ git add backend.py
	$ git add -p common.py

to prepare the index to contain only the changes for the backend part ("add -p" lets you interactively pick and choose the hunks relevant to the backend part), and conclude the commit for the backend part with

	$ git commit ;# no paths arguments

and then when all the remaining changes are for the frontend part, you can follow it with

	$ git commit -a
to make another commit for the frontend part.

A short answer to your question, provided if I understood you correctly, is "no, there is no way to say 'backend.py, backend-2.py, ...' are the backend things and call it a filegroup", accompanied by "a filegroup would only be effective when changes align with file boundary".

And if your response is "but most of the time changes align with file boundary", a counter-response is "and most of the time changes align with directory boundary (in well structured project, at least), so you can do 'git commit backend/' for all the backend part without having to name all the paths anyway".

There is an experimental "attributes-limited pathspec" feature in recent versions of Git, which lets you assign arbitrary sets of labels to each paths, and using that you may be able to do

	$ git commit ':(attr:filegroup=backend).'

and I suspect that would be the closest thing you would want (read about 'pathspec' in 'git help glossary')

Nikolay Shustov· Jul 11, 2017, 18:10 UTC · re: Junio C Hamano · lore

Re: "groups of files" in Git?

Thank you for the explanation. The example about backend and frontend is relevant even though I normally have to deal with more layers at the same time.

However, in my case I have the thing that you have already tried to address, partially: the changes always align with file boundaries BUT not with directory boundaries. Imagine you have the stack of backend, data transport and frontend layers. The feature has to touch all three layers thus resulting in the changes in the apparently different directories. Thus, making the distinction by the pathspec (if I understood it right from reading the documentation) would not help.

The attributes could be a solution, if I could:
1. create attribute designated to the feature
2. "mark" uncommitted files in different directory with that attribute
3. filter the list of unchanged files with such attribute
4. create commit for the files only with the certain attribute

You've kindly demonstrated that #4 is doable; however I could not clearly get for the Git documentation if #1 - #3 are achievable... Could you point me to the right place in the documentation, please?

On Tue, Jul 11, 2017 at 1:27 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 88 quoted lines
> Nikolay Shustov <nikolay.shustov@gmail.com> writes:
>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me ...
>
> Naturally.  If these separate changes need to work together, it is
> way too inconvenient if these changes do not appear in a single
> unified working tree to be built and tested.
>
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> Let's step back and let me make sure if I understand you correctly.
> You want to work on a system with two distinct areas (say, the
> frontend and the backend), that have to work together, but you want
> to make two commits, one for each area.  You make changes for both
> areas in your working tree, build and test them to make sure the
> whole thing works well together, and at the end, you make two
> commits.
>
> In your real project, you may be doing more than two areas and more
> than two commits, but is the above a good degenerative case that
> shows the basic idea?  If not, then please disregard all of the
> following.
>
> You can make partial commits in Git.  In the simplest case, you may
> have two separate files backend.py and frontend.py, you make edits
> to both files and then make two commits:
>
>         $ git commit backend.py
>         $ git commit frontend.py
>
> Changes to some files may contain both changes for the backend and
> for the frontend that does not allow you to separate commits at file
> boundary, and Git even lets you handle such a case.  If you have the
> third file in addition to the above two, called common.py, you could
> instead
>
>         $ git add backend.py
>         $ git add -p common.py
>
> to prepare the index to contain only the changes for the backend
> part ("add -p" lets you interactively pick and choose the hunks
> relevant to the backend part), and conclude the commit for the
> backend part with
>
>         $ git commit ;# no paths arguments
>
> and then when all the remaining changes are for the frontend part,
> you can follow it with
>
>         $ git commit -a
>
> to make another commit for the frontend part.
>
> A short answer to your question, provided if I understood you
> correctly, is "no, there is no way to say 'backend.py, backend-2.py,
> ...' are the backend things and call it a filegroup", accompanied by
> "a filegroup would only be effective when changes align with file
> boundary".
>
> And if your response is "but most of the time changes align with
> file boundary", a counter-response is "and most of the time changes
> align with directory boundary (in well structured project, at
> least), so you can do 'git commit backend/' for all the backend part
> without having to name all the paths anyway".
>
> There is an experimental "attributes-limited pathspec" feature in
> recent versions of Git, which lets you assign arbitrary sets of
> labels to each paths, and using that you may be able to do
>
>         $ git commit ':(attr:filegroup=backend).'
>
> and I suspect that would be the closest thing you would want (read
> about 'pathspec' in 'git help glossary')
>
Stefan Beller· Jul 11, 2017, 18:19 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

On Tue, Jul 11, 2017 at 11:10 AM, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:

Show 15 quoted lines
> Thank you for the explanation. The example about backend and frontend
> is relevant even though I normally have to deal with more layers at
> the same time.
>
> However, in my case I have the thing that you have already tried to
> address, partially: the changes always align with file boundaries BUT
> not with directory boundaries. Imagine you have the stack of backend,
> data transport and frontend layers. The feature has to touch all three
> layers thus resulting in the changes in the apparently different
> directories. Thus, making the distinction by the pathspec (if I
> understood it right from reading the documentation) would not help.
>
> The attributes could be a solution, if I could:
> 1. create attribute designated to the feature
> 2. "mark" uncommitted files in different directory with that attribute

1+2 should be answered by the gitattributes man page https://git-scm.com/docs/gitattributes

> 3. filter the list of unchanged files with such attribute
This sounds like one of
  "git status :(attr:backend) ."
  "git status :(exclude,attr:backend) ."
Show 6 quoted lines
> 4. create commit for the files only with the certain attribute
>
> You've kindly demonstrated that #4 is doable; however I could not
> clearly get for the Git documentation if #1 - #3 are achievable...
> Could you point me to the right place in the documentation, please?
>
Nikolay Shustov· Jul 11, 2017, 18:30 UTC · re: Stefan Beller · lore

Re: "groups of files" in Git?

Thank you for #3. As for 1+2, the documentation says:

"Each line in gitattributes file is of form:
pattern attr1 attr2 ...

... When the pattern matches the path in question, the attributes listed on the line are given to the path."

My understanding is that to have the bunch of the files in the separate directories having the same attribute, I would have to, for each file, create a separate gitattributes line with the exact paths/filename and needed attribute. Is it would you are suggesting or I misunderstood the idea?

On Tue, Jul 11, 2017 at 2:19 PM, Stefan Beller <sbeller@google.com> wrote:
Show 34 quoted lines
> On Tue, Jul 11, 2017 at 11:10 AM, Nikolay Shustov
> <nikolay.shustov@gmail.com> wrote:
>> Thank you for the explanation. The example about backend and frontend
>> is relevant even though I normally have to deal with more layers at
>> the same time.
>>
>> However, in my case I have the thing that you have already tried to
>> address, partially: the changes always align with file boundaries BUT
>> not with directory boundaries. Imagine you have the stack of backend,
>> data transport and frontend layers. The feature has to touch all three
>> layers thus resulting in the changes in the apparently different
>> directories. Thus, making the distinction by the pathspec (if I
>> understood it right from reading the documentation) would not help.
>>
>> The attributes could be a solution, if I could:
>> 1. create attribute designated to the feature
>> 2. "mark" uncommitted files in different directory with that attribute
>
> 1+2 should be answered by the gitattributes man page
> https://git-scm.com/docs/gitattributes
>
>
>> 3. filter the list of unchanged files with such attribute
>
> This sounds like one of
>   "git status :(attr:backend) ."
>   "git status :(exclude,attr:backend) ."
>
>> 4. create commit for the files only with the certain attribute
>>
>> You've kindly demonstrated that #4 is doable; however I could not
>> clearly get for the Git documentation if #1 - #3 are achievable...
>> Could you point me to the right place in the documentation, please?
>>
Lars Schneider· Jul 11, 2017, 17:39 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Show 30 quoted lines
> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
> 
> Hi,
> I have been recently struggling with migrating my development workflow
> from Perforce to Git, all because of the following thing:
> 
> I have to work on several features in the same code tree parallel, in
> the same Perforce workspace. The major reason why I cannot work on one
> feature then on another is just because I have to make sure that the
> changes in the related areas of the product play together well.
> 
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
> 
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me as I would have to re-build code as I re-integrate the
> changes between the branches (or whatever changes separation Git
> feature is used).
> Build takes time and resources and considering that I have to do it on
> multiple platforms (I do cross-platform development) it really
> denominates the option of not having multiple changes in the same code
> tree.
> 
> Am I ignorant about some Git feature/way of using Git that would help?
> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?
Interesting question that came up at my workplace, too.
Here is what I suggested:
1. Keep working on a single branch and make commits for all features
2. If you make a commit, prefix the commit message with the feature name
3. After you are done with a feature create a new feature branch based on
   your combined feature branch. Use `git rebase -i` [1] to remove all
   commits that are not relevant for the feature. Alternatively you could
   cherry pick the relevant commits [2] if this is faster.

I wonder what others think about this solution. Maybe there is a better solution that I overlooked?

- Lars

[1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html

Nikolay Shustov· Jul 11, 2017, 17:54 UTC · re: Lars Schneider · lore

Re: "groups of files" in Git?

Thank you for the idea, however I am having troubles with basically maintaining the uncommitted groups of files: I would prefer the clear distinction that "those files belong to feature A" and "these files belong to feature B", before I commit anything. Committing separately every change for feature A and for feature B would probably a good option unless I have many changes and then cherry-picking the proper commits to create a single changeset for the integration would become a nightmare.

On Tue, Jul 11, 2017 at 1:39 PM, Lars Schneider <larsxschneider@gmail.com> wrote:

Show 50 quoted lines
>
>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>
>> Hi,
>> I have been recently struggling with migrating my development workflow
>> from Perforce to Git, all because of the following thing:
>>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> Interesting question that came up at my workplace, too.
>
> Here is what I suggested:
> 1. Keep working on a single branch and make commits for all features
> 2. If you make a commit, prefix the commit message with the feature name
> 3. After you are done with a feature create a new feature branch based on
>    your combined feature branch. Use `git rebase -i` [1] to remove all
>    commits that are not relevant for the feature. Alternatively you could
>    cherry pick the relevant commits [2] if this is faster.
>
> I wonder what others think about this solution. Maybe there is a better
> solution that I overlooked?
>
> - Lars
>
> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>
Lars Schneider· Jul 11, 2017, 20:20 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Show 52 quoted lines
> On Tue, Jul 11, 2017 at 1:39 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> 
>>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>> 
>>> Hi,
>>> I have been recently struggling with migrating my development workflow
>>> from Perforce to Git, all because of the following thing:
>>> 
>>> I have to work on several features in the same code tree parallel, in
>>> the same Perforce workspace. The major reason why I cannot work on one
>>> feature then on another is just because I have to make sure that the
>>> changes in the related areas of the product play together well.
>>> 
>>> With Perforce, I can have multiple changelists opened, that group the
>>> changed files as needed.
>>> 
>>> With Git I cannot seem to finding the possibility to figure out how to
>>> achieve the same result. And the problem is that putting change sets
>>> on different Git branches (or workdirs, or whatever Git offers that
>>> makes the changes to be NOT in the same source tree) is not a viable
>>> option from me as I would have to re-build code as I re-integrate the
>>> changes between the branches (or whatever changes separation Git
>>> feature is used).
>>> Build takes time and resources and considering that I have to do it on
>>> multiple platforms (I do cross-platform development) it really
>>> denominates the option of not having multiple changes in the same code
>>> tree.
>>> 
>>> Am I ignorant about some Git feature/way of using Git that would help?
>>> Is it worth considering adding to Git a feature like "group of files"
>>> that would offer some virtutal grouping of the locally changed files
>>> in the checked-out branch?
>> 
>> Interesting question that came up at my workplace, too.
>> 
>> Here is what I suggested:
>> 1. Keep working on a single branch and make commits for all features
>> 2. If you make a commit, prefix the commit message with the feature name
>> 3. After you are done with a feature create a new feature branch based on
>>   your combined feature branch. Use `git rebase -i` [1] to remove all
>>   commits that are not relevant for the feature. Alternatively you could
>>   cherry pick the relevant commits [2] if this is faster.
>> 
>> I wonder what others think about this solution. Maybe there is a better
>> solution that I overlooked?
>> 
>> - Lars
>> 
>> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
>> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>>
Show 10 quoted lines
> On 11 Jul 2017, at 19:54, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
> 
> Thank you for the idea, however I am having troubles with basically
> maintaining the uncommitted groups of files: I would prefer the clear
> distinction that "those files belong to feature A" and "these files
> belong to feature B", before I commit anything. Committing separately
> every change for feature A and for feature B would probably a good
> option unless I have many changes and then cherry-picking the proper
> commits to create a single changeset for the integration would become
> a nightmare.
I see. Why so complicated with gitattributes then?

How about this: Let's say you start working on featureX that affects file1 and file2 and featureY that affects file8 and file9

1. Create aliases to add the files:
   $ git config --local alias.featx 'add file1 file2'
   $ git config --local alias.featy 'add file8 file9'
2. Work on the features. Whenever you have something ready for featureX
   run this:
   $ git featx
   $ git commit
   Whenever you have something ready for featureY run this:
   $ git featy
   $ git commit
Wouldn't that work?
- Lars
Nikolay Shustov· Jul 13, 2017, 15:21 UTC · re: Lars Schneider · lore

Re: "groups of files" in Git?

Thank you, this could work, but if I am adding new file to the feature/removing the existing file from the feature, aliases usage for "add" doesn't help much. I would really need to have the lists of files... and attributes look more promising.

On Tue, Jul 11, 2017 at 4:20 PM, Lars Schneider <larsxschneider@gmail.com> wrote:

Show 88 quoted lines
>> On Tue, Jul 11, 2017 at 1:39 PM, Lars Schneider
>> <larsxschneider@gmail.com> wrote:
>>>
>>>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> I have been recently struggling with migrating my development workflow
>>>> from Perforce to Git, all because of the following thing:
>>>>
>>>> I have to work on several features in the same code tree parallel, in
>>>> the same Perforce workspace. The major reason why I cannot work on one
>>>> feature then on another is just because I have to make sure that the
>>>> changes in the related areas of the product play together well.
>>>>
>>>> With Perforce, I can have multiple changelists opened, that group the
>>>> changed files as needed.
>>>>
>>>> With Git I cannot seem to finding the possibility to figure out how to
>>>> achieve the same result. And the problem is that putting change sets
>>>> on different Git branches (or workdirs, or whatever Git offers that
>>>> makes the changes to be NOT in the same source tree) is not a viable
>>>> option from me as I would have to re-build code as I re-integrate the
>>>> changes between the branches (or whatever changes separation Git
>>>> feature is used).
>>>> Build takes time and resources and considering that I have to do it on
>>>> multiple platforms (I do cross-platform development) it really
>>>> denominates the option of not having multiple changes in the same code
>>>> tree.
>>>>
>>>> Am I ignorant about some Git feature/way of using Git that would help?
>>>> Is it worth considering adding to Git a feature like "group of files"
>>>> that would offer some virtutal grouping of the locally changed files
>>>> in the checked-out branch?
>>>
>>> Interesting question that came up at my workplace, too.
>>>
>>> Here is what I suggested:
>>> 1. Keep working on a single branch and make commits for all features
>>> 2. If you make a commit, prefix the commit message with the feature name
>>> 3. After you are done with a feature create a new feature branch based on
>>>   your combined feature branch. Use `git rebase -i` [1] to remove all
>>>   commits that are not relevant for the feature. Alternatively you could
>>>   cherry pick the relevant commits [2] if this is faster.
>>>
>>> I wonder what others think about this solution. Maybe there is a better
>>> solution that I overlooked?
>>>
>>> - Lars
>>>
>>> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
>>> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>>>
>
>> On 11 Jul 2017, at 19:54, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>
>> Thank you for the idea, however I am having troubles with basically
>> maintaining the uncommitted groups of files: I would prefer the clear
>> distinction that "those files belong to feature A" and "these files
>> belong to feature B", before I commit anything. Committing separately
>> every change for feature A and for feature B would probably a good
>> option unless I have many changes and then cherry-picking the proper
>> commits to create a single changeset for the integration would become
>> a nightmare.
>
> I see. Why so complicated with gitattributes then?
>
> How about this:
> Let's say you start working on featureX that affects file1 and file2
> and featureY that affects file8 and file9
>
> 1. Create aliases to add the files:
>    $ git config --local alias.featx 'add file1 file2'
>    $ git config --local alias.featy 'add file8 file9'
>
> 2. Work on the features. Whenever you have something ready for featureX
>    run this:
>    $ git featx
>    $ git commit
>
>    Whenever you have something ready for featureY run this:
>    $ git featy
>    $ git commit
>
> Wouldn't that work?
>
> - Lars
>
>
Igor Djordjevic· Jul 11, 2017, 22:46 UTC · re: Lars Schneider · lore

Re: "groups of files" in Git?

For starters, let me say that I consider myself a mere advanced beginner Git user, and I haven`t used Perforce ever before (did some reading now), but still, for what it`s worth, here are my thoughts, please bare with me :)

Do feel free to correct me if I miss something.
On 11/07/2017 19:39, Lars Schneider wrote:
Show 49 quoted lines
> 
>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>
>> Hi,
>> I have been recently struggling with migrating my development workflow
>> from Perforce to Git, all because of the following thing:
>>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
> 
> Interesting question that came up at my workplace, too.
> 
> Here is what I suggested:
> 1. Keep working on a single branch and make commits for all features
> 2. If you make a commit, prefix the commit message with the feature name
> 3. After you are done with a feature create a new feature branch based on
>    your combined feature branch. Use `git rebase -i` [1] to remove all
>    commits that are not relevant for the feature. Alternatively you could
>    cherry pick the relevant commits [2] if this is faster.
> 
> I wonder what others think about this solution. Maybe there is a better
> solution that I overlooked?
> 
> - Lars
> 
> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html

This "single branch, related commits" approach is exactly what came to my mind as well.

But, isn`t Perforce "changelist" an "atomic" group of changes - like "commit" in Git, "changeset" in Team Foundation Version Control, etc...?

If so, it would mean that this "multiple pending changelists" flow would/should be translated to "multiple pending commits" in Git, where in the end _a single_ Perforce "changelist" is _a single_ Git "commit".

Might be this is where the confusion is coming from, trying to fit natural Git "multiple commits per feature" (but "single feature per branch") concept into a "single changelist per feature" Perforce concept, described/required here?

I don`t think there is a firm concept of such "multiple pending commits" in Git, but the point is the author is having multiple changelists/commits, and it actively updates them (the _same ones_), until they`re ready to be merged (submitted, in Perforce)... which you can do in Git, and might be quite easily :)

So... In Git, you can create a separate commit for each changelist you have in Perforce (all these commits being on the same branch, as desired). Then, when you would have wanted to "update" the pending Perforce changelist (not sure what the corresponding command is in Perforce), you would just `git commit` your current state with additional "--squash" or "--fixup" parameters (depending on if you would like to add more description to existing/original commit message, or not), and the original commit SHA1.

In the end, when everything is tested together and you would like to commit features separately (like submitting changelists in Perforce), you would just need to `git rebase -i --autosquash` your branch, where Git would squash all your "update" commits (fixup/squash ones, that is) to the original/initial ones you made as per your changelists/features. No need for manual rearranging, cherry-picking, or whatever.

An example flow, with two "changelists" for two features (I`ll be using capital letters A, B, C... instead of commit SHA1, for simplicity):

	... do some "Feature 1" work...
	$ git commit -m "Feature 1"
	... do some "Feature 2" work...
	$ git commit -m "Feature 2"
	... do some "Feature 1" work...
	$ git commit --fixup A
	... do some "Feature 1" work...
	$ git commit --fixup A
	... do some "Feature 2" work...
	$ git commit --squash B
	... do some "Feature 1" work...
	$ git commit --fixup A
	... do some "Feature 1" work...
	$ git commit --squash A
	... do some "Feature 2" work...
	$ git commit --fixup B
Branch history would look something like this (H is latest commit):
	H fixup! Feature 2
	G squash! Feature 1
	F fixup! Feature 1
	E squash! Feature 2
	D fixup! Feature 1
	C fixup! Feature 1
	B Feature 2
	A Feature 1

When you finally do `git rebase -i --autosquash A^`, you should get a list like this[1]:

	pick A Feature 1
	fixup C fixup! Feature 1
	fixup D fixup! Feature 1
	fixup F fixup! Feature 1
	squash G squash! Feature 1
	pick B Feature 2
	squash E squash! Feature 2
	fixup H fixup! Feature 2

Once rebase is finished, you`ll end up with branch history looking like this:

	B' Feature 2
	A' Feature 1

... where commits A, C, D, F and G have been squashed into a single commit A', and commits B, E and H have been squashed into a single commit B'. These two single commits should correspond to your two Perforce changelists.

Now you can merge your commits separately, as desired ("submit" the "changelists").

You can even first rearrange/split/squash them further, or make separate branches out of them, whatever you find appropriate - you can do whatever you like to them while they`re your local commits ("pending changelists"), before making them live/visible for other users as well (merge them to a public branch, "submit changelist").

p.s. Doesn`t the flow required here look similar to Mercurial patch "queues" approach (again, resembling "quilt" functionality)? If so, "Guilt"[2] may be an option here as well... if the described flow can`t be altered a bit to align better with Git itself, might be profiting on the side of overall workflow simplicity ;)

[1] Having commits automatically grouped/ordered, you can even 
    replace some "fixup" and "squash" with "reword", for example, so 
    those commits are kept as separate ones, providing you a chance 
    to edit their messages.
[2] http://repo.or.cz/w/guilt.git

Regards, Buga

Nikolay Shustov· Jul 13, 2017, 15:37 UTC · re: Igor Djordjevic · lore

Re: "groups of files" in Git?

Thank you for the detailed explanation, it looks like merging the commits would be helpful in my case. And I think it is a very good analogy that Perforce changelists are like multiple pending committs, if Git were supporting such.

What it won't be achieving by using commits in this schema is the
following thing I can do in Perforce:
In the uncommitted Perforce changelists I can revert the changed file
to the original state and move the files between the changelists.
Quite often, while working on something, in the middle I would decide
to isolate changes to a certain set of files to a separate changelsit
- but then I might change my mind. It is all flexible until I actually
commit my Perforce changelist, after which it becomes very much as
committed changes in any other source control.
This is actual flexibility I am looking for achieving in Git.

On Tue, Jul 11, 2017 at 6:46 PM, Igor Djordjevic <igor.d.djordjevic@gmail.com> wrote:

Show 179 quoted lines
> For starters, let me say that I consider myself a mere advanced
> beginner Git user, and I haven`t used Perforce ever before (did some
> reading now), but still, for what it`s worth, here are my thoughts,
> please bare with me :)
>
> Do feel free to correct me if I miss something.
>
> On 11/07/2017 19:39, Lars Schneider wrote:
>>
>>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>>
>>> Hi,
>>> I have been recently struggling with migrating my development workflow
>>> from Perforce to Git, all because of the following thing:
>>>
>>> I have to work on several features in the same code tree parallel, in
>>> the same Perforce workspace. The major reason why I cannot work on one
>>> feature then on another is just because I have to make sure that the
>>> changes in the related areas of the product play together well.
>>>
>>> With Perforce, I can have multiple changelists opened, that group the
>>> changed files as needed.
>>>
>>> With Git I cannot seem to finding the possibility to figure out how to
>>> achieve the same result. And the problem is that putting change sets
>>> on different Git branches (or workdirs, or whatever Git offers that
>>> makes the changes to be NOT in the same source tree) is not a viable
>>> option from me as I would have to re-build code as I re-integrate the
>>> changes between the branches (or whatever changes separation Git
>>> feature is used).
>>> Build takes time and resources and considering that I have to do it on
>>> multiple platforms (I do cross-platform development) it really
>>> denominates the option of not having multiple changes in the same code
>>> tree.
>>>
>>> Am I ignorant about some Git feature/way of using Git that would help?
>>> Is it worth considering adding to Git a feature like "group of files"
>>> that would offer some virtutal grouping of the locally changed files
>>> in the checked-out branch?
>>
>> Interesting question that came up at my workplace, too.
>>
>> Here is what I suggested:
>> 1. Keep working on a single branch and make commits for all features
>> 2. If you make a commit, prefix the commit message with the feature name
>> 3. After you are done with a feature create a new feature branch based on
>>    your combined feature branch. Use `git rebase -i` [1] to remove all
>>    commits that are not relevant for the feature. Alternatively you could
>>    cherry pick the relevant commits [2] if this is faster.
>>
>> I wonder what others think about this solution. Maybe there is a better
>> solution that I overlooked?
>>
>> - Lars
>>
>> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
>> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>
> This "single branch, related commits" approach is exactly what came
> to my mind as well.
>
> But, isn`t Perforce "changelist" an "atomic" group of changes - like
> "commit" in Git, "changeset" in Team Foundation Version Control,
> etc...?
>
> If so, it would mean that this "multiple pending changelists" flow
> would/should be translated to "multiple pending commits" in Git,
> where in the end _a single_ Perforce "changelist" is _a single_ Git
> "commit".
>
> Might be this is where the confusion is coming from, trying to fit
> natural Git "multiple commits per feature" (but "single feature per
> branch") concept into a "single changelist per feature" Perforce
> concept, described/required here?
>
> I don`t think there is a firm concept of such "multiple pending
> commits" in Git, but the point is the author is having multiple
> changelists/commits, and it actively updates them (the _same ones_),
> until they`re ready to be merged (submitted, in Perforce)... which
> you can do in Git, and might be quite easily :)
>
> So... In Git, you can create a separate commit for each changelist
> you have in Perforce (all these commits being on the same branch, as
> desired). Then, when you would have wanted to "update" the pending
> Perforce changelist (not sure what the corresponding command is in
> Perforce), you would just `git commit` your current state with
> additional "--squash" or "--fixup" parameters (depending on if you
> would like to add more description to existing/original commit
> message, or not), and the original commit SHA1.
>
> In the end, when everything is tested together and you would like to
> commit features separately (like submitting changelists in Perforce),
> you would just need to `git rebase -i --autosquash` your branch,
> where Git would squash all your "update" commits (fixup/squash ones,
> that is) to the original/initial ones you made as per your
> changelists/features. No need for manual rearranging, cherry-picking,
> or whatever.
>
> An example flow, with two "changelists" for two features (I`ll be
> using capital letters A, B, C... instead of commit SHA1, for
> simplicity):
>
>         ... do some "Feature 1" work...
>         $ git commit -m "Feature 1"
>         ... do some "Feature 2" work...
>         $ git commit -m "Feature 2"
>         ... do some "Feature 1" work...
>         $ git commit --fixup A
>         ... do some "Feature 1" work...
>         $ git commit --fixup A
>         ... do some "Feature 2" work...
>         $ git commit --squash B
>         ... do some "Feature 1" work...
>         $ git commit --fixup A
>         ... do some "Feature 1" work...
>         $ git commit --squash A
>         ... do some "Feature 2" work...
>         $ git commit --fixup B
>
>
> Branch history would look something like this (H is latest commit):
>
>         H fixup! Feature 2
>         G squash! Feature 1
>         F fixup! Feature 1
>         E squash! Feature 2
>         D fixup! Feature 1
>         C fixup! Feature 1
>         B Feature 2
>         A Feature 1
>
>
> When you finally do `git rebase -i --autosquash A^`, you should get a
> list like this[1]:
>
>         pick A Feature 1
>         fixup C fixup! Feature 1
>         fixup D fixup! Feature 1
>         fixup F fixup! Feature 1
>         squash G squash! Feature 1
>         pick B Feature 2
>         squash E squash! Feature 2
>         fixup H fixup! Feature 2
>
>
> Once rebase is finished, you`ll end up with branch history looking
> like this:
>
>         B' Feature 2
>         A' Feature 1
>
> ... where commits A, C, D, F and G have been squashed into a single
> commit A', and commits B, E and H have been squashed into a single
> commit B'. These two single commits should correspond to your two
> Perforce changelists.
>
> Now you can merge your commits separately, as desired ("submit" the
> "changelists").
>
> You can even first rearrange/split/squash them further, or make
> separate branches out of them, whatever you find appropriate - you
> can do whatever you like to them while they`re your local commits
> ("pending changelists"), before making them live/visible for other
> users as well (merge them to a public branch, "submit changelist").
>
> p.s. Doesn`t the flow required here look similar to Mercurial patch
> "queues" approach (again, resembling "quilt" functionality)? If so,
> "Guilt"[2] may be an option here as well... if the described flow
> can`t be altered a bit to align better with Git itself, might be
> profiting on the side of overall workflow simplicity ;)
>
> [1] Having commits automatically grouped/ordered, you can even
>     replace some "fixup" and "squash" with "reword", for example, so
>     those commits are kept as separate ones, providing you a chance
>     to edit their messages.
> [2] http://repo.or.cz/w/guilt.git
>
> Regards,
> Buga
Junio C Hamano· Jul 13, 2017, 18:09 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Nikolay Shustov <nikolay.shustov@gmail.com> writes:
Show 15 quoted lines
> Thank you for the detailed explanation, it looks like merging the
> commits would be helpful in my case. And I think it is a very good
> analogy that Perforce changelists are like multiple pending committs,
> if Git were supporting such.
>
> What it won't be achieving by using commits in this schema is the
> following thing I can do in Perforce:
> In the uncommitted Perforce changelists I can revert the changed file
> to the original state and move the files between the changelists.
> Quite often, while working on something, in the middle I would decide
> to isolate changes to a certain set of files to a separate changelsit
> - but then I might change my mind. It is all flexible until I actually
> commit my Perforce changelist, after which it becomes very much as
> committed changes in any other source control.
> This is actual flexibility I am looking for achieving in Git.

I actually think we already have such a flexibility. Unlike Perforce, Git is distributed, and the most important aspect of the distinction is that what happens _in_ your local Git repository may be called "committed" in Git lingo, but not visible to the public.

You can consider these commits you make in your repository "pending" when you think of your workflow in Perforce terms, until you merge and push out the result, which roughly corresponds to "submitting" in Perforce lingo.

Once you start treating your local commits that you haven't pushed out as changes that are still "pending" when observed from the outside world, you'd realize that you have as much flexibilty, if not more, to dice and slice them with the local tools like "rebase -i", "add -p", etc., as you would have in your Perforce workflow, I would think.

Nikolay Shustov· Jul 13, 2017, 19:31 UTC · re: Junio C Hamano · lore

Re: "groups of files" in Git?

Thank you, but I am not sure I quite understand the idea. Could you please elaborate on it for the following example?

I have two Perforce changelists ("A" and "B") that group uncommitted sets of files (paths to each of files could be different):

changelist A: file1 file2

changelist B: file3 file4

In Perforce, I am able to do the following:
- move files between changelists (e.g. file1 could be moved to changelist B)
- add new files to changeslit (e.g. changelist B can get additional file5)
- revert file changes which would effectively remove file from the
changelst (e.g. revert file2 will remove it from changelist A)
How would I do it with sets of files that would belong to Git commit?
On Thu, Jul 13, 2017 at 2:09 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 36 quoted lines
> Nikolay Shustov <nikolay.shustov@gmail.com> writes:
>
>> Thank you for the detailed explanation, it looks like merging the
>> commits would be helpful in my case. And I think it is a very good
>> analogy that Perforce changelists are like multiple pending committs,
>> if Git were supporting such.
>>
>> What it won't be achieving by using commits in this schema is the
>> following thing I can do in Perforce:
>> In the uncommitted Perforce changelists I can revert the changed file
>> to the original state and move the files between the changelists.
>> Quite often, while working on something, in the middle I would decide
>> to isolate changes to a certain set of files to a separate changelsit
>> - but then I might change my mind. It is all flexible until I actually
>> commit my Perforce changelist, after which it becomes very much as
>> committed changes in any other source control.
>> This is actual flexibility I am looking for achieving in Git.
>
> I actually think we already have such a flexibility.  Unlike
> Perforce, Git is distributed, and the most important aspect of the
> distinction is that what happens _in_ your local Git repository may
> be called "committed" in Git lingo, but not visible to the public.
>
> You can consider these commits you make in your repository "pending"
> when you think of your workflow in Perforce terms, until you merge
> and push out the result, which roughly corresponds to "submitting"
> in Perforce lingo.
>
> Once you start treating your local commits that you haven't pushed
> out as changes that are still "pending" when observed from the
> outside world, you'd realize that you have as much flexibilty, if
> not more, to dice and slice them with the local tools like "rebase
> -i", "add -p", etc., as you would have in your Perforce workflow,
> I would think.
>
>
Fredrik Gustafsson· Jul 11, 2017, 20:09 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Hi, I will choose a bit of a less diplomatic path here. Instead of trying to tell you how you can make git fit your needs, I would say that you shouldn't. I've two arguments:

1. It's always painful when you try to use a tool in some way it's not intended or used to work. If you're doing something different than anyone else using that tool, you're probably doing something wrong!

I doubt that your case is so special, so my suggestion is to either use git the way most people use it, with one branch for each feature, or do not use git at all, since perforce seems to be better with your workstyle.

2. Git is a snapshot based SCM system. That means that each commit unique identify a version of the code. With your system (as well as any time you're not commiting all changed files) the commit is never tested. You've no idea of actually knowing if your two changes is dependent or not. Of course you can guess, but it's still a guess and in your current work way with perforce you have no way of knowing if your changesets have a dependency between eachother or not since you never test them individually.

--
Please let me know if you feel that I've missed something.
I can see four solutions:

1. Now I would suggest that you have each feature in a commit and simply run your tests every few commits so you don't have to run it for each commit.

2. Improve your build and test time. I'm sure there's things here to improve.

3. Continue to use perforce. If I recall correctly perforce has even a git integration today.

4. Use integration branches in git and run the tests on that branch. This can be easy todo if you write some scripts for it.

Good luck!
-- 
Fredrik Gustafsson

phone: +46 733-608274
e-mail: iveqy@iveqy.com
website: http://www.iveqy.com
astian· Jul 11, 2017, 22:27 UTC · re: Nikolay Shustov · lore

Re: "groups of files" in Git?

Nikolay Shustov wrote:
Show 19 quoted lines
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
>
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me as I would have to re-build code as I re-integrate the
> changes between the branches (or whatever changes separation Git
> feature is used).
> Build takes time and resources and considering that I have to do it on
> multiple platforms (I do cross-platform development) it really
> denominates the option of not having multiple changes in the same code
> tree.
>
> Am I ignorant about some Git feature/way of using Git that would help?
> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

I never used Perforce and I'm not even sure I understand your problem, but I thought I'd mention something that nobody else seems to have yet (unless I missed it):

First, one thing that seems obvious to me from your description is that these "parallel features" you work on are obviously interdependent, therefore I would rather consider the whole thing as a single feature. Therefore, it makes sense to me to work in a single "topic branch".

This doesn't preclude one from separating the changes in logically sensible pieces. Indeed this is par for the course in Git and people do it all the time by dividing the bulk of changes into a carefully chosen series of commits.

I think the most common way of doing this is to simply work on the whole thing and once you're happy with it you use "git rebase --interative" in order to "prettify" your history.

But, and here comes the part I think nobody mentioned yet, if your feature work is considerably large or spans a considerably long time it may be undesirable to postpone all that work until the very end (perhaps by then you already forgot important information, or perhaps too many changes have accumulated so reviewing them all becomes significantly less efficient). In that case, one solution is to use a "patch management system" which will let you do that work incrementally (going back and forth as needed).

If you know mercurial, this is "hg mq". I don't think Git has any such system built-in, but I know there are at least these external tools that integrate with Git: https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Patch-management_Interface_layers

Feel free to ignore this if I totally misunderstood your use case.
Cheers.
Nikolay Shustov· Jul 13, 2017, 17:04 UTC · re: astian · lore

Re: "groups of files" in Git?

Thank you taking time to think about my issue (I am actually grateful to everyone in this e-mail thread, who did). I looked into Mercurian MQ and it doesn't seem to fit what I need - from what I understood, it speaks about the committed changes and those are not amendable. This is kinda not what I need.

On Tue, Jul 11, 2017 at 6:27 PM, astian <astian@eclipso.at> wrote:
Show 58 quoted lines
> Nikolay Shustov wrote:
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> I never used Perforce and I'm not even sure I understand your problem,
> but I thought I'd mention something that nobody else seems to have yet
> (unless I missed it):
>
> First, one thing that seems obvious to me from your description is that
> these "parallel features" you work on are obviously interdependent,
> therefore I would rather consider the whole thing as a single feature.
> Therefore, it makes sense to me to work in a single "topic branch".
>
> This doesn't preclude one from separating the changes in logically
> sensible pieces.  Indeed this is par for the course in Git and people do
> it all the time by dividing the bulk of changes into a carefully chosen
> series of commits.
>
> I think the most common way of doing this is to simply work on the whole
> thing and once you're happy with it you use "git rebase --interative" in
> order to "prettify" your history.
>
> But, and here comes the part I think nobody mentioned yet, if your
> feature work is considerably large or spans a considerably long time it
> may be undesirable to postpone all that work until the very end (perhaps
> by then you already forgot important information, or perhaps too many
> changes have accumulated so reviewing them all becomes significantly
> less efficient).  In that case, one solution is to use a "patch
> management system" which will let you do that work incrementally (going
> back and forth as needed).
>
> If you know mercurial, this is "hg mq".  I don't think Git has any such
> system built-in, but I know there are at least these external tools that
> integrate with Git:
> https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Patch-management_Interface_layers
>
> Feel free to ignore this if I totally misunderstood your use case.
>
> Cheers.
>
>
Igor Djordjevic· Jul 13, 2017, 23:06 UTC · re: astian · lore

Re: "groups of files" in Git?

Hi astian,
On 12/07/2017 00:27, astian wrote:
Show 56 quoted lines
> Nikolay Shustov wrote:
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
> 
> I never used Perforce and I'm not even sure I understand your problem,
> but I thought I'd mention something that nobody else seems to have yet
> (unless I missed it):
> 
> First, one thing that seems obvious to me from your description is that
> these "parallel features" you work on are obviously interdependent,
> therefore I would rather consider the whole thing as a single feature.
> Therefore, it makes sense to me to work in a single "topic branch".
> 
> This doesn't preclude one from separating the changes in logically
> sensible pieces.  Indeed this is par for the course in Git and people do
> it all the time by dividing the bulk of changes into a carefully chosen
> series of commits.
> 
> I think the most common way of doing this is to simply work on the whole
> thing and once you're happy with it you use "git rebase --interative" in
> order to "prettify" your history.
> 
> But, and here comes the part I think nobody mentioned yet, if your
> feature work is considerably large or spans a considerably long time it
> may be undesirable to postpone all that work until the very end (perhaps
> by then you already forgot important information, or perhaps too many
> changes have accumulated so reviewing them all becomes significantly
> less efficient).  In that case, one solution is to use a "patch
> management system" which will let you do that work incrementally (going
> back and forth as needed).
> 
> If you know mercurial, this is "hg mq".  I don't think Git has any such
> system built-in, but I know there are at least these external tools that
> integrate with Git:
> https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Patch-management_Interface_layers
> 
> Feel free to ignore this if I totally misunderstood your use case.
> 
> Cheers

This message actually creeped me out the first time I read it, after writing an e-mail reply of my own[1].

The tone it`s written in, the points you make, and even the conclusion about "hg mg" -- as if you were reading my mind.

Yours was sent a bit before mine, but I guess we were writing it at the same time as well... Just spooky, lol.

That said, I totally understand what you`re talking about, and I gave an example of the desired (yet missing?) Git work flow here[2] :)

[1] https://public-inbox.org/git/6e4096fd-cbab-68f0-7a23-654382cb810e@gmail.com/ [2] https://public-inbox.org/git/27a3c650-5843-d446-1f59-64fabe5434a3@gmail.com/

Regards, Buga

← back to recent threads