Creating something like increasing revision numbers

Dear all,

(please Cc)

I am managing some of my projects with git and I am quite happy about it.

There is another project I am working that is quite big, the subversion checkout is about 14Gb. Doing svn up is a pain, it has to open tens of thousands of files and directories traversing the whole tree, trashing the fs cache and taking ages.

My idea was to move to git, from what I read it should be more capable in handling these type of projects.

Now, there is one show-stopper I see. From our repository we create a set of "packages", and the maximum of the "last-changed" revisions of the contained files determine the "version" of the package. This guarantees that any change in a file will increase the revision number of the package (some tricks for removals have to be done). This is necessary since we are distributing the packages from servers and the version number pf a package determines whether it should be upgraded (well known concept).

Now my question, is there any way to set up something similar with git?

My idea is that git - like subversion - could (if asked to) count each commit (global to the repository, irrelevant of the branch) and give it a version number. Since we all will use a bare repository on a server and pull/push from/to there, I think that something similar could be possible.

So, before I delve into more gitty-nitty conversion, let me know if there is any chance for that, or we should stay with subversion.

Thanks a lot and all the best

Norbert

PS: for those interested, it is TeX Live: www.tug.org/texlive

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
YONKERS (n.)
(Rare.) The combined thrill of pain and shame when being caught in
public plucking your nostril-hairs and stuffing them into your
side-pocket.
			--- Douglas Adams, The Meaning of Liff

On Sunday 18 October 2009, Norbert Preining wrote:

A global, increasing version number ala SVN is fundamentally impossible in any distributed version control system (like Git).

However, you can get a useful version specifier from the "git describe" command. It will give you back something like the following:

    $ git describe
    v1.0.4-14-g2414721

where the "v1.0.4" part is the last tag that the current state is based on, the "14" part is the number of commit between that tag and the current state, and the "2414721" is the abbreviated object name (SHA1 id) for the current commit itself.

This is somewhat more complex than a simple version number, but guarantees a globally unique name for your current state that works in a distributed environment.

Also, I find "v1.0.4-14..." (i.e. 14 commits since v1.0.4) much more useful than, say, "12534" (i.e. 12534 commits since the start of the project).

See 'git help describe for more info'

Have fun! :)

...Johan

On So, 18 Okt 2009, Johan Herland wrote:

Yes, agreed. 

The point is that I do not actually need the "distributed" part of git. I want one central repository and all collaborators commit to that. Yes, that is subversion, I know.

We have no branches, no tags, nothing of that. Only trunk.

So if we have only one tag (initial) then it would count the number of commits?

And if yes, is it easy to find out at which commit a file has been changed last time (svn status -v).

I have read a lot on the net about the impossibility, and I agree that in a distributed setting it does not work.

And in fact we would not even have revision numbers on our local git repositories. Only one (the master checkout from which our distribution server is updated) needs to have some commit numbers.

THe reason is that we use that as serial number for packages. One packages is roughly on package from CTAN (Comprehensive TeX Archive Network, like CPAN), and we want to make sure that if that is updated on CTAN and we import it into our system, the next time we create a TeX Live package for it (that will be served to quite a lot of users) we have a new version number.

We first thought about using the version number as provided by authors, but that is completely useless, because there are tooo many authors of packages on CTAN where the version numbers are in no way increasing ;-) So we settled for the max of all the last changed revision number of the contained files, whcih is guaranteed to increase.

As a lat resort I will try to use git-svn ...

Again, thanks a lot and all the best

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
QUOYNESS (n.)
The hatefulness of words like 'relionus' and 'easiephit'.
			--- Douglas Adams, The Meaning of Liff

On Sunday 18 October 2009, Norbert Preining wrote:

Yes. You can create the 'initial' tag with

  git rev-list HEAD | tail -n1 | xargs git tag initial

and from then on

  git describe --tags --match initial | cut -d'-' -f2

will give you the increasing "revision" number you're looking for. Just be aware that if you have two parallel branches with the same number of commits, they will give you the same number. I.e. this only works for a single, stable (i.e. no history rewrites), branch of development.

Hope this helps,

...Johan

Johan Herland wrote:

So if you concatenate the branch name with the "revision" number you would have pretty unique tags repo-wide, if you won't rename your branches.

On Sun, 18 Oct 2009, Johan Herland wrote:

Even simpler, without any tag:

	git rev-list HEAD | wc -l

That should roughly give the equivalent of the SVN revision number. Valid only in one specific repository of course.

Nicolas

Norbert Preining <preining@logic.at> writes:

No, it is not subversion at all. Don't say "I know" until you really know. Anybody who thinks that is not distributed does not understand distributedness of git.

Distributedness does _not_ come from not having a central shared repository, nor not using more than one branch.

Distrubutedness comes the moment any and all of your people can make their commits _locally_. And the fundamental lack of "increasing global version number" is one implication of the distributedness.

Suppose you and Alice collaborate that way. You make the initial commit, that is pushed to the central shared repository, and alice clones.

    norb$ git commit -m 'first one'
    norb$ git push
    alice$ git clone $central

Now Norb and Alice share that the 'first one' is the current and only commit at the tip of their history. The central repository also shares that notion. You work and produce a few commits.

    norb$ edit; git commit -a -m 'second norb'
    norb$ edit; git commit -a -m 'third norb'
    norb$ edit; git commit -a -m 'fourth norb'

while Alice does the same.
    
    alice$ edit; git commit -a -m 'second alice'
    alice$ edit; git commit -a -m 'third alice'

You happened to push first:

    norb$ git push

You and the central repository shares the view of the history in which the mapping between "revision sequence number" and commits looks like:

    1. first one
    2. second norb
    3. third norb
    4. fourth norb

Imagine what view of the history Alice has at this point and think for a while. Recall Alice hasn't pulled from the central repository since she cloned. There are two possible things you may want to do.

 #1 Some sequential number is given but that is useless as global
    identifier.

    1. first one
    2. second alice
    3. third alice

 #2 Do not give such sequential number locally; the central repository is
    the _only_ place that assigns such a number.  '?' stands for
    'unnumbered'

    1. first one
    ?. second alice
    ?. third alice

Then Alice fetches from the central and integrates her history with it. She can do one of two things. "pull" to create a non-linear history by merging, or "pull --rebase" to keep the history linear.

Since you are imitating subversion, you may choose the latter route to rebase, and now the linearlized history would become:

    1. first one
    2. second norb
    3. third norb
    4. fourth norb
    ?. second alice
    ?. third alice

Alice's two commits may stay unnumbered (if you chose #2---no local versions), or changes from 2/3 to 5/6 (if you chose #1).

If you instead use merges, then there won't be 'sequence' number you can usefully compare anymore.

                    1 first one
                   /|
  second alice    2 2     second norb
                  | |
                  | 3     third norb
                  | |
   third alice    3 4     fourth norb
                  |/
   fourth alice   5

Scheme #1 inherently cannot give you stable and unique numbers in a distributed environment where you can commit locally without talking to the central "number naming authority". By rebasing, you can keep the long-term numbering unique (by renaming some new ones), but rebase has its own downsides besides the name of the commit.

Scheme #2 is a way to get some stablility; give the authority of numbering to the central repository and commits that haven't hit the central repository are left unnumbered. But that is generally not very useful for your purpose of giving incrementing version number for building (the developers would want to build for testing before finally committing to publish the result to the central place).

Running describe using one tag on the 'initial' would give you a rough equivalent of #1 (i.e. you get tentative numbers on the local commits), both in the case you rebase (i.e. your numbers change) and you merge (i.e. you can have more than one "second" commits and numbers are not unique), which would be the best compromise you can get.

On So, 18 Okt 2009, Junio C Hamano wrote:

[...]

That problem we have anyway with subversion, too, because as long as you do not commit your changes nothing will happen on package rebuild. So that is not a problem here.

Yes, we want server-sided linear numbers and anything *not* pushed to the server is unnumbered.

And that is also fine, because packages are built only from the server.

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
PUDSEY (n.)
The curious-shaped flat wads of dough left on a kitchen table after
someone has been cutting scones out of it.
			--- Douglas Adams, The Meaning of Liff

Norbert Preining wrote:

I was thinking to set a post-commit hook that reads the current version from a file, increment and save it, and also set a tag with that value.

Being a DVCS, this kind of versioning can only be trusted on a single repo, but if you set it on the "main" repo, it should work.

The only drawback could be the ever growing number of tags, I don't know how it will work with thousands of tags or more.

Have a nice day,
  A.

2009/10/18 alexandrul <alexandrul.ct@gmail.com>:

I think the other drawback is that the number would essentially be meaningless and more or less would just be a substitute sha1.

Consider when a remote adds commits and then merges and pushes. What number should those commits have?

Yves

On So, 18 Okt 2009, demerphq wrote:

Well, it would be increasing for that repository. And if we always update our packages from that repository the packages will be guaranteed to have increasing version number, too.

That is the *only* thing I care about. The numbers don't need to have a meaning, nothing else but that in our workflow we guarantee that at the end each package progresses in version numbers.

When using a central repository to which he pushes within that central repository it would give a specific number.

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
WIMBLEDON (n.)
That last drop which, no matter how much you shake it, always goes
down your trouser leg.
			--- Douglas Adams, The Meaning of Liff

2009/10/18 Norbert Preining <preining@logic.at>:

Consider you have A-B-C-D-E in your master repo. So presumably numbered 1..5.

If i then make a trivial comment fix to A and then merge and push we end up with:

A-B-C-D-E-G
 \        /
  F------+

If i understand you right you will set F to 6 and G to 7. Thus youll end up with the problem that F is a descendent of A yet has a higher "version number" than E. You can repeat this process for ever.

If this suits your needs then great.

cheers, Yves

demerphq wrote:

Just like subversion :D

It helps by ordering commits by time, not by parents, and I don't need repo access to figure it out that F is the latest bug/feature/fix.

Think about 2 sql servers, each has different versions of the same stored procedure. Which one is newer, given that you don't have the git repository on the production servers? I can't make any decision on SHA1 only, and if they belong to the same branch ("prod") I'm so out of luck. At the moment I'm saving SHA1, refs, author date, comitter date and last subject. But it's easier for me to compare revision numbers than dates. And don't forget that updating decisions may be taken by people that have nothing to do with development.

Have a nice day,
  A.

On Sun, Oct 18, 2009 at 10:41 AM, Norbert Preining <preining@logic.at> wrote:

There is a large LKML thread discussing this.... http://lwn.net/Articles/355923/

On Sun, 18 Oct 2009, Norbert Preining wrote:

It's possible as long as you don't think of the "version number" as a property of the commit, but rather a property that some commits get by virtue of having been at some time the commit that's what would be found on that particular server at that particular time. Even though the history of the *content* is non-linear, the sequence of values stored in refs/heads/master on your central server is linear, local, and easy to enumerate.

Of course, when someone does a bunch of development in parallel with other people, does a final merge, and pushes it back to the server, this only increases the version by one, and only the final merge actually has a version number at all. For your application, this shouldn't be a problem, because the intermediate commits don't ever get packages created of them to need to be compared to other packages.

	-Daniel
*This .sig left intentionally blank*

Hi all,

thanks everyone for the nice feedback!

On So, 18 Okt 2009, Daniel Barkalow wrote:

Right! That is a good point. In fact I don't care about (local) commits, but about the pushes to the central server.

That is exactely what I need.

As it is now with svn, we have to live with that. The point is that we still would see many different commits pushed to the server, so git log would show the single items, but the "versioning sequence number" is only increased by one. That would be *absolutely*perfect* for me!

Right!

Now my follow-up questions:
- how would one access this "sequence" number on the server
- is there a way to determine at which of this "sequence" numbers a specific
  file has been changed last?

Thanks a lot and all the best

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
DULEEK (n.)
Sudden realisation, as you lie in bed waiting for the alarm to go off,
that it should have gone off an hour ago.
			--- Douglas Adams, The Meaning of Liff

On Mon, 19 Oct 2009, Norbert Preining wrote:

There isn't currently anything built in that counts up like that; however, it shouldn't be too hard to add something, because the reflog gets an entry at the same times the sequence number would increase. In fact, you could disable pruning the reflog, and use its length (in lines), except that would get slow and git doesn't expect you to care about the complete history there (in fact, you only care about the amount of history past some point).

There isn't a built-in way, but you can find the current hash for a filename with "git ls-tree -r <branch> <filename>", and find the hash as of N changes ago with "git ls-tree -r <branch>@{<N>} <filename>". You're looking for the smallest N where they don't match. (And you probably don't want to be a binary search or the like, because that might miss that a file was most recently affected by having a change reverted; you'd want to be sure to report the version that reverted the change, not the version that introduced the content the later one returned to.

	-Daniel
*This .sig left intentionally blank*

Hi Daniel,

On So, 18 Okt 2009, Daniel Barkalow wrote:

Ok.

That sounds like we cannot use that, because we have to do that for about 80k files and that on each (at least daily) rebuilt. That is not feasable.

Again thanks for your helpful comments!

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
BRUMBY
The fake antique plastic seal on a pretentious whisky bottle.
			--- Douglas Adams, The Meaning of Liff

On Mon, 19 Oct 2009, Norbert Preining wrote:

It's likely to be much more efficient (and maybe sufficiently efficient) to do them all in a single pass. In fact, it should be easy enough to keep a cache of the latest numbers for the files, and update that to change the value for those files that differ between the old commit and the new commit in the post-update hook.

	-Daniel
*This .sig left intentionally blank*

On Mon, 19 Oct 2009, Norbert Preining wrote:

you can count how many commits the server has: git rev-list HEAD | wc -l

You can do: git log <path/filename>. That would give you a list of commits that touched that file. But to find out which commit "number" it is you could do:

	git rev-list $(git rev-list -1 HEAD <path/file>) | wc -l

Again, that is true for a particular repository only and may give a different result in another repository with a different state even for the same commit.

However, given what you want to do is really to stick to the SVN way of doing things, then why don't you simply stay with SVN? Git works in a fundamentally different way than SVN, and if you aren't willing/able to change your workflow away from the SVN way then there is really not much for you to gain by switching to Git.

Nicolas

On So, 18 Okt 2009, Nicolas Pitre wrote:

I fear so.

Well, as I said in the beginning, the metadata handling of svn is a pain in huge repositories like ours with thousands of subdirectories of subdirs of subdirs and 80k+ files. That was the reason I thought about git.

But I am coming to the conclusion that I will use git for my other projects, but not for that one.

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining                                        Associate Professor
JAIST Japan Advanced Institute of Science and Technology   preining@jaist.ac.jp
Vienna University of Technology                               preining@logic.at
Debian Developer (Debian TeX Task Force)                    preining@debian.org
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
GLASSEL (n.)
A seaside pebble which was shiny and interesting when wet, and which
is now a lump of rock, which children nevertheless insist on filing
their suitcases with after the holiday.
			--- Douglas Adams, The Meaning of Liff

Norbert Preining schrieb:

You can always run 'git rev-list master | wc -l' to get a sequence number. You can throw in --first-parent if you do not want to count commits that entered master through a merge commit.

You can configure your reflogs such that they do not expire. Then you count the entries in the reflog.

-- Hannes

On Mon, Oct 19, 2009 at 02:44:47AM +0200, Norbert Preining wrote:

If you have any control over how people will use git, then you can give your constantly-incrementing revision number more stability by ensuring that everyone uses 'git pull --rebase'.

That'll literally keep the history completely linear. If someone forgets then it's not a big deal; you'll just get a merge commit and the number will increment by 2 instead of by 1.

If you've done the "tag the initial commit" as suggested elsewhere on this thread:

	git tag projectname $(git rev-list HEAD | tail -n1)

then you can do this with simply:

	git describe --tags

It should output something like:

	projectname-101-g20912df

	commit=$(git log --pretty=%H -1 -- <filename>)
	git describe --tags $commit

Just another happy Debian user here,