git and mtime

> Hi folks,
>
> I'm using git to store some generated files, as well as their sources.
> (This is in the context of Debian package development, where entire
> upstream release tarballs are injected into an upstream branch, with
> Debian releases merging the upstream branch, and adding the Debian
> packaging files.)
>
> The upstream release tarballs contains files such as
> - yacc/lex code, and the corresponding generated sources
> - Docbook/XML code, and corresponding HTML/PDF documentation
>
> These are provided by upstream so that end users don't need these  
> tools
> installed (particularly docbook, since the toolchain is so flaky on
> different systems).  However, the fact that git isn't storing the
> mtime of the files confuses make, so it then tries to regenerate these
> (already up-to-date) files, and fails in the process since the tools
> aren't available.
>
> Would it be possible for git to store the mtime of files in the tree?
>

> Hi,
> 
> On 19.11.2008, at 12:37, Roger Leigh wrote:
> 
>> Hi folks,
>>
>> I'm using git to store some generated files, as well as their sources.
>> (This is in the context of Debian package development, where entire
>> upstream release tarballs are injected into an upstream branch, with
>> Debian releases merging the upstream branch, and adding the Debian
>> packaging files.)
>>
>> The upstream release tarballs contains files such as
>> - yacc/lex code, and the corresponding generated sources
>> - Docbook/XML code, and corresponding HTML/PDF documentation
>>
>> These are provided by upstream so that end users don't need these tools
>> installed (particularly docbook, since the toolchain is so flaky on
>> different systems).  However, the fact that git isn't storing the
>> mtime of the files confuses make, so it then tries to regenerate these
>> (already up-to-date) files, and fails in the process since the tools
>> aren't available.
>>
>> Would it be possible for git to store the mtime of files in the tree?
>>
> 
> This subject comes up from time to time, but the answer always
> stays the same: No. The trees are purely defined by their content, and
> that's by design.
> 
> If you do not want to regenerate files that are already up-to-date,
> you need multiple checkouts of the same repository.
> 

> Matthias Kestenholz wrote:
>> Hi,
>>
>> On 19.11.2008, at 12:37, Roger Leigh wrote:
>>
>>> I'm using git to store some generated files, as well as their sources.
>>> (This is in the context of Debian package development, where entire
>>> upstream release tarballs are injected into an upstream branch, with
>>> Debian releases merging the upstream branch, and adding the Debian
>>> packaging files.)
>>>
>>> The upstream release tarballs contains files such as
>>> - yacc/lex code, and the corresponding generated sources
>>> - Docbook/XML code, and corresponding HTML/PDF documentation
>>>
>>> Would it be possible for git to store the mtime of files in the tree?
>>
>> This subject comes up from time to time, but the answer always
>> stays the same: No. The trees are purely defined by their content, and
>> that's by design.
>>
>> If you do not want to regenerate files that are already up-to-date,
>> you need multiple checkouts of the same repository.
>
> Or a make-rule that touches the files you know are up to date. Since you
> control the build environment, that's probably the simplest solution.

> On Thu, Nov 20, 2008 at 09:38:28AM +0100, Andreas Ericsson wrote:
>> Matthias Kestenholz wrote:
>>> Hi,
>>>
>>> On 19.11.2008, at 12:37, Roger Leigh wrote:
>>>
>>>> I'm using git to store some generated files, as well as their sources.
>>>> (This is in the context of Debian package development, where entire
>>>> upstream release tarballs are injected into an upstream branch, with
>>>> Debian releases merging the upstream branch, and adding the Debian
>>>> packaging files.)
>>>>
>>>> The upstream release tarballs contains files such as
>>>> - yacc/lex code, and the corresponding generated sources
>>>> - Docbook/XML code, and corresponding HTML/PDF documentation
>>>>
>>>> Would it be possible for git to store the mtime of files in the tree?
>>> This subject comes up from time to time, but the answer always
>>> stays the same: No. The trees are purely defined by their content, and
>>> that's by design.
>>>
>>> If you do not want to regenerate files that are already up-to-date,
>>> you need multiple checkouts of the same repository.
>> Or a make-rule that touches the files you know are up to date. Since you
>> control the build environment, that's probably the simplest solution.
> 
> This is the approach I'm currently taking, since it's simple and
> doesn't require any tool changes.  Ideally, I'd like to avoid
> such hackiness, though.
> 
> I understand all the arguments I've seen in favour of not using the
> mtime of the files when checking out.  They make sense.  However,
> in some situations (such as this), they do not--git is breaking
> something that was previously working.  In my case, I'm
> injecting *release tarballs* into git, and the timestamps on the
> files really do matter.  Regarding issues with branching and branch
> switching, I always do builds from clean in this case.
> 
> If an option was added to git-checkout to restore mtimes, it need
> not be the default, but git could record them on commit and then
> restore them if asked /explicitly/.
> 

> For this, and some other uses I have in mind for git, it would be
> great if git could store some more components of the inode
> metadata in the tree, such as:
> - mtime
> - user
> - group
> - full permissions
> - and also allow storage of the full range of file types (i.e.
>   block, character, pipe, etc.)
> 
> This would allow git to be used as the basis for a complete
> functional versioned filesystem (which I'd like to use for my
> lightweight virtualisation tool, schroot, which currently
> uses LVM snapshots for this purpose).
> 

> Roger Leigh wrote:
>> For this, and some other uses I have in mind for git, it would be
>> great if git could store some more components of the inode
>> metadata in the tree, such as:
>> - mtime
>> - user
>> - group
>> - full permissions
>> - and also allow storage of the full range of file types (i.e.
>>   block, character, pipe, etc.)
>>
>> This would allow git to be used as the basis for a complete
>> functional versioned filesystem (which I'd like to use for my
>> lightweight virtualisation tool, schroot, which currently
>> uses LVM snapshots for this purpose).
>>
> 
> I believe someone else has done some work along the way of
> turning git into complete-with-metadata backupsystem before.
> Google might prove beneficial.
> 

> Arafangion <thestar@fussycoder.id.au> writes:
> 
>> You should probably fix your build script,
> 
> ccache should help:
> 
> http://ccache.samba.org/
> 

> Arafangion <thestar@fussycoder.id.au> writes:
> 
> > You should probably fix your build script,
> 
> ccache should help:

> Hi,
>
> On Wed, 19 Nov 2008, Matthieu Moy wrote:
>
>> Arafangion <thestar@fussycoder.id.au> writes:
>> 
>> > You should probably fix your build script,
>> 
>> ccache should help:
>
> Only if you _allow_ the problem that makes ccache necessary.  Which we 
> don't.

> Would it be possible for git to store the mtime of files in the tree?
> 
> This would make it possible to do this type of work in git, since it's
> currently a bit random as to whether it works or not.  This only
> started when I upgraded to an amd64 architecture from powerpc32,
> I guess it's maybe using high-resolution timestamps.

> Would it be possible for git to store the mtime of files in the tree?
>
> This would make it possible to do this type of work in git, since it's
> currently a bit random as to whether it works or not.  This only
> started when I upgraded to an amd64 architecture from powerpc32,
> I guess it's maybe using high-resolution timestamps.
>

> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
> > Would it be possible for git to store the mtime of files in the tree?
> >
> > This would make it possible to do this type of work in git, since it's 
> > currently a bit random as to whether it works or not.  This only 
> > started when I upgraded to an amd64 architecture from powerpc32, I 
> > guess it's maybe using high-resolution timestamps.
> >
> 
> beside the obvious answer it comes back often as a request, it is 
> possible in theory to create a shell script which, for each file present 
> in the sandbox in the current branch, would find the mtime of the last 
> commit on that file (quite an expensive operation) and apply it.
> 
> I had a need for this once, then lost interest since using git as it is 
> is so much better than trying to mimic behaviour of old scm tools and 
> makefiles.

> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
> > Would it be possible for git to store the mtime of files in the tree?
> >
> > This would make it possible to do this type of work in git, since it's
> > currently a bit random as to whether it works or not.  This only
> > started when I upgraded to an amd64 architecture from powerpc32,
> > I guess it's maybe using high-resolution timestamps.
> >
> 
> beside the obvious answer it comes back often as a request, it is
> possible in theory to create a shell script which, for each file
> present in the sandbox in the current branch, would find the mtime of
> the last commit on that file (quite an expensive operation) and apply
> it.

> On Wed, Nov 19, 2008 at 05:18:16PM +0100, Christian MICHON wrote:
>> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
>>> Would it be possible for git to store the mtime of files in the tree?
>>>
>>> This would make it possible to do this type of work in git, since it's
>>> currently a bit random as to whether it works or not.  This only
>>> started when I upgraded to an amd64 architecture from powerpc32,
>>> I guess it's maybe using high-resolution timestamps.
>>>
>> beside the obvious answer it comes back often as a request, it is
>> possible in theory to create a shell script which, for each file
>> present in the sandbox in the current branch, would find the mtime of
>> the last commit on that file (quite an expensive operation) and apply
>> it.
> 
> Surely this is only expensive because you're not already storing the
> information in the tree; if it was there, it would be (relatively)
> cheap?

> Roger Leigh wrote:
>> On Wed, Nov 19, 2008 at 05:18:16PM +0100, Christian MICHON wrote:
>>> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
>>>> Would it be possible for git to store the mtime of files in the tree?
>>>>
>>>> This would make it possible to do this type of work in git, since it's
>>>> currently a bit random as to whether it works or not.  This only
>>>> started when I upgraded to an amd64 architecture from powerpc32,
>>>> I guess it's maybe using high-resolution timestamps.
>>>>
>>> beside the obvious answer it comes back often as a request, it is
>>> possible in theory to create a shell script which, for each file
>>> present in the sandbox in the current branch, would find the mtime of
>>> the last commit on that file (quite an expensive operation) and apply
>>> it.
>>
>> Surely this is only expensive because you're not already storing the
>> information in the tree; if it was there, it would be (relatively)
>> cheap?
>
> No, it's because git is *snapshot* based and doesn't care about anything
> but contents. Storing filestate information in the tree would be a
> backwards incompatible change that would require a major version change.

> Caring about meta-data the way you mean it would mean that
>
>  git add foo.c; git commit -m "kapooie"; touch foo.c; git status
>
> would show "foo.c" as modified. How sane is that?

> On Thu, Nov 20, 2008 at 02:06:13PM +0100, Andreas Ericsson wrote:
>> Roger Leigh wrote:
>>> On Wed, Nov 19, 2008 at 05:18:16PM +0100, Christian MICHON wrote:
>>>> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
>>>>> Would it be possible for git to store the mtime of files in the tree?
>>>>>
>>>>> This would make it possible to do this type of work in git, since it's
>>>>> currently a bit random as to whether it works or not.  This only
>>>>> started when I upgraded to an amd64 architecture from powerpc32,
>>>>> I guess it's maybe using high-resolution timestamps.
>>>>>
>>>> beside the obvious answer it comes back often as a request, it is
>>>> possible in theory to create a shell script which, for each file
>>>> present in the sandbox in the current branch, would find the mtime of
>>>> the last commit on that file (quite an expensive operation) and apply
>>>> it.
>>> Surely this is only expensive because you're not already storing the
>>> information in the tree; if it was there, it would be (relatively)
>>> cheap?
>> No, it's because git is *snapshot* based and doesn't care about anything
>> but contents. Storing filestate information in the tree would be a
>> backwards incompatible change that would require a major version change.
> 
> It's not strictly true that it's only caring about contents.  The
> contents are of course in the blobs, but the tree is already
> effectively storing inode data, since it's a directory of
> filenames/subtrees, just one that only cares to store the
> permissions part of the total inode data.
> 
> I understand that git stored the permissions tacked onto the hash;
> would it be feasable to tack on the other bits as well.

> If I understand correctly, it's binary encoded in the pack format,
> and that would require updating the format to hold the additional
> data?
> 
>> Caring about meta-data the way you mean it would mean that
>>
>>  git add foo.c; git commit -m "kapooie"; touch foo.c; git status
>>
>> would show "foo.c" as modified. How sane is that?
> 
> I've never come close to suggesting we do anything so insane.
> 
> What I am suggesting is that on add/commit, the inode metadata
> be recorded in the tree (like we already store perms), so that
> it can be (**optionally**) reused/restored on checkout.
> 
> Whether it's stored in the tree or not is a separate concern from
> whether to *use* it or not.  For most situations, it won't be
> useful, as has been made quite clear from all of the replies, and I
> don't disagree with this.  However, for some, the ability to have
> this information to hand to make use of would be invaluable.
> 

> 
> There have been quite a few suggestions to look into using hooks,
> and I'll investigate this.  However, I do have some concerns
> about *where* I would store this "extended tree" data, since it
> is implicitly tied to a single tree object, and I wouldn't
> want to store it directly as content.
> 

> Roger Leigh wrote:
>> On Thu, Nov 20, 2008 at 02:06:13PM +0100, Andreas Ericsson wrote:
>>> Roger Leigh wrote:
>>>> On Wed, Nov 19, 2008 at 05:18:16PM +0100, Christian MICHON wrote:
>>>>> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
>>>>>> Would it be possible for git to store the mtime of files in the tree?
>>>>>>
>>>>>> This would make it possible to do this type of work in git, since it's
>>>>>> currently a bit random as to whether it works or not.  This only
>>>>>> started when I upgraded to an amd64 architecture from powerpc32,
>>>>>> I guess it's maybe using high-resolution timestamps.
>>>>>>
>>> Caring about meta-data the way you mean it would mean that
>>>
>>>  git add foo.c; git commit -m "kapooie"; touch foo.c; git status
>>>
>>> would show "foo.c" as modified. How sane is that?
>>
>> I've never come close to suggesting we do anything so insane.
>>
>> What I am suggesting is that on add/commit, the inode metadata
>> be recorded in the tree (like we already store perms), so that
>> it can be (**optionally**) reused/restored on checkout.
>>
>> Whether it's stored in the tree or not is a separate concern from
>> whether to *use* it or not.  For most situations, it won't be
>> useful, as has been made quite clear from all of the replies, and I
>> don't disagree with this.  However, for some, the ability to have
>> this information to hand to make use of would be invaluable.
>>
>
> Then write a hook for it. You agree that for most users this will be
> totally insane, and yet you request that it's added in a place where
> everyone will have to pay the performance/diskspace penalty for it
> but only a handful will get any benefits. That's patently absurd.

>> There have been quite a few suggestions to look into using hooks,
>> and I'll investigate this.  However, I do have some concerns
>> about *where* I would store this "extended tree" data, since it
>> is implicitly tied to a single tree object, and I wouldn't
>> want to store it directly as content.
>
> Store it as a blob targeted by a lightweight tag named
> "metadata.$sha1" and you'll have the easiest time in the world when
> writing the hooks. Also, the tags won't be propagated by default,
> which is a good thing since your timestamps/uid's whatever almost
> certainly will not work well on other developers repositories.

> And yet the fact that it won't propagate makes it totally useless:
> all the other people using the repo won't get the extra metadata
> that will prevent build failures.  Having the extra data locally
> is nice, but not exactly what I'd call a solution.  The whole point
> of what I want is to have it as an integral part of the repo.

> On Thu, Nov 20, 2008 at 03:50:49PM +0100, Andreas Ericsson wrote:
>> Roger Leigh wrote:
>>> On Thu, Nov 20, 2008 at 02:06:13PM +0100, Andreas Ericsson wrote:
>>>> Roger Leigh wrote:
>>>>> On Wed, Nov 19, 2008 at 05:18:16PM +0100, Christian MICHON wrote:
>>>>>> On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
>>>>>>> Would it be possible for git to store the mtime of files in the tree?
>>>>>>>
>>>>>>> This would make it possible to do this type of work in git, since it's
>>>>>>> currently a bit random as to whether it works or not.  This only
>>>>>>> started when I upgraded to an amd64 architecture from powerpc32,
>>>>>>> I guess it's maybe using high-resolution timestamps.
>>>>>>>
>>>> Caring about meta-data the way you mean it would mean that
>>>>
>>>>  git add foo.c; git commit -m "kapooie"; touch foo.c; git status
>>>>
>>>> would show "foo.c" as modified. How sane is that?
>>> I've never come close to suggesting we do anything so insane.
>>>
>>> What I am suggesting is that on add/commit, the inode metadata
>>> be recorded in the tree (like we already store perms), so that
>>> it can be (**optionally**) reused/restored on checkout.
>>>
>>> Whether it's stored in the tree or not is a separate concern from
>>> whether to *use* it or not.  For most situations, it won't be
>>> useful, as has been made quite clear from all of the replies, and I
>>> don't disagree with this.  However, for some, the ability to have
>>> this information to hand to make use of would be invaluable.
>>>
>> Then write a hook for it. You agree that for most users this will be
>> totally insane, and yet you request that it's added in a place where
>> everyone will have to pay the performance/diskspace penalty for it
>> but only a handful will get any benefits. That's patently absurd.
> 
> The cost is tiny.  The extra space would be smaller than a single
> SHA1 hash.
> 
>> Especially since there are such easy workarounds that you can put in
>> place yourself instead.
> 
> 
>>> There have been quite a few suggestions to look into using hooks,
>>> and I'll investigate this.  However, I do have some concerns
>>> about *where* I would store this "extended tree" data, since it
>>> is implicitly tied to a single tree object, and I wouldn't
>>> want to store it directly as content.
>> Store it as a blob targeted by a lightweight tag named
>> "metadata.$sha1" and you'll have the easiest time in the world when
>> writing the hooks. Also, the tags won't be propagated by default,
>> which is a good thing since your timestamps/uid's whatever almost
>> certainly will not work well on other developers repositories.
> 
> And yet the fact that it won't propagate makes it totally useless:
> all the other people using the repo won't get the extra metadata
> that will prevent build failures.  Having the extra data locally
> is nice, but not exactly what I'd call a solution.  The whole point
> of what I want is to have it as an integral part of the repo.
> 

>>
>> Then write a hook for it. You agree that for most users this will be
>> totally insane, and yet you request that it's added in a place where
>> everyone will have to pay the performance/diskspace penalty for it
>> but only a handful will get any benefits. That's patently absurd.
>
> The cost is tiny.  The extra space would be smaller than a single
> SHA1 hash.

> >>>>> "Roger" == Roger Leigh <rleigh@codelibre.net> writes:
> 
> Roger> Except in this case I'm storing the content of *tarballs* (along with
> Roger> pristine-tar).  I'm committing exactly what's in the tarball with
> Roger> no changes (this is a requirement).  I can't change the source prior
> Roger> to commit.
> 
> If you're not doing distributed source code development, why are you using
> git?  It's hard to be angry at a screwdriver for not pounding in nails
> properly.

> On Wed, Nov 19, 2008 at 05:18:16PM +0100, Christian MICHON wrote:
> > On Wed, Nov 19, 2008 at 12:37 PM, Roger Leigh <rleigh@codelibre.net> wrote:
> > > Would it be possible for git to store the mtime of files in the tree?
> > >
> > > This would make it possible to do this type of work in git, since it's
> > > currently a bit random as to whether it works or not.  This only
> > > started when I upgraded to an amd64 architecture from powerpc32,
> > > I guess it's maybe using high-resolution timestamps.
> > >
> > 
> > beside the obvious answer it comes back often as a request, it is
> > possible in theory to create a shell script which, for each file
> > present in the sandbox in the current branch, would find the mtime of
> > the last commit on that file (quite an expensive operation) and apply
> > it.
> 
> Surely this is only expensive because you're not already storing the
> information in the tree; if it was there, it would be (relatively)
> cheap?  You could even compare the old and new trees to see if you
> needed to touch a file at all.
> 
> > You should store mostly content of source files. You should do a make
> > in your first cloned repo at least once before committing anything to
> > the repo. That's what I did and I saved days...
> 
> Except in this case I'm storing the content of *tarballs* (along with
> pristine-tar).  I'm committing exactly what's in the tarball with
> no changes (this is a requirement).  I can't change the source prior
> to commit.

> These are provided by upstream so that end users don't need these tools
> installed (particularly docbook, since the toolchain is so flaky on
> different systems).  However, the fact that git isn't storing the
> mtime of the files confuses make, so it then tries to regenerate these
> (already up-to-date) files, and fails in the process since the tools
> aren't available.

> also sprach Roger Leigh <rleigh@codelibre.net> [2008.11.19.1237 +0100]:
> > These are provided by upstream so that end users don't need these tools
> > installed (particularly docbook, since the toolchain is so flaky on
> > different systems).  However, the fact that git isn't storing the
> > mtime of the files confuses make, so it then tries to regenerate these
> > (already up-to-date) files, and fails in the process since the tools
> > aren't available.
> 
> I don't get it. Why are end users running make in the first place?
> Why aren't those in the build-dependencies?

> By end user, I mean person downloading and building the sources.
> 
> They are optional build depdendencies.  They are provided pre-built,
> and won't be rebuilt unless they get outdated.  In the release
> tarball, the timestamps are correct, ensuring this never happens.
> When checking out with git, the timestamps are incorrect, and it
> attempts to rebuild something that's *already built*.

> They are optional build depdendencies.  They are provided pre-built, and 
> won't be rebuilt unless they get outdated.  In the release tarball, the 
> timestamps are correct, ensuring this never happens. When checking out 
> with git, the timestamps are incorrect, and it attempts to rebuild 
> something that's *already built*.

> Hi,
> 
> On Thu, 20 Nov 2008, Roger Leigh wrote:
> 
> > They are optional build depdendencies.  They are provided pre-built, and 
> > won't be rebuilt unless they get outdated.  In the release tarball, the 
> > timestamps are correct, ensuring this never happens. When checking out 
> > with git, the timestamps are incorrect, and it attempts to rebuild 
> > something that's *already built*.
> 
> I'll try just one more time.  Why don't you teach your build process to 
> check if the generated files can be generated, and if not, fall back to 
> the committed ones?