{"thread":{"id":"14893","subject":"Show of hands, how many set USE_NSEC","startedAt":"2008-08-08T16:34:55Z","lastAt":"2008-08-13T20:01:26Z","messageCount":7,"participants":["Shawn O. Pearce","Johannes Schindelin","Linus Torvalds","Robin Rosenberg"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"86522","messageId":"20080808163455.GE9152@spearce.org","threadId":"14893","inReplyTo":null,"subject":"Show of hands, how many set USE_NSEC","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T16:34:55Z","receivedAt":"2008-08-08T16:34:55Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"How many users really build their Git with USE_NSEC=1?\n\nI'm suspecting a status issue in jgit caused by jgit honoring a\nmillisecond resolution on file modification timestamps, and the\nunderlying filesystem supporting at least a 1/2 second (or finer)\ngranularity, but C Git was built without USE_NSEC so it only honors\n1 second granularity.\n\nThis can cause jgit to think a file is locally modified as the\nmtime has data in the tv_nsec field, but C Git set that to 0 in\nthe index as USE_NSEC wasn't enabled at build time.\n\nI'm trying to come up with a sane way for jgit to realize it should\ntruncate the milliseconds out of a timestamp before it comes to\nthe index record.\n\n-- \nShawn.\n"},{"id":"86526","messageId":"alpine.DEB.1.00.0808081854120.24820@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14893","inReplyTo":"20080808163455.GE9152@spearce.org","subject":"Re: Show of hands, how many set USE_NSEC","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-08-08T16:55:27Z","receivedAt":"2008-08-08T16:55:27Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n\n> How many users really build their Git with USE_NSEC=1?\n\nI don't.\n\n> I'm trying to come up with a sane way for jgit to realize it should \n> truncate the milliseconds out of a timestamp before it comes to the \n> index record.\n\nYou could add a config variable.  I hope that soon, we no longer need to \nshare the same index between C Git and JGit (I hope for a delta \npack implementation in Java...)\n\nCiao,\nDscho\n"},{"id":"86528","messageId":"20080808165718.GG9152@spearce.org","threadId":"14893","inReplyTo":"alpine.DEB.1.00.0808081854120.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: Show of hands, how many set USE_NSEC","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T16:57:18Z","receivedAt":"2008-08-08T16:57:18Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> On Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n> \n> > I'm trying to come up with a sane way for jgit to realize it should \n> > truncate the milliseconds out of a timestamp before it comes to the \n> > index record.\n> \n> You could add a config variable.\n\nI was thinking a gitconfig (e.g. jcore.usensec) to enable the tv_nsec\nusage (ok, well milliseconds only) in that repository, or globally\n(if in ~/.gitconfig).\n\nI also thought about looking at the index records to see if the\ntv_nsec fields were always 0.  If all of them were 0 it would be a\ngood indication that the filesystem doesn't support that level of\ngranularity, or that whoever last wrote this index doesn't support\nthat level of granularity.  But this is a very expensive operation\nto perform, relatively speaking.\n\n> I hope that soon, we no longer need to \n> share the same index between C Git and JGit (I hope for a delta \n> pack implementation in Java...)\n\nI fail to see what the DIRC (.git/index) file format and its cache\nof tv_sec/tv_nsec has to do with delta pack implementation in Java.\nOr are you saying that you could stop using C Git in certain cases\nif you had delta pack generation in Java?\n\nReally I'd just like to scrap the entire DIRC file format and do\nit over again.  Having the flat namespace is nuts.  Linus and I\nreally disagree here, and since I have never produced code for C\nGit to replace it (and prove why its better) I think he has me in\nhis kill file now.  :)\n\n-- \nShawn.\n"},{"id":"86531","messageId":"alpine.LFD.1.10.0808081027590.3462@nehalem.linux-foundation.org","threadId":"14893","inReplyTo":"20080808165718.GG9152@spearce.org","subject":"Re: Show of hands, how many set USE_NSEC","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-08-08T17:42:06Z","receivedAt":"2008-08-08T17:42:06Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n> \n> Really I'd just like to scrap the entire DIRC file format and do\n> it over again.  Having the flat namespace is nuts.  Linus and I\n> really disagree here, and since I have never produced code for C\n> Git to replace it (and prove why its better) I think he has me in\n> his kill file now.  :)\n\nI really disagree, because you have no clue about performance.\n\nThe flat file format is absolutely _critical_ for performance.\n\nTry this:\n\n\ttime cat .git/index > /dev/null\n\ttime git ls-tree -r HEAD > /dev/null\n\nwhich isn't quite fair (but see how it's unfair both ways later), but \ngives you an idea of the cost of recursively reading lots of small files.\n\nNotice how the latter is about an order-and-a-half slower? \n\nThen try it with a cold cache. If you thought the \"ls-tree\" was unfair \nbecause it used things like zlib and pack-files, realize that the ls-tree \nactually almost certainly has _better_ IO patterns than doing individual \nfiles. So when you do the cold-cache case, we're actually giving an unfair \nadvantage to ls-tree (assuming fully packed repo like I have).\n\nAnd it still loses in a big big way. If it was actually one file per \ndirectory, it would be _horrible_. You could kind of approximate the IO \npatterns for that case by doing\n\n\ttime find . -name don-t-exist > /dev/null\n\n(which obviousyl uses \"readdir()\" instead of read(), but the IO patterns \nshould be similar).\n\nThe point is, the index file absolutely *HAS* to be a single file in order \nto perform well in a big project. Otherwise there's no point, and you \nmight as well just use a git \"tree\" object for everything.\n\nNow, if you talk about the _sorting_ order (as opposed to the flatness of \nthe file), I could probably agree. The sort order was probably a mistake. \nThat said, we're stuck with it.  You can't change it without changing the \ntree object format, so it's not just an \"local index file\" format issue.\n\n\t\t\tLinus\n"},{"id":"86532","messageId":"20080808175247.GH9152@spearce.org","threadId":"14893","inReplyTo":"alpine.LFD.1.10.0808081027590.3462@nehalem.linux-foundation.org","subject":"Re: Show of hands, how many set USE_NSEC","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T17:52:47Z","receivedAt":"2008-08-08T17:52:47Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> wrote:\n> On Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n> > \n> > Really I'd just like to scrap the entire DIRC file format and do\n> > it over again.  Having the flat namespace is nuts.\n> \n> I really disagree, because you have no clue about performance.\n\nActually, I do have half a clue.  I just haven't convinced you of\nthat yet.  Something to aspire to.\n \n> The flat file format is absolutely _critical_ for performance.\n \nI know, I agree.\n\n> The point is, the index file absolutely *HAS* to be a single file in order \n> to perform well in a big project. Otherwise there's no point, and you \n> might as well just use a git \"tree\" object for everything.\n\nYes, it _has_ to be a single file, at the root of the directory tree.\nI agree with that design decision entirely.\n\nThat single file however does not need to be structured internallyy\nthe way it is.\n \n> Now, if you talk about the _sorting_ order (as opposed to the flatness of \n> the file), I could probably agree. The sort order was probably a mistake. \n> That said, we're stuck with it.  You can't change it without changing the \n> tree object format, so it's not just an \"local index file\" format issue.\n\nI was talking about something like the 'TREE' extension.  If we\nused a format such as:\n\n\tindex_file:: tree\n\ttree:: entry_count sha1_id record*\n\trecord:: mode pathlen path (tree | file)\n\tfile:: sha1_id ctime mtime ....\n\nAnd sorted the entries within each tree record by their path (or\npath+mode in \"Git style\") then we wind up with all records in the\nindex file in the same order they are today, and we don't have the\nbig redundant path strings like \"src/a.c\", \"src/b.c\", \"src/c.c\".\n\nThis may actually create a _smaller_ index file, resulting in a\nfew less minor page faults as we read it in.\n\n-- \nShawn.\n"},{"id":"86534","messageId":"alpine.LFD.1.10.0808081056090.3462@nehalem.linux-foundation.org","threadId":"14893","inReplyTo":"20080808175247.GH9152@spearce.org","subject":"Re: Show of hands, how many set USE_NSEC","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-08-08T18:00:17Z","receivedAt":"2008-08-08T18:00:17Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n> \n> That single file however does not need to be structured internallyy\n> the way it is.\n\nOk, at that point I certainly agree.  \n\nAs long as we're talking about a single flat file, I don't think it would \nbe all that painful to have a totally new index format. The original \nformat was (obviously) designed to be just mmap'ed and turned into a C \narray with no real parsing.\n\nBut once we started building a separate index data structure with internal \nstructure _anyway_ (for the extended in-memory flags and the filename \nhashing), a lot of the reasons for the original format kind of went away.\n\n\t\t\tLinus\n"},{"id":"87101","messageId":"200808132201.26868.robin.rosenberg.lists@dewire.com","threadId":"14893","inReplyTo":"20080808163455.GE9152@spearce.org","subject":"Re: Show of hands, how many set USE_NSEC","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-08-13T20:01:26Z","receivedAt":"2008-08-13T20:01:26Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"fredagen den 8 augusti 2008 18.34.55 skrev Shawn O. Pearce:\n> How many users really build their Git with USE_NSEC=1?\n> \n> I'm suspecting a status issue in jgit caused by jgit honoring a\n> millisecond resolution on file modification timestamps, and the\n> underlying filesystem supporting at least a 1/2 second (or finer)\n> granularity, but C Git was built without USE_NSEC so it only honors\n> 1 second granularity.\n> \n> This can cause jgit to think a file is locally modified as the\n> mtime has data in the tv_nsec field, but C Git set that to 0 in\n> the index as USE_NSEC wasn't enabled at build time.\nWhen jgit finds an index entry with zero nsec it ignores the subsecond\nportion of the file timestamp when comparing.\n\n-- robin\n"}]}