{"thread":{"id":"33224","subject":"FW: Windows. Git, and Dedupe","startedAt":"2013-03-18T21:20:48Z","lastAt":"2013-03-20T21:45:20Z","messageCount":6,"participants":["Josh Rowe","René Scharfe"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"211608","messageId":"02803146dc74414191fab53b0de33fa1@BLUPR03MB019.namprd03.prod.outlook.com","threadId":"33224","inReplyTo":null,"subject":"FW: Windows. Git, and Dedupe","fromName":"Josh Rowe","fromEmail":"joshua.rowe@microsoft.com","sentAt":"2013-03-18T21:20:48Z","receivedAt":"2013-03-18T21:20:48Z","isPatch":false,"sender":{"key":"joshua.rowe@microsoft.com","avatar":null},"body":"Windows probably isn’t the most popular platform for Git developers ☺, but here goes…\n\nOn Windows with an NTFS volume with Deduplication enabled, Git believes that deduplicated files are symlinks.  It then fails to be able to do anything with the file.  This can be repro-ed by creating an NTFS volume with dedup, creating some duplicate files, verifying that a few files are deduped, and trying to add and commit the files via git.\n\nJmr\n\n"},{"id":"211709","messageId":"5148D3BD.4040608@lsrfire.ath.cx","threadId":"33224","inReplyTo":"02803146dc74414191fab53b0de33fa1@BLUPR03MB019.namprd03.prod.outlook.com","subject":"Re: FW: Windows. Git, and Dedupe","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2013-03-19T21:08:13Z","receivedAt":"2013-03-19T21:08:13Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.03.2013 22:20, schrieb Josh Rowe:\n> On Windows with an NTFS volume with Deduplication enabled, Git\n> believes that deduplicated files are symlinks.  It then fails to be\n> able to do anything with the file.  This can be repro-ed by creating\n> an NTFS volume with dedup, creating some duplicate files, verifying\n> that a few files are deduped, and trying to add and commit the files\n> via git.\n\nBoth Single Instance Storage[1] and Data Deduplication[2] (introduced\nwith Windows Server 2012) seem to be server-only features.  How about\nkeeping regular git repositories with checked-out files on client\ndisks and use the server only for bare repositories (without working\ntree)?\n\nWhen I tried to add a symbolic link created with mklink on Windows 8,\nthe mingw version of git refused because readlink(2) is not\nsupported.  This seems to be sufficient to reproduce the issue.\n\nI couldn't test the Cygwin version, though, because http://cygwin.com\ndoesn't respond at the moment.\n\nBut a working readlink(2) wouldn't help anyway, I guess.  I imagine\nthat the reparse points used for deduplication point into a magic\nblock store which performs garbage collection of content that is no\nlonger referenced -- which probably means that a recreated \"symlink\"\nmay point to blocks that have been deleted in the meantime.\n\nPerhaps you need a way to ask git to always follow symlinks instead\nof trying to store their target specification.\n\nRené\n\n\n[1] http://technet.microsoft.com/en-us/library/dd573308%28v=ws.10%29.aspx\n[2] http://msdn.microsoft.com/en-us/library/windows/desktop/hh769303%28v=vs.85%29.aspx\n"},{"id":"211711","messageId":"614e174a0b4548a1bcdc6c24c2682829@BLUPR03MB019.namprd03.prod.outlook.com","threadId":"33224","inReplyTo":"5148D3BD.4040608@lsrfire.ath.cx","subject":"RE: FW: Windows. Git, and Dedupe","fromName":"Josh Rowe","fromEmail":"joshua.rowe@microsoft.com","sentAt":"2013-03-19T21:36:12Z","receivedAt":"2013-03-19T21:36:12Z","isPatch":false,"sender":{"key":"joshua.rowe@microsoft.com","avatar":null},"body":"Yes, Dedup is in fact a Server-only feature.  However, there are lots of people using the Server SKU as development workstations (especially here at Microsoft <g>).  There are also some sysadmins that I know of who use git and download sysadmin scripts via git to Servers.  Finally, I would hazard a guess that it's possible to mount an NTFS filesystem containing deduped files from a Server machine onto a client SKU and access those files.  (I'm not on the NTFS team, and haven't tried it.)  So I think there are good reasons to support reparse points on Windows.  \n\nThe reparse point could be decoded as being a non-symlink reparse item using; in those cases, treating the file as an \"ordinary\" file would be appropriate.\n\nFor example, see the following.  The reparse tag value for symlinks is IO_REPARSE_TAG_SYMLINK (0xa000000c) and for deduped files is (IO_REPARSE_TAG_DEDUP) 0x80000013.  The value can be discovered from the information at [1].  \n\nI admin to not having looked at the git code nor being familiar with mingw.  Are native Win32 calls supported in the git codebase?\n\nJmr\n\n\n[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365740(v=vs.85).aspx\n\n\nPS I:\\temp> cmd /c mklink x y\nsymbolic link created for x <<===>> y\nPS I:\\temp> fsutil reparsepoint query x\nReparse Tag Value : 0xa000000c\nTag value: Microsoft\nTag value: Name Surrogate\nTag value: Symbolic Link\n\nReparse Data Length: 0x00000010\nReparse Data:\n0000:  02 00 02 00 00 00 02 00  01 00 00 00 79 00 79 00  ............y.y.\nPS I:\\temp> fsutil reparsepoint query x.txt\nReparse Tag Value : 0x80000013\nTag value: Microsoft\n\nReparse Data Length: 0x0000007c\nReparse Data:\n0000:  01 02 7c 00 00 00 00 00  66 9c 1a 01 00 00 00 00  ..|.....f.......\n0010:  00 00 01 00 00 00 00 00  cb eb c5 00 6a 97 63 4d  ............j.cM\n0020:  97 9c 13 0c 41 8e ed 8b  40 00 40 00 40 00 00 00  ....A...@.@.@...\n0030:  d3 b9 a8 d4 e4 c6 cd 01  55 ca 02 00 00 00 05 00  ........U.......\n0040:  70 ac 21 04 00 00 05 00  01 00 00 00 88 8d 00 00  p.!.............\n0050:  c8 30 00 00 00 00 00 00  c8 44 db 94 6c 88 9a d4  .0.......D..l...\n0060:  0a a9 01 3a 1f 80 80 8d  ea 0d 53 d7 36 49 b9 a4  ...:......S.6I..\n0070:  82 a2 b9 4e 2a 16 4b a1  2e d9 f3 dd              ...N*.K.....\n\n-----Original Message-----\nFrom: René Scharfe [mailto:rene.scharfe@lsrfire.ath.cx] \nSent: Tuesday, March 19, 2013 2:08 PM\nTo: Josh Rowe\nCc: git@vger.kernel.org; msysgit@googlegroups.com\nSubject: Re: FW: Windows. Git, and Dedupe\n\nAm 18.03.2013 22:20, schrieb Josh Rowe:\n> On Windows with an NTFS volume with Deduplication enabled, Git \n> believes that deduplicated files are symlinks.  It then fails to be \n> able to do anything with the file.  This can be repro-ed by creating \n> an NTFS volume with dedup, creating some duplicate files, verifying \n> that a few files are deduped, and trying to add and commit the files \n> via git.\n\nBoth Single Instance Storage[1] and Data Deduplication[2] (introduced with Windows Server 2012) seem to be server-only features.  How about keeping regular git repositories with checked-out files on client disks and use the server only for bare repositories (without working tree)?\n\nWhen I tried to add a symbolic link created with mklink on Windows 8, the mingw version of git refused because readlink(2) is not supported.  This seems to be sufficient to reproduce the issue.\n\nI couldn't test the Cygwin version, though, because http://cygwin.com doesn't respond at the moment.\n\nBut a working readlink(2) wouldn't help anyway, I guess.  I imagine that the reparse points used for deduplication point into a magic block store which performs garbage collection of content that is no longer referenced -- which probably means that a recreated \"symlink\"\nmay point to blocks that have been deleted in the meantime.\n\nPerhaps you need a way to ask git to always follow symlinks instead of trying to store their target specification.\n\nRené\n\n\n[1] http://technet.microsoft.com/en-us/library/dd573308%28v=ws.10%29.aspx\n[2] http://msdn.microsoft.com/en-us/library/windows/desktop/hh769303%28v=vs.85%29.aspx\n\n\n\n"},{"id":"211816","messageId":"514A140C.1040901@lsrfire.ath.cx","threadId":"33224","inReplyTo":"614e174a0b4548a1bcdc6c24c2682829@BLUPR03MB019.namprd03.prod.outlook.com","subject":"Re: FW: Windows. Git, and Dedupe","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2013-03-20T19:54:52Z","receivedAt":"2013-03-20T19:54:52Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 19.03.2013 22:36, schrieb Josh Rowe:\n> Yes, Dedup is in fact a Server-only feature.\n\nIs there an easier way to reproduce the issue than registering and \ndownloading the Windows Server 2012 evaluation version?  It's not that \nhard, admittedly, but still.\n\n> The reparse point could be decoded as being a non-symlink reparse\n> itemusing; in those cases, treating the file as an \"ordinary\"\n> file would be appropriate.\n>\n> For example, see the following. The reparse tag value for symlinks\n> isIO_REPARSE_TAG_SYMLINK (0xa000000c) and for deduped files is\n > (IO_REPARSE_TAG_DEDUP) 0x80000013.\n\nThat's interesting and invalidates my initial checks with mklink, \nbecause if I read compat/mingw.c [1] correctly then git handles symlinks \non Windows in a special way, but should treat dedup reparse points as \nnormal files already.\n\nHrm, but probably st_size is set to zero for them.  Do the deduped files \nappear as empty?  \"git ls-tree -r HEAD\" would show them with a hash of \ne69de29bb2d1d6434b8b29ae775ad8c2e48c5391.  If true then how do we get \ntheir real content sizes using Win32 API calls?\n\nBy the way, what does the command \"git version\" return for you?\n\nThanks,\nRené\n\n\n[1] https://git.kernel.org/cgit/git/git.git/tree/compat/mingw.c#n427\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/groups/opt_out.\n"},{"id":"211825","messageId":"2156b5b7b012490490c1b52637a6164a@BLUPR03MB019.namprd03.prod.outlook.com","threadId":"33224","inReplyTo":"514A140C.1040901@lsrfire.ath.cx","subject":"RE: FW: Windows. Git, and Dedupe","fromName":"Josh Rowe","fromEmail":"joshua.rowe@microsoft.com","sentAt":"2013-03-20T20:43:03Z","receivedAt":"2013-03-20T20:43:03Z","isPatch":false,"sender":{"key":"joshua.rowe@microsoft.com","avatar":null},"body":"If you have Win8 or HyperV 2012, I can ship you a small NTFS .vhd with some deduped files.  I'm not sure if that will be readable, but I would hazard a guess that it would be.  It definitely will not be readable on Win7.  \n\nI'm using:\n\nPS C:\\> git version\ngit version 1.8.0.msysgit.0\n\nI don't see any changes related to this in the file log since the original code was added in 2010.  I do notice that mingw_fstat doesn't do anything special with symlinks; I don't know where that is used.  \n\nThe file sizes show up as their original size with Windows tools (powershell, Win32, cmd, .Net, etc).  git ls-tree -r HEAD does not show that hash code for files that are not intentionally empty.  \n\nJmr\n\n\n-----Original Message-----\nFrom: René Scharfe [mailto:rene.scharfe@lsrfire.ath.cx] \nSent: Wednesday, March 20, 2013 12:55 PM\nTo: Josh Rowe\nCc: git@vger.kernel.org; msysgit@googlegroups.com\nSubject: Re: FW: Windows. Git, and Dedupe\n\nAm 19.03.2013 22:36, schrieb Josh Rowe:\n> Yes, Dedup is in fact a Server-only feature.\n\nIs there an easier way to reproduce the issue than registering and downloading the Windows Server 2012 evaluation version?  It's not that hard, admittedly, but still.\n\n> The reparse point could be decoded as being a non-symlink reparse \n> itemusing; in those cases, treating the file as an \"ordinary\"\n> file would be appropriate.\n>\n> For example, see the following. The reparse tag value for symlinks \n> isIO_REPARSE_TAG_SYMLINK (0xa000000c) and for deduped files is\n > (IO_REPARSE_TAG_DEDUP) 0x80000013.\n\nThat's interesting and invalidates my initial checks with mklink, because if I read compat/mingw.c [1] correctly then git handles symlinks on Windows in a special way, but should treat dedup reparse points as normal files already.\n\nHrm, but probably st_size is set to zero for them.  Do the deduped files appear as empty?  \"git ls-tree -r HEAD\" would show them with a hash of e69de29bb2d1d6434b8b29ae775ad8c2e48c5391.  If true then how do we get their real content sizes using Win32 API calls?\n\nBy the way, what does the command \"git version\" return for you?\n\nThanks,\nRené\n\n\n[1] https://git.kernel.org/cgit/git/git.git/tree/compat/mingw.c#n427\n\n\n\n"},{"id":"211830","messageId":"514A2DF0.2080708@lsrfire.ath.cx","threadId":"33224","inReplyTo":"2156b5b7b012490490c1b52637a6164a@BLUPR03MB019.namprd03.prod.outlook.com","subject":"Re: FW: Windows. Git, and Dedupe","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2013-03-20T21:45:20Z","receivedAt":"2013-03-20T21:45:20Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 20.03.2013 21:43, schrieb Josh Rowe:\n> If you have Win8 or HyperV 2012, I can ship you a small NTFS .vhd\n> with some deduped files.  I'm not sure if that will be readable, but\n> I would hazard a guess that it would be.  It definitely will not be\n> readable on Win7.\n\nIt would be nice if you could upload it to an FTP server or website and \npost a public link so that the real git-on-Windows developers can get it \nas well.  You can also send it to me personally and I'll see if I can \nmount it using Windows 8 and where I get from there.  In any case, \nplease make sure there's no sensitive or private data in the VHD file.\n\nHow big is it after compression using (preferably) 7-Zip or ZIP?\n\n> I'm using:\n>\n> PS C:\\> git version git version 1.8.0.msysgit.0\n\nThat means compat/mingw.c is directly relevant to you; more about MinGW, \nMSys and git at http://msysgit.github.com/ and http://mingw.org/.\n\n> The file sizes show up as their original size with Windows tools\n> (powershell, Win32, cmd, .Net, etc).  git ls-tree -r HEAD does not\n> show that hash code for files that are not intentionally empty.\n\nSo we can likely (hopefully) get the sizes of deduped files with the \nsame API calls as for regular ones.  Which makes me even more puzzled \nover the question of why git makes a difference between the two kinds.\n\nRené\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/groups/opt_out.\n"}]}