{"thread":{"id":"52394","subject":"Git as data archive","startedAt":"2019-12-06T18:54:11Z","lastAt":"2019-12-09T16:39:21Z","messageCount":6,"participants":["Andreas Kalz","Philip Oakley","Thomas Braun"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"387637","messageId":"21fcdc5f-955a-e027-0d71-15f476f6162c@gmx.de","threadId":"52394","inReplyTo":null,"subject":"Git as data archive","fromName":"Andreas Kalz","fromEmail":"andreas-kalz@gmx.de","sentAt":"2019-12-06T18:54:08Z","receivedAt":"2019-12-06T18:54:11Z","isPatch":false,"sender":{"key":"andreas-kalz@gmx.de","avatar":null},"body":"Hello,\nI am using git as archive and versioning also for photos. Apart from\nperformance issues, I wanted to ask if there are hard limits and\nconfigurable limits (how to configure?) for maximum single file size and\nmaximum .git archive size (Windows 64 Bit system)?\nThanks in advance for your answer.\nAll the best,\nAndreas\n"},{"id":"387678","messageId":"60945f9c-d79c-7806-0c5e-3e2affcea001@iee.email","threadId":"52394","inReplyTo":"21fcdc5f-955a-e027-0d71-15f476f6162c@gmx.de","subject":"Re: Git as data archive","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2019-12-07T16:54:37Z","receivedAt":"2019-12-07T16:54:40Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Andreas,\n\nOn 06/12/2019 18:54, Andreas Kalz wrote:\n> Hello,\n> I am using git as archive and versioning also for photos. Apart from\n> performance issues, I wanted to ask if there are hard limits and\n> configurable limits (how to configure?) for maximum single file size and\n> maximum .git archive size (Windows 64 Bit system)?\n> Thanks in advance for your answer.\n> All the best,\n> Andreas\n\nOn Git the file size is currently limited to size of `long`, rather than \n`size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB\n\nAny change will be a big change as it ripples through many places in the \ncode base and, for some, will feel 'wrong'. I did some work [1-4] on top \nof those of many others that was almost there, but...\n\nThe alternative is git-lfs, which I don't personally use (see [4]).\n\nPhilip\n\n[1] https://github.com/git-for-windows/git/pull/2179\n[2] https://github.com/gitgitgadget/git/pull/115\n[3] https://github.com/git-for-windows/git/issues/1063\n[4] https://github.com/git-lfs/git-lfs/issues/2434\n\n"},{"id":"387695","messageId":"1c81112d-5a1b-a791-b43f-d6a67d5b2e04@virtuell-zuhause.de","threadId":"52394","inReplyTo":"60945f9c-d79c-7806-0c5e-3e2affcea001@iee.email","subject":"Re: Git as data archive","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2019-12-07T18:04:45Z","receivedAt":"2019-12-07T18:04:54Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"On 07.12.2019 17:54, Philip Oakley wrote:\n> Hi Andreas,\n> \n> On 06/12/2019 18:54, Andreas Kalz wrote:\n>> Hello,\n>> I am using git as archive and versioning also for photos. Apart from\n>> performance issues, I wanted to ask if there are hard limits and\n>> configurable limits (how to configure?) for maximum single file size and\n>> maximum .git archive size (Windows 64 Bit system)?\n>> Thanks in advance for your answer.\n>> All the best,\n>> Andreas\n> \n> On Git the file size is currently limited to size of `long`, rather than\n> `size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB\n> \n> Any change will be a big change as it ripples through many places in the\n> code base and, for some, will feel 'wrong'. I did some work [1-4] on top\n> of those of many others that was almost there, but...\n\nAdding to what Philip said. On Windows the size of exported archives\n(git archive) is currently also limited to 4GB. The reason being also\nthe long vs size_t issue (which is not present on linux though).\n\nSo if you can switch to Linux or even MacOSX these issues are gone.\n\nThe number of files in .git, only the number packfiles would be of\ninterest here I guess, do not have the long vs size_t issue. So\npackfiles can be larger than 4GB on 64bit Windows (with 64bit git of\ncourse).\n\nBut depending on how large the biggest files are, it might be worth\ntweaking some of the settings, so that the created packfiles are\nreadable on all platforms. I once created a repo on linux which could\nnot be checked on windows, and that is a bit annoying.\n\nSo the questions are how large is each file? And what repository size do\nyou expect? Are we talking about 20MB files and 10GB repository? Or a\nfactor 100 more? And are you just adding files or are you modifying the\nadded files? Depending on the file sizes it might then also be\nbeneficial to tweak the delta compression settings and/or the big file\nthreshold limits.\n\nThomas\n\n> The alternative is git-lfs, which I don't personally use (see [4]).\n> \n> Philip\n> \n> [1] https://github.com/git-for-windows/git/pull/2179\n> [2] https://github.com/gitgitgadget/git/pull/115\n> [3] https://github.com/git-for-windows/git/issues/1063\n> [4] https://github.com/git-lfs/git-lfs/issues/2434\n> \n> \n\n"},{"id":"387720","messageId":"76f7d2f5-45fe-a43f-2121-954e1a497b4f@gmx.de","threadId":"52394","inReplyTo":"1c81112d-5a1b-a791-b43f-d6a67d5b2e04@virtuell-zuhause.de","subject":"Re: Git as data archive","fromName":"Andreas Kalz","fromEmail":"andreas-kalz@gmx.de","sentAt":"2019-12-08T18:44:30Z","receivedAt":"2019-12-08T18:44:37Z","isPatch":false,"sender":{"key":"andreas-kalz@gmx.de","avatar":null},"body":"Hi,\n\nthanks to you both.\n\n@Thomas: are you Thomas Braun who studied at FH Regensburg?\n\nWell, currently the .git repository is 715GB and the maximum file size\nis 9.5GB, but I did not get error messages due to that even if the\nperformance is quite low. The biggest pack* file is 24GB. There are some\nfiles which are modified, but most are not modified.\n\nMy question came up as I did not find a documentation about limits of\ngit, only a lot of entries about github and forum users who are\ndiscussing about old bugs of git. I read about git-lfs and also that it\nis not working very stable, due to that I did not use it yet.\n\nHow can the delta compression settings and/or the big filethreshold\nlimits be modified?\nThanks in advance.\n\nAll the best,\nAndreas\n\n\nAm 07.12.2019 um 19:04 schrieb Thomas Braun:\n> On 07.12.2019 17:54, Philip Oakley wrote:\n>> Hi Andreas,\n>>\n>> On 06/12/2019 18:54, Andreas Kalz wrote:\n>>> Hello,\n>>> I am using git as archive and versioning also for photos. Apart from\n>>> performance issues, I wanted to ask if there are hard limits and\n>>> configurable limits (how to configure?) for maximum single file size and\n>>> maximum .git archive size (Windows 64 Bit system)?\n>>> Thanks in advance for your answer.\n>>> All the best,\n>>> Andreas\n>> On Git the file size is currently limited to size of `long`, rather than\n>> `size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB\n>>\n>> Any change will be a big change as it ripples through many places in the\n>> code base and, for some, will feel 'wrong'. I did some work [1-4] on top\n>> of those of many others that was almost there, but...\n> Adding to what Philip said. On Windows the size of exported archives\n> (git archive) is currently also limited to 4GB. The reason being also\n> the long vs size_t issue (which is not present on linux though).\n>\n> So if you can switch to Linux or even MacOSX these issues are gone.\n>\n> The number of files in .git, only the number packfiles would be of\n> interest here I guess, do not have the long vs size_t issue. So\n> packfiles can be larger than 4GB on 64bit Windows (with 64bit git of\n> course).\n>\n> But depending on how large the biggest files are, it might be worth\n> tweaking some of the settings, so that the created packfiles are\n> readable on all platforms. I once created a repo on linux which could\n> not be checked on windows, and that is a bit annoying.\n>\n> So the questions are how large is each file? And what repository size do\n> you expect? Are we talking about 20MB files and 10GB repository? Or a\n> factor 100 more? And are you just adding files or are you modifying the\n> added files? Depending on the file sizes it might then also be\n> beneficial to tweak the delta compression settings and/or the big file\n> threshold limits.\n>\n> Thomas\n>\n>> The alternative is git-lfs, which I don't personally use (see [4]).\n>>\n>> Philip\n>>\n>> [1] https://github.com/git-for-windows/git/pull/2179\n>> [2] https://github.com/gitgitgadget/git/pull/115\n>> [3] https://github.com/git-for-windows/git/issues/1063\n>> [4] https://github.com/git-lfs/git-lfs/issues/2434\n>>\n>>\n\n"},{"id":"387724","messageId":"d4e34fa6-e92c-5178-c61e-e4b87dab7e09@virtuell-zuhause.de","threadId":"52394","inReplyTo":"76f7d2f5-45fe-a43f-2121-954e1a497b4f@gmx.de","subject":"Re: Git as data archive","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2019-12-09T01:18:37Z","receivedAt":"2019-12-09T01:18:43Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"On 08.12.2019 19:44, Andreas Kalz wrote:\n\nHi Andreas,\n\n> @Thomas: are you Thomas Braun who studied at FH Regensburg?\n\nnope, sorry.\n\n> Well, currently the .git repository is 715GB and the maximum file size\n> is 9.5GB, but I did not get error messages due to that even if the\n> performance is quite low. The biggest pack* file is 24GB. There are some\n> files which are modified, but most are not modified.\n\nOkay that is kind-of-large. How did you add the 9.5GB file? AFAIK this\ncould not have be done on windows.\n\nDo you push that to a remote repository as well?\n\n> My question came up as I did not find a documentation about limits of\n> git, only a lot of entries about github and forum users who are\n> discussing about old bugs of git. I read about git-lfs and also that it\n> is not working very stable, due to that I did not use it yet.\n\nAlthough I'm not using git-lfs myself, from what I know it works well.\nBut it does have the same limitation as stock git for windows as Philip\npointed out already.\n\n> How can the delta compression settings and/or the big filethreshold\n> limits be modified?\n\nThese are plain git config settings. Have a look at [1]. The attributes\nare explained in [2-3]. Basically you can set in .gitattributes\n\n*.bin -delta, -diff\n\nwhich would tell git that files with suffix bin should not be delta\ncompressed and are always binary.\n\nYou could also play around with turning compression completely off via\ncore.compression or pack.compression.\n\nHope that helps,\nThomas\n\nPS: If you have resources to help fixing that long-standing bug in git\nfor windows, there is a PR open [4] which has a WIP version. But beware\nyou need good C skills and better-than-average git skills, or a\nSanta-Claus-style bag with monetary resources.\n\n[1]:\nhttps://git-scm.com/docs/git-config#Documentation/git-config.txt-corebigFileThreshold\n[2]: https://git-scm.com/docs/gitattributes#_code_delta_code\n[3]: https://git-scm.com/docs/gitattributes#_marking_files_as_binary\n[4]: https://github.com/git-for-windows/git/pull/2179\n\n> Am 07.12.2019 um 19:04 schrieb Thomas Braun:\n>> On 07.12.2019 17:54, Philip Oakley wrote:\n>>> Hi Andreas,\n>>>\n>>> On 06/12/2019 18:54, Andreas Kalz wrote:\n>>>> Hello,\n>>>> I am using git as archive and versioning also for photos. Apart from\n>>>> performance issues, I wanted to ask if there are hard limits and\n>>>> configurable limits (how to configure?) for maximum single file size\n>>>> and\n>>>> maximum .git archive size (Windows 64 Bit system)?\n>>>> Thanks in advance for your answer.\n>>>> All the best,\n>>>> Andreas\n>>> On Git the file size is currently limited to size of `long`, rather than\n>>> `size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB\n>>>\n>>> Any change will be a big change as it ripples through many places in the\n>>> code base and, for some, will feel 'wrong'. I did some work [1-4] on top\n>>> of those of many others that was almost there, but...\n>> Adding to what Philip said. On Windows the size of exported archives\n>> (git archive) is currently also limited to 4GB. The reason being also\n>> the long vs size_t issue (which is not present on linux though).\n>>\n>> So if you can switch to Linux or even MacOSX these issues are gone.\n>>\n>> The number of files in .git, only the number packfiles would be of\n>> interest here I guess, do not have the long vs size_t issue. So\n>> packfiles can be larger than 4GB on 64bit Windows (with 64bit git of\n>> course).\n>>\n>> But depending on how large the biggest files are, it might be worth\n>> tweaking some of the settings, so that the created packfiles are\n>> readable on all platforms. I once created a repo on linux which could\n>> not be checked on windows, and that is a bit annoying.\n>>\n>> So the questions are how large is each file? And what repository size do\n>> you expect? Are we talking about 20MB files and 10GB repository? Or a\n>> factor 100 more? And are you just adding files or are you modifying the\n>> added files? Depending on the file sizes it might then also be\n>> beneficial to tweak the delta compression settings and/or the big file\n>> threshold limits.\n>>\n>> Thomas\n>>\n>>> The alternative is git-lfs, which I don't personally use (see [4]).\n>>>\n>>> Philip\n>>>\n>>> [1] https://github.com/git-for-windows/git/pull/2179\n>>> [2] https://github.com/gitgitgadget/git/pull/115\n>>> [3] https://github.com/git-for-windows/git/issues/1063\n>>> [4] https://github.com/git-lfs/git-lfs/issues/2434\n>>>\n>>>\n> \n> \n\n"},{"id":"387806","messageId":"fc38fbbe-e481-6a36-162e-34465f0a5bb0@gmx.de","threadId":"52394","inReplyTo":"d4e34fa6-e92c-5178-c61e-e4b87dab7e09@virtuell-zuhause.de","subject":"Re: Git as data archive","fromName":"Andreas Kalz","fromEmail":"andreas-kalz@gmx.de","sentAt":"2019-12-09T16:39:10Z","receivedAt":"2019-12-09T16:39:21Z","isPatch":false,"sender":{"key":"andreas-kalz@gmx.de","avatar":null},"body":"Hi Thomas,\nI committed it only on a local repository (git add . / git commit\n-m\"...\"). But I never tested to restore the big files from the archive\n:( and then I stepped over the bug description.\n\nNow I tried it out and something bad happened:\nE:\\bilder_git>git checkout -- Hochzeitsmesse.mp4\nerror: bad object header\nfatal: packed object 5c1403a85829c1c9e03bf04ac814d65bb72b617f (stored in\n.git/objects/pack/pack-00246783dc8e6b7365220e75563b5cecfa358e11.pack) is\ncorrupt\n\nDuring add / commit there was no problem, but now this is not a good\nthing...\n\nMy C-Skills are not bad - I worked about 10 years in embedded SW\ndevelopment. But, currently my time is limited as I have a 3 week old\nbaby child :)\n\nAll the best,\nAndreas\n\n\n\nAm 09.12.2019 um 02:18 schrieb Thomas Braun:\n> On 08.12.2019 19:44, Andreas Kalz wrote:\n>\n> Hi Andreas,\n>\n>> @Thomas: are you Thomas Braun who studied at FH Regensburg?\n> nope, sorry.\n>\n>> Well, currently the .git repository is 715GB and the maximum file size\n>> is 9.5GB, but I did not get error messages due to that even if the\n>> performance is quite low. The biggest pack* file is 24GB. There are some\n>> files which are modified, but most are not modified.\n> Okay that is kind-of-large. How did you add the 9.5GB file? AFAIK this\n> could not have be done on windows.\n>\n> Do you push that to a remote repository as well?\n>\n>> My question came up as I did not find a documentation about limits of\n>> git, only a lot of entries about github and forum users who are\n>> discussing about old bugs of git. I read about git-lfs and also that it\n>> is not working very stable, due to that I did not use it yet.\n> Although I'm not using git-lfs myself, from what I know it works well.\n> But it does have the same limitation as stock git for windows as Philip\n> pointed out already.\n>\n>> How can the delta compression settings and/or the big filethreshold\n>> limits be modified?\n> These are plain git config settings. Have a look at [1]. The attributes\n> are explained in [2-3]. Basically you can set in .gitattributes\n>\n> *.bin -delta, -diff\n>\n> which would tell git that files with suffix bin should not be delta\n> compressed and are always binary.\n>\n> You could also play around with turning compression completely off via\n> core.compression or pack.compression.\n>\n> Hope that helps,\n> Thomas\n>\n> PS: If you have resources to help fixing that long-standing bug in git\n> for windows, there is a PR open [4] which has a WIP version. But beware\n> you need good C skills and better-than-average git skills, or a\n> Santa-Claus-style bag with monetary resources.\n>\n> [1]:\n> https://git-scm.com/docs/git-config#Documentation/git-config.txt-corebigFileThreshold\n> [2]: https://git-scm.com/docs/gitattributes#_code_delta_code\n> [3]: https://git-scm.com/docs/gitattributes#_marking_files_as_binary\n> [4]: https://github.com/git-for-windows/git/pull/2179\n>\n>> Am 07.12.2019 um 19:04 schrieb Thomas Braun:\n>>> On 07.12.2019 17:54, Philip Oakley wrote:\n>>>> Hi Andreas,\n>>>>\n>>>> On 06/12/2019 18:54, Andreas Kalz wrote:\n>>>>> Hello,\n>>>>> I am using git as archive and versioning also for photos. Apart from\n>>>>> performance issues, I wanted to ask if there are hard limits and\n>>>>> configurable limits (how to configure?) for maximum single file size\n>>>>> and\n>>>>> maximum .git archive size (Windows 64 Bit system)?\n>>>>> Thanks in advance for your answer.\n>>>>> All the best,\n>>>>> Andreas\n>>>> On Git the file size is currently limited to size of `long`, rather than\n>>>> `size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB\n>>>>\n>>>> Any change will be a big change as it ripples through many places in the\n>>>> code base and, for some, will feel 'wrong'. I did some work [1-4] on top\n>>>> of those of many others that was almost there, but...\n>>> Adding to what Philip said. On Windows the size of exported archives\n>>> (git archive) is currently also limited to 4GB. The reason being also\n>>> the long vs size_t issue (which is not present on linux though).\n>>>\n>>> So if you can switch to Linux or even MacOSX these issues are gone.\n>>>\n>>> The number of files in .git, only the number packfiles would be of\n>>> interest here I guess, do not have the long vs size_t issue. So\n>>> packfiles can be larger than 4GB on 64bit Windows (with 64bit git of\n>>> course).\n>>>\n>>> But depending on how large the biggest files are, it might be worth\n>>> tweaking some of the settings, so that the created packfiles are\n>>> readable on all platforms. I once created a repo on linux which could\n>>> not be checked on windows, and that is a bit annoying.\n>>>\n>>> So the questions are how large is each file? And what repository size do\n>>> you expect? Are we talking about 20MB files and 10GB repository? Or a\n>>> factor 100 more? And are you just adding files or are you modifying the\n>>> added files? Depending on the file sizes it might then also be\n>>> beneficial to tweak the delta compression settings and/or the big file\n>>> threshold limits.\n>>>\n>>> Thomas\n>>>\n>>>> The alternative is git-lfs, which I don't personally use (see [4]).\n>>>>\n>>>> Philip\n>>>>\n>>>> [1] https://github.com/git-for-windows/git/pull/2179\n>>>> [2] https://github.com/gitgitgadget/git/pull/115\n>>>> [3] https://github.com/git-for-windows/git/issues/1063\n>>>> [4] https://github.com/git-lfs/git-lfs/issues/2434\n>>>>\n>>>>\n>>\n\n"}]}