{"thread":{"id":"49804","subject":"Migration to Git LFS inflates repository multiple times","startedAt":"2018-11-11T23:48:13Z","lastAt":"2018-11-13T20:24:47Z","messageCount":5,"participants":["Mateusz Loskot","Jeff King","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"362950","messageId":"CABUeae-Bk5hxEsLuDi=MytzN-kBG0D4psLbc_1=B+=70D=JKCQ@mail.gmail.com","threadId":"49804","inReplyTo":null,"subject":"Migration to Git LFS inflates repository multiple times","fromName":"Mateusz Loskot","fromEmail":"mateusz@loskot.net","sentAt":"2018-11-11T23:47:42Z","receivedAt":"2018-11-11T23:48:13Z","isPatch":false,"sender":{"key":"mateusz@loskot.net","avatar":"https://gravatar.com/avatar/fe5a0ce8995d4986f5c46aad6353e21717808da58e4bc710da1a100e247be239?d=mp&s=160"},"body":"Hi,\n\nI'm posting here for the first time and I hope it's the right place to ask\nquestions about Git LFS.\n\nTL;TR: Is this normal a repository migrated to Git LFS inflates multiple times\nand how to deal with it?\n\nI'm migrating a big SVN repository to Git.\nIn SVN, a collection of third-party SDKs is maintained along with codebase.\nMany of the third-party libraries come in binary form.\nSo, I'm migrating binary files of those to Git LFS.\n\nI'm following the Git LFS tutorial,\nsection \"Migrating existing repository data to LFS\"\nhttps://github.com/git-lfs/git-lfs/wiki/Tutorial\n\nFirst, I run initial translation of the SVN reoi into Git..\nThe new repository is a Git bare repository.\nThere are 5 branches and 10+ tags in the proj.git repo.\n\nIt is quite large:\n\nproj.git (BARE:master) $ du -sh\n19G\n\nNext, I performed the following sequence of steps to optimise it\nand migrate to Git LFS:\n\n1. Optimise the repo\n\nproj.git (BARE:master) $ git gc\nEnumerating objects: 1432599, done.\nCounting objects: 100% (1432599/1432599), done.\nDelta compression using up to 48 threads\nCompressing objects: 100% (864524/864524), done.\nWriting objects: 100% (1432599/1432599), done.\nTotal 1432599 (delta 541698), reused 1405922 (delta 525738)\nRemoving duplicate objects: 100% (256/256), done.\nChecking connectivity: 1432599, done.\n\nproj.git (BARE:master) $ du -sh\n11G\n\n2. List the file types taking up the most space in the repo\n\nproj.git (BARE:master) $ git lfs migrate info --everything\nmigrate: Sorting commits: ..., done\nmigrate: Examining commits: 100% (29412/29412), done\n*.lib   27 GB       3524/3524 files(s)  100%\n*.pdb   5.6 GB      1412/1412 files(s)  100%\n*.cpp   4.8 GB  131848/131854 files(s)  100%\n*.exe   2.3 GB        798/798 files(s)  100%\n*.dll   2.0 GB      1000/1000 files(s)  100%\n\n3. Migrate the repo to Git LFS\n\nproj.git (BARE:master) $ git lfs migrate import\n--include=\"*.exe,*.dll,*.lib,*.pdb,*.zip\" --everything\n\n4. Check size of the repo after migration to Git LFS\n\nproj.git (BARE:master) $ du -sh\n47G\n\n5. Cleaning up the `.git` directory after migration to Git LFS\n\nproj.git (BARE:master) $ git reflog expire --expire-unreachable=now --all\n\nproj.git (BARE:master) $ git gc --prune=now --aggressive\nEnumerating objects: 1462310, done.\nCounting objects: 100% (1462310/1462310), done.\nDelta compression using up to 48 threads\nCompressing objects: 100% (1422322/1422322), done.\nWriting objects: 100% (1462310/1462310), done.\nTotal 1462310 (delta 577640), reused 845097 (delta 0)\nRemoving duplicate objects: 100% (256/256), done.\nChecking connectivity: 1462310, done.\n\n6. Check final disk size of the repo\n\nproj.git (BARE:master) $ du -sh\n39G\n\n7. List the file types taking up the most space in the repository\nafter migration to Git LFS\n\nproj.git (BARE:master) $ git lfs migrate info --everything\nmigrate: Sorting commits: ..., done\nmigrate: Examining commits: 100% (29412/29412), done\n*.cpp   4.8 GB  131848/131854 files(s)  100%\n*.png   1.1 GB  696499/696499 files(s)  100%\n*.h     828 MB    86386/86471 files(s)  100%\n*.csv   820 MB        939/939 files(s)  100%\n*.html  686 MB    34126/34126 files(s)  100%\n\n\nNow, I'm looking for anaswers to the following questions:\n\n1. Is the procedure presented above correct to migrate (SVN ->) Git -> Git LFS?\n\n2. Given the initial translation to Git generated 19 GB repo\n(optimised to 11 GB)\nis this normal Git LFS migration inflates the repository\nto 47 GB (optimised ot 39 GB)?\n\n3. Why the inflation happens? Is this a function of number of branches?\n   How to understand the jump from 11 GB to 39 GB?\n\n4. How to optimise the repository to cut the size down further?\n\nMy next step is to somehow push the fat pig into GitHub, Bitbucket or\nAzure DevOps ;-)\n\nI've used Git for a few years, but I'm pretty newbie regarding low-level\nor administration tasks, so I might have made basic errors.\nI'll be thankful for any feedback.\n\nBest regards,\n-- \nMateusz Loskot, http://mateusz.loskot.net\n"},{"id":"363000","messageId":"20181112123058.GE3956@sigill.intra.peff.net","threadId":"49804","inReplyTo":"CABUeae-Bk5hxEsLuDi=MytzN-kBG0D4psLbc_1=B+=70D=JKCQ@mail.gmail.com","subject":"Re: Migration to Git LFS inflates repository multiple times","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T12:30:59Z","receivedAt":"2018-11-12T12:31:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 12:47:42AM +0100, Mateusz Loskot wrote:\n\n> Hi,\n> \n> I'm posting here for the first time and I hope it's the right place to ask\n> questions about Git LFS.\n> \n> TL;TR: Is this normal a repository migrated to Git LFS inflates multiple times\n> and how to deal with it?\n\nThat does sound odd to me. People with more LFS experience can probably\ngive you a better answers, but one thought occurred to me: does LFS\nstore backup copies of the original refs that it rewrites (similar to\nthe way filter-branch stores refs/original)?\n\nIf so, then the resulting repo has the new history _and_ the old\nhistory. Which might mean storing those large blobs both as Git objects\n(for the old history) and in an LFS cache directory (for the new\nhistory).\n\nAnd the right next step is probably to delete those backup refs, and\nthen \"git gc --prune=now\". Hmm, actually thinking about it, reflogs\ncould be making the old history reachable, too.\n\nTry looking at the output of \"git for-each-ref\" and seeing if there are\nany backup refs. After deleting them (or confirming that there aren't),\nprune the reflogs with:\n\n  git reflog expire --expire-unreachable=now --all\n\nand then \"git gc --prune=now\".\n\n-Peff\n"},{"id":"363002","messageId":"87k1li31si.fsf@evledraar.gmail.com","threadId":"49804","inReplyTo":"20181112123058.GE3956@sigill.intra.peff.net","subject":"Re: Migration to Git LFS inflates repository multiple times","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-12T12:42:21Z","receivedAt":"2018-11-12T12:42:27Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, Jeff King wrote:\n\n> On Mon, Nov 12, 2018 at 12:47:42AM +0100, Mateusz Loskot wrote:\n>\n>> Hi,\n>>\n>> I'm posting here for the first time and I hope it's the right place to ask\n>> questions about Git LFS.\n>>\n>> TL;TR: Is this normal a repository migrated to Git LFS inflates multiple times\n>> and how to deal with it?\n>\n> That does sound odd to me. People with more LFS experience can probably\n> give you a better answers, but one thought occurred to me: does LFS\n> store backup copies of the original refs that it rewrites (similar to\n> the way filter-branch stores refs/original)?\n>\n> If so, then the resulting repo has the new history _and_ the old\n> history. Which might mean storing those large blobs both as Git objects\n> (for the old history) and in an LFS cache directory (for the new\n> history).\n>\n> And the right next step is probably to delete those backup refs, and\n> then \"git gc --prune=now\". Hmm, actually thinking about it, reflogs\n> could be making the old history reachable, too.\n>\n> Try looking at the output of \"git for-each-ref\" and seeing if there are\n> any backup refs. After deleting them (or confirming that there aren't),\n> prune the reflogs with:\n>\n>   git reflog expire --expire-unreachable=now --all\n>\n> and then \"git gc --prune=now\".\n\nEven if it's only the most recent version of each file this could also\nbe explained by LFS storing each file inflated as-is on disk, whereas\ngit will store them delta-compressed.\n\nAccording to the initial E-Mail \"*.exe,*.dll,*.lib,*.pdb,*.zip\" was\nadded to LFS. Depending on the content of those they might be delta\ncompressing somewhat better than random data.\n"},{"id":"363109","messageId":"CABUeae895z=UCr35VVUkMYz9nA7osVGSSCnMOUvEZRChtvz_vw@mail.gmail.com","threadId":"49804","inReplyTo":"20181112123058.GE3956@sigill.intra.peff.net","subject":"Re: Migration to Git LFS inflates repository multiple times","fromName":"Mateusz Loskot","fromEmail":"mateusz@loskot.net","sentAt":"2018-11-13T00:39:51Z","receivedAt":"2018-11-13T00:40:23Z","isPatch":false,"sender":{"key":"mateusz@loskot.net","avatar":"https://gravatar.com/avatar/fe5a0ce8995d4986f5c46aad6353e21717808da58e4bc710da1a100e247be239?d=mp&s=160"},"body":"On Mon, 12 Nov 2018 at 13:31, Jeff King <peff@peff.net> wrote:\n> On Mon, Nov 12, 2018 at 12:47:42AM +0100, Mateusz Loskot wrote:\n> >\n> > TL;TR: Is this normal a repository migrated to Git LFS inflates multiple times\n> > and how to deal with it?\n>\n> That does sound odd to me. People with more LFS experience can probably\n> give you a better answers\n\nFYI, I forwarded my question to https://github.com/git-lfs/git-lfs/issues/3374\n\n> but one thought occurred to me: does LFS\n> store backup copies of the original refs that it rewrites (similar to\n> the way filter-branch stores refs/original)?\n\nI don't think I see any backup refs (see below for full list).\nBut, I may be misunderstanding what they are, how to look for them.\n\n> history. Which might mean storing those large blobs both as Git objects\n> (for the old history) and in an LFS cache directory (for the new\n> history).\n\nYes, it makes sense.\n\n> And the right next step is probably to delete those backup refs, and\n> then \"git gc --prune=now\". Hmm, actually thinking about it, reflogs\n> could be making the old history reachable, too.\n>\n> Try looking at the output of \"git for-each-ref\" and seeing if there are\n> any backup refs.\n\nI see. Here is the list (long!) of all I found:\n\nproj.git (BARE:master) $ git for-each-ref\nc718eadcf8d09d68c385f0a9355a2c871474fb43 commit refs/heads/1.0\ndaa75889053b70515179e334cbe3fe6fc7873ff3 commit refs/heads/1.1\ncb70db292c1f0c62170d05ffa8dad3c87a6f8ebd commit refs/heads/2.0\nf1597e80fcea16bec96dc43f7ab706616126305b commit refs/heads/3.0\n1d9e4813ae2fdc5c2b52f7115facda9059b009dc commit refs/heads/master\nf41edf37e9a4120bc5d5d66b29d110d403b8db9b commit\nrefs/svn/attic/tags/1.0.1006/6674\n850166b21f27447c6b503bb753c454ccedcea8ef commit refs/svn/attic/tags/1.0.216/1291\n8a24407f2df0ea7a401fbc08b387387538912642 commit refs/svn/attic/tags/1.0.252/1543\n771d81b0756d6ff7d73779ed79f49a607bffb80e commit refs/svn/attic/tags/1.0.299/1883\n10925d0fe0de090d4de109fd6403b86d014d6a21 commit refs/svn/attic/tags/1.0.342/2288\nca8dc7b243d002ac6b27f219a6172d36e7885ac1 commit refs/svn/attic/tags/1.0.391/2470\n79ebabed25d31dfa34ad68e58fa9327f71928df1 commit refs/svn/attic/tags/1.0.433/2657\nd3ed45804843c5aa153810b7494e2c7c0b842c82 commit refs/svn/attic/tags/1.0.450/2724\n088af5dbb225cb8dfbeafb5b63158234f4e4017d commit refs/svn/attic/tags/1.0.502/2967\n5da8598ed98a5a6610108d67849955da14f9d5b8 commit refs/svn/attic/tags/1.0.546/3212\nc3463397337f9f6e5f9d8e64cd79c013fd798bc8 commit refs/svn/attic/tags/1.0.615/3470\n673b2f93fda830cc8f28d436abead5fd54baa361 commit refs/svn/attic/tags/1.0.657/3704\n247cf24b90afd39f4f5dd7a27cf6b74483215285 commit refs/svn/attic/tags/1.0.662/3725\ne2a1609bb6b15ee767565ca0ff152eae3f72a76b commit refs/svn/attic/tags/1.0.673/3820\n48033fb9046b1ee60ad9b73073e9185c31ee4568 commit refs/svn/attic/tags/1.0.742/4325\nd7b566a275209d0aebba39c7a4028c9dcfb8a468 commit\nrefs/svn/attic/tags/1.1.1141/13525\n80f922becfc406420d2f14543e6da684f7377504 commit\nrefs/svn/attic/tags/1.1.1535/16534\n252481191cacdef0e77eb6ec02c98b07fca7bc77 commit\nrefs/svn/attic/tags/1.1.1582/16435\n601f072d559b664c101d89f3445ed3d00f4ef5dd commit\nrefs/svn/attic/tags/1.1.939/12077\n417fb23d71cab30e2c6218faed6f86021b67ca25 commit\nrefs/svn/attic/tags/2.0.1156/21143\n5539fbbe0078b782af02d46e0c1abc86ce3d5902 commit refs/svn/map\nc718eadcf8d09d68c385f0a9355a2c871474fb43 commit refs/svn/root/branches/1.0\ndaa75889053b70515179e334cbe3fe6fc7873ff3 commit refs/svn/root/branches/1.1\ncb70db292c1f0c62170d05ffa8dad3c87a6f8ebd commit refs/svn/root/branches/2.0\nf1597e80fcea16bec96dc43f7ab706616126305b commit refs/svn/root/branches/3.0\ne946e96ce2b37a771769196027ae87b8f24181e0 commit refs/svn/root/tags/1.0.1058\n06928f42664384bd5e24c115f9c23acc2fd949da commit refs/svn/root/tags/1.0.1240\n9f059337974aa195386c5f3ee21957551624aa27 commit refs/svn/root/tags/1.0.1653\ndb98d68f93d2e9127ab766d3bbe6c933ec169d29 commit refs/svn/root/tags/1.0.764\n20ad69c6b94bc8b73b613b5c21d367f22f423501 commit refs/svn/root/tags/1.1.1163\n458be535d7f0bc512e800759759e86a211a418b6 commit refs/svn/root/tags/1.1.1290\ndf045ab97bee94e8cfe72b70802b837719899587 commit refs/svn/root/tags/1.1.1556\ncd8ce83868016a0854c0f3cf1b23ea68a32674a2 commit refs/svn/root/tags/1.1.1706\nfcd0801b93f48bd46b276ccb82678a70f11fc3ca commit refs/svn/root/tags/1.1.1809\n5df0902cfe973b0a041409ec2e8d2314f2b8031e commit refs/svn/root/tags/1.1.2368\nc9a06b4f43bab77ed283fe2736ab5c865e03026e commit refs/svn/root/tags/1.1.2417\n32e505f8c4deadb73c63bd20a598481cd164541d commit refs/svn/root/tags/1.1.947\nef9c3667ec419bcb6d5eb5b9dbacb1cff0b1051e commit refs/svn/root/tags/2.0.1187\na1a6a5bedb8949eb91f3509929edb9efa9ad2875 commit refs/svn/root/tags/2.0.1198\n33a8f49da311caecdb5521759251bbcb78e3bff2 commit refs/svn/root/tags/2.0.1338\n63e59278131281858296b56f4ef5dd91c332941a commit refs/svn/root/tags/2.0.1481\nd23a1c662f772a7fc0d23a07794b57cfd9eff064 commit refs/svn/root/tags/2.0.1835\nc53e2cc4660a9e3121dff33c28c1383766fda39b commit refs/svn/root/tags/2.0.2148\nc7e0293ec09fee809fba707054cd1fd8fe492664 commit refs/svn/root/tags/2.0.2580\n1d9e4813ae2fdc5c2b52f7115facda9059b009dc commit refs/svn/root/trunk\ne946e96ce2b37a771769196027ae87b8f24181e0 commit refs/tags/1.0.1058\n06928f42664384bd5e24c115f9c23acc2fd949da commit refs/tags/1.0.1240\n9f059337974aa195386c5f3ee21957551624aa27 commit refs/tags/1.0.1653\ndb98d68f93d2e9127ab766d3bbe6c933ec169d29 commit refs/tags/1.0.764\n20ad69c6b94bc8b73b613b5c21d367f22f423501 commit refs/tags/1.1.1163\n458be535d7f0bc512e800759759e86a211a418b6 commit refs/tags/1.1.1290\ndf045ab97bee94e8cfe72b70802b837719899587 commit refs/tags/1.1.1556\ncd8ce83868016a0854c0f3cf1b23ea68a32674a2 commit refs/tags/1.1.1706\nfcd0801b93f48bd46b276ccb82678a70f11fc3ca commit refs/tags/1.1.1809\n5df0902cfe973b0a041409ec2e8d2314f2b8031e commit refs/tags/1.1.2368\nc9a06b4f43bab77ed283fe2736ab5c865e03026e commit refs/tags/1.1.2417\n32e505f8c4deadb73c63bd20a598481cd164541d commit refs/tags/1.1.947\nef9c3667ec419bcb6d5eb5b9dbacb1cff0b1051e commit refs/tags/2.0.1187\na1a6a5bedb8949eb91f3509929edb9efa9ad2875 commit refs/tags/2.0.1198\n33a8f49da311caecdb5521759251bbcb78e3bff2 commit refs/tags/2.0.1338\n63e59278131281858296b56f4ef5dd91c332941a commit refs/tags/2.0.1481\nd23a1c662f772a7fc0d23a07794b57cfd9eff064 commit refs/tags/2.0.1835\nc53e2cc4660a9e3121dff33c28c1383766fda39b commit refs/tags/2.0.2148\nc7e0293ec09fee809fba707054cd1fd8fe492664 commit refs/tags/2.0.2580\n\nAFAIS, there are only ones related to SVN to Git translation, and Git natives.\n\n> After deleting them (or confirming that there aren't),\n> prune the reflogs with:\n>\n>   git reflog expire --expire-unreachable=now --all\n>\n> and then \"git gc --prune=now\".\n\nAlthough there seem to be no backup refs in my repo, I decided to get\nrid of all refs/svn/*\n\nproj.git (BARE:master) $ du -sh\n39G     .\n\nproj.git (BARE:master) $ git for-each-ref --format=\"%(refname)\"\nrefs/svn/ | xargs -n 1 git update-ref -d\n\nproj.git (BARE:master) $ git for-each-ref\nc718eadcf8d09d68c385f0a9355a2c871474fb43 commit refs/heads/1.0\ndaa75889053b70515179e334cbe3fe6fc7873ff3 commit refs/heads/1.1\ncb70db292c1f0c62170d05ffa8dad3c87a6f8ebd commit refs/heads/2.0\nf1597e80fcea16bec96dc43f7ab706616126305b commit refs/heads/3.0\n1d9e4813ae2fdc5c2b52f7115facda9059b009dc commit refs/heads/master\ne946e96ce2b37a771769196027ae87b8f24181e0 commit refs/tags/1.0.1058\n06928f42664384bd5e24c115f9c23acc2fd949da commit refs/tags/1.0.1240\n9f059337974aa195386c5f3ee21957551624aa27 commit refs/tags/1.0.1653\ndb98d68f93d2e9127ab766d3bbe6c933ec169d29 commit refs/tags/1.0.764\n20ad69c6b94bc8b73b613b5c21d367f22f423501 commit refs/tags/1.1.1163\n458be535d7f0bc512e800759759e86a211a418b6 commit refs/tags/1.1.1290\ndf045ab97bee94e8cfe72b70802b837719899587 commit refs/tags/1.1.1556\ncd8ce83868016a0854c0f3cf1b23ea68a32674a2 commit refs/tags/1.1.1706\nfcd0801b93f48bd46b276ccb82678a70f11fc3ca commit refs/tags/1.1.1809\n5df0902cfe973b0a041409ec2e8d2314f2b8031e commit refs/tags/1.1.2368\nc9a06b4f43bab77ed283fe2736ab5c865e03026e commit refs/tags/1.1.2417\n32e505f8c4deadb73c63bd20a598481cd164541d commit refs/tags/1.1.947\nef9c3667ec419bcb6d5eb5b9dbacb1cff0b1051e commit refs/tags/2.0.1187\na1a6a5bedb8949eb91f3509929edb9efa9ad2875 commit refs/tags/2.0.1198\n33a8f49da311caecdb5521759251bbcb78e3bff2 commit refs/tags/2.0.1338\n63e59278131281858296b56f4ef5dd91c332941a commit refs/tags/2.0.1481\nd23a1c662f772a7fc0d23a07794b57cfd9eff064 commit refs/tags/2.0.1835\nc53e2cc4660a9e3121dff33c28c1383766fda39b commit refs/tags/2.0.2148\nc7e0293ec09fee809fba707054cd1fd8fe492664 commit refs/tags/2.0.2580\n\nproj.git (BARE:master) $ git reflog expire --expire-unreachable=now --all\n\nproj.git (BARE:master) $ git gc --prune=now\nEnumerating objects: 1315030, done.\nCounting objects: 100% (1315030/1315030), done.\nDelta compression using up to 48 threads\nCompressing objects: 100% (809937/809937), done.\nWriting objects: 100% (1315030/1315030), done.\nTotal 1315030 (delta 495565), reused 1314739 (delta 495312)\nChecking connectivity: 1315030, done.\n\nproj.git (BARE:master)\n$ du -sh\n38G     .\n\nThe size decrease, but not much.\n\nThanks for the ideas though.\n\nBest regards,\n-- \nMateusz Loskot, http://mateusz.loskot.net\n"},{"id":"363214","messageId":"CABUeae8pVeK+t2z1e5cvu65Ek7oEuo49M2tnovYvDEx7ruGt5Q@mail.gmail.com","threadId":"49804","inReplyTo":"CABUeae-Bk5hxEsLuDi=MytzN-kBG0D4psLbc_1=B+=70D=JKCQ@mail.gmail.com","subject":"Re: Migration to Git LFS inflates repository multiple times","fromName":"Mateusz Loskot","fromEmail":"mateusz@loskot.net","sentAt":"2018-11-13T20:24:15Z","receivedAt":"2018-11-13T20:24:47Z","isPatch":false,"sender":{"key":"mateusz@loskot.net","avatar":"https://gravatar.com/avatar/fe5a0ce8995d4986f5c46aad6353e21717808da58e4bc710da1a100e247be239?d=mp&s=160"},"body":"On Mon, 12 Nov 2018 at 00:47, Mateusz Loskot <mateusz@loskot.net> wrote:\n>\n> Hi,\n>\n> I'm posting here for the first time and I hope it's the right place to ask\n> questions about Git LFS.\n>\n> TL;TR: Is this normal a repository migrated to Git LFS inflates multiple times\n> and how to deal with it?\n\nFYI, answers to my questions have been completed via GitHub\nhttps://github.com/git-lfs/git-lfs/issues/3374\n\nI'd like to thank Jeff and Ævar here for help too.\n\nBest regards,\n-- \nMateusz Loskot, http://mateusz.loskot.net\n"}]}