{"thread":{"id":"16107","subject":"Are binary xdeltas only used if you use git-gc?","startedAt":"2008-10-31T09:43:43Z","lastAt":"2008-11-04T03:17:56Z","messageCount":23,"participants":["Thanassis Tsiodras","Pierre Habouzit","Jakub Narebski","Matthieu Moy","Jean-Luc Herren","Nicolas Pitre","Pieter de Bie","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"94385","messageId":"f1d2d9ca0810310243r669840bbj2c5ee7183e0caaed@mail.gmail.com","threadId":"16107","inReplyTo":null,"subject":"Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-10-31T09:43:43Z","receivedAt":"2008-10-31T09:43:43Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"Hi everyone.\n\nI've been usig Git for the last couple of months and am quite happy with it.\nIn one of my Git repositories, I am storing uncompressed .tar files\n(since being uncompressed allows git to detect and store\nonly their \"real\"differences).\n\nHowever, when I introduce a new filename in the repos (with a minor\nset of differences compared to an existing file with a different filename)\nI've been unsuccessful in finding a way to tell Git to do it efficiently...\n\nThis is what I mean:\n\nbash$ mkdir -p /var/tmp/tst\nbash$ cd /var/tmp/tst\nbash$ git init\nbash$ cp /var/www/renderer-2.0e.tar .\nbash$ git add renderer-2.0e.tar\nbash$ git commit -m \"First version\"\nbash$ du -s -k .git/\n1724 .git/\nbash$ cp renderer-2.0e.tar renderer-2.0f.tar\nbash$ git add renderer-2.0f.tar\nbash$ git commit -m \"To add new version, first copy the first, so Git\ndetects it\"\nbash$ du -s -k .git/\n1740 .git/\nbash$ echo Good, Git detected it is the same\nbash$ cp /var/www/renderer-2.0f.tar .\nbash$ git add renderer-2.0f.tar\nbash$ git commit -m \"Real new version, slightly different to first\"\nbash$ du -s -k .git/\n3344 .git/\nbash$ echo What... did I do something wrong\nbash$ xdelta delta renderer-2.0e.tar renderer-2.0f.tar delta\nbash$ ls -l\ntotal 7788\n-rw-r--r-- 1 ttsiod ttsiod 8181 2008-10-31 11:27 delta\n-rw-r--r-- 1 ttsiod ttsiod 3962880 2008-10-31 11:23 renderer-2.0e.tar\n-rw-r--r-- 1 ttsiod ttsiod 3993600 2008-10-31 11:25 renderer-2.0f.tar\nbash$ git-gc\nbash$ du -s -k .git/\n1660 .git/\n\nSo even though the xdelta is just 8KB, and git-gc actually finds out\nthat indeed\nthe new file is very similar to the old one, the initial commit of the\nnew version\nin the repos is not taking advantage.\n\nI found out about this when I tried to \"git push\" over a PSTN modem...\n\nThen again, I must confess I only did the git-gc after I pushed.\nDoes the git-push actually take advantage of the similarities only if\nI do a git-gc first?\n\nIf that is the case, I will create an alias to always git-gc after commits...\n\n--\nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94390","messageId":"20081031110245.GA22633@artemis.corp","threadId":"16107","inReplyTo":"f1d2d9ca0810310243r669840bbj2c5ee7183e0caaed@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-10-31T11:02:45Z","receivedAt":"2008-10-31T11:02:45Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Fri, Oct 31, 2008 at 09:43:43AM +0000, Thanassis Tsiodras wrote:\n> So even though the xdelta is just 8KB, and git-gc actually finds out\n> that indeed\n> the new file is very similar to the old one, the initial commit of the\n> new version\n> in the repos is not taking advantage.\n\nHave you tried to git repack with aggressive options, like:\n\n git repack --window=500 --depth=500 \\\n --window-memory=\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"94392","messageId":"m37i7pggnk.fsf@localhost.localdomain","threadId":"16107","inReplyTo":"f1d2d9ca0810310243r669840bbj2c5ee7183e0caaed@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-10-31T11:15:32Z","receivedAt":"2008-10-31T11:15:32Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Thanassis Tsiodras\" writes:\n\n> I've been usig Git for the last couple of months and am quite happy with it.\n> In one of my Git repositories, I am storing uncompressed .tar files\n> (since being uncompressed allows git to detect and store\n> only their \"real\"differences).\n\nI think you can use clean / smudge filter in gitattributes for that.\n\n[...]\n\n> Then again, I must confess I only did the git-gc after I pushed.\n> Does the git-push actually take advantage of the similarities only if\n> I do a git-gc first?\n\nGit does deltification _only_ in packfiles. But when you push via SSH\ngit would generate a pack file with commits the other side doesn't\nhave, and those packs are thin packs, so they also have deltas... but\nthe remote side then adds bases to those thin packs making them\nstandalone: you would have to git-gc on remote.\n\nHTH\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"94393","messageId":"f1d2d9ca0810310416n1e9f3700if49951a05cae9a37@mail.gmail.com","threadId":"16107","inReplyTo":"20081031110245.GA22633@artemis.corp","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-10-31T11:16:05Z","receivedAt":"2008-10-31T11:16:05Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"Actually, after using git-gc, git-repack isn't really needed...\ngit-gc identifies that the two files are very similar and re-deltifies\n(see the du -s -k outputs in the original mail, after git-gc we have\nin fact lower usage than the first commit).\n\nMy question is basically...\n(a) why doesn't git detect this during commit and needs a git-gc\n(b) whether after git-gc I would have seen the massive difference\nduring a subsequent git-push or not\n\nThanassis.\n\n\n> Have you tried to git repack with aggressive options, like:\n>\n> git repack --window=500 --depth=500 \\\n> --window-memory=\n\n-- \nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94394","messageId":"f1d2d9ca0810310428o166dc075wbb43c00c1a555350@mail.gmail.com","threadId":"16107","inReplyTo":"m37i7pggnk.fsf@localhost.localdomain","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-10-31T11:28:44Z","receivedAt":"2008-10-31T11:28:44Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"On Fri, Oct 31, 2008 at 1:15 PM, Jakub Narebski wrote:\n\n> I think you can use clean / smudge filter in gitattributes for that.\n\nThanks, I didn't know about that. Will look into it\n\n> Git does deltification _only_ in packfiles. But when you push via SSH\n> git would generate a pack file with commits the other side doesn't\n> have, and those packs are thin packs, so they also have deltas... but\n> the remote side then adds bases to those thin packs making them\n> standalone: you would have to git-gc on remote.\n\nSo I have to git-gc on my side (after the commits), git-gc on the remote,\nand then git-push?\n\nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94401","messageId":"vpqy705rl5u.fsf@bauges.imag.fr","threadId":"16107","inReplyTo":"f1d2d9ca0810310243r669840bbj2c5ee7183e0caaed@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2008-10-31T12:42:05Z","receivedAt":"2008-10-31T12:42:05Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"\"Thanassis Tsiodras\" writes:\n\n> If that is the case, I will create an alias to always git-gc after commits...\n\nIf you have a decent version of git, it already does \"git gc --auto\"\nregularly. With --auto, git gc will do nothing if you don't have too\nmany unpacked objects, and will try to do the right thing otherwise\n(incremental packs, see man git gc). The idea is that \"git gc\" is a\ncostly operation, and git prefers to waste a bit of disk space to make\nmost \"commit\" really fast, and to take time to optimize the repository\nonly when it grew too much.\n\nIf you're worried about repository size and you have a permanently\nrunning machine, a good idea is to run git gc in a cron job, so that\nyou work fast in daytime, and your computer optimizes hard at night\ntime ;-) (I have gic gc + git fsck in a cron job, so I'll also know if\na repository gets corrupted).\n\n-- \nMatthieu\n"},{"id":"94404","messageId":"f1d2d9ca0810310722s4338a245wb4bb1c300d701332@mail.gmail.com","threadId":"16107","inReplyTo":"vpqy705rl5u.fsf@bauges.imag.fr","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-10-31T14:22:57Z","receivedAt":"2008-10-31T14:22:57Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"Actually, I am not so worried about disk size - I am far more worried\nabout how long it takes to git-push over my PSTN modem connection.\n\nI'll try Jakub's suggestion (git-gc on both my machine and the remote\nmachine hosting the repos) and report back.\n\nOn Fri, Oct 31, 2008 at 2:42 PM, Matthieu Moy wrote:\n> If you're worried about repository size and you have a permanently\n> running machine, a good idea is to run git gc in a cron job, so that\n> you work fast in daytime, and your computer optimizes hard at night\n> time ;-) (I have gic gc + git fsck in a cron job, so I'll also know if\n> a repository gets corrupted).\n>\n> --\n> Matthieu\n>\n\n\n\n-- \nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94414","messageId":"200810311726.57122.jnareb@gmail.com","threadId":"16107","inReplyTo":"f1d2d9ca0810310428o166dc075wbb43c00c1a555350@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-10-31T16:26:56Z","receivedAt":"2008-10-31T16:26:56Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Thanassis Tsiodras wrote:\n> On Fri, Oct 31, 2008 at 1:15 PM, Jakub Narebski wrote:\n \n> > Git does deltification _only_ in packfiles. But when you push via SSH\n> > git would generate a pack file with commits the other side doesn't\n> > have, and those packs are thin packs, so they also have deltas... but\n> > the remote side then adds bases to those thin packs making them\n> > standalone: you would have to git-gc on remote.\n> \n> So I have to git-gc on my side (after the commits), git-gc on the remote,\n> and then git-push?\n\nPerhaps I haven't made myself clear.\n\nOn the local side: git-commit creates loose (compressed, but not\ndeltified) objects. git-gc packs and deltifies.\n\nOn the remote side (for smart protocols, i.e. git and ssh): git\ncreates _thin_ pack, deltified; on the remote side git either makes\npack thick/self contained by adding base objects (object + deltas),\nor explodes pack into loose object (object). You need git-gc on\nremote server to fully deltify on remote side. But transfer is fully\ndeltified.\n\nOn the remote side (for dumb protocols, i.e. rsync and http): git\nfinds required packs and transfers them whole. So the situation is\nlike on local side, but git might transfer more than really needed\nbecause it transfers packs in full.\n\nHTH.\n-- \nJakub Narebski\nPoland\n"},{"id":"94417","messageId":"vpqej1wra1c.fsf@bauges.imag.fr","threadId":"16107","inReplyTo":"200810311726.57122.jnareb@gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2008-10-31T16:42:23Z","receivedAt":"2008-10-31T16:42:23Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Jakub Narebski writes:\n\n> Thanassis Tsiodras wrote:\n> \n>> So I have to git-gc on my side (after the commits), git-gc on the remote,\n>> and then git-push?\n>\n> Perhaps I haven't made myself clear.\n>\n> On the local side: git-commit creates loose (compressed, but not\n> deltified) objects. git-gc packs and deltifies.\n>\n> On the remote side (for smart protocols, i.e. git and ssh): git\n> creates _thin_ pack, deltified; \n\nI don't understand this point: the OP talks about pushing, so isn't\nthe pack created on the _local_ machine (and then sent to the remote)?\n\n-- \nMatthieu\n"},{"id":"94423","messageId":"490B3A56.3040601@gmx.ch","threadId":"16107","inReplyTo":"m37i7pggnk.fsf@localhost.localdomain","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Jean-Luc Herren","fromEmail":"jlh@gmx.ch","sentAt":"2008-10-31T17:03:18Z","receivedAt":"2008-10-31T17:03:18Z","isPatch":false,"sender":{"key":"jlh@gmx.ch","avatar":null},"body":"Jakub Narebski wrote:\n> \"Thanassis Tsiodras\" writes:\n>> Then again, I must confess I only did the git-gc after I pushed.\n>> Does the git-push actually take advantage of the similarities only if\n>> I do a git-gc first?\n> \n> Git does deltification _only_ in packfiles. But when you push via SSH\n> git would generate a pack file with commits the other side doesn't\n> have, and those packs are thin packs, so they also have deltas...\n\nAFAICT, git stopped pushing thin packs by default with 1.5.3.2, so\nyou have to explicitely ask for it. The original poster might not\nbe clear about this (or even know what a thin pack is).\n\nThanassis, try to use \"git push --thin\". 'man git-push' says:\n\n --thin, --no-thin\n These options are passed to git-send-pack. Thin transfer spends\n extra cycles to minimize the number of objects to be sent and meant\n to be used on slower connection.\n\nI did a quick test with big random files and it indeed only sends\nsmall deltas on small changes, but if you don't pass --thin, it\nwill send the full objects.\n\nI didn't find a configuration variable to change that default. It\nwould make sense for people that regularly push over slow lines.\n\nHope this helps,\njlh\n"},{"id":"94441","messageId":"alpine.LFD.2.00.0810311530290.13034@xanadu.home","threadId":"16107","inReplyTo":"20081031110245.GA22633@artemis.corp","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-10-31T19:31:05Z","receivedAt":"2008-10-31T19:31:05Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 31 Oct 2008, Pierre Habouzit wrote:\n\n> On Fri, Oct 31, 2008 at 09:43:43AM +0000, Thanassis Tsiodras wrote:\n> > So even though the xdelta is just 8KB, and git-gc actually finds out\n> > that indeed\n> > the new file is very similar to the old one, the initial commit of the\n> > new version\n> > in the repos is not taking advantage.\n> \n> Have you tried to git repack with aggressive options, like:\n> \n> git repack --window=500 --depth=500 \\\n> --window-memory=\n\nThat wouldn't bring any benefit in this case.\n\n\nNicolas\n"},{"id":"94443","messageId":"alpine.LFD.2.00.0810311531540.13034@xanadu.home","threadId":"16107","inReplyTo":"f1d2d9ca0810310416n1e9f3700if49951a05cae9a37@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-10-31T19:47:59Z","receivedAt":"2008-10-31T19:47:59Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 31 Oct 2008, Thanassis Tsiodras wrote:\n\n> Actually, after using git-gc, git-repack isn't really needed...\n> git-gc identifies that the two files are very similar and re-deltifies\n> (see the du -s -k outputs in the original mail, after git-gc we have\n> in fact lower usage than the first commit).\n\ngit-gc does call git-repack already. In fact, git-gc is only a \nconvenience wrapper for a couple maintenance commands.\n\n> My question is basically...\n> (a) why doesn't git detect this during commit and needs a git-gc\n\nBecause we want commit operations to be fast. One of many usage \nscenarios for git is to apply a large amount of patches in one go, \nmeaning many commits per second. The gc operation is potentially long, \ncan be done unfrequently and deferred any time, like when you don't have \nto wait for it.\n\n> (b) whether after git-gc I would have seen the massive difference\n> during a subsequent git-push or not\n\nNo. A push does create a pack with best size reduction already, whether \nor not your local or remote repositories are already packed. The only \nadvantage for having your local repository packed is in the time \nrequired to create that same pack to be pushed.\n\nAs mentioned already, you should consider the --thin switch if you are \npushing over a slow link. If the remote repository already has \nnecessary objects then all the pushed pack will contain is all deltas.\n\nThe reason why --thin is not activated by default is because most people \ndo pulls from a central server and only few people do pushes to such a \nserver, and while thin packs do reduce the transmission size they do \ncreate slightly bigger packs on the receiving end which is best avoided \non a busy server.\n\n\nNicolas\n"},{"id":"94445","messageId":"alpine.LFD.2.00.0810311549570.13034@xanadu.home","threadId":"16107","inReplyTo":"vpqej1wra1c.fsf@bauges.imag.fr","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-10-31T19:53:50Z","receivedAt":"2008-10-31T19:53:50Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 31 Oct 2008, Matthieu Moy wrote:\n\n> Jakub Narebski writes:\n> \n> > Thanassis Tsiodras wrote:\n> > \n> >> So I have to git-gc on my side (after the commits), git-gc on the remote,\n> >> and then git-push?\n> >\n> > Perhaps I haven't made myself clear.\n> >\n> > On the local side: git-commit creates loose (compressed, but not\n> > deltified) objects. git-gc packs and deltifies.\n> >\n> > On the remote side (for smart protocols, i.e. git and ssh): git\n> > creates _thin_ pack, deltified; \n> \n> I don't understand this point: the OP talks about pushing, so isn't\n> the pack created on the _local_ machine (and then sent to the remote)?\n\nYes, the pack is created on the fly when pushing, regardless if the repo \nis already packed or not locally. The only difference a locally packed \nrepo provides is a shorter \"Compressing objects\" phase when pushing \nthat's all. The packedness of the remote has no effect at all.\n\n\nNicolas\n"},{"id":"94554","messageId":"f1d2d9ca0811010454u203a7c88x1e09735b3fc1358f@mail.gmail.com","threadId":"16107","inReplyTo":"alpine.LFD.2.00.0810311549570.13034@xanadu.home","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-11-01T11:54:53Z","receivedAt":"2008-11-01T11:54:53Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"Thanks to everybody for your help.\n\nI will setup an alias to always use \"git push --thin\".\nFor the reverse direction, I don't see a --thin for \"git pull\",\n\nMy understanding is that \"git pull\" is optimal,\nand does what --thin does for push anyway, right?\n\nOn 10/31/08, Nicolas Pitre wrote:\n> On Fri, 31 Oct 2008, Matthieu Moy wrote:\n>\n>> Jakub Narebski writes:\n>>\n>> > Thanassis Tsiodras wrote:\n>> >\n>> >> So I have to git-gc on my side (after the commits), git-gc on the\n>> >> remote,\n>> >> and then git-push?\n>> >\n>> > Perhaps I haven't made myself clear.\n>> >\n>> > On the local side: git-commit creates loose (compressed, but not\n>> > deltified) objects. git-gc packs and deltifies.\n>> >\n>> > On the remote side (for smart protocols, i.e. git and ssh): git\n>> > creates _thin_ pack, deltified;\n>>\n>> I don't understand this point: the OP talks about pushing, so isn't\n>> the pack created on the _local_ machine (and then sent to the remote)?\n>\n> Yes, the pack is created on the fly when pushing, regardless if the repo\n> is already packed or not locally. The only difference a locally packed\n> repo provides is a shorter \"Compressing objects\" phase when pushing\n> that's all. The packedness of the remote has no effect at all.\n>\n>\n> Nicolas\n>\n\n\n-- \nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94557","messageId":"alpine.LFD.2.00.0811010924550.13034@xanadu.home","threadId":"16107","inReplyTo":"f1d2d9ca0811010454u203a7c88x1e09735b3fc1358f@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-11-01T13:25:17Z","receivedAt":"2008-11-01T13:25:17Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 1 Nov 2008, Thanassis Tsiodras wrote:\n\n> Thanks to everybody for your help.\n> \n> I will setup an alias to always use \"git push --thin\".\n> For the reverse direction, I don't see a --thin for \"git pull\",\n> \n> My understanding is that \"git pull\" is optimal,\n> and does what --thin does for push anyway, right?\n\nExact.\n\n\nNicolas\n"},{"id":"94805","messageId":"f1d2d9ca0811031235w3581f7ffnc7380b4cb488e71a@mail.gmail.com","threadId":"16107","inReplyTo":"alpine.LFD.2.00.0811010924550.13034@xanadu.home","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-11-03T20:35:07Z","receivedAt":"2008-11-03T20:35:07Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"Despair...\n\nI just tested \"git push --thin\"...\nDoesn't work.\n\nIt still sends the complete object, not a tiny pack as it could (should).\n\nBut perhaps I now understand why:\n\nI run git-gc on both the remote end and the working end (before\nchanging anything,\ni.e. with both repos being in sync - \"git pull\" and \"git push\" report all OK).\nI then noticed that on the remote side, .git/objects/pack had one big pack file,\nbut on the local one I have two .pack files...!\n\nI proceeded to try (many combinations of params on) git-repack in a vain attempt\nto make my local repos also have one single .pack file (presumably, it\nshould be able\nto exactly mirror the remote one, since it has the same objects inside\nit!). No way...\ngit-prune and \"git-fsck --full --strict --unreachable\" report no errors either.\n\nI'm at a loss as to why the two repos are having different \"pack\nrepresentation\" of the\nsame objects and why git-gc and git-repack fail to create a single\npack on my working\nside, but I'm guessing that this is why \"git push --thin\" fails to\nsend small xdeltas...\n\nAny help/advice on what to try next would be most welcome...\n\nThanassis.\n\n\n\n\n\nOn 11/1/08, Nicolas Pitre wrote:\n> On Sat, 1 Nov 2008, Thanassis Tsiodras wrote:\n>\n>> Thanks to everybody for your help.\n>>\n>> I will setup an alias to always use \"git push --thin\".\n>> For the reverse direction, I don't see a --thin for \"git pull\",\n>>\n>> My understanding is that \"git pull\" is optimal,\n>> and does what --thin does for push anyway, right?\n>\n> Exact.\n>\n>\n> Nicolas\n>\n\n\n-- \nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94811","messageId":"EC47BA70-B80D-4D27-A8C2-26C3E2C31382@ai.rug.nl","threadId":"16107","inReplyTo":"f1d2d9ca0811031235w3581f7ffnc7380b4cb488e71a@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Pieter de Bie","fromEmail":"pdebie@ai.rug.nl","sentAt":"2008-11-03T20:52:05Z","receivedAt":"2008-11-03T20:52:05Z","isPatch":false,"sender":{"key":"pdebie@ai.rug.nl","avatar":null},"body":"\nOn 3 nov 2008, at 21:35, Thanassis Tsiodras wrote:\n\n> Any help/advice on what to try next would be most welcome...\n\nPerhaps this is one of the cases that can be handled by the tellme- \nmore extension?\n\nhttp://repo.or.cz/w/git.git?a=commitdiff;h=5a9574c0100d287a4f2729dbaa64d057a5ee02e7\n\n- Pieter\n"},{"id":"94814","messageId":"alpine.LFD.2.00.0811031611060.13034@xanadu.home","threadId":"16107","inReplyTo":"f1d2d9ca0811031235w3581f7ffnc7380b4cb488e71a@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-11-03T21:42:12Z","receivedAt":"2008-11-03T21:42:12Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 3 Nov 2008, Thanassis Tsiodras wrote:\n\n> Despair...\n> \n> I just tested \"git push --thin\"...\n> Doesn't work.\n> \n> It still sends the complete object, not a tiny pack as it could (should).\n> \n> But perhaps I now understand why:\n> \n> I run git-gc on both the remote end and the working end (before\n> changing anything,\n> i.e. with both repos being in sync - \"git pull\" and \"git push\" report all OK).\n> I then noticed that on the remote side, .git/objects/pack had one big pack file,\n> but on the local one I have two .pack files...!\n> \n> I proceeded to try (many combinations of params on) git-repack in a vain attempt\n> to make my local repos also have one single .pack file (presumably, it\n> should be able\n> to exactly mirror the remote one, since it has the same objects inside\n> it!). No way...\n\nPlease stop thinking that your repository layout has anything to do with \nwhat is actually transferred on a push. It has not.\n\nHere's a small test that you can do locally:\n\n\tmkdir repo_a\n\tmkdir repo_b\n\tcd repo_a\n\tgit init\n\tseq 1000000 > data\n\tgit add data\n\tgit commit -m \"initial commit\"\n\tcd ../repo_b\n\tgit init\n\tcd ../repo_a\n\tgit push ../repo_b master:master\n\nHere you should see a line that says:\n\n\tWriting objects: 100% (3/3), 2.01 MiB, done.\n\nTherefore 2.1 MiB were transferred. Now let's continue:\n\n\techo \"foo\" >> data\n\tgit add data\n\tgit commit -m \"second commit\"\n\tgit push ../repo_b master:master\n\nYou should get:\n\n\tWriting objects: 100% (3/3), 423 bytes, done.\n\nAnd this means that you even don't need the --thin switch (which is \nwrong -- this has been broken before but that's another story) for the \ntransfer to actually send only the difference and not the whole file \nagain. And note that none of those repositoryes actually contain any \npack as everything is still loose objects.\n\n> I'm at a loss as to why the two repos are having different \"pack \n> representation\" of the same objects\n\nThat's only because those objects entered each repositories in a \ndifferent way.\n\n> and why git-gc and git-repack fail \n> to create a single pack on my working side,\n\nMaybe you have a .keep file in .git/objects/pack/ ? If so delete it and \nrun 'git repack -a -d'.\n\n> but I'm guessing that this is why \"git push --thin\" fails to send \n> small xdeltas...\n\nNot at all.\n\nPlease provide a complete log of your tests and maybe we could find \nsomething.\n\n\nNicolas\n"},{"id":"94822","messageId":"f1d2d9ca0811031453p49390911p956149ca76b9b80d@mail.gmail.com","threadId":"16107","inReplyTo":"alpine.LFD.2.00.0811031611060.13034@xanadu.home","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Thanassis Tsiodras","fromEmail":"ttsiodras@gmail.com","sentAt":"2008-11-03T22:53:42Z","receivedAt":"2008-11-03T22:53:42Z","isPatch":false,"sender":{"key":"ttsiodras@gmail.com","avatar":null},"body":"RESOLVED!!!\n\nFinally...\nWhat happened was actually quite reasonable, in hindsight...\nAs I said in the original mail, this was what I did:\n\ncp version7.1.tar version7.2.tar\ngit add version7.2.tar\ngit commit -m \"same data as old, so git will use old blob\"\necho MAGICPLACE read below...\ncp /path/to/work/realNewVersion7.2.tar version7.2.tar\ngit add version7.2.tar\ngit commit -m \"and now, commit the really new version, so git can xdelta\"\ngit push --thin\n\nThe problem was solved (that is, the \"git push\" became optimal,\nwhen I added a \"git push\" right after the MAGICPLACE mark above...\nIn that way, the remote repo learned about the \"dummy\" commit that\nreferenced the old blob... and when I did the subsequent \"git push\"\nat the end, the remote side could see that it already had this \"dummy\"\ncommit to \"xdelta on\", and that it only needed the delta...\n\nOriginally, when I used only one \"git push --thin\" at the end, the remote\nside didn't have the \"dummy\" commit, so it probably said: \"I can't\napply a delta, give me the full object\".\n\nPhew.\n\nSo it seems that if you must introduce a new file that is\nvery similar to an existing one (in my case, a new version\nof software kept in an uncompressed .tar file),\nyou have to do what I did above to allow for optimal \"git push\"es:\nthat is:\n\n1. Create the new filename by just copying the old\n (so the old blob is used)\n2. commit\n3. PUSH\n4. copy the real new file\n5. commit\n6. PUSH.\n\nIf you omit the middle PUSH in step 3, neither \"git push\", nor \"git push --thin\"\ncan realize that this new file can be \"incrementally built\" on the remote side\n(even though git-gc totally squashes it in the pack).\n\nThanks to all the people who responded, and especially Nicolas...\nMerci!\n\n\nOn 11/3/08, Nicolas Pitre wrote:\n> On Mon, 3 Nov 2008, Thanassis Tsiodras wrote:\n>\n>> Despair...\n>>\n>> I just tested \"git push --thin\"...\n>> Doesn't work.\n>>\n>> It still sends the complete object, not a tiny pack as it could (should).\n>>\n>> But perhaps I now understand why:\n>>\n>> I run git-gc on both the remote end and the working end (before\n>> changing anything,\n>> i.e. with both repos being in sync - \"git pull\" and \"git push\" report all\n>> OK).\n>> I then noticed that on the remote side, .git/objects/pack had one big pack\n>> file,\n>> but on the local one I have two .pack files...!\n>>\n>> I proceeded to try (many combinations of params on) git-repack in a vain\n>> attempt\n>> to make my local repos also have one single .pack file (presumably, it\n>> should be able\n>> to exactly mirror the remote one, since it has the same objects inside\n>> it!). No way...\n>\n> Please stop thinking that your repository layout has anything to do with\n> what is actually transferred on a push. It has not.\n>\n> Here's a small test that you can do locally:\n>\n> \tmkdir repo_a\n> \tmkdir repo_b\n> \tcd repo_a\n> \tgit init\n> \tseq 1000000 > data\n> \tgit add data\n> \tgit commit -m \"initial commit\"\n> \tcd ../repo_b\n> \tgit init\n> \tcd ../repo_a\n> \tgit push ../repo_b master:master\n>\n> Here you should see a line that says:\n>\n> \tWriting objects: 100% (3/3), 2.01 MiB, done.\n>\n> Therefore 2.1 MiB were transferred. Now let's continue:\n>\n> \techo \"foo\" >> data\n> \tgit add data\n> \tgit commit -m \"second commit\"\n> \tgit push ../repo_b master:master\n>\n> You should get:\n>\n> \tWriting objects: 100% (3/3), 423 bytes, done.\n>\n> And this means that you even don't need the --thin switch (which is\n> wrong -- this has been broken before but that's another story) for the\n> transfer to actually send only the difference and not the whole file\n> again. And note that none of those repositoryes actually contain any\n> pack as everything is still loose objects.\n>\n>> I'm at a loss as to why the two repos are having different \"pack\n>> representation\" of the same objects\n>\n> That's only because those objects entered each repositories in a\n> different way.\n>\n>> and why git-gc and git-repack fail\n>> to create a single pack on my working side,\n>\n> Maybe you have a .keep file in .git/objects/pack/ ? If so delete it and\n> run 'git repack -a -d'.\n>\n>> but I'm guessing that this is why \"git push --thin\" fails to send\n>> small xdeltas...\n>\n> Not at all.\n>\n> Please provide a complete log of your tests and maybe we could find\n> something.\n>\n>\n> Nicolas\n>\n\n\n-- \nWhat I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph\n"},{"id":"94836","messageId":"alpine.LFD.2.00.0811031959070.13034@xanadu.home","threadId":"16107","inReplyTo":"f1d2d9ca0811031453p49390911p956149ca76b9b80d@mail.gmail.com","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-11-04T01:18:37Z","receivedAt":"2008-11-04T01:18:37Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 4 Nov 2008, Thanassis Tsiodras wrote:\n\n> RESOLVED!!!\n> \n> Finally...\n> What happened was actually quite reasonable, in hindsight...\n> As I said in the original mail, this was what I did:\n> \n> cp version7.1.tar version7.2.tar\n> git add version7.2.tar\n> git commit -m \"same data as old, so git will use old blob\"\n> echo MAGICPLACE read below...\n> cp /path/to/work/realNewVersion7.2.tar version7.2.tar\n> git add version7.2.tar\n> git commit -m \"and now, commit the really new version, so git can xdelta\"\n> git push --thin\n> \n> The problem was solved (that is, the \"git push\" became optimal,\n> when I added a \"git push\" right after the MAGICPLACE mark above...\n> In that way, the remote repo learned about the \"dummy\" commit that\n> referenced the old blob... and when I did the subsequent \"git push\"\n> at the end, the remote side could see that it already had this \"dummy\"\n> commit to \"xdelta on\", and that it only needed the delta...\n> \n> Originally, when I used only one \"git push --thin\" at the end, the remote\n> side didn't have the \"dummy\" commit, so it probably said: \"I can't\n> apply a delta, give me the full object\".\n\nOh! But of course...\n\nIn fact, the way thin packs work is to store delta against a base object \nwhich is not included in the pack. Those objects which are not included \nbut used as delta base are currently only the previous version of a file \nwhich is part of the update to be pushed/fetched. In other words, there \nmust be a previous version under the same name for this to work. Doing \notherwise wouldn't scale if the previous commit had thousands of files \nto test against.\n\nBut this particularity had escaped my mind somehow.\n\n> Phew.\n> \n> So it seems that if you must introduce a new file that is\n> very similar to an existing one (in my case, a new version\n> of software kept in an uncompressed .tar file),\n> you have to do what I did above to allow for optimal \"git push\"es:\n> that is:\n> \n> 1. Create the new filename by just copying the old\n> (so the old blob is used)\n> 2. commit\n> 3. PUSH\n> 4. copy the real new file\n> 5. commit\n> 6. PUSH.\n> \n> If you omit the middle PUSH in step 3, neither \"git push\", nor \"git push --thin\"\n> can realize that this new file can be \"incrementally built\" on the remote side\n> (even though git-gc totally squashes it in the pack).\n\nRight. Those thin packs were designed for different versions of the \nsame file in mind, not different files with almost the same content. \nThis could possibly be improved at some point...\n\n\nNicolas\n"},{"id":"94837","messageId":"7v3ai8tgq9.fsf@gitster.siamese.dyndns.org","threadId":"16107","inReplyTo":"alpine.LFD.2.00.0811031959070.13034@xanadu.home","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-11-04T01:36:14Z","receivedAt":"2008-11-04T01:36:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> Right. Those thin packs were designed for different versions of the \n> same file in mind, not different files with almost the same content. \n> This could possibly be improved at some point...\n\nWouldn't using a large --window help by going across name-hash boundaries?\n"},{"id":"94839","messageId":"alpine.LFD.2.00.0811032053410.13034@xanadu.home","threadId":"16107","inReplyTo":"7v3ai8tgq9.fsf@gitster.siamese.dyndns.org","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-11-04T01:57:04Z","receivedAt":"2008-11-04T01:57:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 3 Nov 2008, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > Right. Those thin packs were designed for different versions of the \n> > same file in mind, not different files with almost the same content. \n> > This could possibly be improved at some point...\n> \n> Wouldn't using a large --window help by going across name-hash boundaries?\n\nThe issue is to decide what preferred delta base to add to the list of \nobjects. Currently only objects with the same path as those being \nmodified are considered.\n\n\nNicolas\n"},{"id":"94840","messageId":"7vtzaorxgb.fsf@gitster.siamese.dyndns.org","threadId":"16107","inReplyTo":"alpine.LFD.2.00.0811032053410.13034@xanadu.home","subject":"Re: Are binary xdeltas only used if you use git-gc?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-11-04T03:17:56Z","receivedAt":"2008-11-04T03:17:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> The issue is to decide what preferred delta base to add to the list of \n> objects. Currently only objects with the same path as those being \n> modified are considered.\n\nAh, I was blind (even though that part is my code). Thanks.\n"}]}