{"thread":{"id":"13673","subject":"sync() slowdown","startedAt":"2008-05-26T14:26:07Z","lastAt":"2008-05-26T16:32:17Z","messageCount":4,"participants":["Sebastien Gross","Kenneth P. Turvey","Matthieu Moy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"77779","messageId":"20080526142607.GA3082@kali.thurne.chezwam.org","threadId":"13673","inReplyTo":null,"subject":"sync() slowdown","fromName":"Sebastien Gross","fromEmail":"seb-git@chezwam.org","sentAt":"2008-05-26T14:26:07Z","receivedAt":"2008-05-26T14:26:07Z","isPatch":false,"sender":{"key":"seb-git@chezwam.org","avatar":null},"body":"Hi git users,\n\nI use git (a very basic usage) every day and I noticed a big slowdown\nwhen I do a \"git repack -a -d\".\n\nI noticed that it only happens when I do backup to an usb stick.\n\nAfter a few investigation, I noticed that sync() is call when repacking\nobjects (from both builtin-prune.c and builtin-prune-packed.c).\n\nI do understand that syncing filesystem is usefull and needed.\n\nBut is there a good idea to add a --no-sync option to prevent that\nbehaviour ?\n\nI think this might be useful if you repack many repositories.\nIf you call the sync command before looping the repacks I guess this\ncould be equivalent (modulo changes done in repositories during that\ntime).\n\nAny idea suggestions ?\n\n\nThanks lot\n\nCheers.\n\n-- \nSébastien Gross\n"},{"id":"77781","messageId":"g1en6g$vuc$1@ger.gmane.org","threadId":"13673","inReplyTo":"20080526142607.GA3082@kali.thurne.chezwam.org","subject":"Re: sync() slowdown","fromName":"Kenneth P. Turvey","fromEmail":"kt-usenet@squeakydolphin.com","sentAt":"2008-05-26T16:06:41Z","receivedAt":"2008-05-26T16:06:41Z","isPatch":false,"sender":{"key":"kt-usenet@squeakydolphin.com","avatar":null},"body":"On Mon, 26 May 2008 16:26:07 +0200, Sebastien Gross wrote:\n\n> I do understand that syncing filesystem is usefull and needed.\n> \n> But is there a good idea to add a --no-sync option to prevent that\n> behaviour ?\n\nJust a user here, but I would prefer it if it didn't sync at all. If I \nwant to sync it, I will, or the operating system will handle it like it \ndoes with all other file accesses. \n\nJust my 2 cents. \n\nI was just editing my backup script the other day and part of the problem \nwith it was that I was syncing too often. What I needed was a single \nsync when everything was done. \n\nThis was possible because I was just doing copies and tars. It should be \npossible with git too. \n\n-- \nKenneth P. Turvey \nhttp://www.electricsenator.net\n\n There are two major products that come out of Berkeley: LSD and UNIX.\n We don't believe this to be a coincidence.\n -- Jeremy S. Anderson\n"},{"id":"77782","messageId":"vpq1w3p59ns.fsf@bauges.imag.fr","threadId":"13673","inReplyTo":"20080526142607.GA3082@kali.thurne.chezwam.org","subject":"Re: sync() slowdown","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2008-05-26T16:09:59Z","receivedAt":"2008-05-26T16:09:59Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Sebastien Gross writes:\n\n> I think this might be useful if you repack many repositories.\n> If you call the sync command before looping the repacks I guess this\n> could be equivalent (modulo changes done in repositories during that\n> time).\n\nI suppose git-repack does something like\n\nwrite_new_data();\nsync();\ndelete_old_data();\n\nAnd if you remove the \"sync\" and your system crashes (or you eject\nyour USB key, or ...) while \"delete_old_data\" is done, but\n\"write_new_data\" hasn't been sync-ed to the hard disk, you're in\ntrouble.\n\nIf you repack many repositories, I guess the first time is expansive,\nbut the next ones pay only for what they just did.\n\nMy 2 cents,\n\n-- \nMatthieu\n"},{"id":"77783","messageId":"20080526163217.GB3082@kali.thurne.chezwam.org","threadId":"13673","inReplyTo":"g1en6g$vuc$1@ger.gmane.org","subject":"Re: sync() slowdown","fromName":"Sebastien Gross","fromEmail":"seb-git@chezwam.org","sentAt":"2008-05-26T16:32:17Z","receivedAt":"2008-05-26T16:32:17Z","isPatch":false,"sender":{"key":"seb-git@chezwam.org","avatar":null},"body":"On Mon, May 26, 2008 at 04:06:41PM +0000, kt-usenet@squeakydolphin.com wrote:\n> On Mon, 26 May 2008 16:26:07 +0200, Sebastien Gross wrote:\n> \n> > I do understand that syncing filesystem is usefull and needed.\n> > \n> > But is there a good idea to add a --no-sync option to prevent that\n> > behaviour ?\n> \n> Just a user here, but I would prefer it if it didn't sync at all. If I \n> want to sync it, I will, or the operating system will handle it like it \n> does with all other file accesses. \n\nWell I guess I missed something in my explanation.\n\nI do my backup to an usb stick (somewhere like /media/usb0) and I work\nin git dirs (somewhere in /srv/git-repo). Obviously these 2 mount points\nare in different physical devices.\n\nIn a common run the system would sync cache and storage media when\nneeded.\nBut git (both prune and prune-packed command) call the sync() function\nbefore pruning objects and packs:\n\nbuiltin-prune-packed.c:\n\nint cmd_prune_packed(int argc, const char **argv, const char *prefix)\n...\n sync();\n prune_packed_objects(opts);\n return 0;\n}\n\nThe code is exactly the same in builtin-prune.c.\n\ncalling sync is a good way to be sure that no unsaved data remains in\nram and then everything would be included in the packs.\n\nThis must remain the default behaviour.\n\nBut in some case, sync() would also act on usb storage (which is my\ncase) and would be very slow.\n\nI do repack a lot of repositories something such as:\nfor d in *.git; do cd $d; git repack -a -d; cd ..; done\n\nIn the same time if I use the usb stick to do some backup on it, it\nwould change all the time then sync() would flush a changed cache for\neach call.\n\nThat's why I suggested to add a --no-sync option to bypass the sync()\ncall.\n\nIn any case this would be a dangerous option to not use unless you know\nwhat you are doing.\n\nCheers\n\n\n-- \nSébastien Gross\n"}]}