{"thread":{"id":"679","subject":"checkout-cache -f: a better way?","startedAt":"2005-05-20T21:05:06Z","lastAt":"2005-05-21T01:45:53Z","messageCount":11,"participants":["Jeff Garzik","Junio C Hamano","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"3647","messageId":"428E5102.60003@pobox.com","threadId":"679","inReplyTo":null,"subject":"checkout-cache -f: a better way?","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-05-20T21:05:06Z","receivedAt":"2005-05-20T21:05:06Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"\nBeing a weirdo, I don't use cogito for kernel development, just git \nitself.  I store branches in .git/refs/heads/ per the defacto standard, \nand use the attached script to switch the working directory from one \nbranch to another.\n\nProblem is, 'git-checkout-cache -q -f -a' really pounds the disk, and \ntakes quite a while.\n\nIs there any way to avoid -f, while ensuring that the working directory \ntruly represents the new branch?\n\nBitKeeper has a secret checkout arg '-S', which will leave files \nuntouched if the mtime/size information is unchanged.\n\n\tJeff\n\n\n\n\n\n#!/bin/sh\n\nif [ \"x$1\" != \"x\" ]\nthen\n\tif [ \"$1\" == \"master\" ]\n\tthen\n\t\t( cd .git && rm -f HEAD && ln -s refs/heads/master HEAD )\n\telse\n\t\tif [ ! -f .git/refs/heads/$1 ]\n\t\tthen\n\t\t\techo Branch $1 not found.\n\t\t\texit 1\n\t\tfi\n\n\t\t( cd .git && rm -f HEAD && ln -s refs/heads/$1 HEAD )\n\tfi\nfi\n\ngit-read-tree $(cat .git/HEAD) && \\\n\tgit-checkout-cache -q -f -a && \\\n\tgit-update-cache --refresh\n\n"},{"id":"3652","messageId":"7vacmpsetb.fsf@assigned-by-dhcp.cox.net","threadId":"679","inReplyTo":"428E5102.60003@pobox.com","subject":"Re: checkout-cache -f: a better way?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-05-20T22:38:56Z","receivedAt":"2005-05-20T22:38:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":">>>>> \"JG\" == Jeff Garzik <jgarzik@pobox.com> writes:\n\nJG> Being a weirdo, I don't use cogito for kernel development, just git\nJG> itself.\n\nMy customer, in other words ;-).\n\nJG> git-read-tree $(cat .git/HEAD) && \\\nJG> \tgit-checkout-cache -q -f -a && \\\nJG> \tgit-update-cache --refresh\n\nI have to check checkout-cache.c, but assuming that you start\nfrom an already populated work tree with a valid cache when you\ndo the git-read-tree at the third line from the last, using\n\"git-read-tree -m HEAD\" (you do not need to say $(cat .git/HEAD)\nin the modern git anymore) would be a good place to start.\n\nAlso the modern git-checkout-cache has a '-u' option and with it\nyou should not need 'git-update-cache --refresh' after that.\n\nLet me know if you have any problems.  Single tree '-m' is what\nLinus did and '-u' option to git-checkout-cache is mine.\n\n\n\n"},{"id":"3655","messageId":"428E73B9.1080907@pobox.com","threadId":"679","inReplyTo":"7vacmpsetb.fsf@assigned-by-dhcp.cox.net","subject":"Re: checkout-cache -f: a better way?","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-05-20T23:33:13Z","receivedAt":"2005-05-20T23:33:13Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Junio C Hamano wrote:\n>>>>>>\"JG\" == Jeff Garzik <jgarzik@pobox.com> writes:\n> \n> \n> JG> Being a weirdo, I don't use cogito for kernel development, just git\n> JG> itself.\n> \n> My customer, in other words ;-).\n> \n> JG> git-read-tree $(cat .git/HEAD) && \\\n> JG> \tgit-checkout-cache -q -f -a && \\\n> JG> \tgit-update-cache --refresh\n> \n> I have to check checkout-cache.c, but assuming that you start\n> from an already populated work tree with a valid cache when you\n> do the git-read-tree at the third line from the last, using\n> \"git-read-tree -m HEAD\" (you do not need to say $(cat .git/HEAD)\n> in the modern git anymore) would be a good place to start.\n> \n> Also the modern git-checkout-cache has a '-u' option and with it\n> you should not need 'git-update-cache --refresh' after that.\n> \n> Let me know if you have any problems.  Single tree '-m' is what\n> Linus did and '-u' option to git-checkout-cache is mine.\n\nPardon my ignorance (I'm slow :)), but how do those changes address the \nfact that git-checkout-cache appears to checkout the entire kernel tree \n(over 100MB of writes) when using '-f' ?\n\ngit-checkout-cache -f writes out every file, even if it exists, correct?\n\n\tJeff\n\n\n"},{"id":"3654","messageId":"Pine.LNX.4.58.0505201626560.2206@ppc970.osdl.org","threadId":"679","inReplyTo":"428E5102.60003@pobox.com","subject":"Re: checkout-cache -f: a better way?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-05-20T23:33:14Z","receivedAt":"2005-05-20T23:33:14Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 20 May 2005, Jeff Garzik wrote:\n> \n> Problem is, 'git-checkout-cache -q -f -a' really pounds the disk, and \n> takes quite a while.\n\nNo. \"git\" is perfect, and \"git-checkout-cache -f\" already does exactly \nwhat you want.\n\n> Is there any way to avoid -f, while ensuring that the working directory \n> truly represents the new branch?\n\nYou don't need to avoid -f, it already has the logic to avoid writing \nfiles that are already up-to-date.\n\nHOWEVER, your script is broken:\n\n\tgit-read-tree $(cat .git/HEAD) && \\\n\t        git-checkout-cache -q -f -a && \\\n\t        git-update-cache --refresh\n\nyou need to use the \"-m\" switch to git-read-tree to tell it to merge the \nindex information from your previous tree with the new one.\n\nAlso, don't do the \"$(cat .git/HEAD)\" thing any more, since modern git \ndoes this so much more nicely, and allows you to use your branch names \ndirectly.\n\nFinally, use the new \"-u\" flag to git-checkout-cache, which will update \nthe cache as it goes along. \n\nIn other words, those lines in your script should look like this:\n\n\tgit-read-tree -m HEAD && git-checkout-cache -q -f -u -a\n\nand you'll be a lot happier.\n\n\t\t\tLinus\n"},{"id":"3656","messageId":"7vvf5dqxfq.fsf@assigned-by-dhcp.cox.net","threadId":"679","inReplyTo":"428E73B9.1080907@pobox.com","subject":"Re: checkout-cache -f: a better way?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-05-20T23:39:37Z","receivedAt":"2005-05-20T23:39:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":">>>>> \"JG\" == Jeff Garzik <jgarzik@pobox.com> writes:\n\nJG> git-checkout-cache -f writes out every file, even if it exists, correct?\n\nNo, that's not correct.  To translate my prose, you would want\nthis:\n\n    git-read-tree -m HEAD && git-checkout-cache -q -f -u -a\n\n(notice that I do not have git-update-cache --refresh after\nthat).\n\n\n\n\n"},{"id":"3657","messageId":"Pine.LNX.4.58.0505201641160.2206@ppc970.osdl.org","threadId":"679","inReplyTo":"Pine.LNX.4.58.0505201626560.2206@ppc970.osdl.org","subject":"Re: checkout-cache -f: a better way?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-05-20T23:51:00Z","receivedAt":"2005-05-20T23:51:00Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 20 May 2005, Linus Torvalds wrote:\n> \n> In other words, those lines in your script should look like this:\n> \n> \tgit-read-tree -m HEAD && git-checkout-cache -q -f -u -a\n> \n> and you'll be a lot happier.\n\nBtw, I do realize that I'm a total wiener, and that my inability to use \n\"getopt_long()\" is shameful and stupid. \n\nWhat can I say? I'm easily confused, and besides, I really seldom program \nin user mode.\n\nSo if somebody were to getopt'ify git, _without_ adding crapola like\nautoconf (which probably implies that git would just require GNU getopt),\nand others agree that it's ok to just say that we expect getopt_long() to\nexist, then I'd not have any objections to making the above just be\n\n\tgit-read-tree -m HEAD | git-checkout-cache -fqua\n\n(to which the beavis-and-butthead in me says \"hehhehhehh.. He said fqua.  \nHehhehh. fire fire fire.\")\n\n\t\tLinus\n"},{"id":"3658","messageId":"428E78E0.4010503@pobox.com","threadId":"679","inReplyTo":"Pine.LNX.4.58.0505201641160.2206@ppc970.osdl.org","subject":"Re: checkout-cache -f: a better way?","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-05-20T23:55:12Z","receivedAt":"2005-05-20T23:55:12Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> On Fri, 20 May 2005, Linus Torvalds wrote:\n> \n>>In other words, those lines in your script should look like this:\n>>\n>>\tgit-read-tree -m HEAD && git-checkout-cache -q -f -u -a\n>>\n>>and you'll be a lot happier.\n> \n> \n> Btw, I do realize that I'm a total wiener, and that my inability to use \n> \"getopt_long()\" is shameful and stupid. \n\ninfo libc argp :)  argp is a lot more flexible, but with the same basic \nstructure as getopt_long().\n\nIf you pick a random git program, I would be willing to convert it as an \nexample.  I attached my implementation of ipcrm[1] as an example.\n\n\tJeff\n\n\n[1] from 'posixutils', my project to implement all the POSIX command \nline utilities.  Yes, I'm crazy too.\n\n\n/*\n * Copyright 2004-2005 Jeff Garzik <jgarzik@pobox.com>\n *\n * This program is free software; you can redistribute it and/or modify\n * it under the terms of the GNU General Public License as published by\n * the Free Software Foundation.\n *\n * This program is distributed in the hope that it will be useful,\n * but WITHOUT ANY WARRANTY; without even the implied warranty of\n * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n * GNU General Public License for more details.\n *\n * You should have received a copy of the GNU General Public License\n * along with this program; see the file COPYING.  If not, write to\n * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.\n *\n */\n\n\n#ifndef HAVE_CONFIG_H\n#error missing autoconf-generated config.h.\n#endif\n#include \"posixutils-config.h\"\n\n#include <sys/types.h>\n#include <stdlib.h>\n#include <string.h>\n#include <sys/ipc.h>\n#include <sys/msg.h>\n#include <sys/sem.h>\n#include <sys/shm.h>\n#include <argp.h>\n#include <libpu.h>\n\n\nstatic const char doc[] =\nN_(\"ipcrm - remove a message queue, semaphore set or shared memory id\");\n\nstatic struct argp_option options[] = {\n\t{ NULL, 'q', \"msgid\", 0,\n\t  N_(\"Remove message queue identifier msgid from system\") },\n\t{ NULL, 'm', \"shmid\", 0,\n\t  N_(\"Remove shared memory identifier shmid from system\") },\n\t{ NULL, 's', \"semid\", 0,\n\t  N_(\"Remove semaphore identifier semid from system\") },\n\t{ NULL, 'Q', \"msgkey\", 0,\n\t  N_(\"Remove message queue identifier, created with key msgkey, from system\") },\n\t{ NULL, 'M', \"shmkey\", 0,\n\t  N_(\"Remove shared memory identifier, created with key shmkey, from system\") },\n\t{ NULL, 'S', \"semkey\", 0,\n\t  N_(\"Remove semaphore identifier, created with key semkey, from system\") },\n\t{ }\n};\n\nstatic error_t parse_opt (int key, char *arg, struct argp_state *state);\nstatic const struct argp argp = { options, parse_opt, NULL, doc };\n\nenum parse_options_bits {\n\tOPT_MSG\t\t\t= (1 << 0),\n\tOPT_SHM\t\t\t= (1 << 1),\n\tOPT_SEM\t\t\t= (1 << 2),\n\tOPT_KEY\t\t\t= (1 << 3),\n};\n\nstruct arglist {\n\tstruct arglist\t\t*next;\n\tint\t\t\tmask;\n\tunsigned long\t\targ;\n};\n\n#ifdef _SEM_SEMUN_UNDEFINED\n   union semun\n   {\n     int val;\n     struct semid_ds *buf;\n     unsigned short int *array;\n     struct seminfo *__buf;\n   };\n#endif\n\nstatic int exit_status = EXIT_SUCCESS;\nstatic struct arglist *arglist;\n\n\nstatic const char *arg_name(int mask)\n{\n\tif (mask & OPT_MSG) return \"msg\";\n\tif (mask & OPT_SHM) return \"shm\";\n\tif (mask & OPT_SEM) return \"sem\";\n\treturn NULL;\n}\n\nstatic void push_opt(int mask, unsigned long arg)\n{\n\tstruct arglist *tmp, *node = xcalloc(1, sizeof(struct arglist));\n\n\tnode->mask = mask;\n\tnode->arg = arg;\n\n\ttmp = arglist;\n\tif (!tmp) {\n\t\targlist = node;\n\t} else {\n\t\twhile (tmp->next)\n\t\t\ttmp = tmp->next;\n\t\ttmp->next = node;\n\t}\n}\n\nstatic void push_arg_opt(int mask, const char *arg)\n{\n\tint base = (mask & OPT_KEY) ? 0 : 10;\n\tchar *end = NULL;\n\tunsigned long l;\n\n\tl = strtoul(arg, &end, base);\n\n\tif ((*end != 0) ||\t/* entire string is -not- valid */\n\t    ((mask & OPT_KEY) && (l == IPC_PRIVATE))) {\n\t\tfprintf(stderr, \"%s%s '%s' invalid\\n\",\n\t\t\targ_name(mask),\n\t\t\tmask & OPT_KEY ? \"key\" : \"id\",\n\t\t\targ);\n\t\texit_status = EXIT_FAILURE;\n\t\treturn;\n\t}\n\n\tpush_opt(mask, l);\n}\n\nstatic error_t parse_opt (int key, char *arg, struct argp_state *state)\n{\n\tswitch (key) {\n\tcase 'q': push_arg_opt(OPT_MSG, arg); break;\n\tcase 'm': push_arg_opt(OPT_SHM, arg); break;\n\tcase 's': push_arg_opt(OPT_SEM, arg); break;\n\tcase 'Q': push_arg_opt(OPT_MSG | OPT_KEY, arg); break;\n\tcase 'M': push_arg_opt(OPT_SHM | OPT_KEY, arg); break;\n\tcase 'S': push_arg_opt(OPT_SEM | OPT_KEY, arg); break;\n\n\tdefault:\n\t\treturn ARGP_ERR_UNKNOWN;\n\t}\n\n\treturn 0;\n}\n\nstatic void pinterr(const char *msg, long l)\n{\n\tfprintf(stderr, msg, l, strerror(errno));\n\texit_status = 1;\n}\n\nstatic void remove_one(int mask, unsigned long arg)\n{\n\tint rc;\n\tint id = (int) arg;\n\tconst char *errmsg = NULL;\n\n\tif (mask & OPT_KEY) {\n\t\tif (mask & OPT_MSG)\n\t\t\tid = msgget(arg, 0);\n\t\telse if (mask & OPT_SHM)\n\t\t\tid = shmget(arg, 0, 0);\n\t\telse if (mask & OPT_SEM)\n\t\t\tid = semget(arg, 0, 0);\n\t\telse\n\t\t\tabort();\t/* should never happen */\n\t}\n\n\tif (id < 0) {\n\t\tpinterr(\"key 0x%lx lookup failed: %s\\n\", arg);\n\t\treturn;\n\t}\n\n\tif (mask & OPT_MSG) {\n\t\trc = msgctl(id, IPC_RMID, NULL);\n\t\terrmsg = \"msgctl(0x%x): %s\\n\";\n\t}\n\telse if (mask & OPT_SHM) {\n\t\trc = shmctl(id, IPC_RMID, NULL);\n\t\terrmsg = \"shmctl(0x%x): %s\\n\";\n\t}\n\telse if (mask & OPT_SEM) {\n\t\tunion semun dummy;\n\t\tdummy.val = 0;\n\n\t\trc = semctl(id, 0, IPC_RMID, dummy);\n\t\terrmsg = \"semctl(0x%x): %s\\n\";\n\t}\n\n\telse\n\t\tabort();\t/* should never happen */\n\n\tif (rc < 0) {\n\t\tfprintf(stderr, errmsg, id, strerror(errno));\n\t\texit_status = 1;\n\t}\n}\n\nstatic void remove_stuff(void)\n{\n\tstruct arglist *tmp = arglist;\n\n\twhile (tmp) {\n\t\tremove_one(tmp->mask, tmp->arg);\n\t\ttmp = tmp->next;\n\t}\n}\n\nint main (int argc, char *argv[])\n{\n\terror_t rc;\n\n\tpu_init();\n\n\trc = argp_parse(&argp, argc, argv, 0, NULL, NULL);\n\tif (rc) {\n\t\tfprintf(stderr, \"argp_parse failed: %s\\n\", strerror(rc));\n\t\treturn 1;\n\t}\n\n\tremove_stuff();\n\n\treturn exit_status;\n}\n\n"},{"id":"3659","messageId":"428E7994.1090402@pobox.com","threadId":"679","inReplyTo":"7vvf5dqxfq.fsf@assigned-by-dhcp.cox.net","subject":"Re: checkout-cache -f: a better way?","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-05-20T23:58:12Z","receivedAt":"2005-05-20T23:58:12Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Junio C Hamano wrote:\n>>>>>>\"JG\" == Jeff Garzik <jgarzik@pobox.com> writes:\n> \n> \n> JG> git-checkout-cache -f writes out every file, even if it exists, correct?\n> \n> No, that's not correct.  To translate my prose, you would want\n> this:\n\nThanks, I stand corrected :)\n\n\n>     git-read-tree -m HEAD && git-checkout-cache -q -f -u -a\n> \n> (notice that I do not have git-update-cache --refresh after\n> that).\n\nYep, thanks.  Script does seem faster now.  Numbers for hot cache (first \nis pre-modification, post is your mod):\n\n[jgarzik@pretzel libata-dev]$ time git-switch-tree adma-mwi\n\nreal    0m7.069s\nuser    0m4.183s\nsys     0m2.817s\n[jgarzik@pretzel libata-dev]$ time git-switch-tree adma\n\nreal    0m0.389s\nuser    0m0.294s\nsys     0m0.094s\n\n"},{"id":"3660","messageId":"7voeb5qwio.fsf@assigned-by-dhcp.cox.net","threadId":"679","inReplyTo":"Pine.LNX.4.58.0505201641160.2206@ppc970.osdl.org","subject":"Re: checkout-cache -f: a better way?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-05-20T23:59:27Z","receivedAt":"2005-05-20T23:59:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":">>>>> \"LT\" == Linus Torvalds <torvalds@osdl.org> writes:\n\nLT> (to which the beavis-and-butthead in me says \"hehhehhehh.. He said fqua.  \nLT> Hehhehh. fire fire fire.\")\n\nEarlier this week I've sent out a \"Request for Help\" listing\nsome janitorial work, on which this was one of the item.  I\nbelieve Jeff suggested use of argp over GNU getopt(), but other\nthan that I do not think we had any volunteers (hint hint).  I\nhaven't looked into any of the RFH items myself yet.\n\n"},{"id":"3668","messageId":"Pine.LNX.4.58.0505201841550.2206@ppc970.osdl.org","threadId":"679","inReplyTo":"428E7994.1090402@pobox.com","subject":"Re: checkout-cache -f: a better way?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-05-21T01:44:30Z","receivedAt":"2005-05-21T01:44:30Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 20 May 2005, Jeff Garzik wrote:\n>\n> Yep, thanks.  Script does seem faster now. \n\nYeah. They \"seem faster\".\n\n> real    0m7.069s\n\nto\n\n> real    0m0.389s\n\nThat's what, 20 times faster? \n\nMore, actually, I suspect, since the \"-m\" version is not only faster, but\nit doesn't do much IO, so you'll not have tons of dirty pages/inodes etc\nafterwards.\n\n\t\tLinus\n"},{"id":"3670","messageId":"428E92D1.80408@pobox.com","threadId":"679","inReplyTo":"Pine.LNX.4.58.0505201841550.2206@ppc970.osdl.org","subject":"Re: checkout-cache -f: a better way?","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-05-21T01:45:53Z","receivedAt":"2005-05-21T01:45:53Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Linus Torvalds wrote:\n> That's what, 20 times faster? \n\n:)\n\n\n> More, actually, I suspect, since the \"-m\" version is not only faster, but\n> it doesn't do much IO, so you'll not have tons of dirty pages/inodes etc\n> afterwards.\n\nYep.  A -lot- of writeback would occur, a few seconds after my original \nscript completed.\n\n\tJeff\n\n\n"}]}