{"thread":{"id":"43209","subject":"Re: Adding glob support to remotes","startedAt":"2006-11-22T09:04:20Z","lastAt":"2006-11-23T08:44:27Z","messageCount":5,"participants":["Junio C Hamano","Andy Parkins"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"296410","messageId":"200611220904.21850.andyparkins@gmail.com","threadId":"43209","inReplyTo":null,"subject":"Adding glob support to remotes","fromName":"Andy Parkins","fromEmail":"andyparkins@gmail.com","sentAt":"2006-11-22T09:04:20Z","receivedAt":"2006-11-22T09:04:20Z","isPatch":false,"sender":{"key":"andyparkins@gmail.com","avatar":null},"body":"Hello,\n\nI'm working on adding (basic) glob support to remote fetch definitions.  This \nis to allow you to write things like\n\n[remote \"origin\"]\n  fetch = refs/heads/*:refs/remotes/upstream/*\n\nI started to add code to git-parse-remote.sh:canon_refs_list_for_fetch() to \npreprocess the reflist to catch lines with a \"*\" in them then use the remote \npattern to filter the output of from \"git-ls-remote -h\", blah, blah, you get \nthe idea...\n\nHowever, git-ls-remote needs the name of the remote repository (of course), \nbut that isn't directly available in git-parse-remote.sh.  Should I\n a) pass it as a parameter from git-fetch.sh right through each intermediate \nfunction\n b) create a global?\n c) change git-check-ref-format to allow \"*\" in the name, then put the \ngit-ls-remote call in git-fetch instead.\n\nI don't like to do (b) as it's nasty programming behaviour; however passing a \nparameter is fairly intrusive to the existing code.  Similarly (c) means I'm \nmessing in places I suspect I shouldn't be (git-check-ref-format).\n\ngit-gods - what do I do?\n\n\n\nAndy\n-- \nDr Andy Parkins, M Eng (hons), MIEE\n"},{"id":"294036","messageId":"7v7ixnskql.fsf@assigned-by-dhcp.cox.net","threadId":"43209","inReplyTo":"200611220904.21850.andyparkins@gmail.com","subject":"Re: Adding glob support to remotes","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-22T12:56:34Z","receivedAt":"2006-11-22T12:56:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Parkins <andyparkins@gmail.com> writes:\n\n> I started to add code to git-parse-remote.sh:canon_refs_list_for_fetch() to \n> preprocess the reflist to catch lines with a \"*\" in them then use the remote \n> pattern to filter the output of from \"git-ls-remote -h\", blah, blah, you get \n> the idea...\n>\n> However, git-ls-remote needs the name of the remote repository (of course), \n> but that isn't directly available in git-parse-remote.sh.  \n\nIs it really the case?  I do not remember the details offhand,\nbut I do not think canon_refs_list_for_fetch is the function you\nshould be messing with to implement the remote.\"origin\".fetch\nstuff.  It should be get_remote_default_refs_for_fetch().  The\nfunction returns the list based on which remote, so it surely\nknows which remote the caller is talking about.\n\nHowever, I would recommend against actually running ls-remote to\nhelp \"git-fetch\" inside git-parse-remote.sh.  I think you should\nrun ls-remote upfront early in git-fetch because there are at\nleast two other parts in git-fetch that wants the same ls-remote\noutput:\n\n (1) dumb protocols currently cannot deal with a remote that has\n     run \"packed-ref --prune\" because git-fetch.sh first uses\n     curl executable to download the loose ref, read it and then\n     use the object name read from that to drive git-http-fetch.\n     We can and should get rid of the /max_depth=5/,/done/ loop\n     there and replace it with a grep of ls-remote output to\n     make them work against such a remote.  When tracking many\n     branches from the remote, this would reduce the number of\n     http requests (one per branch vs a ls-remote which is just\n     a single download of info/refs).\n\n (2) when doing a fetch with tracking branches (which is what\n     your change is about), we would need to run ls-remote to\n     find out the remote tags for tag following purposes anyway.\n     Running \"ls-remote -h\" once for your purpose and then\n     \"ls-remote -t\" for tag following later is obviously very\n     wasteful.\n\n"},{"id":"295358","messageId":"200611221441.02459.andyparkins@gmail.com","threadId":"43209","inReplyTo":"7v7ixnskql.fsf@assigned-by-dhcp.cox.net","subject":"Re: Adding glob support to remotes","fromName":"Andy Parkins","fromEmail":"andyparkins@gmail.com","sentAt":"2006-11-22T14:41:00Z","receivedAt":"2006-11-22T14:41:00Z","isPatch":false,"sender":{"key":"andyparkins@gmail.com","avatar":null},"body":"On Wednesday 2006 November 22 12:56, Junio C Hamano wrote:\n\n> > However, git-ls-remote needs the name of the remote repository (of\n> > course), but that isn't directly available in git-parse-remote.sh.\n>\n> Is it really the case?  I do not remember the details offhand,\n> but I do not think canon_refs_list_for_fetch is the function you\n> should be messing with to implement the remote.\"origin\".fetch\n> stuff.  It should be get_remote_default_refs_for_fetch().  The\n> function returns the list based on which remote, so it surely\n> knows which remote the caller is talking about.\n\nThe problem is that canon_refs_list_for_fetch bombs out too early because \"*\" \nis not an acceptable name for a ref.\n\n> However, I would recommend against actually running ls-remote to\n> help \"git-fetch\" inside git-parse-remote.sh.  I think you should\n> run ls-remote upfront early in git-fetch because there are at\n> least two other parts in git-fetch that wants the same ls-remote\n> output:\n\nOkay.  That's what I'll do.  It means altering git-check-ref-format to prevent \nthe early bomb out.  Perhaps I should move this check to somewhere after I've \ndone the reflist expansion?\n\n>  (1) dumb protocols currently cannot deal with a remote that has\n\nI'm not sure I've understood this point.  I shall look at git-fetch.sh more \nclosely to try and address this though.\n\n>  (2) when doing a fetch with tracking branches (which is what\n\nAccepted.\n\n\n\nAndy\n\n-- \nDr Andy Parkins, M Eng (hons), MIEE\n"},{"id":"297236","messageId":"7v3b8bryt6.fsf@assigned-by-dhcp.cox.net","threadId":"43209","inReplyTo":"200611221441.02459.andyparkins@gmail.com","subject":"Re: Adding glob support to remotes","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-22T20:50:13Z","receivedAt":"2006-11-22T20:50:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Parkins <andyparkins@gmail.com> writes:\n\n> On Wednesday 2006 November 22 12:56, Junio C Hamano wrote:\n>\n>> > However, git-ls-remote needs the name of the remote repository (of\n>> > course), but that isn't directly available in git-parse-remote.sh.\n>>\n>> Is it really the case?  I do not remember the details offhand,\n>> but I do not think canon_refs_list_for_fetch is the function you\n>> should be messing with to implement the remote.\"origin\".fetch\n>> stuff.  It should be get_remote_default_refs_for_fetch().  The\n>> function returns the list based on which remote, so it surely\n>> knows which remote the caller is talking about.\n>\n> The problem is that canon_refs_list_for_fetch bombs out too early because \"*\" \n> is not an acceptable name for a ref.\n\nI do not understand and that is why I do not understand why you\nwould need to touch check-ref-format.\n\n\"remote.$1.fetch\" is read by get_remote_default_refs_for_fetch\nand currently the value is given canon_refs_list_for_fetch\nwithout any preprocessing.  My suggestion is to catch the\n\"wildcard\" in get_remote_default_refs_for_fetch, do your\nreplacement based on an earlier ls-remote output, and give\nreplaced values to canon_refs_list_for_fetch.  That way I do not\nthink check-ref-format that is used by canon_refs_list_for_fetch\nneeds to see any wildcard.\n\nBoth config and remotes but not branches _could_ have wildcard\nso you would need to do the same wildcard replacement in two\ncase arms of get_remote_default_refs_for_fetch but I think that\ncan be done in a common helper function.\n\nIf on the other hand if you want to do this in canon_*, then you\ncould do that before the \"for ref\" loop to set \"$@\" to the\nwildcard-expanded form.  But there are other codepaths that use\nthis function and I do not know if you want to apply the\nwildcarding to all of them (I haven't checked all the callers).\nIf you are sure all the callers want the wildcarding the same\nway, it would be more appropriate to do so there than doing it\nin get_remote_default_refs_for_fetch as I suggested in the\nabove.  I.e. something like this perhaps.\n\n canon_refs_list_for_fetch () {\n ...\n \t\t\t    --get-all \"branch.${curr_branch}.merge\")\n \t\tfi\n \tfi\n+\n+\texpanded_ref_list=\n+\tfor ref\n+\tdo\n+\t\tif ref is wildcard\n+\t\tthen\n+\t\t\tcompute wildcard replacement and set it to nref\n+\t\t\texpanded_ref_list=\"$expanded_ref_list $nref\"\n+\t\telse\n+\t\t\texpanded_ref_list=\"$expanded_ref_list $ref\"\n+\t\tfi\n+\tdone\n+\tset x $expanded_ref_list\n+\tshift\n+\n \tfor ref\n \tdo\n \t\tforce=\n\n\n>>  (1) dumb protocols currently cannot deal with a remote that has\n>\n> I'm not sure I've understood this point.  I shall look at git-fetch.sh more \n> closely to try and address this though.\n\nI was talking about this part of the code:\n\n      case \"$remote\" in\n      http://* | https://* | ftp://*)\n\t  if [ -n \"$GIT_SSL_NO_VERIFY\" ]; then\n\t      curl_extra_args=\"-k\"\n\t  fi\n\t  if [ -n \"$GIT_CURL_FTP_NO_EPSV\" -o \\\n\t\t\"`git-repo-config --bool http.noEPSV`\" = true ]; then\n\t      noepsv_opt=\"--disable-epsv\"\n\t  fi\n\t  max_depth=5\n\t  depth=0\n\t  head=\"ref: $remote_name\"\n\t  while (expr \"z$head\" : \"zref:\" && expr $depth \\< $max_depth) >/dev/null\n\t  do\n\t    remote_name_quoted=$(@@PERL@@ -e '\n\t      my $u = $ARGV[0];\n              $u =~ s/^ref:\\s*//;\n\t      $u =~ s{([^-a-zA-Z0-9/.])}{sprintf\"%%%02x\",ord($1)}eg;\n\t      print \"$u\";\n\t  ' \"$head\")\n\t    head=$(curl -nsfL $curl_extra_args $noepsv_opt \"$remote/$remote_name_quoted\")\n\t    depth=$( expr \\( $depth + 1 \\) )\n\t  done\n\t  expr \"z$head\" : \"z$_x40\\$\" >/dev/null ||\n\t      die \"Failed to fetch $remote_name from $remote\"\n\t  echo >&2 Fetching \"$remote_name from $remote\" using http\n\t  git-http-fetch -v -a \"$head\" \"$remote/\" || exit\n\t  ;;\n\nThe while loop is initialized with \"ref: refs/heads/$branch\", \nextracts the \"refs/heads/$branch\" from it by hand, and runs\n\"curl -nsfL\" to get http://host/repo.git/refs/heads/$branch,\nuntil it is not a symref anymore (refs/heads/$branch _could_\nbe a symref that points at refs/heads/$another, and that is the\nloop is about).  At the end of the loop, $head would contain\nwhat is read from the ref -- 40-byte object name.  That is given\nto git-http-fetch.\n\nHowever, the above assumes $GIT_DIR/refs/heads/$branch file\nexists at the remote end for \"curl -nsfL\" to fetch.  When refs\nare packed-and-pruned, refs/heads/$branch may not exist as a\nstandalone file; the right way to learn \"what object does a ref\npoint at\" is to ask ls-remote, and if you do one ls-remote\nupfront then all you need to do here is a single grep.\n\n"},{"id":"298649","messageId":"200611230844.29061.andyparkins@gmail.com","threadId":"43209","inReplyTo":"7v3b8bryt6.fsf@assigned-by-dhcp.cox.net","subject":"Re: Adding glob support to remotes","fromName":"Andy Parkins","fromEmail":"andyparkins@gmail.com","sentAt":"2006-11-23T08:44:27Z","receivedAt":"2006-11-23T08:44:27Z","isPatch":false,"sender":{"key":"andyparkins@gmail.com","avatar":null},"body":"On Wednesday 2006 November 22 20:50, Junio C Hamano wrote:\n\n> I do not understand and that is why I do not understand why you\n> would need to touch check-ref-format.\n\nThe problem is that I didn't want to call git-ls-remote in git-parse-remote, \nas git-parse-remote doesn't know about $uploadpack or what the actual remote \nin use is.  It seemed fairly intrusive to completely alter the functions in \ngit-parse-remote; which is obviously a set of library functions.\n\nInstead I did the expansion in git-fetch.sh.  Unfortunately that meant \ndisabling the check-ref-format call in canon_refs_list_for_fetch.\n\n> point at\" is to ask ls-remote, and if you do one ls-remote\n> upfront then all you need to do here is a single grep.\n\nI understand now.  I'll look again at this tonight and send a further patch.\n\n\nAndy\n-- \nDr Andy Parkins, M Eng (hons), MIEE\n"}]}