{"thread":{"id":"50591","subject":"[WIP RFC PATCH v2 0/5] clone: dir iterator refactoring with tests","startedAt":"2019-02-26T05:18:16Z","lastAt":"2019-07-11T15:25:00Z","messageCount":127,"participants":["Matheus Tavares","Ævar Arnfjörð Bjarmason","Duy Nguyen","Matheus Tavares Bernardino","Thomas Gummerer","SZEDER Gábor","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":5},"messages":[{"id":"370201","messageId":"20190226051804.10631-1-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":null,"subject":"[WIP RFC PATCH v2 0/5] clone: dir iterator refactoring with tests","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T05:17:59Z","receivedAt":"2019-02-26T05:18:16Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Ævar sent v1: https://public-inbox.org/git/20190226002625.13022-1-avarab@gmail.com/\n\nBased on Ævar's work and comments, I refactored my previous series[1],\nso that clone's original behaviour regarding symlinks is kept untouched.\nSo, keeping this behaviour, having tests for this, changing local clone\nto use dir-iterator's API and modifing hidden paths behaviour (which we\nagreed to seem like unintentional), I think this version addresses all\nthe comments and concerns raised in this thread:\nhttps://public-inbox.org/git/CAP8UFD2xrfMHNxcmeYf8G+d53SL26N07FFAoDP+e0h3r-tvKQw@mail.gmail.com/\nAnd also addresses the comments I made in WIP's v1.\n\n[1]: https://public-inbox.org/git/20190223190309.6728-1-matheus.bernardino@usp.br/\n\nMatheus Tavares (4):\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n builtin/clone.c | 70 ++++++++++++++++++++++----------------\n dir-iterator.c | 28 +++++++++++++--\n dir-iterator.h | 40 +++++++++++++++++-----\n refs/files-backend.c | 2 +-\n t/t5604-clone-reference.sh | 69 +++++++++++++++++++++++++++++++++++++\n 5 files changed, 168 insertions(+), 41 deletions(-)\n\n-- \n2.20.1\n\n"},{"id":"370202","messageId":"20190226051804.10631-2-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[WIP RFC PATCH v2 1/5] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T05:18:00Z","receivedAt":"2019-02-26T05:18:21Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are DIR_ITERATOR_PEDANTIC, which makes\ndir_iterator_advance abort imediatelly in the case of an error while\ntrying to fetch next entry; and DIR_ITERATOR_FOLLOW_SYMLINKS, which\nmakes the iteration follow symlinks to directories and include its\ncontents in the iteration. These new flags will be used in a subsequent\npatch.\n\nAlso adjust refs/files-backend.c to the new dir_iterator_begin\nsignature.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 28 +++++++++++++++++++++++++---\n dir-iterator.h | 40 ++++++++++++++++++++++++++++++++--------\n refs/files-backend.c | 2 +-\n 3 files changed, 58 insertions(+), 12 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..17aca8ea41 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -48,12 +48,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned flags;\n };\n \n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n+\tint ret;\n \n \twhile (1) {\n \t\tstruct dir_iterator_level *level =\n@@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n \t\t\t\twarning(\"error opening directory %s: %s\",\n \t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t/* Popping the level is handled below */\n@@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading directory %s: %s\",\n \t\t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t} else if (closedir(level->dir))\n@@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\t\tcontinue;\n \n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n+\n+\t\t\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\t\t\tret = stat(iter->base.path.buf, &iter->base.st);\n+\t\t\telse\n+\t\t\t\tret = lstat(iter->base.path.buf, &iter->base.st);\n+\n+\t\t\tif (ret < 0) {\n+\t\t\t\tif (errno != ENOENT) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading path '%s': %s\",\n \t\t\t\t\t\titer->base.path.buf,\n \t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t}\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\treturn ITER_OK;\n \t\t}\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -182,7 +203,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -195,6 +216,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n+\titer->flags = flags;\n \titer->levels_nr = 1;\n \titer->levels[0].initialized = 0;\n \ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..fe9eb9a04b 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -6,9 +6,10 @@\n /*\n * Iterate over a directory tree.\n *\n- * Iterate over a directory tree, recursively, including paths of all\n- * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n- * symlinks except for the original path.\n+ * With no flags to modify behaviour, iterate over a directory tree,\n+ * recursively, including paths of all types and hidden paths. Skip\n+ * \".\" and \"..\" entries and don't follow symlinks except for the\n+ * original path.\n *\n * Every time dir_iterator_advance() is called, update the members of\n * the dir_iterator structure to reflect the next path in the\n@@ -19,7 +20,7 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * struct iterator *iter = dir_iterator_begin(path, 0);\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n@@ -40,6 +41,20 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error while trying to fetch the next entry, which is\n+ * to emit a warning and keep going. With this flag, resouces are\n+ * freed and ITER_ERROR is return immediately.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n+ * directories, i.e., iterate over linked directories' contents.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -59,15 +74,19 @@ struct dir_iterator {\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. Return a dir_iterator that holds the\n+ * internal state of the iteration.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behaviour.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n@@ -76,6 +95,11 @@ struct dir_iterator *dir_iterator_begin(const char *path);\n * dir_iterator and associated resources and return ITER_ERROR. It is\n * a bug to use iterator or call this function again after it has\n * returned ITER_DONE or ITER_ERROR.\n+ *\n+ * Note that whether dir-iterator will return ITER_ERROR when failing\n+ * to fetch the next entry or just emit a warning and try to fetch the\n+ * next is defined by the 'pedantic' option at dir-iterator's\n+ * initialization.\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex dd8abe9185..c3d3b6c454 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,7 +2143,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\titer->dir_iterator = dir_iterator_begin(sb.buf, 0);\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \n-- \n2.20.1\n\n"},{"id":"370203","messageId":"20190226051804.10631-4-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[WIP RFC PATCH v2 3/5] clone: copy hidden paths at local clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T05:18:02Z","receivedAt":"2019-02-26T05:18:27Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make the copy_or_link_directory function no longer skip hidden paths.\nThis function, used to copy .git/objects, currently skips all hidden\ndirectories but not hidden files, which is an odd behaviour. The reason\nfor that could be unintentional: probably the intention was to skip '.'\nand '..' only but it ended up accidentally skipping all directories\nstarting with '.'. Besides being more natural, the new behaviour is more\npermissive to the user.\n\nAlso adjusted tests to reflect this change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 50bde99618..cae069f03b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -428,7 +428,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 6f9c77049e..f1a8e74c44 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -262,16 +262,25 @@ test_expect_success SHA1,SYMLINKS 'clone repo with manually symlinked objects/*'\n \ttest_cmp expected actual &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.20.1\n\n"},{"id":"370204","messageId":"20190226051804.10631-3-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[WIP RFC PATCH v2 2/5] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T05:18:01Z","receivedAt":"2019-02-26T05:18:29Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nWe've implicitly supported .git/objects/* content of symlinks since\napproximately forever, and when we do a copy of the repo we transfer\nthose over, but aren't very consistent about other random stuff we\nfind depending on if it's a \"hidden\" file or not.\n\nLet's add a test for that, which shouldn't read as an endorsement of\nwhat we're doing now, just asserts current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5604-clone-reference.sh | 60 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 60 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..6f9c77049e 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,64 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success SHA1,SYMLINKS 'setup repo with manually symlinked objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\ttest_commit B &&\n+\t\t(\n+\t\t\tcd .git/objects &&\n+\t\t\tmv 22/3b7836fb19fdf64ba2d3cd6173c6a283141f78 . &&\n+\t\t\tln -s ../3b7836fb19fdf64ba2d3cd6173c6a283141f78 22/ &&\n+\t\t\tmv 40 forty &&\n+\t\t\tln -s forty 40 &&\n+\t\t\tmv pack packs &&\n+\t\t\tln -s packs pack &&\n+\t\t\t>.some-hidden-file &&\n+\t\t\t>some-file &&\n+\t\t\tmkdir .some-hidden-dir &&\n+\t\t\t>.some-hidden-dir/some-file &&\n+\t\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\t\tmkdir some-dir &&\n+\t\t\t>some-dir/some-file &&\n+\t\t\t>some-dir/.some-dot-file\n+\t\t)\n+\t)\n+'\n+\n+test_expect_success SHA1,SYMLINKS 'clone repo with manually symlinked objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -type l | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/22/3b7836fb19fdf64ba2d3cd6173c6a283141f78\n+\tS--local/.git/objects/22/3b7836fb19fdf64ba2d3cd6173c6a283141f78\n+\tEOF\n+\ttest_cmp expected actual &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.20.1\n\n"},{"id":"370205","messageId":"20190226051804.10631-5-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[WIP RFC PATCH v2 4/5] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T05:18:03Z","receivedAt":"2019-02-26T05:18:31Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Extract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help removing\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 27 +++++++++++++++++++--------\n 1 file changed, 19 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex cae069f03b..fd580fa98d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -392,6 +392,24 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\t/*\n+\t * Create a dir at pathname unless there's already one.\n+\t */\n+\tstruct stat st;\n+\n+\tif (mkdir(pathname, mode)) {\n+\t\tif (errno != EEXIST)\n+\t\t\tdie_errno(_(\"failed to create directory '%s'\"),\n+\t\t\t\t pathname);\n+\t\telse if (stat(pathname, &st))\n+\t\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\t\telse if (!S_ISDIR(st.st_mode))\n+\t\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+\t}\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -404,14 +422,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.20.1\n\n"},{"id":"370206","messageId":"20190226051804.10631-6-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T05:18:04Z","receivedAt":"2019-02-26T05:18:33Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoid recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat, inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone would end up\nsuccessfully even though the .git/objects copy didn't fully succeeded.\n\nSigned-off-by: Matheus Tavares \n---\nI can also make the change described in the last paragraph in a separate\npatch before this one, but I would have to undo it in this patch because\ndir-iterator already implements it. So, IMHO, it would be just noise\nand not worthy.\n\n builtin/clone.c | 45 +++++++++++++++++++++++----------------------\n 1 file changed, 23 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd580fa98d..b23ba64c94 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -411,42 +413,37 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tstruct stat st;\n+\tunsigned flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -463,7 +460,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -481,7 +482,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.20.1\n\n"},{"id":"370220","messageId":"87wolmvl2f.fsf@evledraar.gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-6-matheus.bernardino@usp.br","subject":"Re: [WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T11:35:52Z","receivedAt":"2019-02-26T11:35:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 26 2019, Matheus Tavares wrote:\n\n> +\tint iter_status;\n> +\tstruct stat st;\n> +\tunsigned flags;\n\nIf you compile with DEVELOPER=1 you'll get compile-time errors when\nvariables aren't used. Like \"st\" here.\n"},{"id":"370221","messageId":"87va16vl1f.fsf@evledraar.gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"Re: [WIP RFC PATCH v2 0/5] clone: dir iterator refactoring with tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T11:36:28Z","receivedAt":"2019-02-26T11:36:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 26 2019, Matheus Tavares wrote:\n\n> Ævar sent v1: https://public-inbox.org/git/20190226002625.13022-1-avarab@gmail.com/\n\nTip. Use --in-reply-to with git format-patch, then the whole discussion\ngoes in the same thread on the ML.\n"},{"id":"370222","messageId":"CACsJy8DCpA3yvSjWiNDPtqMxSo9wNUobCkBHXOnJsG-+6C2Eew@mail.gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-2-matheus.bernardino@usp.br","subject":"Re: [WIP RFC PATCH v2 1/5] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-26T12:01:58Z","receivedAt":"2019-02-26T12:02:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n wrote:\n> int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> {\n> struct dir_iterator_int *iter =\n> (struct dir_iterator_int *)dir_iterator;\n> + int ret;\n>\n> while (1) {\n> struct dir_iterator_level *level =\n> @@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n>\n> level->dir = opendir(iter->base.path.buf);\n> if (!level->dir && errno != ENOENT) {\n> + if (iter->flags & DIR_ITERATOR_PEDANTIC)\n> + goto error_out;\n> warning(\"error opening directory %s: %s\",\n> iter->base.path.buf, strerror(errno));\n> /* Popping the level is handled below */\n> @@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> if (!de) {\n> /* This level is exhausted; pop up a level. */\n> if (errno) {\n> + if (iter->flags & DIR_ITERATOR_PEDANTIC)\n> + goto error_out;\n> warning(\"error reading directory %s: %s\",\n> iter->base.path.buf, strerror(errno));\n> } else if (closedir(level->dir))\n> @@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> continue;\n>\n> strbuf_addstr(&iter->base.path, de->d_name);\n> - if (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n> - if (errno != ENOENT)\n> +\n> + if (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n> + ret = stat(iter->base.path.buf, &iter->base.st);\n> + else\n> + ret = lstat(iter->base.path.buf, &iter->base.st);\n> +\n> + if (ret < 0) {\n> + if (errno != ENOENT) {\n> + if (iter->flags & DIR_ITERATOR_PEDANTIC)\n> + goto error_out;\n> warning(\"error reading path '%s': %s\",\n> iter->base.path.buf,\n> strerror(errno));\n> + }\n> continue;\n> }\n>\n> @@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> return ITER_OK;\n> }\n> }\n> +\n> +error_out:\n> + dir_iterator_abort(dir_iterator);\n\nShould this function call this or leaveit to the caller? The\ndescription says \"free resources\" which to me sounds like something\nthe caller should be aware of. Although if double dir_iterator_abort()\nhas no bad side effects then I guess we can even leave it here.\n\nPS. files-backend.c does call dir_iterator_abort() unconditionally.\nWhich sounds like this double-abort pattern should be dealt with even\nif that call site does not use the pedantic flag (it could later on,\nwho knows; don't leave traps behind).\n\n> + return ITER_ERROR;\n> }\n>\n> diff --git a/dir-iterator.h b/dir-iterator.h\n> index 970793d07a..fe9eb9a04b 100644\n> --- a/dir-iterator.h\n> +++ b/dir-iterator.h\n> @@ -6,9 +6,10 @@\n> /*\n> * Iterate over a directory tree.\n> *\n> - * Iterate over a directory tree, recursively, including paths of all\n> - * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n> - * symlinks except for the original path.\n> + * With no flags to modify behaviour, iterate over a directory tree,\n\nNit but I think we can just skip \"With no flags to modify behavior\". It's given.\n\n> + * recursively, including paths of all types and hidden paths. Skip\n> + * \".\" and \"..\" entries and don't follow symlinks except for the\n> + * original path.\n> *\n> * Every time dir_iterator_advance() is called, update the members of\n> * the dir_iterator structure to reflect the next path in the\n> @@ -19,7 +20,7 @@\n> * A typical iteration looks like this:\n> *\n> * int ok;\n> - * struct iterator *iter = dir_iterator_begin(path);\n> + * struct iterator *iter = dir_iterator_begin(path, 0);\n> *\n> * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n> * if (want_to_stop_iteration()) {\n> @@ -40,6 +41,20 @@\n> * dir_iterator_advance() again.\n> */\n>\n> +/*\n> + * Flags for dir_iterator_begin:\n> + *\n> + * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n> + * in case of an error while trying to fetch the next entry, which is\n> + * to emit a warning and keep going. With this flag, resouces are\n> + * freed and ITER_ERROR is return immediately.\n> + *\n> + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n> + * directories, i.e., iterate over linked directories' contents.\n\nDo we really need this flag? If dir-iterator does not follow symlinks,\nthe caller _can_ check stat data to detect symlinks and look inside\nanyway. So this flag is more about convenience (_if_ it has more than\none call site, convenience for one call site is just not worth it).\n\nOr is there something else I'm missing?\n\n> + */\n> +#define DIR_ITERATOR_PEDANTIC (1 << 0)\n> +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n> +\n-- \nDuy\n"},{"id":"370223","messageId":"CACsJy8B=NT8F4-gHwomSu39iY0pbJqJjCi+yqgPs_D_6ELx5+Q@mail.gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-4-matheus.bernardino@usp.br","subject":"Re: [WIP RFC PATCH v2 3/5] clone: copy hidden paths at local clone","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-26T12:13:45Z","receivedAt":"2019-02-26T12:14:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n wrote:\n>\n> Make the copy_or_link_directory function no longer skip hidden paths.\n\nIt's actually only hidden directories because of the S_ISDIR check\nright above. Not that it matters much...\n\n> This function, used to copy .git/objects, currently skips all hidden\n> directories but not hidden files, which is an odd behaviour. The reason\n> for that could be unintentional:\n\nThis goes back to the very first version of clone.c in 8434c2f1af\n(Build in clone - 2008-04-27). If you look at git-clone.sh back then,\nwhich is the version before the C conversion, it does something like\nthis\n\n find objects -depth -print | cpio $cpio_quiet_flag -pumd$l \"$GIT_DIR/\"\n\nand I'm pretty sure 'find' will not attempt to hide anything. So yes I\nthink this is just for skipping '.' and '..' and accidentally skips\nmore. From that view, it's actually a regresssion but nobody ever\nbothers to hide anything in 'objects' directory to notice.\n\n> probably the intention was to skip '.'\n> and '..' only but it ended up accidentally skipping all directories\n> starting with '.'. Besides being more natural, the new behaviour is more\n> permissive to the user.\n>\n> Also adjusted tests to reflect this change.\n>\n> Signed-off-by: Matheus Tavares \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/clone.c | 2 +-\n> t/t5604-clone-reference.sh | 9 +++++++++\n> 2 files changed, 10 insertions(+), 1 deletion(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 50bde99618..cae069f03b 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -428,7 +428,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> continue;\n> }\n> if (S_ISDIR(buf.st_mode)) {\n> - if (de->d_name[0] != '.')\n> + if (!is_dot_or_dotdot(de->d_name))\n> copy_or_link_directory(src, dest,\n> src_repo, src_baselen);\n> continue;\n> diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n> index 6f9c77049e..f1a8e74c44 100755\n> --- a/t/t5604-clone-reference.sh\n> +++ b/t/t5604-clone-reference.sh\n> @@ -262,16 +262,25 @@ test_expect_success SHA1,SYMLINKS 'clone repo with manually symlinked objects/*'\n> test_cmp expected actual &&\n> find S-* -name \"*some*\" | sort >actual &&\n> cat >expected <<-EOF &&\n> + S--dissociate/.git/objects/.some-hidden-dir\n> + S--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n> + S--dissociate/.git/objects/.some-hidden-dir/some-file\n> S--dissociate/.git/objects/.some-hidden-file\n> S--dissociate/.git/objects/some-dir\n> S--dissociate/.git/objects/some-dir/.some-dot-file\n> S--dissociate/.git/objects/some-dir/some-file\n> S--dissociate/.git/objects/some-file\n> + S--local/.git/objects/.some-hidden-dir\n> + S--local/.git/objects/.some-hidden-dir/.some-dot-file\n> + S--local/.git/objects/.some-hidden-dir/some-file\n> S--local/.git/objects/.some-hidden-file\n> S--local/.git/objects/some-dir\n> S--local/.git/objects/some-dir/.some-dot-file\n> S--local/.git/objects/some-dir/some-file\n> S--local/.git/objects/some-file\n> + S--no-hardlinks/.git/objects/.some-hidden-dir\n> + S--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n> + S--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n> S--no-hardlinks/.git/objects/.some-hidden-file\n> S--no-hardlinks/.git/objects/some-dir\n> S--no-hardlinks/.git/objects/some-dir/.some-dot-file\n> --\n> 2.20.1\n>\n\n\n-- \nDuy\n"},{"id":"370224","messageId":"CACsJy8BztC=GNuPU_CuLbPSmhsjrSk8MmkyeK6ahgjEOzY=yOQ@mail.gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-5-matheus.bernardino@usp.br","subject":"Re: [WIP RFC PATCH v2 4/5] clone: extract function from copy_or_link_directory","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-26T12:18:28Z","receivedAt":"2019-02-26T12:18:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n wrote:\n>\n> Extract dir creation code snippet from copy_or_link_directory to its own\n> function named mkdir_if_missing. This change will help removing\n> copy_or_link_directory's explicit recursion, which will be done in a\n> following patch. Also makes code more readable.\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> builtin/clone.c | 27 +++++++++++++++++++--------\n> 1 file changed, 19 insertions(+), 8 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index cae069f03b..fd580fa98d 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -392,6 +392,24 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n> fclose(in);\n> }\n>\n> +static void mkdir_if_missing(const char *pathname, mode_t mode)\n> +{\n> + /*\n> + * Create a dir at pathname unless there's already one.\n\nThis confused me for a second because I thought it described \"st\"\nvariable. I think we usually put the description of the function on\ntop (before the \"void mkdir_if..\" line). But with a such a short\nfunction and clear name like this, I don't think we need any comments.\n\n> + */\n> + struct stat st;\n> +\n> + if (mkdir(pathname, mode)) {\n\nGood opportunity to unindent this by doing\n\n if (!mkdir(...\n return;\n\nbut it's up to you.\n\n> + if (errno != EEXIST)\n> + die_errno(_(\"failed to create directory '%s'\"),\n> + pathname);\n> + else if (stat(pathname, &st))\n> + die_errno(_(\"failed to stat '%s'\"), pathname);\n> + else if (!S_ISDIR(st.st_mode))\n> + die(_(\"%s exists and is not a directory\"), pathname);\n> + }\n> +}\n> +\n> static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> const char *src_repo, int src_baselen)\n> {\n> @@ -404,14 +422,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> if (!dir)\n> die_errno(_(\"failed to open '%s'\"), src->buf);\n>\n> - if (mkdir(dest->buf, 0777)) {\n> - if (errno != EEXIST)\n> - die_errno(_(\"failed to create directory '%s'\"), dest->buf);\n> - else if (stat(dest->buf, &buf))\n> - die_errno(_(\"failed to stat '%s'\"), dest->buf);\n> - else if (!S_ISDIR(buf.st_mode))\n> - die(_(\"%s exists and is not a directory\"), dest->buf);\n> - }\n> + mkdir_if_missing(dest->buf, 0777);\n>\n> strbuf_addch(src, '/');\n> src_len = src->len;\n> --\n> 2.20.1\n>\n\n\n-- \nDuy\n"},{"id":"370225","messageId":"CACsJy8BWmCGim1_8P6j_iYCvPALdTAVhqqsBdR+ktfN6XmFSkQ@mail.gmail.com","threadId":"50591","inReplyTo":"87va16vl1f.fsf@evledraar.gmail.com","subject":"Re: [WIP RFC PATCH v2 0/5] clone: dir iterator refactoring with tests","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-26T12:20:34Z","receivedAt":"2019-02-26T12:21:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 26, 2019 at 6:36 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Tue, Feb 26 2019, Matheus Tavares wrote:\n>\n> > Ævar sent v1: https://public-inbox.org/git/20190226002625.13022-1-avarab@gmail.com/\n>\n> Tip. Use --in-reply-to with git format-patch, then the whole discussion\n> goes in the same thread on the ML.\n\nBetter tip. Click on v1 link, look for the \"reply\" link, click it and\ncopy the whole format-patch command :D\n-- \nDuy\n"},{"id":"370226","messageId":"20190226122829.19178-1-avarab@gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[RFC PATCH v3 0/5] clone: dir iterator refactoring with tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:28:24Z","receivedAt":"2019-02-26T12:28:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There's still active review going on for the \"v2\"[1], in particular\nDuy's comments coming in as I write this. This doesn't address any of\nthat.\n\nWhat it does do is have a better version of my patch to add tests for\nthe current behavior. It now doesn't reply the on SHA1 prereq anymore,\nand we can test the dotfiles without needing the SYMLINK prereq.\n\nI also moved it to the front of the series just to document/make sure\nthat we start by asserting testing functionality. I ran a full test\nsuite run for each of these patches and they all pass.\n\nThe only other change is getting rid of an unused \"struct stat st\"\nvariable which errored out under DEVELOPER=1.\n\n1. https://public-inbox.org/git/20190226051804.10631-1-matheus.bernardino@usp.br/\n\nMatheus Tavares (4):\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n builtin/clone.c | 69 ++++++++++-------\n dir-iterator.c | 28 ++++++-\n dir-iterator.h | 40 ++++++++--\n refs/files-backend.c | 2 +-\n t/t5604-clone-reference.sh | 151 +++++++++++++++++++++++++++++++++++++\n 5 files changed, 249 insertions(+), 41 deletions(-)\n\n-- \n2.21.0.rc2.261.ga7da99ff1b\n\n"},{"id":"370227","messageId":"20190226122829.19178-2-avarab@gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[RFC PATCH v3 1/5] clone: test for our behavior on odd objects/* content","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:28:25Z","receivedAt":"2019-02-26T12:28:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add tests for what happens when we locally clone .git/objects\ndirectories where some of the loose objects or packs are symlinked, or\nwhen when there's unknown files there.\n\nI'm bending over backwards here to avoid a SHA1 dependency. See [1]\nfor an earlier and simpler version that hardcoded a SHA-1s.\n\nThis behavior has been the same for a *long* time, but hasn't been\ntested for.\n\nThere's a good post-hoc argument to be made for copying over unknown\nthings, e.g. I'd like a git version that doesn't know about the\ncommit-graph to copy it under \"clone --local\" so a newer git version\ncan make use of it.\n\nBut the behavior showed where with symlinks seems pretty\nrandom. E.g. if \"pack\" is a symlink we end up with two copies of the\ncontents, and only transfer some symlinks as-is.\n\nIn follow-up commits we'll look at changing some of this behavior, but\nfor now let's just assert it as-is so we'll notice what we'll change\nlater.\n\n1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5604-clone-reference.sh | 142 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 142 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..cb0dc22d14 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,146 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success 'setup repo with garbage in objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\n+\t\tcd .git/objects &&\n+\t\t>.some-hidden-file &&\n+\t\t>some-file &&\n+\t\tmkdir .some-hidden-dir &&\n+\t\t>.some-hidden-dir/some-file &&\n+\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\tmkdir some-dir &&\n+\t\t>some-dir/some-file &&\n+\t\t>some-dir/.some-dot-file\n+\t)\n+'\n+\n+test_expect_success 'clone a repo with garbage in objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success SYMLINKS 'setup repo with manually symlinked objects/*' '\n+\tgit init T &&\n+\t(\n+\t\tcd T &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\t(\n+\t\t\tcd .git/objects &&\n+\t\t\tmv pack packs &&\n+\t\t\tln -s packs pack\n+\t\t) &&\n+\t\ttest_commit B &&\n+\t\t(\n+\t\t\tcd .git/objects &&\n+\t\t\tfind ?? -type d >loose-dirs &&\n+\t\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n+\t\t\tmv $last_loose a-loose-dir &&\n+\t\t\tln -s a-loose-dir $last_loose &&\n+\t\t\tfirst_loose=$(head -n 1 loose-dirs) &&\n+\t\t\t(\n+\t\t\t\tcd $first_loose &&\n+\t\t\t\tobj=$(ls *) &&\n+\t\t\t\tmv $obj ../an-object &&\n+\t\t\t\tln -s ../an-object $obj\n+\t\t\t) &&\n+\t\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T.objects-links.raw\n+\t\t)\n+\t) &&\n+\tgit -C T fsck &&\n+\tgit -C T rev-list --all --objects >T.objects\n+'\n+\n+\n+test_expect_success SYMLINKS 'clone repo with symlinked objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option T T$option || return 1 &&\n+\t\tgit -C T$option fsck || return 1 &&\n+\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n+\t\ttest_cmp T.objects T$option.objects &&\n+\t\t(\n+\t\t\tcd T$option/.git/objects &&\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T$option.objects-links.raw\n+\t\t)\n+\tdone &&\n+\n+\tfor raw in $(ls T*.raw)\n+\tdo\n+\t\tsed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" <$raw >$raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./an-object\n+\t./Y/Z\n+\t./info/packs\n+\t./loose-dirs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\tEOF\n+\tcat >expected-links <<-EOF &&\n+\t./Y/Z\n+\tEOF\n+\tfor option in --local --dissociate\n+\tdo\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n+\t\ttest_cmp expected-links T$option.objects-links.raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./an-object\n+\t./Y/Z\n+\t./info/packs\n+\t./loose-dirs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\tEOF\n+\ttest_cmp expected-files T--no-hardlinks.objects-files.raw.de-sha &&\n+\ttest_must_be_empty T--no-hardlinks.objects-links.raw.de-sha &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./info/alternates\n+\tEOF\n+\ttest_cmp expected-files T--shared.objects-files.raw &&\n+\ttest_must_be_empty T--shared.objects-links.raw\n+'\n+\n test_done\n-- \n2.21.0.rc2.261.ga7da99ff1b\n\n"},{"id":"370228","messageId":"20190226122829.19178-3-avarab@gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[RFC PATCH v3 2/5] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:28:26Z","receivedAt":"2019-02-26T12:28:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Matheus Tavares \n\nAdd the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are DIR_ITERATOR_PEDANTIC, which makes\ndir_iterator_advance abort imediatelly in the case of an error while\ntrying to fetch next entry; and DIR_ITERATOR_FOLLOW_SYMLINKS, which\nmakes the iteration follow symlinks to directories and include its\ncontents in the iteration. These new flags will be used in a subsequent\npatch.\n\nAlso adjust refs/files-backend.c to the new dir_iterator_begin\nsignature.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 28 +++++++++++++++++++++++++---\n dir-iterator.h | 40 ++++++++++++++++++++++++++++++++--------\n refs/files-backend.c | 2 +-\n 3 files changed, 58 insertions(+), 12 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..17aca8ea41 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -48,12 +48,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned flags;\n };\n \n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n+\tint ret;\n \n \twhile (1) {\n \t\tstruct dir_iterator_level *level =\n@@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n \t\t\t\twarning(\"error opening directory %s: %s\",\n \t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t/* Popping the level is handled below */\n@@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading directory %s: %s\",\n \t\t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t} else if (closedir(level->dir))\n@@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\t\tcontinue;\n \n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n+\n+\t\t\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\t\t\tret = stat(iter->base.path.buf, &iter->base.st);\n+\t\t\telse\n+\t\t\t\tret = lstat(iter->base.path.buf, &iter->base.st);\n+\n+\t\t\tif (ret < 0) {\n+\t\t\t\tif (errno != ENOENT) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading path '%s': %s\",\n \t\t\t\t\t\titer->base.path.buf,\n \t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t}\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\treturn ITER_OK;\n \t\t}\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -182,7 +203,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -195,6 +216,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n+\titer->flags = flags;\n \titer->levels_nr = 1;\n \titer->levels[0].initialized = 0;\n \ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..fe9eb9a04b 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -6,9 +6,10 @@\n /*\n * Iterate over a directory tree.\n *\n- * Iterate over a directory tree, recursively, including paths of all\n- * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n- * symlinks except for the original path.\n+ * With no flags to modify behaviour, iterate over a directory tree,\n+ * recursively, including paths of all types and hidden paths. Skip\n+ * \".\" and \"..\" entries and don't follow symlinks except for the\n+ * original path.\n *\n * Every time dir_iterator_advance() is called, update the members of\n * the dir_iterator structure to reflect the next path in the\n@@ -19,7 +20,7 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * struct iterator *iter = dir_iterator_begin(path, 0);\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n@@ -40,6 +41,20 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error while trying to fetch the next entry, which is\n+ * to emit a warning and keep going. With this flag, resouces are\n+ * freed and ITER_ERROR is return immediately.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n+ * directories, i.e., iterate over linked directories' contents.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -59,15 +74,19 @@ struct dir_iterator {\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. Return a dir_iterator that holds the\n+ * internal state of the iteration.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behaviour.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n@@ -76,6 +95,11 @@ struct dir_iterator *dir_iterator_begin(const char *path);\n * dir_iterator and associated resources and return ITER_ERROR. It is\n * a bug to use iterator or call this function again after it has\n * returned ITER_DONE or ITER_ERROR.\n+ *\n+ * Note that whether dir-iterator will return ITER_ERROR when failing\n+ * to fetch the next entry or just emit a warning and try to fetch the\n+ * next is defined by the 'pedantic' option at dir-iterator's\n+ * initialization.\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex dd8abe9185..c3d3b6c454 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,7 +2143,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\titer->dir_iterator = dir_iterator_begin(sb.buf, 0);\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \n-- \n2.21.0.rc2.261.ga7da99ff1b\n\n"},{"id":"370229","messageId":"20190226122829.19178-4-avarab@gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[RFC PATCH v3 3/5] clone: copy hidden paths at local clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:28:27Z","receivedAt":"2019-02-26T12:28:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Matheus Tavares \n\nMake the copy_or_link_directory function no longer skip hidden paths.\nThis function, used to copy .git/objects, currently skips all hidden\ndirectories but not hidden files, which is an odd behaviour. The reason\nfor that could be unintentional: probably the intention was to skip '.'\nand '..' only but it ended up accidentally skipping all directories\nstarting with '.'. Besides being more natural, the new behaviour is more\npermissive to the user.\n\nAlso adjusted tests to reflect this change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 50bde99618..cae069f03b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -428,7 +428,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex cb0dc22d14..d650f67ca5 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \tdone &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.21.0.rc2.261.ga7da99ff1b\n\n"},{"id":"370230","messageId":"20190226122829.19178-5-avarab@gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[RFC PATCH v3 4/5] clone: extract function from copy_or_link_directory","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:28:28Z","receivedAt":"2019-02-26T12:29:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Matheus Tavares \n\nExtract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help removing\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 27 +++++++++++++++++++--------\n 1 file changed, 19 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex cae069f03b..fd580fa98d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -392,6 +392,24 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\t/*\n+\t * Create a dir at pathname unless there's already one.\n+\t */\n+\tstruct stat st;\n+\n+\tif (mkdir(pathname, mode)) {\n+\t\tif (errno != EEXIST)\n+\t\t\tdie_errno(_(\"failed to create directory '%s'\"),\n+\t\t\t\t pathname);\n+\t\telse if (stat(pathname, &st))\n+\t\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\t\telse if (!S_ISDIR(st.st_mode))\n+\t\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+\t}\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -404,14 +422,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.21.0.rc2.261.ga7da99ff1b\n\n"},{"id":"370231","messageId":"20190226122829.19178-6-avarab@gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-1-matheus.bernardino@usp.br","subject":"[RFC PATCH v3 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:28:29Z","receivedAt":"2019-02-26T12:29:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Matheus Tavares \n\nReplace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoid recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat, inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone would end up\nsuccessfully even though the .git/objects copy didn't fully succeeded.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 44 ++++++++++++++++++++++----------------------\n 1 file changed, 22 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd580fa98d..6c07648e49 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -411,42 +413,36 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tunsigned flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -463,7 +459,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -481,7 +481,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.21.0.rc2.261.ga7da99ff1b\n\n"},{"id":"370232","messageId":"CACsJy8B1asF3i+G-C1aZRw7QTW7jS+a4VkCbg-17YOTyYHuw5w@mail.gmail.com","threadId":"50591","inReplyTo":"20190226051804.10631-6-matheus.bernardino@usp.br","subject":"Re: [WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-26T12:32:05Z","receivedAt":"2019-02-26T12:32:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n wrote:\n>\n> Replace usage of opendir/readdir/closedir API to traverse directories\n> recursively, at copy_or_link_directory function, by the dir-iterator\n> API. This simplifies the code and avoid recursive calls to\n> copy_or_link_directory.\n>\n> This process also makes copy_or_link_directory call die() in case of an\n> error on readdir or stat, inside dir_iterator_advance. Previously it\n> would just print a warning for errors on stat and ignore errors on\n> readdir, which isn't nice because a local git clone would end up\n> successfully even though the .git/objects copy didn't fully succeeded.\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> I can also make the change described in the last paragraph in a separate\n> patch before this one, but I would have to undo it in this patch because\n> dir-iterator already implements it. So, IMHO, it would be just noise\n> and not worthy.\n>\n> builtin/clone.c | 45 +++++++++++++++++++++++----------------------\n> 1 file changed, 23 insertions(+), 22 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index fd580fa98d..b23ba64c94 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -23,6 +23,8 @@\n> #include \"transport.h\"\n> #include \"strbuf.h\"\n> #include \"dir.h\"\n> +#include \"dir-iterator.h\"\n> +#include \"iterator.h\"\n> #include \"sigchain.h\"\n> #include \"branch.h\"\n> #include \"remote.h\"\n> @@ -411,42 +413,37 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n> }\n>\n> static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> - const char *src_repo, int src_baselen)\n> + const char *src_repo)\n> {\n> - struct dirent *de;\n> - struct stat buf;\n> int src_len, dest_len;\n> - DIR *dir;\n> -\n> - dir = opendir(src->buf);\n> - if (!dir)\n> - die_errno(_(\"failed to open '%s'\"), src->buf);\n> + struct dir_iterator *iter;\n> + int iter_status;\n> + struct stat st;\n> + unsigned flags;\n>\n> mkdir_if_missing(dest->buf, 0777);\n>\n> + flags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n> + iter = dir_iterator_begin(src->buf, flags);\n> +\n> strbuf_addch(src, '/');\n> src_len = src->len;\n> strbuf_addch(dest, '/');\n> dest_len = dest->len;\n>\n> - while ((de = readdir(dir)) != NULL) {\n> + while ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> strbuf_setlen(src, src_len);\n> - strbuf_addstr(src, de->d_name);\n> + strbuf_addstr(src, iter->relative_path);\n> strbuf_setlen(dest, dest_len);\n> - strbuf_addstr(dest, de->d_name);\n> - if (stat(src->buf, &buf)) {\n> - warning (_(\"failed to stat %s\\n\"), src->buf);\n> - continue;\n> - }\n> - if (S_ISDIR(buf.st_mode)) {\n> - if (!is_dot_or_dotdot(de->d_name))\n> - copy_or_link_directory(src, dest,\n> - src_repo, src_baselen);\n> + strbuf_addstr(dest, iter->relative_path);\n> +\n> + if (S_ISDIR(iter->st.st_mode)) {\n> + mkdir_if_missing(dest->buf, 0777);\n\nI wonder if this mkdir_if_missing is sufficient. What if you have to\ncreate multiple directories?\n\nLet's say the first advance, we hit \"a\". The the second advance we hit\ndirectory \"b/b/b/b\", we would need to mkdir recursively and something\nlike safe_create_leading_directories() would be a better fit.\n\nI'm not sure if it can happen though. I haven't re-read dir-iterator\ncode carefully.\n\n> continue;\n> }\n>\n> /* Files that cannot be copied bit-for-bit... */\n> - if (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n> + if (!strcmp(iter->relative_path, \"info/alternates\")) {\n\nWhile we're here, this should be fspathcmp to be friendlier to\ncase-insensitive filesystems. You probably should fix it in a separate\npatch though.\n\n> copy_alternates(src, dest, src_repo);\n> continue;\n> }\n> @@ -463,7 +460,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> if (copy_file_with_time(dest->buf, src->buf, 0666))\n> die_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n> }\n> - closedir(dir);\n> +\n> + if (iter_status != ITER_DONE) {\n> + strbuf_setlen(src, src_len);\n> + die(_(\"failed to iterate over '%s'\"), src->buf);\n> + }\n\nI think you need to abort the iterator even when it returns ITER_DONE.\nAt least that's how the first caller in files-backend.c does it.\n\n> }\n>\n> static void clone_local(const char *src_repo, const char *dest_repo)\n> @@ -481,7 +482,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n> get_common_dir(&dest, dest_repo);\n> strbuf_addstr(&src, \"/objects\");\n> strbuf_addstr(&dest, \"/objects\");\n> - copy_or_link_directory(&src, &dest, src_repo, src.len);\n> + copy_or_link_directory(&src, &dest, src_repo);\n> strbuf_release(&src);\n> strbuf_release(&dest);\n> }\n> --\n> 2.20.1\n>\n\n\n-- \nDuy\n"},{"id":"370235","messageId":"87tvgqvhlr.fsf@evledraar.gmail.com","threadId":"50591","inReplyTo":"CACsJy8B1asF3i+G-C1aZRw7QTW7jS+a4VkCbg-17YOTyYHuw5w@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-26T12:50:40Z","receivedAt":"2019-02-26T12:50:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 26 2019, Duy Nguyen wrote:\n\n> On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n> wrote:\n>>\n>> Replace usage of opendir/readdir/closedir API to traverse directories\n>> recursively, at copy_or_link_directory function, by the dir-iterator\n>> API. This simplifies the code and avoid recursive calls to\n>> copy_or_link_directory.\n>>\n>> This process also makes copy_or_link_directory call die() in case of an\n>> error on readdir or stat, inside dir_iterator_advance. Previously it\n>> would just print a warning for errors on stat and ignore errors on\n>> readdir, which isn't nice because a local git clone would end up\n>> successfully even though the .git/objects copy didn't fully succeeded.\n>>\n>> Signed-off-by: Matheus Tavares \n>> ---\n>> I can also make the change described in the last paragraph in a separate\n>> patch before this one, but I would have to undo it in this patch because\n>> dir-iterator already implements it. So, IMHO, it would be just noise\n>> and not worthy.\n>>\n>> builtin/clone.c | 45 +++++++++++++++++++++++----------------------\n>> 1 file changed, 23 insertions(+), 22 deletions(-)\n>>\n>> diff --git a/builtin/clone.c b/builtin/clone.c\n>> index fd580fa98d..b23ba64c94 100644\n>> --- a/builtin/clone.c\n>> +++ b/builtin/clone.c\n>> @@ -23,6 +23,8 @@\n>> #include \"transport.h\"\n>> #include \"strbuf.h\"\n>> #include \"dir.h\"\n>> +#include \"dir-iterator.h\"\n>> +#include \"iterator.h\"\n>> #include \"sigchain.h\"\n>> #include \"branch.h\"\n>> #include \"remote.h\"\n>> @@ -411,42 +413,37 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n>> }\n>>\n>> static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n>> - const char *src_repo, int src_baselen)\n>> + const char *src_repo)\n>> {\n>> - struct dirent *de;\n>> - struct stat buf;\n>> int src_len, dest_len;\n>> - DIR *dir;\n>> -\n>> - dir = opendir(src->buf);\n>> - if (!dir)\n>> - die_errno(_(\"failed to open '%s'\"), src->buf);\n>> + struct dir_iterator *iter;\n>> + int iter_status;\n>> + struct stat st;\n>> + unsigned flags;\n>>\n>> mkdir_if_missing(dest->buf, 0777);\n>>\n>> + flags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n>> + iter = dir_iterator_begin(src->buf, flags);\n>> +\n>> strbuf_addch(src, '/');\n>> src_len = src->len;\n>> strbuf_addch(dest, '/');\n>> dest_len = dest->len;\n>>\n>> - while ((de = readdir(dir)) != NULL) {\n>> + while ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n>> strbuf_setlen(src, src_len);\n>> - strbuf_addstr(src, de->d_name);\n>> + strbuf_addstr(src, iter->relative_path);\n>> strbuf_setlen(dest, dest_len);\n>> - strbuf_addstr(dest, de->d_name);\n>> - if (stat(src->buf, &buf)) {\n>> - warning (_(\"failed to stat %s\\n\"), src->buf);\n>> - continue;\n>> - }\n>> - if (S_ISDIR(buf.st_mode)) {\n>> - if (!is_dot_or_dotdot(de->d_name))\n>> - copy_or_link_directory(src, dest,\n>> - src_repo, src_baselen);\n>> + strbuf_addstr(dest, iter->relative_path);\n>> +\n>> + if (S_ISDIR(iter->st.st_mode)) {\n>> + mkdir_if_missing(dest->buf, 0777);\n>\n> I wonder if this mkdir_if_missing is sufficient. What if you have to\n> create multiple directories?\n>\n> Let's say the first advance, we hit \"a\". The the second advance we hit\n> directory \"b/b/b/b\", we would need to mkdir recursively and something\n> like safe_create_leading_directories() would be a better fit.\n>\n> I'm not sure if it can happen though. I haven't re-read dir-iterator\n> code carefully.\n\nThis part isn't a problem. It iterates one level at a time. So given a\nstructure like a/b/c/d/e/f/g/h/i/j/k/some-l you'll find that if you\ninstrument the loop in clone.c you get:\n\n dir = a\n dir = a/b\n dir = a/b/c\n dir = a/b/c/d\n dir = a/b/c/d/e\n dir = a/b/c/d/e/f\n dir = a/b/c/d/e/f/g\n dir = a/b/c/d/e/f/g/h\n dir = a/b/c/d/e/f/g/h/i\n dir = a/b/c/d/e/f/g/h/i/j\n dir = a/b/c/d/e/f/g/h/i/j/k\n dir = a/b/c/d/e/f/g/h/i/j/k/some-l\n\nSo it's like the old implementation in that way. It readdir()'s and\nwalks directories one level at a time.\n"},{"id":"370274","messageId":"CAHd-oW6+Q86F3D8wzVJngRU8CFq5qXPfiV+t_0kuCR1RioSmHQ@mail.gmail.com","threadId":"50591","inReplyTo":"20190226122829.19178-1-avarab@gmail.com","subject":"Re: [RFC PATCH v3 0/5] clone: dir iterator refactoring with tests","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-26T20:56:41Z","receivedAt":"2019-02-26T20:56:55Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Ævar\n\nThank you for helping me out on this series and, especially, for the tests part.\n\nNow that we've come to a consensus on what the overall series' \"shape\"\nshould be, can I refine what's still needed and resubmit it, in the\nupcoming days, as a patch set with the test included?\n\nBest,\nMatheus Tavares\n\n\nOn Tue, Feb 26, 2019 at 9:28 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> There's still active review going on for the \"v2\"[1], in particular\n> Duy's comments coming in as I write this. This doesn't address any of\n> that.\n>\n> What it does do is have a better version of my patch to add tests for\n> the current behavior. It now doesn't reply the on SHA1 prereq anymore,\n> and we can test the dotfiles without needing the SYMLINK prereq.\n>\n> I also moved it to the front of the series just to document/make sure\n> that we start by asserting testing functionality. I ran a full test\n> suite run for each of these patches and they all pass.\n>\n> The only other change is getting rid of an unused \"struct stat st\"\n> variable which errored out under DEVELOPER=1.\n>\n> 1. https://public-inbox.org/git/20190226051804.10631-1-matheus.bernardino@usp.br/\n>\n> Matheus Tavares (4):\n> dir-iterator: add flags parameter to dir_iterator_begin\n> clone: copy hidden paths at local clone\n> clone: extract function from copy_or_link_directory\n> clone: use dir-iterator to avoid explicit dir traversal\n>\n> Ævar Arnfjörð Bjarmason (1):\n> clone: test for our behavior on odd objects/* content\n>\n> builtin/clone.c | 69 ++++++++++-------\n> dir-iterator.c | 28 ++++++-\n> dir-iterator.h | 40 ++++++++--\n> refs/files-backend.c | 2 +-\n> t/t5604-clone-reference.sh | 151 +++++++++++++++++++++++++++++++++++++\n> 5 files changed, 249 insertions(+), 41 deletions(-)\n>\n> --\n> 2.21.0.rc2.261.ga7da99ff1b\n>\n"},{"id":"370305","messageId":"CAHd-oW6c6JMqH7czjpYkg7Z5NJgrnqWpcCmZq4Mf+pj4tjgYUw@mail.gmail.com","threadId":"50591","inReplyTo":"CACsJy8DCpA3yvSjWiNDPtqMxSo9wNUobCkBHXOnJsG-+6C2Eew@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 1/5] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-27T13:59:53Z","receivedAt":"2019-02-27T14:00:08Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Feb 26, 2019 at 9:02 AM Duy Nguyen wrote:\n>\n> On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n> wrote:\n> > int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> > {\n> > struct dir_iterator_int *iter =\n> > (struct dir_iterator_int *)dir_iterator;\n> > + int ret;\n> >\n> > while (1) {\n> > struct dir_iterator_level *level =\n> > @@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> >\n> > level->dir = opendir(iter->base.path.buf);\n> > if (!level->dir && errno != ENOENT) {\n> > + if (iter->flags & DIR_ITERATOR_PEDANTIC)\n> > + goto error_out;\n> > warning(\"error opening directory %s: %s\",\n> > iter->base.path.buf, strerror(errno));\n> > /* Popping the level is handled below */\n> > @@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> > if (!de) {\n> > /* This level is exhausted; pop up a level. */\n> > if (errno) {\n> > + if (iter->flags & DIR_ITERATOR_PEDANTIC)\n> > + goto error_out;\n> > warning(\"error reading directory %s: %s\",\n> > iter->base.path.buf, strerror(errno));\n> > } else if (closedir(level->dir))\n> > @@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> > continue;\n> >\n> > strbuf_addstr(&iter->base.path, de->d_name);\n> > - if (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n> > - if (errno != ENOENT)\n> > +\n> > + if (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n> > + ret = stat(iter->base.path.buf, &iter->base.st);\n> > + else\n> > + ret = lstat(iter->base.path.buf, &iter->base.st);\n> > +\n> > + if (ret < 0) {\n> > + if (errno != ENOENT) {\n> > + if (iter->flags & DIR_ITERATOR_PEDANTIC)\n> > + goto error_out;\n> > warning(\"error reading path '%s': %s\",\n> > iter->base.path.buf,\n> > strerror(errno));\n> > + }\n> > continue;\n> > }\n> >\n> > @@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> > return ITER_OK;\n> > }\n> > }\n> > +\n> > +error_out:\n> > + dir_iterator_abort(dir_iterator);\n>\n> Should this function call this or leaveit to the caller? The\n> description says \"free resources\" which to me sounds like something\n> the caller should be aware of. Although if double dir_iterator_abort()\n> has no bad side effects then I guess we can even leave it here.\n>\n\nThe documentation at dir-iterator.h says that in case of errors (or\niteration exhaustion) at dir_iterator_advance(), the dir-iterator and\nassociated resources will be freed, so dir_iterator_abort() must be\ncalled here, not left to the user. (Also, the use-case example at the\ntop of dir-iterator.h confirms this.)\n\n> PS. files-backend.c does call dir_iterator_abort() unconditionally.\n> Which sounds like this double-abort pattern should be dealt with even\n> if that call site does not use the pedantic flag (it could later on,\n> who knows; don't leave traps behind).\n>\n\nI have read the code at files-backend.c again but couldn't find where\nit could call dir_iterator_abort() after dir_iterator_advance() have\nalready called it. Could you please point me out that?\n\nA double-abort would certainly lead to 'double free or corruption'\nerror, so API users must only call it to abort iteration for some\nreason when treating the iteration files, not because of iteration\nerrors. Also, this behaviour is not only for the pedantic flag. When\nan iteration is complete, for example, a call to\ndir_iterator_advance() will free the resources and return ITER_DONE.\n\n> > + return ITER_ERROR;\n> > }\n> >\n> > diff --git a/dir-iterator.h b/dir-iterator.h\n> > index 970793d07a..fe9eb9a04b 100644\n> > --- a/dir-iterator.h\n> > +++ b/dir-iterator.h\n> > @@ -6,9 +6,10 @@\n> > /*\n> > * Iterate over a directory tree.\n> > *\n> > - * Iterate over a directory tree, recursively, including paths of all\n> > - * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n> > - * symlinks except for the original path.\n> > + * With no flags to modify behaviour, iterate over a directory tree,\n>\n> Nit but I think we can just skip \"With no flags to modify behavior\". It's given.\n>\n\nOk!\n\n> > + * recursively, including paths of all types and hidden paths. Skip\n> > + * \".\" and \"..\" entries and don't follow symlinks except for the\n> > + * original path.\n> > *\n> > * Every time dir_iterator_advance() is called, update the members of\n> > * the dir_iterator structure to reflect the next path in the\n> > @@ -19,7 +20,7 @@\n> > * A typical iteration looks like this:\n> > *\n> > * int ok;\n> > - * struct iterator *iter = dir_iterator_begin(path);\n> > + * struct iterator *iter = dir_iterator_begin(path, 0);\n> > *\n> > * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n> > * if (want_to_stop_iteration()) {\n> > @@ -40,6 +41,20 @@\n> > * dir_iterator_advance() again.\n> > */\n> >\n> > +/*\n> > + * Flags for dir_iterator_begin:\n> > + *\n> > + * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n> > + * in case of an error while trying to fetch the next entry, which is\n> > + * to emit a warning and keep going. With this flag, resouces are\n> > + * freed and ITER_ERROR is return immediately.\n> > + *\n> > + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n> > + * directories, i.e., iterate over linked directories' contents.\n>\n> Do we really need this flag? If dir-iterator does not follow symlinks,\n> the caller _can_ check stat data to detect symlinks and look inside\n> anyway. So this flag is more about convenience (_if_ it has more than\n> one call site, convenience for one call site is just not worth it).\n>\n\nI don't think is just a convenience. If the API user wants to follow\nsymlinks a simple check at stat data to detect symlinks, wouldn't be\nenough, since the user would want to traverse through all contents of\na symlinked directory and its subdirectories, recursively. Without\nthis flag, the caller would need to manually detect symlinks to\ndirectories and start a new iteration on them (probably using a new\ndir-iterator?). As that could lead to a more unnecessarily complex\ncode, and being this flag implementation so simple, I think it is\nworthy.\n\n> Or is there something else I'm missing?\n>\n> > + */\n> > +#define DIR_ITERATOR_PEDANTIC (1 << 0)\n> > +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n> > +\n> --\n> Duy\n"},{"id":"370327","messageId":"CAHd-oW5aOf+TE_AeA=VkaZUt7LTeSK_buojsEgb3NYMVVm1B_g@mail.gmail.com","threadId":"50591","inReplyTo":"CACsJy8BztC=GNuPU_CuLbPSmhsjrSk8MmkyeK6ahgjEOzY=yOQ@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 4/5] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-27T17:30:27Z","receivedAt":"2019-02-27T17:30:41Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Feb 26, 2019 at 9:18 AM Duy Nguyen wrote:\n>\n> On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n> wrote:\n> >\n> > Extract dir creation code snippet from copy_or_link_directory to its own\n> > function named mkdir_if_missing. This change will help removing\n> > copy_or_link_directory's explicit recursion, which will be done in a\n> > following patch. Also makes code more readable.\n> >\n> > Signed-off-by: Matheus Tavares \n> > ---\n> > builtin/clone.c | 27 +++++++++++++++++++--------\n> > 1 file changed, 19 insertions(+), 8 deletions(-)\n> >\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index cae069f03b..fd580fa98d 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -392,6 +392,24 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n> > fclose(in);\n> > }\n> >\n> > +static void mkdir_if_missing(const char *pathname, mode_t mode)\n> > +{\n> > + /*\n> > + * Create a dir at pathname unless there's already one.\n>\n> This confused me for a second because I thought it described \"st\"\n> variable. I think we usually put the description of the function on\n> top (before the \"void mkdir_if..\" line). But with a such a short\n> function and clear name like this, I don't think we need any comments.\n>\n\nYes, I also don't like the description being after the function\ndeclaration, but I did this to follow the pattern from other functions\non the same file (e.g. copy_alternates). Anyway, I do agree with you\nthat this function don't need a description, so I'm removing it for\nthe next version. Thanks!\n\n> > + */\n> > + struct stat st;\n> > +\n> > + if (mkdir(pathname, mode)) {\n>\n> Good opportunity to unindent this by doing\n>\n> if (!mkdir(...\n> return;\n>\n> but it's up to you.\n>\n\nOk. But being such a small snippet, is the indentation really a code\nsmell here? (sorry, I'm still getting used to git's coding guidelines)\n\n> > + if (errno != EEXIST)\n> > + die_errno(_(\"failed to create directory '%s'\"),\n> > + pathname);\n> > + else if (stat(pathname, &st))\n> > + die_errno(_(\"failed to stat '%s'\"), pathname);\n> > + else if (!S_ISDIR(st.st_mode))\n> > + die(_(\"%s exists and is not a directory\"), pathname);\n> > + }\n> > +}\n> > +\n> > static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > const char *src_repo, int src_baselen)\n> > {\n> > @@ -404,14 +422,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > if (!dir)\n> > die_errno(_(\"failed to open '%s'\"), src->buf);\n> >\n> > - if (mkdir(dest->buf, 0777)) {\n> > - if (errno != EEXIST)\n> > - die_errno(_(\"failed to create directory '%s'\"), dest->buf);\n> > - else if (stat(dest->buf, &buf))\n> > - die_errno(_(\"failed to stat '%s'\"), dest->buf);\n> > - else if (!S_ISDIR(buf.st_mode))\n> > - die(_(\"%s exists and is not a directory\"), dest->buf);\n> > - }\n> > + mkdir_if_missing(dest->buf, 0777);\n> >\n> > strbuf_addch(src, '/');\n> > src_len = src->len;\n> > --\n> > 2.20.1\n> >\n>\n>\n> --\n> Duy\n"},{"id":"370330","messageId":"CAHd-oW6WJ5JSRAbcy+5kcEA4V8qKEUc9B=6WQZvdqaHz4XHBTA@mail.gmail.com","threadId":"50591","inReplyTo":"CACsJy8B1asF3i+G-C1aZRw7QTW7jS+a4VkCbg-17YOTyYHuw5w@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-27T17:40:16Z","receivedAt":"2019-02-27T17:40:31Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Feb 26, 2019 at 9:32 AM Duy Nguyen wrote:\n>\n> On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n> wrote:\n> >\n> > Replace usage of opendir/readdir/closedir API to traverse directories\n> > recursively, at copy_or_link_directory function, by the dir-iterator\n> > API. This simplifies the code and avoid recursive calls to\n> > copy_or_link_directory.\n> >\n> > This process also makes copy_or_link_directory call die() in case of an\n> > error on readdir or stat, inside dir_iterator_advance. Previously it\n> > would just print a warning for errors on stat and ignore errors on\n> > readdir, which isn't nice because a local git clone would end up\n> > successfully even though the .git/objects copy didn't fully succeeded.\n> >\n> > Signed-off-by: Matheus Tavares \n> > ---\n> > I can also make the change described in the last paragraph in a separate\n> > patch before this one, but I would have to undo it in this patch because\n> > dir-iterator already implements it. So, IMHO, it would be just noise\n> > and not worthy.\n> >\n> > builtin/clone.c | 45 +++++++++++++++++++++++----------------------\n> > 1 file changed, 23 insertions(+), 22 deletions(-)\n> >\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index fd580fa98d..b23ba64c94 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -23,6 +23,8 @@\n> > #include \"transport.h\"\n> > #include \"strbuf.h\"\n> > #include \"dir.h\"\n> > +#include \"dir-iterator.h\"\n> > +#include \"iterator.h\"\n> > #include \"sigchain.h\"\n> > #include \"branch.h\"\n> > #include \"remote.h\"\n> > @@ -411,42 +413,37 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n> > }\n> >\n> > static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > - const char *src_repo, int src_baselen)\n> > + const char *src_repo)\n> > {\n> > - struct dirent *de;\n> > - struct stat buf;\n> > int src_len, dest_len;\n> > - DIR *dir;\n> > -\n> > - dir = opendir(src->buf);\n> > - if (!dir)\n> > - die_errno(_(\"failed to open '%s'\"), src->buf);\n> > + struct dir_iterator *iter;\n> > + int iter_status;\n> > + struct stat st;\n> > + unsigned flags;\n> >\n> > mkdir_if_missing(dest->buf, 0777);\n> >\n> > + flags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n> > + iter = dir_iterator_begin(src->buf, flags);\n> > +\n> > strbuf_addch(src, '/');\n> > src_len = src->len;\n> > strbuf_addch(dest, '/');\n> > dest_len = dest->len;\n> >\n> > - while ((de = readdir(dir)) != NULL) {\n> > + while ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> > strbuf_setlen(src, src_len);\n> > - strbuf_addstr(src, de->d_name);\n> > + strbuf_addstr(src, iter->relative_path);\n> > strbuf_setlen(dest, dest_len);\n> > - strbuf_addstr(dest, de->d_name);\n> > - if (stat(src->buf, &buf)) {\n> > - warning (_(\"failed to stat %s\\n\"), src->buf);\n> > - continue;\n> > - }\n> > - if (S_ISDIR(buf.st_mode)) {\n> > - if (!is_dot_or_dotdot(de->d_name))\n> > - copy_or_link_directory(src, dest,\n> > - src_repo, src_baselen);\n> > + strbuf_addstr(dest, iter->relative_path);\n> > +\n> > + if (S_ISDIR(iter->st.st_mode)) {\n> > + mkdir_if_missing(dest->buf, 0777);\n>\n> I wonder if this mkdir_if_missing is sufficient. What if you have to\n> create multiple directories?\n>\n> Let's say the first advance, we hit \"a\". The the second advance we hit\n> directory \"b/b/b/b\", we would need to mkdir recursively and something\n> like safe_create_leading_directories() would be a better fit.\n>\n> I'm not sure if it can happen though. I haven't re-read dir-iterator\n> code carefully.\n>\n> > continue;\n> > }\n> >\n> > /* Files that cannot be copied bit-for-bit... */\n> > - if (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n> > + if (!strcmp(iter->relative_path, \"info/alternates\")) {\n>\n> While we're here, this should be fspathcmp to be friendlier to\n> case-insensitive filesystems. You probably should fix it in a separate\n> patch though.\n>\n\nNice! I will make this change in a separate patch in the series. Thanks!\n\n> > copy_alternates(src, dest, src_repo);\n> > continue;\n> > }\n> > @@ -463,7 +460,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > if (copy_file_with_time(dest->buf, src->buf, 0666))\n> > die_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n> > }\n> > - closedir(dir);\n> > +\n> > + if (iter_status != ITER_DONE) {\n> > + strbuf_setlen(src, src_len);\n> > + die(_(\"failed to iterate over '%s'\"), src->buf);\n> > + }\n>\n> I think you need to abort the iterator even when it returns ITER_DONE.\n> At least that's how the first caller in files-backend.c does it.\n>\n\nHm, I don't think so, since dir_iterator_advance() already frees the\nresources before returning ITER_DONE. Also, I may be wrong, but it\ndoesn't seem to me, that files-backend.c does it. The function\nfiles_reflog_iterator_advance() that calls dir_iterator_advance() even\nsets the dir-iterator pointer to NULL as soon as ITER_DONE is\nreturned.\n\n\n> > }\n> >\n> > static void clone_local(const char *src_repo, const char *dest_repo)\n> > @@ -481,7 +482,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n> > get_common_dir(&dest, dest_repo);\n> > strbuf_addstr(&src, \"/objects\");\n> > strbuf_addstr(&dest, \"/objects\");\n> > - copy_or_link_directory(&src, &dest, src_repo, src.len);\n> > + copy_or_link_directory(&src, &dest, src_repo);\n> > strbuf_release(&src);\n> > strbuf_release(&dest);\n> > }\n> > --\n> > 2.20.1\n> >\n>\n>\n> --\n> Duy\n"},{"id":"370344","messageId":"20190227224553.GS6085@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"CAHd-oW5aOf+TE_AeA=VkaZUt7LTeSK_buojsEgb3NYMVVm1B_g@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 4/5] clone: extract function from copy_or_link_directory","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-02-27T22:45:53Z","receivedAt":"2019-02-27T22:45:59Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 02/27, Matheus Tavares Bernardino wrote:\n> On Tue, Feb 26, 2019 at 9:18 AM Duy Nguyen wrote:\n> >\n> > On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n> > wrote:\n> > > + */\n> > > + struct stat st;\n> > > +\n> > > + if (mkdir(pathname, mode)) {\n> >\n> > Good opportunity to unindent this by doing\n> >\n> > if (!mkdir(...\n> > return;\n> >\n> > but it's up to you.\n> >\n> \n> Ok. But being such a small snippet, is the indentation really a code\n> smell here? (sorry, I'm still getting used to git's coding guidelines)\n\nI don't think the indentation here is too bad here, but I think the\ncode is slightly easier to read with less indentation, and it's easier\nto see what's happening in the success case as well without reading\nthe whole method.\n\nAnd since this patch is already refactoring code we could do it here.\nI don't think it's a very big deal either way, which is why Duy left\nthe decision on whether to use the suggestion or not up to you.\n"},{"id":"370345","messageId":"CAHd-oW70pZaMX80TarpMd2rOtR_ONOikwUOO8n6ZD_vfKhN5yQ@mail.gmail.com","threadId":"50591","inReplyTo":"20190227224553.GS6085@hank.intra.tgummerer.com","subject":"Re: [WIP RFC PATCH v2 4/5] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-27T22:50:49Z","receivedAt":"2019-02-27T22:51:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Feb 27, 2019 at 7:45 PM Thomas Gummerer wrote:\n>\n> On 02/27, Matheus Tavares Bernardino wrote:\n> > On Tue, Feb 26, 2019 at 9:18 AM Duy Nguyen wrote:\n> > >\n> > > On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n> > > wrote:\n> > > > + */\n> > > > + struct stat st;\n> > > > +\n> > > > + if (mkdir(pathname, mode)) {\n> > >\n> > > Good opportunity to unindent this by doing\n> > >\n> > > if (!mkdir(...\n> > > return;\n> > >\n> > > but it's up to you.\n> > >\n> >\n> > Ok. But being such a small snippet, is the indentation really a code\n> > smell here? (sorry, I'm still getting used to git's coding guidelines)\n>\n> I don't think the indentation here is too bad here, but I think the\n> code is slightly easier to read with less indentation, and it's easier\n> to see what's happening in the success case as well without reading\n> the whole method.\n>\n> And since this patch is already refactoring code we could do it here.\n> I don't think it's a very big deal either way, which is why Duy left\n> the decision on whether to use the suggestion or not up to you.\n\nOk, so I will do it for the next version! I just asked because it was\na good chance for me to learn a bit more about git's code style :)\n"},{"id":"370350","messageId":"CACsJy8BV8CHOnbfDysgdo-md0pXLg05rKpMP5a8TUkUcWw0G8g@mail.gmail.com","threadId":"50591","inReplyTo":"CAHd-oW6WJ5JSRAbcy+5kcEA4V8qKEUc9B=6WQZvdqaHz4XHBTA@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-28T07:13:01Z","receivedAt":"2019-02-28T07:13:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Feb 28, 2019 at 12:40 AM Matheus Tavares Bernardino\n wrote:\n> > > @@ -463,7 +460,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > > if (copy_file_with_time(dest->buf, src->buf, 0666))\n> > > die_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n> > > }\n> > > - closedir(dir);\n> > > +\n> > > + if (iter_status != ITER_DONE) {\n> > > + strbuf_setlen(src, src_len);\n> > > + die(_(\"failed to iterate over '%s'\"), src->buf);\n> > > + }\n> >\n> > I think you need to abort the iterator even when it returns ITER_DONE.\n> > At least that's how the first caller in files-backend.c does it.\n> >\n>\n> Hm, I don't think so, since dir_iterator_advance() already frees the\n> resources before returning ITER_DONE. Also, I may be wrong, but it\n> doesn't seem to me, that files-backend.c does it. The function\n> files_reflog_iterator_advance() that calls dir_iterator_advance() even\n> sets the dir-iterator pointer to NULL as soon as ITER_DONE is\n> returned.\n\nArghhh.. I read the ref_iterator_abort and thought it was\ndir_iterator_abort! Sorry for the noise, too many iterators.\n-- \nDuy\n"},{"id":"370353","messageId":"87pnrcuz54.fsf@evledraar.gmail.com","threadId":"50591","inReplyTo":"CAHd-oW6WJ5JSRAbcy+5kcEA4V8qKEUc9B=6WQZvdqaHz4XHBTA@mail.gmail.com","subject":"Re: [WIP RFC PATCH v2 5/5] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-28T07:53:59Z","receivedAt":"2019-02-28T07:54:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 27 2019, Matheus Tavares Bernardino wrote:\n\n> On Tue, Feb 26, 2019 at 9:32 AM Duy Nguyen wrote:\n>>\n>> On Tue, Feb 26, 2019 at 12:18 PM Matheus Tavares\n>> wrote:\n>> >\n>> > Replace usage of opendir/readdir/closedir API to traverse directories\n>> > recursively, at copy_or_link_directory function, by the dir-iterator\n>> > API. This simplifies the code and avoid recursive calls to\n>> > copy_or_link_directory.\n>> >\n>> > This process also makes copy_or_link_directory call die() in case of an\n>> > error on readdir or stat, inside dir_iterator_advance. Previously it\n>> > would just print a warning for errors on stat and ignore errors on\n>> > readdir, which isn't nice because a local git clone would end up\n>> > successfully even though the .git/objects copy didn't fully succeeded.\n>> >\n>> > Signed-off-by: Matheus Tavares \n>> > ---\n>> > I can also make the change described in the last paragraph in a separate\n>> > patch before this one, but I would have to undo it in this patch because\n>> > dir-iterator already implements it. So, IMHO, it would be just noise\n>> > and not worthy.\n>> >\n>> > builtin/clone.c | 45 +++++++++++++++++++++++----------------------\n>> > 1 file changed, 23 insertions(+), 22 deletions(-)\n>> >\n>> > diff --git a/builtin/clone.c b/builtin/clone.c\n>> > index fd580fa98d..b23ba64c94 100644\n>> > --- a/builtin/clone.c\n>> > +++ b/builtin/clone.c\n>> > @@ -23,6 +23,8 @@\n>> > #include \"transport.h\"\n>> > #include \"strbuf.h\"\n>> > #include \"dir.h\"\n>> > +#include \"dir-iterator.h\"\n>> > +#include \"iterator.h\"\n>> > #include \"sigchain.h\"\n>> > #include \"branch.h\"\n>> > #include \"remote.h\"\n>> > @@ -411,42 +413,37 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n>> > }\n>> >\n>> > static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n>> > - const char *src_repo, int src_baselen)\n>> > + const char *src_repo)\n>> > {\n>> > - struct dirent *de;\n>> > - struct stat buf;\n>> > int src_len, dest_len;\n>> > - DIR *dir;\n>> > -\n>> > - dir = opendir(src->buf);\n>> > - if (!dir)\n>> > - die_errno(_(\"failed to open '%s'\"), src->buf);\n>> > + struct dir_iterator *iter;\n>> > + int iter_status;\n>> > + struct stat st;\n>> > + unsigned flags;\n>> >\n>> > mkdir_if_missing(dest->buf, 0777);\n>> >\n>> > + flags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n>> > + iter = dir_iterator_begin(src->buf, flags);\n>> > +\n>> > strbuf_addch(src, '/');\n>> > src_len = src->len;\n>> > strbuf_addch(dest, '/');\n>> > dest_len = dest->len;\n>> >\n>> > - while ((de = readdir(dir)) != NULL) {\n>> > + while ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n>> > strbuf_setlen(src, src_len);\n>> > - strbuf_addstr(src, de->d_name);\n>> > + strbuf_addstr(src, iter->relative_path);\n>> > strbuf_setlen(dest, dest_len);\n>> > - strbuf_addstr(dest, de->d_name);\n>> > - if (stat(src->buf, &buf)) {\n>> > - warning (_(\"failed to stat %s\\n\"), src->buf);\n>> > - continue;\n>> > - }\n>> > - if (S_ISDIR(buf.st_mode)) {\n>> > - if (!is_dot_or_dotdot(de->d_name))\n>> > - copy_or_link_directory(src, dest,\n>> > - src_repo, src_baselen);\n>> > + strbuf_addstr(dest, iter->relative_path);\n>> > +\n>> > + if (S_ISDIR(iter->st.st_mode)) {\n>> > + mkdir_if_missing(dest->buf, 0777);\n>>\n>> I wonder if this mkdir_if_missing is sufficient. What if you have to\n>> create multiple directories?\n>>\n>> Let's say the first advance, we hit \"a\". The the second advance we hit\n>> directory \"b/b/b/b\", we would need to mkdir recursively and something\n>> like safe_create_leading_directories() would be a better fit.\n>>\n>> I'm not sure if it can happen though. I haven't re-read dir-iterator\n>> code carefully.\n>>\n>> > continue;\n>> > }\n>> >\n>> > /* Files that cannot be copied bit-for-bit... */\n>> > - if (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n>> > + if (!strcmp(iter->relative_path, \"info/alternates\")) {\n>>\n>> While we're here, this should be fspathcmp to be friendlier to\n>> case-insensitive filesystems. You probably should fix it in a separate\n>> patch though.\n>>\n>\n> Nice! I will make this change in a separate patch in the series. Thanks!\n>\n>> > copy_alternates(src, dest, src_repo);\n>> > continue;\n>> > }\n>> > @@ -463,7 +460,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n>> > if (copy_file_with_time(dest->buf, src->buf, 0666))\n>> > die_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n>> > }\n>> > - closedir(dir);\n>> > +\n>> > + if (iter_status != ITER_DONE) {\n>> > + strbuf_setlen(src, src_len);\n>> > + die(_(\"failed to iterate over '%s'\"), src->buf);\n>> > + }\n>>\n>> I think you need to abort the iterator even when it returns ITER_DONE.\n>> At least that's how the first caller in files-backend.c does it.\n>>\n>\n> Hm, I don't think so, since dir_iterator_advance() already frees the\n> resources before returning ITER_DONE. Also, I may be wrong, but it\n> doesn't seem to me, that files-backend.c does it. The function\n> files_reflog_iterator_advance() that calls dir_iterator_advance() even\n> sets the dir-iterator pointer to NULL as soon as ITER_DONE is\n> returned.\n\nAs Duy notes you're right about this. Just to add: This pattern is\nusually something we avoid in the git codebase, i.e. we try not to make\nit an error to call the whatever_utility_free() function twice.\n\nSee e.g. stop_progress_msg for such an implementation, i.e. we'll check\nif it's NULL already and exit early, and maybe use FREE_AND_NULL()\ninstead of NULL.\n\nIt means that for the cost of trivial overhead you don't need to worry\nabout double freeing or maintaining a \"was this freed?\" state machine.\n\nNow, whether you want to fix that while you're at it is another matter,\njust pointing out that we usually try to avoid this problem entirely...\n"},{"id":"370381","messageId":"CAHd-oW71N94vBvN4wYQVzXyUZjvVs3Ca9tM3VUEuqB66YNGDtA@mail.gmail.com","threadId":"50591","inReplyTo":"20190226122829.19178-2-avarab@gmail.com","subject":"Re: [RFC PATCH v3 1/5] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-02-28T21:19:19Z","receivedAt":"2019-02-28T21:19:33Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Ævar\n\nI'm finishing the required changes in this series to send a v4, but\nwhen submitting to travis ci, I got some errors on the\nt5604-clone-reference test:\nhttps://travis-ci.org/MatheusBernardino/git/builds/500007587\n\nDo you have any idea why?\n\nBest,\nMatheus Tavares\n\nOn Tue, Feb 26, 2019 at 9:28 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> Add tests for what happens when we locally clone .git/objects\n> directories where some of the loose objects or packs are symlinked, or\n> when when there's unknown files there.\n>\n> I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n> for an earlier and simpler version that hardcoded a SHA-1s.\n>\n> This behavior has been the same for a *long* time, but hasn't been\n> tested for.\n>\n> There's a good post-hoc argument to be made for copying over unknown\n> things, e.g. I'd like a git version that doesn't know about the\n> commit-graph to copy it under \"clone --local\" so a newer git version\n> can make use of it.\n>\n> But the behavior showed where with symlinks seems pretty\n> random. E.g. if \"pack\" is a symlink we end up with two copies of the\n> contents, and only transfer some symlinks as-is.\n>\n> In follow-up commits we'll look at changing some of this behavior, but\n> for now let's just assert it as-is so we'll notice what we'll change\n> later.\n>\n> 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t5604-clone-reference.sh | 142 +++++++++++++++++++++++++++++++++++++\n> 1 file changed, 142 insertions(+)\n>\n> diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n> index 4320082b1b..cb0dc22d14 100755\n> --- a/t/t5604-clone-reference.sh\n> +++ b/t/t5604-clone-reference.sh\n> @@ -221,4 +221,146 @@ test_expect_success 'clone, dissociate from alternates' '\n> ( cd C && git fsck )\n> '\n>\n> +test_expect_success 'setup repo with garbage in objects/*' '\n> + git init S &&\n> + (\n> + cd S &&\n> + test_commit A &&\n> +\n> + cd .git/objects &&\n> + >.some-hidden-file &&\n> + >some-file &&\n> + mkdir .some-hidden-dir &&\n> + >.some-hidden-dir/some-file &&\n> + >.some-hidden-dir/.some-dot-file &&\n> + mkdir some-dir &&\n> + >some-dir/some-file &&\n> + >some-dir/.some-dot-file\n> + )\n> +'\n> +\n> +test_expect_success 'clone a repo with garbage in objects/*' '\n> + for option in --local --no-hardlinks --shared --dissociate\n> + do\n> + git clone $option S S$option || return 1 &&\n> + git -C S$option fsck || return 1\n> + done &&\n> + find S-* -name \"*some*\" | sort >actual &&\n> + cat >expected <<-EOF &&\n> + S--dissociate/.git/objects/.some-hidden-file\n> + S--dissociate/.git/objects/some-dir\n> + S--dissociate/.git/objects/some-dir/.some-dot-file\n> + S--dissociate/.git/objects/some-dir/some-file\n> + S--dissociate/.git/objects/some-file\n> + S--local/.git/objects/.some-hidden-file\n> + S--local/.git/objects/some-dir\n> + S--local/.git/objects/some-dir/.some-dot-file\n> + S--local/.git/objects/some-dir/some-file\n> + S--local/.git/objects/some-file\n> + S--no-hardlinks/.git/objects/.some-hidden-file\n> + S--no-hardlinks/.git/objects/some-dir\n> + S--no-hardlinks/.git/objects/some-dir/.some-dot-file\n> + S--no-hardlinks/.git/objects/some-dir/some-file\n> + S--no-hardlinks/.git/objects/some-file\n> + EOF\n> + test_cmp expected actual\n> +'\n> +\n> +test_expect_success SYMLINKS 'setup repo with manually symlinked objects/*' '\n> + git init T &&\n> + (\n> + cd T &&\n> + test_commit A &&\n> + git gc &&\n> + (\n> + cd .git/objects &&\n> + mv pack packs &&\n> + ln -s packs pack\n> + ) &&\n> + test_commit B &&\n> + (\n> + cd .git/objects &&\n> + find ?? -type d >loose-dirs &&\n> + last_loose=$(tail -n 1 loose-dirs) &&\n> + mv $last_loose a-loose-dir &&\n> + ln -s a-loose-dir $last_loose &&\n> + first_loose=$(head -n 1 loose-dirs) &&\n> + (\n> + cd $first_loose &&\n> + obj=$(ls *) &&\n> + mv $obj ../an-object &&\n> + ln -s ../an-object $obj\n> + ) &&\n> + find . -type f | sort >../../../T.objects-files.raw &&\n> + find . -type l | sort >../../../T.objects-links.raw\n> + )\n> + ) &&\n> + git -C T fsck &&\n> + git -C T rev-list --all --objects >T.objects\n> +'\n> +\n> +\n> +test_expect_success SYMLINKS 'clone repo with symlinked objects/*' '\n> + for option in --local --no-hardlinks --shared --dissociate\n> + do\n> + git clone $option T T$option || return 1 &&\n> + git -C T$option fsck || return 1 &&\n> + git -C T$option rev-list --all --objects >T$option.objects &&\n> + test_cmp T.objects T$option.objects &&\n> + (\n> + cd T$option/.git/objects &&\n> + find . -type f | sort >../../../T$option.objects-files.raw &&\n> + find . -type l | sort >../../../T$option.objects-links.raw\n> + )\n> + done &&\n> +\n> + for raw in $(ls T*.raw)\n> + do\n> + sed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" <$raw >$raw.de-sha || return 1\n> + done &&\n> +\n> + cat >expected-files <<-EOF &&\n> + ./Y/Z\n> + ./a-loose-dir/Z\n> + ./an-object\n> + ./Y/Z\n> + ./info/packs\n> + ./loose-dirs\n> + ./pack/pack-Z.idx\n> + ./pack/pack-Z.pack\n> + ./packs/pack-Z.idx\n> + ./packs/pack-Z.pack\n> + EOF\n> + cat >expected-links <<-EOF &&\n> + ./Y/Z\n> + EOF\n> + for option in --local --dissociate\n> + do\n> + test_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n> + test_cmp expected-links T$option.objects-links.raw.de-sha || return 1\n> + done &&\n> +\n> + cat >expected-files <<-EOF &&\n> + ./Y/Z\n> + ./Y/Z\n> + ./a-loose-dir/Z\n> + ./an-object\n> + ./Y/Z\n> + ./info/packs\n> + ./loose-dirs\n> + ./pack/pack-Z.idx\n> + ./pack/pack-Z.pack\n> + ./packs/pack-Z.idx\n> + ./packs/pack-Z.pack\n> + EOF\n> + test_cmp expected-files T--no-hardlinks.objects-files.raw.de-sha &&\n> + test_must_be_empty T--no-hardlinks.objects-links.raw.de-sha &&\n> +\n> + cat >expected-files <<-EOF &&\n> + ./info/alternates\n> + EOF\n> + test_cmp expected-files T--shared.objects-files.raw &&\n> + test_must_be_empty T--shared.objects-links.raw\n> +'\n> +\n> test_done\n> --\n> 2.21.0.rc2.261.ga7da99ff1b\n>\n"},{"id":"370417","messageId":"87o96uvh4x.fsf@evledraar.gmail.com","threadId":"50591","inReplyTo":"CAHd-oW71N94vBvN4wYQVzXyUZjvVs3Ca9tM3VUEuqB66YNGDtA@mail.gmail.com","subject":"Re: [RFC PATCH v3 1/5] clone: test for our behavior on odd objects/* content","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-01T13:49:50Z","receivedAt":"2019-03-01T13:49:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 28 2019, Matheus Tavares Bernardino wrote:\n\n> Hi, Ævar\n>\n> I'm finishing the required changes in this series to send a v4, but\n> when submitting to travis ci, I got some errors on the\n> t5604-clone-reference test:\n> https://travis-ci.org/MatheusBernardino/git/builds/500007587\n\nI don't have access to an OSX box, but could reproduce the failure on\nNetBSD.\n\nIt's because there link() when faced with a symlink behaves\ndifferently. On GNU/Linux link()-ing a symlink will produce another\nsymlink like it, on NetBSD (and presumably OSX) doing that will produce\na hardlink to the file the existing symlink points to.\n\nI've pushed out a version of mine here which you might want to pull in:\nhttps://github.com/git/git/compare/master...avar:clone-dir-iterator-3\n\nI.e. this whole thing is silly, but just preserving the notion that\nwe're not going to introduce behavior changes as we're refactoring.\n\nSo it adds a commit right after the tests I added to detect this case,\nand use symlink() or link() as appropriate instead of link().\n\nThere's then a commit at the end you might want to squash in that\nreproduces this behavior on top of your iterator refactoring.\n\nOf course the DIR_ITERATOR_FOLLOW_SYMLINKS flag at this point is rather\nsilly. We're telling it to stat(), and then end up needing both stat()\nand lstat() data.\n\nI'm starting to think that this interface which previously only had one\ncaller, but now has two exists at the wrong abstraction level. I.e. it\nitself needs to call lstat(). Seems sensible to always do that and leave\nit to the caller to call stat() if they need, as I believe Duy pointed\nout. Also noticed that dir-iterator.h still has a comment to the effect\nthat it'll call \"lstat()\", even though we now have a \"stat() or\nlstat()?\" flag.\n\n> On Tue, Feb 26, 2019 at 9:28 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> Add tests for what happens when we locally clone .git/objects\n>> directories where some of the loose objects or packs are symlinked, or\n>> when when there's unknown files there.\n>>\n>> I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n>> for an earlier and simpler version that hardcoded a SHA-1s.\n>>\n>> This behavior has been the same for a *long* time, but hasn't been\n>> tested for.\n>>\n>> There's a good post-hoc argument to be made for copying over unknown\n>> things, e.g. I'd like a git version that doesn't know about the\n>> commit-graph to copy it under \"clone --local\" so a newer git version\n>> can make use of it.\n>>\n>> But the behavior showed where with symlinks seems pretty\n>> random. E.g. if \"pack\" is a symlink we end up with two copies of the\n>> contents, and only transfer some symlinks as-is.\n>>\n>> In follow-up commits we'll look at changing some of this behavior, but\n>> for now let's just assert it as-is so we'll notice what we'll change\n>> later.\n>>\n>> 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n>>\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> t/t5604-clone-reference.sh | 142 +++++++++++++++++++++++++++++++++++++\n>> 1 file changed, 142 insertions(+)\n>>\n>> diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n>> index 4320082b1b..cb0dc22d14 100755\n>> --- a/t/t5604-clone-reference.sh\n>> +++ b/t/t5604-clone-reference.sh\n>> @@ -221,4 +221,146 @@ test_expect_success 'clone, dissociate from alternates' '\n>> ( cd C && git fsck )\n>> '\n>>\n>> +test_expect_success 'setup repo with garbage in objects/*' '\n>> + git init S &&\n>> + (\n>> + cd S &&\n>> + test_commit A &&\n>> +\n>> + cd .git/objects &&\n>> + >.some-hidden-file &&\n>> + >some-file &&\n>> + mkdir .some-hidden-dir &&\n>> + >.some-hidden-dir/some-file &&\n>> + >.some-hidden-dir/.some-dot-file &&\n>> + mkdir some-dir &&\n>> + >some-dir/some-file &&\n>> + >some-dir/.some-dot-file\n>> + )\n>> +'\n>> +\n>> +test_expect_success 'clone a repo with garbage in objects/*' '\n>> + for option in --local --no-hardlinks --shared --dissociate\n>> + do\n>> + git clone $option S S$option || return 1 &&\n>> + git -C S$option fsck || return 1\n>> + done &&\n>> + find S-* -name \"*some*\" | sort >actual &&\n>> + cat >expected <<-EOF &&\n>> + S--dissociate/.git/objects/.some-hidden-file\n>> + S--dissociate/.git/objects/some-dir\n>> + S--dissociate/.git/objects/some-dir/.some-dot-file\n>> + S--dissociate/.git/objects/some-dir/some-file\n>> + S--dissociate/.git/objects/some-file\n>> + S--local/.git/objects/.some-hidden-file\n>> + S--local/.git/objects/some-dir\n>> + S--local/.git/objects/some-dir/.some-dot-file\n>> + S--local/.git/objects/some-dir/some-file\n>> + S--local/.git/objects/some-file\n>> + S--no-hardlinks/.git/objects/.some-hidden-file\n>> + S--no-hardlinks/.git/objects/some-dir\n>> + S--no-hardlinks/.git/objects/some-dir/.some-dot-file\n>> + S--no-hardlinks/.git/objects/some-dir/some-file\n>> + S--no-hardlinks/.git/objects/some-file\n>> + EOF\n>> + test_cmp expected actual\n>> +'\n>> +\n>> +test_expect_success SYMLINKS 'setup repo with manually symlinked objects/*' '\n>> + git init T &&\n>> + (\n>> + cd T &&\n>> + test_commit A &&\n>> + git gc &&\n>> + (\n>> + cd .git/objects &&\n>> + mv pack packs &&\n>> + ln -s packs pack\n>> + ) &&\n>> + test_commit B &&\n>> + (\n>> + cd .git/objects &&\n>> + find ?? -type d >loose-dirs &&\n>> + last_loose=$(tail -n 1 loose-dirs) &&\n>> + mv $last_loose a-loose-dir &&\n>> + ln -s a-loose-dir $last_loose &&\n>> + first_loose=$(head -n 1 loose-dirs) &&\n>> + (\n>> + cd $first_loose &&\n>> + obj=$(ls *) &&\n>> + mv $obj ../an-object &&\n>> + ln -s ../an-object $obj\n>> + ) &&\n>> + find . -type f | sort >../../../T.objects-files.raw &&\n>> + find . -type l | sort >../../../T.objects-links.raw\n>> + )\n>> + ) &&\n>> + git -C T fsck &&\n>> + git -C T rev-list --all --objects >T.objects\n>> +'\n>> +\n>> +\n>> +test_expect_success SYMLINKS 'clone repo with symlinked objects/*' '\n>> + for option in --local --no-hardlinks --shared --dissociate\n>> + do\n>> + git clone $option T T$option || return 1 &&\n>> + git -C T$option fsck || return 1 &&\n>> + git -C T$option rev-list --all --objects >T$option.objects &&\n>> + test_cmp T.objects T$option.objects &&\n>> + (\n>> + cd T$option/.git/objects &&\n>> + find . -type f | sort >../../../T$option.objects-files.raw &&\n>> + find . -type l | sort >../../../T$option.objects-links.raw\n>> + )\n>> + done &&\n>> +\n>> + for raw in $(ls T*.raw)\n>> + do\n>> + sed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" <$raw >$raw.de-sha || return 1\n>> + done &&\n>> +\n>> + cat >expected-files <<-EOF &&\n>> + ./Y/Z\n>> + ./a-loose-dir/Z\n>> + ./an-object\n>> + ./Y/Z\n>> + ./info/packs\n>> + ./loose-dirs\n>> + ./pack/pack-Z.idx\n>> + ./pack/pack-Z.pack\n>> + ./packs/pack-Z.idx\n>> + ./packs/pack-Z.pack\n>> + EOF\n>> + cat >expected-links <<-EOF &&\n>> + ./Y/Z\n>> + EOF\n>> + for option in --local --dissociate\n>> + do\n>> + test_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n>> + test_cmp expected-links T$option.objects-links.raw.de-sha || return 1\n>> + done &&\n>> +\n>> + cat >expected-files <<-EOF &&\n>> + ./Y/Z\n>> + ./Y/Z\n>> + ./a-loose-dir/Z\n>> + ./an-object\n>> + ./Y/Z\n>> + ./info/packs\n>> + ./loose-dirs\n>> + ./pack/pack-Z.idx\n>> + ./pack/pack-Z.pack\n>> + ./packs/pack-Z.idx\n>> + ./packs/pack-Z.pack\n>> + EOF\n>> + test_cmp expected-files T--no-hardlinks.objects-files.raw.de-sha &&\n>> + test_must_be_empty T--no-hardlinks.objects-links.raw.de-sha &&\n>> +\n>> + cat >expected-files <<-EOF &&\n>> + ./info/alternates\n>> + EOF\n>> + test_cmp expected-files T--shared.objects-files.raw &&\n>> + test_must_be_empty T--shared.objects-links.raw\n>> +'\n>> +\n>> test_done\n>> --\n>> 2.21.0.rc2.261.ga7da99ff1b\n>>\n"},{"id":"371325","messageId":"2e2b8ac6-86e8-1820-89fa-2b04a9ce8151@usp.br","threadId":"50591","inReplyTo":"87o96uvh4x.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH v3 1/5] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-13T03:17:47Z","receivedAt":"2019-03-13T03:17:55Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Ævar\n\nFirst of all, I must apologize for my very late reply. I just got back \nfrom a trip and only now have been able to look again at this series.\n\nOn Fri, Mar 1, 2019 at 10:49 AM Ævar Arnfjörð Bjarmason \n wrote:\n >\n >\n > On Thu, Feb 28 2019, Matheus Tavares Bernardino wrote:\n >\n > > Hi, Ævar\n > >\n > > I'm finishing the required changes in this series to send a v4, but\n > > when submitting to travis ci, I got some errors on the\n > > t5604-clone-reference test:\n > > https://travis-ci.org/MatheusBernardino/git/builds/500007587\n >\n > I don't have access to an OSX box, but could reproduce the failure on\n > NetBSD.\n >\n > It's because there link() when faced with a symlink behaves\n > differently. On GNU/Linux link()-ing a symlink will produce another\n > symlink like it, on NetBSD (and presumably OSX) doing that will produce\n > a hardlink to the file the existing symlink points to.\n\nHm, interesting. I installed NetBSD here and played with it a little: It \nseems that the inconsistency comes from the fact that link() follows \nsymlinks on NetBSD but not on Linux. i.e., if you have a file \"C\" \nlink()-ed to a file \"B\" which, in turn, is a symlink to a file \"A\", \nrunning \"ls -li A B C\" we can see that:\n- On linux, C points to B's inode\n- On NetBSD, C points to A's inode\n\n > I've pushed out a version of mine here which you might want to pull in:\n > https://github.com/git/git/compare/master...avar:clone-dir-iterator-3\n >\n > I.e. this whole thing is silly, but just preserving the notion that\n > we're not going to introduce behavior changes as we're refactoring.\n >\n > So it adds a commit right after the tests I added to detect this case,\n > and use symlink() or link() as appropriate instead of link().\n\nI think this still modifies [a little] the current behaviour, since \nsymlink() will make dest->buf point to a new inode (which will be a \nsymlink just as src->buf, but on a different inode) while the current \nbehaviour, on Linux, is to have dest->buf being a hardlink to src->buf \n(same inode). I don't know if this sentence got too confuse, but what I \nmeant is that symlink() will make a symlink at dest->buf while link(), \non linux, will make a hardlink to the given symlink.\n\n > There's then a commit at the end you might want to squash in that\n > reproduces this behavior on top of your iterator refactoring.\n >\n > Of course the DIR_ITERATOR_FOLLOW_SYMLINKS flag at this point is rather\n > silly. We're telling it to stat(), and then end up needing both stat()\n > and lstat() data.\n >\n > I'm starting to think that this interface which previously only had one\n > caller, but now has two exists at the wrong abstraction level. I.e. it\n > itself needs to call lstat(). Seems sensible to always do that and leave\n > it to the caller to call stat() if they need, as I believe Duy pointed\n > out.\n\nI see what you mean, but if the caller needs to call stat() itself, in \nthe occurrence of a symlink to a directory, it would have to start a new \ndirectory iteration upon the symlinked dir and its subdirectories (if it \nwants to follow symlinks). This approach could became a little messy, \nIMHO. And just by calling stat() at dir-iterator we already get the \nsymlinked directories iterated \"for free\", without having to modify \nanything else in the code. So I still think it is a good idea to have \nthe DIR_ITERATOR_FOLLOW_SYMLINKS flag at dir-iterator, making it call \nstat() instead of lstat().\n\n > Also noticed that dir-iterator.h still has a comment to the effect\n > that it'll call \"lstat()\", even though we now have a \"stat() or\n > lstat()?\" flag.\n\nThanks for noticing it, I will fix it in v4.\n\n > > On Tue, Feb 26, 2019 at 9:28 AM Ævar Arnfjörð Bjarmason\n > > wrote:\n > >>\n > >> Add tests for what happens when we locally clone .git/objects\n > >> directories where some of the loose objects or packs are symlinked, or\n > >> when when there's unknown files there.\n > >>\n > >> I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n > >> for an earlier and simpler version that hardcoded a SHA-1s.\n > >>\n > >> This behavior has been the same for a *long* time, but hasn't been\n > >> tested for.\n > >>\n > >> There's a good post-hoc argument to be made for copying over unknown\n > >> things, e.g. I'd like a git version that doesn't know about the\n > >> commit-graph to copy it under \"clone --local\" so a newer git version\n > >> can make use of it.\n > >>\n > >> But the behavior showed where with symlinks seems pretty\n > >> random. E.g. if \"pack\" is a symlink we end up with two copies of the\n > >> contents, and only transfer some symlinks as-is.\n > >>\n > >> In follow-up commits we'll look at changing some of this behavior, but\n > >> for now let's just assert it as-is so we'll notice what we'll change\n > >> later.\n > >>\n > >> 1. \nhttps://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n > >>\n > >> Signed-off-by: Ævar Arnfjörð Bjarmason \n > >> ---\n > >> t/t5604-clone-reference.sh | 142 \n+++++++++++++++++++++++++++++++++++++\n > >> 1 file changed, 142 insertions(+)\n > >>\n > >> diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n > >> index 4320082b1b..cb0dc22d14 100755\n > >> --- a/t/t5604-clone-reference.sh\n > >> +++ b/t/t5604-clone-reference.sh\n > >> @@ -221,4 +221,146 @@ test_expect_success 'clone, dissociate from \nalternates' '\n > >> ( cd C && git fsck )\n > >> '\n > >>\n > >> +test_expect_success 'setup repo with garbage in objects/*' '\n > >> + git init S &&\n > >> + (\n > >> + cd S &&\n > >> + test_commit A &&\n > >> +\n > >> + cd .git/objects &&\n > >> + >.some-hidden-file &&\n > >> + >some-file &&\n > >> + mkdir .some-hidden-dir &&\n > >> + >.some-hidden-dir/some-file &&\n > >> + >.some-hidden-dir/.some-dot-file &&\n > >> + mkdir some-dir &&\n > >> + >some-dir/some-file &&\n > >> + >some-dir/.some-dot-file\n > >> + )\n > >> +'\n > >> +\n > >> +test_expect_success 'clone a repo with garbage in objects/*' '\n > >> + for option in --local --no-hardlinks --shared --dissociate\n > >> + do\n > >> + git clone $option S S$option || return 1 &&\n > >> + git -C S$option fsck || return 1\n > >> + done &&\n > >> + find S-* -name \"*some*\" | sort >actual &&\n > >> + cat >expected <<-EOF &&\n > >> + S--dissociate/.git/objects/.some-hidden-file\n > >> + S--dissociate/.git/objects/some-dir\n > >> + S--dissociate/.git/objects/some-dir/.some-dot-file\n > >> + S--dissociate/.git/objects/some-dir/some-file\n > >> + S--dissociate/.git/objects/some-file\n > >> + S--local/.git/objects/.some-hidden-file\n > >> + S--local/.git/objects/some-dir\n > >> + S--local/.git/objects/some-dir/.some-dot-file\n > >> + S--local/.git/objects/some-dir/some-file\n > >> + S--local/.git/objects/some-file\n > >> + S--no-hardlinks/.git/objects/.some-hidden-file\n > >> + S--no-hardlinks/.git/objects/some-dir\n > >> + S--no-hardlinks/.git/objects/some-dir/.some-dot-file\n > >> + S--no-hardlinks/.git/objects/some-dir/some-file\n > >> + S--no-hardlinks/.git/objects/some-file\n > >> + EOF\n > >> + test_cmp expected actual\n > >> +'\n > >> +\n > >> +test_expect_success SYMLINKS 'setup repo with manually symlinked \nobjects/*' '\n > >> + git init T &&\n > >> + (\n > >> + cd T &&\n > >> + test_commit A &&\n > >> + git gc &&\n > >> + (\n > >> + cd .git/objects &&\n > >> + mv pack packs &&\n > >> + ln -s packs pack\n > >> + ) &&\n > >> + test_commit B &&\n > >> + (\n > >> + cd .git/objects &&\n > >> + find ?? -type d >loose-dirs &&\n > >> + last_loose=$(tail -n 1 loose-dirs) &&\n > >> + mv $last_loose a-loose-dir &&\n > >> + ln -s a-loose-dir $last_loose &&\n > >> + first_loose=$(head -n 1 loose-dirs) &&\n > >> + (\n > >> + cd $first_loose &&\n > >> + obj=$(ls *) &&\n > >> + mv $obj ../an-object &&\n > >> + ln -s ../an-object $obj\n > >> + ) &&\n > >> + find . -type f | sort \n >../../../T.objects-files.raw &&\n > >> + find . -type l | sort \n >../../../T.objects-links.raw\n > >> + )\n > >> + ) &&\n > >> + git -C T fsck &&\n > >> + git -C T rev-list --all --objects >T.objects\n > >> +'\n > >> +\n > >> +\n > >> +test_expect_success SYMLINKS 'clone repo with symlinked objects/*' '\n > >> + for option in --local --no-hardlinks --shared --dissociate\n > >> + do\n > >> + git clone $option T T$option || return 1 &&\n > >> + git -C T$option fsck || return 1 &&\n > >> + git -C T$option rev-list --all --objects \n >T$option.objects &&\n > >> + test_cmp T.objects T$option.objects &&\n > >> + (\n > >> + cd T$option/.git/objects &&\n > >> + find . -type f | sort \n >../../../T$option.objects-files.raw &&\n > >> + find . -type l | sort \n >../../../T$option.objects-links.raw\n > >> + )\n > >> + done &&\n > >> +\n > >> + for raw in $(ls T*.raw)\n > >> + do\n > >> + sed -e \"s!/..\\$!/X!; s!/../!/Y/!; \ns![0-9a-f]\\{38,\\}!Z!\" <$raw >$raw.de-sha || return 1\n > >> + done &&\n > >> +\n > >> + cat >expected-files <<-EOF &&\n > >> + ./Y/Z\n > >> + ./a-loose-dir/Z\n > >> + ./an-object\n > >> + ./Y/Z\n > >> + ./info/packs\n > >> + ./loose-dirs\n > >> + ./pack/pack-Z.idx\n > >> + ./pack/pack-Z.pack\n > >> + ./packs/pack-Z.idx\n > >> + ./packs/pack-Z.pack\n > >> + EOF\n > >> + cat >expected-links <<-EOF &&\n > >> + ./Y/Z\n > >> + EOF\n > >> + for option in --local --dissociate\n > >> + do\n > >> + test_cmp expected-files \nT$option.objects-files.raw.de-sha || return 1 &&\n > >> + test_cmp expected-links \nT$option.objects-links.raw.de-sha || return 1\n > >> + done &&\n > >> +\n > >> + cat >expected-files <<-EOF &&\n > >> + ./Y/Z\n > >> + ./Y/Z\n > >> + ./a-loose-dir/Z\n > >> + ./an-object\n > >> + ./Y/Z\n > >> + ./info/packs\n > >> + ./loose-dirs\n > >> + ./pack/pack-Z.idx\n > >> + ./pack/pack-Z.pack\n > >> + ./packs/pack-Z.idx\n > >> + ./packs/pack-Z.pack\n > >> + EOF\n > >> + test_cmp expected-files \nT--no-hardlinks.objects-files.raw.de-sha &&\n > >> + test_must_be_empty T--no-hardlinks.objects-links.raw.de-sha &&\n > >> +\n > >> + cat >expected-files <<-EOF &&\n > >> + ./info/alternates\n > >> + EOF\n > >> + test_cmp expected-files T--shared.objects-files.raw &&\n > >> + test_must_be_empty T--shared.objects-links.raw\n > >> +'\n > >> +\n > >> test_done\n > >> --\n > >> 2.21.0.rc2.261.ga7da99ff1b\n > >>\n"},{"id":"372285","messageId":"20190322232237.13293-1-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190226122829.19178-1-avarab@gmail.com","subject":"[GSoC][PATCH v4 0/7] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:30Z","receivedAt":"2019-03-22T23:22:51Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This patchset contains:\n- a replacement of explicit recursive dir iteration at\n copy_or_link_directory for the dir-iterator API;\n- some refactoring and behaviour changes at local clone, mainly to\n take care of symlinks and hidden files at .git/objects; and\n- tests for this type of files\n\nChanges since v3 includes:\n- Addressed Duy's and Ævar's comments and suggestions in v2,\n including but not limited to:\n - Add patch to replace strcmp for fspathcmp\n - Code comments refactoring\n - Unident snippet at mkdir_if_missing\n- Made t5604 added subtests pass under GIT_TEST_MULTI_PACK_INDEX=1\n and GIT_TEST_COMMIT_GRAPH=1\n- Re-implemented patch 2 with linkat(), to be simpler and have a safer\n behaviour when clonning repos with symlinks at .git/objects\n- Split first patch's tests into patches 1 and 2, tweaked it a little\n to reflect the previous item changes, and replaced some usages of the\n string 'link' for 'symlink' just to avoid confusion with 'hardlinks'\n which are also known just by 'links'.\n\nv3: https://public-inbox.org/git/20190226122829.19178-1-avarab@gmail.com/\n\nMatheus Tavares (6):\n clone: better handle symlinked files at .git/objects/\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n clone: Replace strcmp by fspathcmp\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n builtin/clone.c | 72 ++++++++++---------\n dir-iterator.c | 28 +++++++-\n dir-iterator.h | 39 +++++++++--\n refs/files-backend.c | 2 +-\n t/t5604-clone-reference.sh | 137 +++++++++++++++++++++++++++++++++++++\n 5 files changed, 236 insertions(+), 42 deletions(-)\n\n-- \n2.20.1\n\n"},{"id":"372286","messageId":"20190322232237.13293-2-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:31Z","receivedAt":"2019-03-22T23:22:54Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nAdd tests for what happens when we perform a local clone on a repo\ncontaining odd files at .git/object directory, such as symlinks to other\ndirs, or unknown files.\n\nI'm bending over backwards here to avoid a SHA1 dependency. See [1]\nfor an earlier and simpler version that hardcoded a SHA-1s.\n\nThis behavior has been the same for a *long* time, but hasn't been\ntested for.\n\nThere's a good post-hoc argument to be made for copying over unknown\nthings, e.g. I'd like a git version that doesn't know about the\ncommit-graph to copy it under \"clone --local\" so a newer git version\ncan make use of it.\n\nIn follow-up commits we'll look at changing some of this behavior, but\nfor now let's just assert it as-is so we'll notice what we'll change\nlater.\n\n1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Matheus Tavares \nHelped-by: Matheus Tavares \n---\n t/t5604-clone-reference.sh | 116 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 116 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..708b1a2c66 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,120 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success 'setup repo with garbage in objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\n+\t\tcd .git/objects &&\n+\t\t>.some-hidden-file &&\n+\t\t>some-file &&\n+\t\tmkdir .some-hidden-dir &&\n+\t\t>.some-hidden-dir/some-file &&\n+\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\tmkdir some-dir &&\n+\t\t>some-dir/some-file &&\n+\t\t>some-dir/.some-dot-file\n+\t)\n+'\n+\n+test_expect_success 'clone a repo with garbage in objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+\tgit init T &&\n+\t(\n+\t\tcd T &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\t(\n+\t\t\tcd .git/objects &&\n+\t\t\tmv pack packs &&\n+\t\t\tln -s packs pack\n+\t\t) &&\n+\t\ttest_commit B &&\n+\t\t(\n+\t\t\tcd .git/objects &&\n+\t\t\tfind ?? -type d >loose-dirs &&\n+\t\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n+\t\t\trm -f loose-dirs &&\n+\t\t\tmv $last_loose a-loose-dir &&\n+\t\t\tln -s a-loose-dir $last_loose &&\n+\t\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\t\techo unknown_content> unknown_file\n+\t\t)\n+\t) &&\n+\tgit -C T fsck &&\n+\tgit -C T rev-list --all --objects >T.objects\n+'\n+\n+\n+test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option T T$option || return 1 &&\n+\t\tgit -C T$option fsck || return 1 &&\n+\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n+\t\ttest_cmp T.objects T$option.objects &&\n+\t\t(\n+\t\t\tcd T$option/.git/objects &&\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t)\n+\tdone &&\n+\n+\tfor raw in $(ls T*.raw)\n+\tdo\n+\t\tsed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" \\\n+\t\t -e \"/multi-pack-index/d\" -e \"/commit-graph/d\" <$raw >$raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./Y/Z\n+\t./info/packs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\t./unknown_file\n+\tEOF\n+\n+\tfor option in --local --dissociate --no-hardlinks\n+\tdo\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./info/alternates\n+\tEOF\n+\ttest_cmp expected-files T--shared.objects-files.raw\n+'\n+\n test_done\n-- \n2.20.1\n\n"},{"id":"372287","messageId":"20190322232237.13293-3-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:32Z","receivedAt":"2019-03-22T23:22:57Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is currently an odd behaviour when locally clonning a repository\nwith symlinks at .git/objects: using --no-hardlinks all symlinks are\ndereferenced but without it Git will try to hardlink the files with the\nlink() function, which has an OS-specific behaviour on symlinks. On OSX\nand NetBSD, it creates a hardlink to the file pointed by the symlink\nwhilst on GNU/Linux, it creates a hardlink to the symlink itself.\n\nOn Manjaro GNU/Linux:\n $ touch a\n $ ln -s a b\n $ link b c\n $ ls -li a b c\n 155 [...] a\n 156 [...] b -> a\n 156 [...] c -> a\n\nBut on NetBSD:\n $ ls -li a b c\n 2609160 [...] a\n 2609164 [...] b -> a\n 2609160 [...] c\n\nIt's not good to have the result of a local clone to be OS-dependent and\nsince the behaviour on GNU/Linux may result in broken symlinks, let's\nre-implement it with linkat() instead of link() using a flag to always\nfollow symlinks and make the hardlink be to the pointed file. With this,\nbesides standardizing the behaviour, no broken symlinks will be\nproduced. Also, add tests for symlinked files at .git/objects/.\n\nNote: Git won't create symlinks at .git/objects itself, but it's better\nto handle this case and be friendly with users who manually create them.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 26 +++++++++++++++++++-------\n 2 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 50bde99618..b76f33c635 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -443,7 +443,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (unlink(dest->buf) && errno != ENOENT)\n \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n \t\tif (!option_no_hardlinks) {\n-\t\t\tif (!link(src->buf, dest->buf))\n+\t\t\tif (!linkat(AT_FDCWD, src->buf, AT_FDCWD, dest->buf, AT_SYMLINK_FOLLOW))\n \t\t\t\tcontinue;\n \t\t\tif (option_local > 0)\n \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 708b1a2c66..76d45f1187 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -266,7 +266,7 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'setup repo with manually symlinked or unknown files at objects/' '\n \tgit init T &&\n \t(\n \t\tcd T &&\n@@ -282,10 +282,18 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n \t\t\tcd .git/objects &&\n \t\t\tfind ?? -type d >loose-dirs &&\n \t\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n-\t\t\trm -f loose-dirs &&\n \t\t\tmv $last_loose a-loose-dir &&\n \t\t\tln -s a-loose-dir $last_loose &&\n+\t\t\tfirst_loose=$(head -n 1 loose-dirs) &&\n+\t\t\trm -f loose-dirs &&\n+\t\t\t(\n+\t\t\t\tcd $first_loose &&\n+\t\t\t\tobj=$(ls *) &&\n+\t\t\t\tmv $obj ../an-object &&\n+\t\t\t\tln -s ../an-object $obj\n+\t\t\t) &&\n \t\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T.objects-symlinks.raw &&\n \t\t\techo unknown_content> unknown_file\n \t\t)\n \t) &&\n@@ -294,7 +302,7 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n '\n \n \n-test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at objects/' '\n \tfor option in --local --no-hardlinks --shared --dissociate\n \tdo\n \t\tgit clone $option T T$option || return 1 &&\n@@ -303,7 +311,8 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t\ttest_cmp T.objects T$option.objects &&\n \t\t(\n \t\t\tcd T$option/.git/objects &&\n-\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n \t\t)\n \tdone &&\n \n@@ -317,6 +326,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./Y/Z\n \t./Y/Z\n \t./a-loose-dir/Z\n+\t./an-object\n \t./Y/Z\n \t./info/packs\n \t./pack/pack-Z.idx\n@@ -326,15 +336,17 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./unknown_file\n \tEOF\n \n-\tfor option in --local --dissociate --no-hardlinks\n+\tfor option in --local --no-hardlinks --dissociate\n \tdo\n-\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n+\t\ttest_must_be_empty T$option.objects-symlinks.raw.de-sha || return 1\n \tdone &&\n \n \tcat >expected-files <<-EOF &&\n \t./info/alternates\n \tEOF\n-\ttest_cmp expected-files T--shared.objects-files.raw\n+\ttest_cmp expected-files T--shared.objects-files.raw &&\n+\ttest_must_be_empty T--shared.objects-symlinks.raw\n '\n \n test_done\n-- \n2.20.1\n\n"},{"id":"372288","messageId":"20190322232237.13293-4-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:33Z","receivedAt":"2019-03-22T23:23:01Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are DIR_ITERATOR_PEDANTIC, which makes\ndir_iterator_advance abort imediatelly in the case of an error while\ntrying to fetch next entry; and DIR_ITERATOR_FOLLOW_SYMLINKS, which\nmakes the iteration follow symlinks to directories and include its\ncontents in the iteration. These new flags will be used in a subsequent\npatch.\n\nAlso adjust refs/files-backend.c to the new dir_iterator_begin\nsignature.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 28 +++++++++++++++++++++++++---\n dir-iterator.h | 39 +++++++++++++++++++++++++++++++++------\n refs/files-backend.c | 2 +-\n 3 files changed, 59 insertions(+), 10 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..17aca8ea41 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -48,12 +48,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned flags;\n };\n \n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n+\tint ret;\n \n \twhile (1) {\n \t\tstruct dir_iterator_level *level =\n@@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n \t\t\t\twarning(\"error opening directory %s: %s\",\n \t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t/* Popping the level is handled below */\n@@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading directory %s: %s\",\n \t\t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t} else if (closedir(level->dir))\n@@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\t\tcontinue;\n \n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n+\n+\t\t\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\t\t\tret = stat(iter->base.path.buf, &iter->base.st);\n+\t\t\telse\n+\t\t\t\tret = lstat(iter->base.path.buf, &iter->base.st);\n+\n+\t\t\tif (ret < 0) {\n+\t\t\t\tif (errno != ENOENT) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading path '%s': %s\",\n \t\t\t\t\t\titer->base.path.buf,\n \t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t}\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\treturn ITER_OK;\n \t\t}\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -182,7 +203,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -195,6 +216,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n+\titer->flags = flags;\n \titer->levels_nr = 1;\n \titer->levels[0].initialized = 0;\n \ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..890d5d8dbb 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -19,7 +19,7 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * struct iterator *iter = dir_iterator_begin(path, 0);\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n@@ -40,6 +40,20 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error while trying to fetch the next entry, which is\n+ * to emit a warning and keep going. With this flag, resouces are\n+ * freed and ITER_ERROR is return immediately.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n+ * directories, i.e., iterate over linked directories' contents.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -54,20 +68,28 @@ struct dir_iterator {\n \t/* The current basename: */\n \tconst char *basename;\n \n-\t/* The result of calling lstat() on path: */\n+\t/*\n+\t * The result of calling lstat() on path or stat(), if the\n+\t * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at\n+\t * dir_iterator's initialization.\n+\t */\n \tstruct stat st;\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. Return a dir_iterator that holds the\n+ * internal state of the iteration.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behaviour.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n@@ -76,6 +98,11 @@ struct dir_iterator *dir_iterator_begin(const char *path);\n * dir_iterator and associated resources and return ITER_ERROR. It is\n * a bug to use iterator or call this function again after it has\n * returned ITER_DONE or ITER_ERROR.\n+ *\n+ * Note that whether dir-iterator will return ITER_ERROR when failing\n+ * to fetch the next entry or just emit a warning and try to fetch the\n+ * next is defined by the 'pedantic' option at dir-iterator's\n+ * initialization.\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex ef053f716c..2ce9783097 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,7 +2143,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\titer->dir_iterator = dir_iterator_begin(sb.buf, 0);\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \n-- \n2.20.1\n\n"},{"id":"372289","messageId":"20190322232237.13293-5-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 4/7] clone: copy hidden paths at local clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:34Z","receivedAt":"2019-03-22T23:23:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make the copy_or_link_directory function no longer skip hidden\ndirectories. This function, used to copy .git/objects, currently skips\nall hidden directories but not hidden files, which is an odd behaviour.\nThe reason for that could be unintentional: probably the intention was\nto skip '.' and '..' only but it ended up accidentally skipping all\ndirectories starting with '.'. Besides being more natural, the new\nbehaviour is more permissive to the user.\n\nAlso adjust tests to reflect this behaviour change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex b76f33c635..60c6780c06 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -428,7 +428,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 76d45f1187..0992baa5ac 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \tdone &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.20.1\n\n"},{"id":"372290","messageId":"20190322232237.13293-6-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 5/7] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:35Z","receivedAt":"2019-03-22T23:23:06Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Extract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help removing\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 60c6780c06..c17bbf1bfc 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -392,6 +392,21 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\tstruct stat st;\n+\n+\tif (!mkdir(pathname, mode))\n+\t\treturn;\n+\n+\tif (errno != EEXIST)\n+\t\tdie_errno(_(\"failed to create directory '%s'\"), pathname);\n+\telse if (stat(pathname, &st))\n+\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\telse if (!S_ISDIR(st.st_mode))\n+\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -404,14 +419,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.20.1\n\n"},{"id":"372291","messageId":"20190322232237.13293-7-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 6/7] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:36Z","receivedAt":"2019-03-22T23:23:10Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoid recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat, inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone would end up\nsuccessfully even though the .git/objects copy didn't fully succeeded.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 44 ++++++++++++++++++++++----------------------\n 1 file changed, 22 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex c17bbf1bfc..4ee45e7862 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -408,42 +410,36 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tunsigned flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -460,7 +456,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -478,7 +478,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.20.1\n\n"},{"id":"372292","messageId":"20190322232237.13293-8-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v4 7/7] clone: Replace strcmp by fspathcmp","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-22T23:22:37Z","receivedAt":"2019-03-22T23:23:13Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace the use of strcmp by fspathcmp at copy_or_link_directory, which\nis more permissive/friendly to case-insensitive file systems.\n\nSigned-off-by: Matheus Tavares \nSuggested-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4ee45e7862..763ad5e31f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -439,7 +439,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n+\t\tif (!fspathcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n-- \n2.20.1\n\n"},{"id":"372374","messageId":"CAHd-oW5Yg=dLfLhVmwkOLC=OW=SYRCXCq+pqn=bKNDY=qydfDg@mail.gmail.com","threadId":"50591","inReplyTo":"20190322232237.13293-2-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-24T18:09:32Z","receivedAt":"2019-03-24T18:09:46Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Mar 22, 2019 at 8:22 PM Matheus Tavares\n wrote:\n>\n> From: Ævar Arnfjörð Bjarmason \n>\n> Add tests for what happens when we perform a local clone on a repo\n> containing odd files at .git/object directory, such as symlinks to other\n> dirs, or unknown files.\n>\n> I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n> for an earlier and simpler version that hardcoded a SHA-1s.\n>\n> This behavior has been the same for a *long* time, but hasn't been\n> tested for.\n>\n> There's a good post-hoc argument to be made for copying over unknown\n> things, e.g. I'd like a git version that doesn't know about the\n> commit-graph to copy it under \"clone --local\" so a newer git version\n> can make use of it.\n>\n> In follow-up commits we'll look at changing some of this behavior, but\n> for now let's just assert it as-is so we'll notice what we'll change\n> later.\n>\n> 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Matheus Tavares \n> Helped-by: Matheus Tavares \n> ---\n> t/t5604-clone-reference.sh | 116 +++++++++++++++++++++++++++++++++++++\n> 1 file changed, 116 insertions(+)\n>\n> diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n> index 4320082b1b..708b1a2c66 100755\n> --- a/t/t5604-clone-reference.sh\n> +++ b/t/t5604-clone-reference.sh\n> @@ -221,4 +221,120 @@ test_expect_success 'clone, dissociate from alternates' '\n> ( cd C && git fsck )\n> '\n>\n> +test_expect_success 'setup repo with garbage in objects/*' '\n> + git init S &&\n> + (\n> + cd S &&\n> + test_commit A &&\n> +\n> + cd .git/objects &&\n> + >.some-hidden-file &&\n> + >some-file &&\n> + mkdir .some-hidden-dir &&\n> + >.some-hidden-dir/some-file &&\n> + >.some-hidden-dir/.some-dot-file &&\n> + mkdir some-dir &&\n> + >some-dir/some-file &&\n> + >some-dir/.some-dot-file\n> + )\n> +'\n> +\n> +test_expect_success 'clone a repo with garbage in objects/*' '\n> + for option in --local --no-hardlinks --shared --dissociate\n> + do\n> + git clone $option S S$option || return 1 &&\n> + git -C S$option fsck || return 1\n> + done &&\n> + find S-* -name \"*some*\" | sort >actual &&\n> + cat >expected <<-EOF &&\n> + S--dissociate/.git/objects/.some-hidden-file\n> + S--dissociate/.git/objects/some-dir\n> + S--dissociate/.git/objects/some-dir/.some-dot-file\n> + S--dissociate/.git/objects/some-dir/some-file\n> + S--dissociate/.git/objects/some-file\n> + S--local/.git/objects/.some-hidden-file\n> + S--local/.git/objects/some-dir\n> + S--local/.git/objects/some-dir/.some-dot-file\n> + S--local/.git/objects/some-dir/some-file\n> + S--local/.git/objects/some-file\n> + S--no-hardlinks/.git/objects/.some-hidden-file\n> + S--no-hardlinks/.git/objects/some-dir\n> + S--no-hardlinks/.git/objects/some-dir/.some-dot-file\n> + S--no-hardlinks/.git/objects/some-dir/some-file\n> + S--no-hardlinks/.git/objects/some-file\n> + EOF\n> + test_cmp expected actual\n> +'\n> +\n> +test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n> + git init T &&\n> + (\n> + cd T &&\n> + test_commit A &&\n> + git gc &&\n> + (\n> + cd .git/objects &&\n> + mv pack packs &&\n> + ln -s packs pack\n> + ) &&\n> + test_commit B &&\n> + (\n> + cd .git/objects &&\n> + find ?? -type d >loose-dirs &&\n> + last_loose=$(tail -n 1 loose-dirs) &&\n> + rm -f loose-dirs &&\n> + mv $last_loose a-loose-dir &&\n> + ln -s a-loose-dir $last_loose &&\n> + find . -type f | sort >../../../T.objects-files.raw &&\n> + echo unknown_content> unknown_file\n> + )\n> + ) &&\n> + git -C T fsck &&\n> + git -C T rev-list --all --objects >T.objects\n> +'\n> +\n> +\n> +test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n> + for option in --local --no-hardlinks --shared --dissociate\n> + do\n> + git clone $option T T$option || return 1 &&\n> + git -C T$option fsck || return 1 &&\n> + git -C T$option rev-list --all --objects >T$option.objects &&\n> + test_cmp T.objects T$option.objects &&\n> + (\n> + cd T$option/.git/objects &&\n> + find . -type f | sort >../../../T$option.objects-files.raw\n> + )\n> + done &&\n> +\n> + for raw in $(ls T*.raw)\n> + do\n> + sed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" \\\n> + -e \"/multi-pack-index/d\" -e \"/commit-graph/d\" <$raw >$raw.de-sha || return 1\n\nÆvar, maybe I'm missing something here, but do we really need the\nfirst sed command (\"s!/..\\$!/X!\") ?\n\n\n> + done &&\n> +\n> + cat >expected-files <<-EOF &&\n> + ./Y/Z\n> + ./Y/Z\n> + ./a-loose-dir/Z\n> + ./Y/Z\n> + ./info/packs\n> + ./pack/pack-Z.idx\n> + ./pack/pack-Z.pack\n> + ./packs/pack-Z.idx\n> + ./packs/pack-Z.pack\n> + ./unknown_file\n> + EOF\n> +\n> + for option in --local --dissociate --no-hardlinks\n> + do\n> + test_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n> + done &&\n> +\n> + cat >expected-files <<-EOF &&\n> + ./info/alternates\n> + EOF\n> + test_cmp expected-files T--shared.objects-files.raw\n> +'\n> +\n> test_done\n> --\n> 2.20.1\n>\n"},{"id":"372377","messageId":"20190324205624.GJ22459@szeder.dev","threadId":"50591","inReplyTo":"20190322232237.13293-2-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-24T20:56:24Z","receivedAt":"2019-03-24T20:56:33Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Mar 22, 2019 at 08:22:31PM -0300, Matheus Tavares wrote:\n> From: Ævar Arnfjörð Bjarmason \n> \n> Add tests for what happens when we perform a local clone on a repo\n> containing odd files at .git/object directory, such as symlinks to other\n> dirs, or unknown files.\n> \n> I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n\ns/SHA1/SHA-1/\n\n> for an earlier and simpler version that hardcoded a SHA-1s.\n\ns/SHA-1s/SHA-1/ or s/a SHA-1s/SHA-1s/, depending on what you consider\nmultiple occurrances of the same SHA-1.\n\n> This behavior has been the same for a *long* time, but hasn't been\n> tested for.\n> \n> There's a good post-hoc argument to be made for copying over unknown\n> things, e.g. I'd like a git version that doesn't know about the\n> commit-graph to copy it under \"clone --local\" so a newer git version\n> can make use of it.\n> \n> In follow-up commits we'll look at changing some of this behavior, but\n> for now let's just assert it as-is so we'll notice what we'll change\n> later.\n> \n> 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Matheus Tavares \n> Helped-by: Matheus Tavares \n\n\n> +test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n> +\tgit init T &&\n> +\t(\n> +\t\tcd T &&\n> +\t\ttest_commit A &&\n> +\t\tgit gc &&\n> +\t\t(\n> +\t\t\tcd .git/objects &&\n> +\t\t\tmv pack packs &&\n> +\t\t\tln -s packs pack\n> +\t\t) &&\n> +\t\ttest_commit B &&\n> +\t\t(\n> +\t\t\tcd .git/objects &&\n> +\t\t\tfind ?? -type d >loose-dirs &&\n> +\t\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n> +\t\t\trm -f loose-dirs &&\n> +\t\t\tmv $last_loose a-loose-dir &&\n> +\t\t\tln -s a-loose-dir $last_loose &&\n> +\t\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n> +\t\t\techo unknown_content> unknown_file\n> +\t\t)\n\nPlease drop these inner subshells. They are unnecessary, because the\nouter subshell alone is sufficient to ensure that the test script\nreturns to the original directory if one of the commands were to fail.\n\n> +\t) &&\n> +\tgit -C T fsck &&\n> +\tgit -C T rev-list --all --objects >T.objects\n> +'\n> +\n> +\n> +test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n> +\tfor option in --local --no-hardlinks --shared --dissociate\n> +\tdo\n> +\t\tgit clone $option T T$option || return 1 &&\n> +\t\tgit -C T$option fsck || return 1 &&\n> +\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n> +\t\ttest_cmp T.objects T$option.objects &&\n> +\t\t(\n> +\t\t\tcd T$option/.git/objects &&\n> +\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n> +\t\t)\n\nNit: this might be a bit easier on the eyes when written as\n\n ( \n cd T$option/.git/objects &&\n find . -type f\n ) | sort >T$option.objects-files.raw\n\nbecause it would avoid that '../../../'.\n\n> +\tdone &&\n> +\n> +\tfor raw in $(ls T*.raw)\n> +\tdo\n> +\t\tsed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" \\\n> +\t\t -e \"/multi-pack-index/d\" -e \"/commit-graph/d\" <$raw >$raw.de-sha || return 1\n> +\tdone &&\n> +\n> +\tcat >expected-files <<-EOF &&\n> +\t./Y/Z\n> +\t./Y/Z\n> +\t./a-loose-dir/Z\n> +\t./Y/Z\n> +\t./info/packs\n> +\t./pack/pack-Z.idx\n> +\t./pack/pack-Z.pack\n> +\t./packs/pack-Z.idx\n> +\t./packs/pack-Z.pack\n> +\t./unknown_file\n> +\tEOF\n> +\n> +\tfor option in --local --dissociate --no-hardlinks\n> +\tdo\n> +\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n> +\tdone &&\n> +\n> +\tcat >expected-files <<-EOF &&\n> +\t./info/alternates\n> +\tEOF\n\nPerhaps\n\n echo ./info/alternates >expected-files\n\n> +\ttest_cmp expected-files T--shared.objects-files.raw\n> +'\n> +\n> test_done\n> -- \n> 2.20.1\n> \n"},{"id":"372545","messageId":"CAHd-oW4qZWhHJVnnOpdUnN1J9r9V=LZ+dEdceWD=o==OwTwFRg@mail.gmail.com","threadId":"50591","inReplyTo":"20190324205624.GJ22459@szeder.dev","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-26T19:43:24Z","receivedAt":"2019-03-26T19:43:38Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sun, Mar 24, 2019 at 5:56 PM SZEDER Gábor wrote:\n>\n> On Fri, Mar 22, 2019 at 08:22:31PM -0300, Matheus Tavares wrote:\n> > From: Ævar Arnfjörð Bjarmason \n> >\n> > Add tests for what happens when we perform a local clone on a repo\n> > containing odd files at .git/object directory, such as symlinks to other\n> > dirs, or unknown files.\n> >\n> > I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n>\n> s/SHA1/SHA-1/\n>\n\nThanks, nice catch.\n\n> > for an earlier and simpler version that hardcoded a SHA-1s.\n>\n> s/SHA-1s/SHA-1/ or s/a SHA-1s/SHA-1s/, depending on what you consider\n> multiple occurrances of the same SHA-1.\n>\n\nYes, I think it should be just \"SHA-1s\". Thanks.\n\n> > This behavior has been the same for a *long* time, but hasn't been\n> > tested for.\n> >\n> > There's a good post-hoc argument to be made for copying over unknown\n> > things, e.g. I'd like a git version that doesn't know about the\n> > commit-graph to copy it under \"clone --local\" so a newer git version\n> > can make use of it.\n> >\n> > In follow-up commits we'll look at changing some of this behavior, but\n> > for now let's just assert it as-is so we'll notice what we'll change\n> > later.\n> >\n> > 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n> >\n> > Signed-off-by: Ævar Arnfjörð Bjarmason \n> > Signed-off-by: Matheus Tavares \n> > Helped-by: Matheus Tavares \n>\n>\n> > +test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n> > + git init T &&\n> > + (\n> > + cd T &&\n> > + test_commit A &&\n> > + git gc &&\n> > + (\n> > + cd .git/objects &&\n> > + mv pack packs &&\n> > + ln -s packs pack\n> > + ) &&\n> > + test_commit B &&\n> > + (\n> > + cd .git/objects &&\n> > + find ?? -type d >loose-dirs &&\n> > + last_loose=$(tail -n 1 loose-dirs) &&\n> > + rm -f loose-dirs &&\n> > + mv $last_loose a-loose-dir &&\n> > + ln -s a-loose-dir $last_loose &&\n> > + find . -type f | sort >../../../T.objects-files.raw &&\n> > + echo unknown_content> unknown_file\n> > + )\n>\n> Please drop these inner subshells. They are unnecessary, because the\n> outer subshell alone is sufficient to ensure that the test script\n> returns to the original directory if one of the commands were to fail.\n\nOk!\n\n> > + ) &&\n> > + git -C T fsck &&\n> > + git -C T rev-list --all --objects >T.objects\n> > +'\n> > +\n> > +\n> > +test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n> > + for option in --local --no-hardlinks --shared --dissociate\n> > + do\n> > + git clone $option T T$option || return 1 &&\n> > + git -C T$option fsck || return 1 &&\n> > + git -C T$option rev-list --all --objects >T$option.objects &&\n> > + test_cmp T.objects T$option.objects &&\n> > + (\n> > + cd T$option/.git/objects &&\n> > + find . -type f | sort >../../../T$option.objects-files.raw\n> > + )\n>\n> Nit: this might be a bit easier on the eyes when written as\n>\n> (\n> cd T$option/.git/objects &&\n> find . -type f\n> ) | sort >T$option.objects-files.raw\n>\n> because it would avoid that '../../../'.\n\nSounds good, but in the next patch of this series, another 'find'\nstatement will be added inside this subshell, so I think that change\nis not really possible, unfortunately.\n\n> > + done &&\n> > +\n> > + for raw in $(ls T*.raw)\n> > + do\n> > + sed -e \"s!/..\\$!/X!; s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" \\\n> > + -e \"/multi-pack-index/d\" -e \"/commit-graph/d\" <$raw >$raw.de-sha || return 1\n> > + done &&\n> > +\n> > + cat >expected-files <<-EOF &&\n> > + ./Y/Z\n> > + ./Y/Z\n> > + ./a-loose-dir/Z\n> > + ./Y/Z\n> > + ./info/packs\n> > + ./pack/pack-Z.idx\n> > + ./pack/pack-Z.pack\n> > + ./packs/pack-Z.idx\n> > + ./packs/pack-Z.pack\n> > + ./unknown_file\n> > + EOF\n> > +\n> > + for option in --local --dissociate --no-hardlinks\n> > + do\n> > + test_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n> > + done &&\n> > +\n> > + cat >expected-files <<-EOF &&\n> > + ./info/alternates\n> > + EOF\n>\n> Perhaps\n>\n> echo ./info/alternates >expected-files\n\nIndeed, much simpler. Thanks.\n\n> > + test_cmp expected-files T--shared.objects-files.raw\n> > +'\n> > +\n> > test_done\n> > --\n> > 2.20.1\n> >\n"},{"id":"372650","messageId":"20190328214950.GJ32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"20190322232237.13293-2-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-28T21:49:50Z","receivedAt":"2019-03-28T21:49:55Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/22, Matheus Tavares wrote:\n> From: Ævar Arnfjörð Bjarmason \n> \n> Add tests for what happens when we perform a local clone on a repo\n> containing odd files at .git/object directory, such as symlinks to other\n> dirs, or unknown files.\n> \n> I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n> for an earlier and simpler version that hardcoded a SHA-1s.\n> \n> This behavior has been the same for a *long* time, but hasn't been\n> tested for.\n> \n> There's a good post-hoc argument to be made for copying over unknown\n> things, e.g. I'd like a git version that doesn't know about the\n> commit-graph to copy it under \"clone --local\" so a newer git version\n> can make use of it.\n> \n> In follow-up commits we'll look at changing some of this behavior, but\n> for now let's just assert it as-is so we'll notice what we'll change\n> later.\n> \n> 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Matheus Tavares \n> Helped-by: Matheus Tavares \n\nThe trailers should be in the order things have happened usually. So\nhaving Ævar's S-o-b first makes sense, but the Helped-by should come\nbefore your S-o-b, as you made the changes first before sending out\nthe patch series.\n\nWhen sending someone elses patch in a slightly modified version, it\nmay also be useful to add which parts you changed, as it was done in\ne8dfcace31 (\"poll: use GetTickCount64() to avoid wrap-around issues\",\n2018-10-31) for example.\n\nIirc, the test that is added in this patch does not work on some\nplatforms, notably MacOS. That would mean that we would break\nbisectability at this patch on some platforms if we were to introduce\nit here. Therefore I think it would be better to squash this patch\ninto the next one which fixes these inconsistencies.\n\nNote that I can't test this at the moment, so this concern is only\nbased on previous discussions that I remember. If that's already\naddressed somehow, all the better!\n"},{"id":"372651","messageId":"20190328221049.GK32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"20190322232237.13293-3-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-28T22:10:49Z","receivedAt":"2019-03-28T22:10:55Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/22, Matheus Tavares wrote:\n> There is currently an odd behaviour when locally clonning a repository\n> with symlinks at .git/objects: using --no-hardlinks all symlinks are\n> dereferenced but without it Git will try to hardlink the files with the\n> link() function, which has an OS-specific behaviour on symlinks. On OSX\n> and NetBSD, it creates a hardlink to the file pointed by the symlink\n> whilst on GNU/Linux, it creates a hardlink to the symlink itself.\n> \n> On Manjaro GNU/Linux:\n> $ touch a\n> $ ln -s a b\n> $ link b c\n> $ ls -li a b c\n> 155 [...] a\n> 156 [...] b -> a\n> 156 [...] c -> a\n> \n> But on NetBSD:\n> $ ls -li a b c\n> 2609160 [...] a\n> 2609164 [...] b -> a\n> 2609160 [...] c\n> \n> It's not good to have the result of a local clone to be OS-dependent and\n> since the behaviour on GNU/Linux may result in broken symlinks, let's\n> re-implement it with linkat() instead of link() using a flag to always\n> follow symlinks and make the hardlink be to the pointed file. With this,\n> besides standardizing the behaviour, no broken symlinks will be\n> produced. Also, add tests for symlinked files at .git/objects/.\n> \n> Note: Git won't create symlinks at .git/objects itself, but it's better\n> to handle this case and be friendly with users who manually create them.\n> \n> Signed-off-by: Matheus Tavares \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Co-authored-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/clone.c | 2 +-\n> t/t5604-clone-reference.sh | 26 +++++++++++++++++++-------\n> 2 files changed, 20 insertions(+), 8 deletions(-)\n> \n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 50bde99618..b76f33c635 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -443,7 +443,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> \t\tif (unlink(dest->buf) && errno != ENOENT)\n> \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n> \t\tif (!option_no_hardlinks) {\n> -\t\t\tif (!link(src->buf, dest->buf))\n> +\t\t\tif (!linkat(AT_FDCWD, src->buf, AT_FDCWD, dest->buf, AT_SYMLINK_FOLLOW))\n\nThis line is starting to get a bit long, might be worth breaking it up\nto keep to 80 characters per line.\n\nI notice that we are currently not using 'linkat()' anywhere else in\nour codebase. It looks like it has been introduced in POSIX.1-2008,\nwhich sounds fairly recent by git's standards. So I wonder if this is\nreally supported on all platforms that git is being built on.\n\nI also wonder what would need to be done on Windows if we were to\nintroduce this. I see we define the 'link()' function in\n'compat/mingw.c' for that currently, so I guess something similar\nwould be needed for 'linkat()'. I added Dscho to Cc for Windows\nexpertise.\n\nWhile I agree with the goal of consistency accross all platforms here,\nI don't know if it's actually worth going through the pain of doing\nthat, especially for somewhat of an edge case in local clones.\n\nIf the test in the previous patch passes on all platforms, I'd be okay\nwith just calling the behaviour here undefined, especially as git\nwould never actually create symlinks in the .git/objects directory.\n"},{"id":"372653","messageId":"20190328221910.GL32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"20190322232237.13293-4-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v4 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-28T22:19:10Z","receivedAt":"2019-03-28T22:19:15Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/22, Matheus Tavares wrote:\n> Add the possibility of giving flags to dir_iterator_begin to initialize\n> a dir-iterator with special options.\n> \n> Currently possible flags are DIR_ITERATOR_PEDANTIC, which makes\n> dir_iterator_advance abort imediatelly in the case of an error while\n\ns/imediatelly/immediately/\n\n"},{"id":"372655","messageId":"87sgv69itl.fsf@evledraar.gmail.com","threadId":"50591","inReplyTo":"20190328221049.GK32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-29T08:38:14Z","receivedAt":"2019-03-29T08:38:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 28 2019, Thomas Gummerer wrote:\n\n> On 03/22, Matheus Tavares wrote:\n>> There is currently an odd behaviour when locally clonning a repository\n>> with symlinks at .git/objects: using --no-hardlinks all symlinks are\n>> dereferenced but without it Git will try to hardlink the files with the\n>> link() function, which has an OS-specific behaviour on symlinks. On OSX\n>> and NetBSD, it creates a hardlink to the file pointed by the symlink\n>> whilst on GNU/Linux, it creates a hardlink to the symlink itself.\n>>\n>> On Manjaro GNU/Linux:\n>> $ touch a\n>> $ ln -s a b\n>> $ link b c\n>> $ ls -li a b c\n>> 155 [...] a\n>> 156 [...] b -> a\n>> 156 [...] c -> a\n>>\n>> But on NetBSD:\n>> $ ls -li a b c\n>> 2609160 [...] a\n>> 2609164 [...] b -> a\n>> 2609160 [...] c\n>>\n>> It's not good to have the result of a local clone to be OS-dependent and\n>> since the behaviour on GNU/Linux may result in broken symlinks, let's\n>> re-implement it with linkat() instead of link() using a flag to always\n>> follow symlinks and make the hardlink be to the pointed file. With this,\n>> besides standardizing the behaviour, no broken symlinks will be\n>> produced. Also, add tests for symlinked files at .git/objects/.\n>>\n>> Note: Git won't create symlinks at .git/objects itself, but it's better\n>> to handle this case and be friendly with users who manually create them.\n>>\n>> Signed-off-by: Matheus Tavares \n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> Co-authored-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> builtin/clone.c | 2 +-\n>> t/t5604-clone-reference.sh | 26 +++++++++++++++++++-------\n>> 2 files changed, 20 insertions(+), 8 deletions(-)\n>>\n>> diff --git a/builtin/clone.c b/builtin/clone.c\n>> index 50bde99618..b76f33c635 100644\n>> --- a/builtin/clone.c\n>> +++ b/builtin/clone.c\n>> @@ -443,7 +443,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n>> \t\tif (unlink(dest->buf) && errno != ENOENT)\n>> \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n>> \t\tif (!option_no_hardlinks) {\n>> -\t\t\tif (!link(src->buf, dest->buf))\n>> +\t\t\tif (!linkat(AT_FDCWD, src->buf, AT_FDCWD, dest->buf, AT_SYMLINK_FOLLOW))\n>\n> This line is starting to get a bit long, might be worth breaking it up\n> to keep to 80 characters per line.\n>\n> I notice that we are currently not using 'linkat()' anywhere else in\n> our codebase. It looks like it has been introduced in POSIX.1-2008,\n> which sounds fairly recent by git's standards. So I wonder if this is\n> really supported on all platforms that git is being built on.\n>\n> I also wonder what would need to be done on Windows if we were to\n> introduce this. I see we define the 'link()' function in\n> 'compat/mingw.c' for that currently, so I guess something similar\n> would be needed for 'linkat()'. I added Dscho to Cc for Windows\n> expertise.\n\nFor better of worse this particular quest started because I pointed out\n(with some WIP patches) that for understanding this change we should\ntest whatever we did now, to ensure that the refactoring didn't have\nunintended side-effects.\n\nBut that's a separate question from whether or not we want to keep the\ncurrent behavior.\n\nI think the current behavior is clearly insane, so I think we should\nchange it with some follow-up patches. In particular options like\n--dissociate should clearly (in my mind at least) have behavior similar\nto \"cp -L\", and --local should hardlink to the *target* of the symlink,\nif anything, at least for objects/{??,pack,info}\n\nI think that changes the portability story with linkat(), since it's not\nsomething we should be planning to keep, just an intermediate step so we\ndon't have a gigantic patch that both adds tests, refactors and changes\nthe behavior.\n\n> While I agree with the goal of consistency accross all platforms here,\n> I don't know if it's actually worth going through the pain of doing\n> that, especially for somewhat of an edge case in local clones.\n\nNote that we explicitly clone everything under objects/, including\nrecursively cloning unknown directories and their files.\n\nSo this is not just say about how we handle symlinks that we don't\nexpect now (nothing uses them), but if we want to make the promise that\nnothing in objects/ will ever use symlinks. Or more specifically, that\nif a new version of git starts using it that something doing local\nclones might produce a broken copy of such a repo.\n\nMaybe we'll still say \"we don't care\". Just saying it's a slightly\ndifferent question...\n\n> If the test in the previous patch passes on all platforms, I'd be okay\n> with just calling the behaviour here undefined, especially as git\n> would never actually create symlinks in the .git/objects directory.\n\n"},{"id":"372697","messageId":"CAHd-oW4dpr9bp2fgYzwzz1NcpXGnD3o6phcY8Dek5T6rahMX3Q@mail.gmail.com","threadId":"50591","inReplyTo":"20190328221910.GL32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-29T13:16:01Z","receivedAt":"2019-03-29T13:16:15Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Mar 28, 2019 at 7:19 PM Thomas Gummerer wrote:\n>\n> On 03/22, Matheus Tavares wrote:\n> > Add the possibility of giving flags to dir_iterator_begin to initialize\n> > a dir-iterator with special options.\n> >\n> > Currently possible flags are DIR_ITERATOR_PEDANTIC, which makes\n> > dir_iterator_advance abort imediatelly in the case of an error while\n>\n> s/imediatelly/immediately/\n>\n\nThanks!\n"},{"id":"372703","messageId":"CAHd-oW70VNgYmX9=5x0CwoSD2a0ik2AoJyd7T3xiO4O030c5yQ@mail.gmail.com","threadId":"50591","inReplyTo":"20190328214950.GJ32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-29T14:06:20Z","receivedAt":"2019-03-29T14:06:34Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Mar 28, 2019 at 6:49 PM Thomas Gummerer wrote:\n>\n> On 03/22, Matheus Tavares wrote:\n> > From: Ævar Arnfjörð Bjarmason \n> >\n> > Add tests for what happens when we perform a local clone on a repo\n> > containing odd files at .git/object directory, such as symlinks to other\n> > dirs, or unknown files.\n> >\n> > I'm bending over backwards here to avoid a SHA1 dependency. See [1]\n> > for an earlier and simpler version that hardcoded a SHA-1s.\n> >\n> > This behavior has been the same for a *long* time, but hasn't been\n> > tested for.\n> >\n> > There's a good post-hoc argument to be made for copying over unknown\n> > things, e.g. I'd like a git version that doesn't know about the\n> > commit-graph to copy it under \"clone --local\" so a newer git version\n> > can make use of it.\n> >\n> > In follow-up commits we'll look at changing some of this behavior, but\n> > for now let's just assert it as-is so we'll notice what we'll change\n> > later.\n> >\n> > 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n> >\n> > Signed-off-by: Ævar Arnfjörð Bjarmason \n> > Signed-off-by: Matheus Tavares \n> > Helped-by: Matheus Tavares \n>\n> The trailers should be in the order things have happened usually. So\n> having Ævar's S-o-b first makes sense, but the Helped-by should come\n> before your S-o-b, as you made the changes first before sending out\n> the patch series.\n\nOk, thanks for letting me know. I'll fix it.\n\n> When sending someone elses patch in a slightly modified version, it\n> may also be useful to add which parts you changed, as it was done in\n> e8dfcace31 (\"poll: use GetTickCount64() to avoid wrap-around issues\",\n> 2018-10-31) for example.\n\nThanks, I didn't know about that! I searched the log and didn't see\nmany of this on patches with 'Helped-by' tags, is there a particular\ncase to use it or not?\n\n> Iirc, the test that is added in this patch does not work on some\n> platforms, notably MacOS. That would mean that we would break\n> bisectability at this patch on some platforms if we were to introduce\n> it here. Therefore I think it would be better to squash this patch\n> into the next one which fixes these inconsistencies.\n> Note that I can't test this at the moment, so this concern is only\n> based on previous discussions that I remember. If that's already\n> addressed somehow, all the better!\n\nYes, it is already addressed :) The section of these tests that used\nto break on some platforms is now moved to the next patch which also\nfixes the platform inconsistencies. Now both patches (this and the\nnext) work on macOS, NetBSD and GNU/Linux. Also every test and job is\npassing at travis-ci, except by the job named \"Documentation\"[1]. But,\nit's weird since these patches don't even touch Documentation/... And\nmaster is failing the same job at my fork as well [2]... Any thoughts\non that?\n\n[1] https://travis-ci.org/MatheusBernardino/git/builds/512713775\n[2] https://travis-ci.org/MatheusBernardino/git/builds/513028692\n"},{"id":"372704","messageId":"CAHd-oW78Cwhq8sFFjOX4c6kS8JsEUUd_RGgheJhdN6MLAr+G2Q@mail.gmail.com","threadId":"50591","inReplyTo":"20190328221049.GK32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-29T14:27:54Z","receivedAt":"2019-03-29T14:28:09Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Mar 28, 2019 at 7:10 PM Thomas Gummerer wrote:\n>\n> On 03/22, Matheus Tavares wrote:\n> > There is currently an odd behaviour when locally clonning a repository\n> > with symlinks at .git/objects: using --no-hardlinks all symlinks are\n> > dereferenced but without it Git will try to hardlink the files with the\n> > link() function, which has an OS-specific behaviour on symlinks. On OSX\n> > and NetBSD, it creates a hardlink to the file pointed by the symlink\n> > whilst on GNU/Linux, it creates a hardlink to the symlink itself.\n> >\n> > On Manjaro GNU/Linux:\n> > $ touch a\n> > $ ln -s a b\n> > $ link b c\n> > $ ls -li a b c\n> > 155 [...] a\n> > 156 [...] b -> a\n> > 156 [...] c -> a\n> >\n> > But on NetBSD:\n> > $ ls -li a b c\n> > 2609160 [...] a\n> > 2609164 [...] b -> a\n> > 2609160 [...] c\n> >\n> > It's not good to have the result of a local clone to be OS-dependent and\n> > since the behaviour on GNU/Linux may result in broken symlinks, let's\n> > re-implement it with linkat() instead of link() using a flag to always\n> > follow symlinks and make the hardlink be to the pointed file. With this,\n> > besides standardizing the behaviour, no broken symlinks will be\n> > produced. Also, add tests for symlinked files at .git/objects/.\n> >\n> > Note: Git won't create symlinks at .git/objects itself, but it's better\n> > to handle this case and be friendly with users who manually create them.\n> >\n> > Signed-off-by: Matheus Tavares \n> > Signed-off-by: Ævar Arnfjörð Bjarmason \n> > Co-authored-by: Ævar Arnfjörð Bjarmason \n> > ---\n> > builtin/clone.c | 2 +-\n> > t/t5604-clone-reference.sh | 26 +++++++++++++++++++-------\n> > 2 files changed, 20 insertions(+), 8 deletions(-)\n> >\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index 50bde99618..b76f33c635 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -443,7 +443,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > if (unlink(dest->buf) && errno != ENOENT)\n> > die_errno(_(\"failed to unlink '%s'\"), dest->buf);\n> > if (!option_no_hardlinks) {\n> > - if (!link(src->buf, dest->buf))\n> > + if (!linkat(AT_FDCWD, src->buf, AT_FDCWD, dest->buf, AT_SYMLINK_FOLLOW))\n>\n> This line is starting to get a bit long, might be worth breaking it up\n> to keep to 80 characters per line.\n>\n> I notice that we are currently not using 'linkat()' anywhere else in\n> our codebase. It looks like it has been introduced in POSIX.1-2008,\n> which sounds fairly recent by git's standards. So I wonder if this is\n> really supported on all platforms that git is being built on.\n>\n> I also wonder what would need to be done on Windows if we were to\n> introduce this. I see we define the 'link()' function in\n> 'compat/mingw.c' for that currently, so I guess something similar\n> would be needed for 'linkat()'. I added Dscho to Cc for Windows\n> expertise.\n\nOk, what if instead of using linkat() we use 'realpath(const char\n*path, char *resolved_path)', which will resolve any symlinks at\n'path' and store the canonical path at 'resolved_path'? Then, we can\nstill keep using link() but now, with the certainty that all platforms\nwill have a consistent behaviour? (also, realpath() is POSIX.1-2001)\nWould that be a better idea?\n\n> While I agree with the goal of consistency accross all platforms here,\n> I don't know if it's actually worth going through the pain of doing\n> that, especially for somewhat of an edge case in local clones.\n>\n> If the test in the previous patch passes on all platforms, I'd be okay\n> with just calling the behaviour here undefined, especially as git\n> would never actually create symlinks in the .git/objects directory.\n"},{"id":"372707","messageId":"nycvar.QRO.7.76.6.1903291635050.41@tvgsbejvaqbjf.bet","threadId":"50591","inReplyTo":"20190328221049.GK32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-03-29T15:40:43Z","receivedAt":"2019-03-29T15:40:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Thomas,\n\nOn Thu, 28 Mar 2019, Thomas Gummerer wrote:\n\n> On 03/22, Matheus Tavares wrote:\n> >\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index 50bde99618..b76f33c635 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -443,7 +443,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > \t\tif (unlink(dest->buf) && errno != ENOENT)\n> > \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n> > \t\tif (!option_no_hardlinks) {\n> > -\t\t\tif (!link(src->buf, dest->buf))\n> > +\t\t\tif (!linkat(AT_FDCWD, src->buf, AT_FDCWD, dest->buf, AT_SYMLINK_FOLLOW))\n>\n> [...]\n>\n> I notice that we are currently not using 'linkat()' anywhere else in\n> our codebase. It looks like it has been introduced in POSIX.1-2008,\n> which sounds fairly recent by git's standards. So I wonder if this is\n> really supported on all platforms that git is being built on.\n\nI bet you it isn't.\n\n> I also wonder what would need to be done on Windows if we were to\n> introduce this. I see we define the 'link()' function in\n> 'compat/mingw.c' for that currently, so I guess something similar\n> would be needed for 'linkat()'. I added Dscho to Cc for Windows\n> expertise.\n\nIndeed, `linkat()` would have to be implemented in `compat/mingw.c`. It\nwould be a bit involved because the last parameter of that function\nchanges behavior noticeably, but the main difficulty (to determine the\npath from a file descriptor) should be overcome using\n`HANDLE olddirhandle = _get_osfhandle(olddirfd);` and the calling\n`GetFinalPathNameByHandleW(olddirhandle, wbuf, sizeof(wbuf));`.\n\nSo yes, this is *not* something I'd do lightly.\n\nThe bigger problem will be to continue to support older Unices such as\nSunOS and AIX. I highly doubt that they have that function. You should\nfind out, Matheus.\n\nCiao,\nJohannes\n"},{"id":"372726","messageId":"20190329193158.GN32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"CAHd-oW70VNgYmX9=5x0CwoSD2a0ik2AoJyd7T3xiO4O030c5yQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-29T19:31:58Z","receivedAt":"2019-03-29T19:32:04Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/29, Matheus Tavares Bernardino wrote:\n> On Thu, Mar 28, 2019 at 6:49 PM Thomas Gummerer wrote:\n> > When sending someone elses patch in a slightly modified version, it\n> > may also be useful to add which parts you changed, as it was done in\n> > e8dfcace31 (\"poll: use GetTickCount64() to avoid wrap-around issues\",\n> > 2018-10-31) for example.\n> \n> Thanks, I didn't know about that! I searched the log and didn't see\n> many of this on patches with 'Helped-by' tags, is there a particular\n> case to use it or not?\n\nHelped-by tags are usually used when you want to give someone credit\nfor help you got on a patch that you originally authored. It's up to\nyou at which point of involvement you actually want to add the tag, I\ntend to add them whenever someones input significantly\nchanges/improves the patch. I think adding it here might be okay,\nit's just less common when sending a patch that someone else authored\noriginally.\n\n> > Iirc, the test that is added in this patch does not work on some\n> > platforms, notably MacOS. That would mean that we would break\n> > bisectability at this patch on some platforms if we were to introduce\n> > it here. Therefore I think it would be better to squash this patch\n> > into the next one which fixes these inconsistencies.\n> > Note that I can't test this at the moment, so this concern is only\n> > based on previous discussions that I remember. If that's already\n> > addressed somehow, all the better!\n> \n> Yes, it is already addressed :) The section of these tests that used\n> to break on some platforms is now moved to the next patch which also\n> fixes the platform inconsistencies. Now both patches (this and the\n> next) work on macOS, NetBSD and GNU/Linux.\n\nGreat!\n\n> Also every test and job is\n> passing at travis-ci, except by the job named \"Documentation\"[1]. But,\n> it's weird since these patches don't even touch Documentation/... And\n> master is failing the same job at my fork as well [2]... Any thoughts\n> on that?\n\nYeah, this error seems to have nothing to do with your patch series.\nSince the last run of travis on master [*1*] at least the asciidoc\npackage doesn't seem to have changed, so from a first look I don't\nquite understand what's going on there. In any case, I don't think\nyou need to worry about that for now, as it hasn't been triggered by\nyour changes (I won't discourage you from looking at why it is failing\nand to try and fix that, but I think your time is probably better\nspent looking at this patch series and the proposal for GSoC for\nnow).\n\n*1*: https://travis-ci.org/git/git/builds/508784487\n\n> [1] https://travis-ci.org/MatheusBernardino/git/builds/512713775\n> [2] https://travis-ci.org/MatheusBernardino/git/builds/513028692\n"},{"id":"372727","messageId":"20190329194202.GG32732@szeder.dev","threadId":"50591","inReplyTo":"20190329193158.GN32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-29T19:42:02Z","receivedAt":"2019-03-29T19:42:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Mar 29, 2019 at 07:31:58PM +0000, Thomas Gummerer wrote:\n> > Also every test and job is\n> > passing at travis-ci, except by the job named \"Documentation\"[1]. But,\n> > it's weird since these patches don't even touch Documentation/... And\n> > master is failing the same job at my fork as well [2]... Any thoughts\n> > on that?\n> \n> Yeah, this error seems to have nothing to do with your patch series.\n> Since the last run of travis on master [*1*] at least the asciidoc\n> package doesn't seem to have changed, so from a first look I don't\n> quite understand what's going on there.\n\nhttps://public-inbox.org/git/20190329123520.27549-6-szeder.dev@gmail.com/\n\n"},{"id":"372729","messageId":"20190329200517.GO32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"CAHd-oW78Cwhq8sFFjOX4c6kS8JsEUUd_RGgheJhdN6MLAr+G2Q@mail.gmail.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-29T20:05:17Z","receivedAt":"2019-03-29T20:05:22Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/29, Matheus Tavares Bernardino wrote:\n> On Thu, Mar 28, 2019 at 7:10 PM Thomas Gummerer wrote:\n> > I notice that we are currently not using 'linkat()' anywhere else in\n> > our codebase. It looks like it has been introduced in POSIX.1-2008,\n> > which sounds fairly recent by git's standards. So I wonder if this is\n> > really supported on all platforms that git is being built on.\n> >\n> > I also wonder what would need to be done on Windows if we were to\n> > introduce this. I see we define the 'link()' function in\n> > 'compat/mingw.c' for that currently, so I guess something similar\n> > would be needed for 'linkat()'. I added Dscho to Cc for Windows\n> > expertise.\n> \n> Ok, what if instead of using linkat() we use 'realpath(const char\n> *path, char *resolved_path)', which will resolve any symlinks at\n> 'path' and store the canonical path at 'resolved_path'? Then, we can\n> still keep using link() but now, with the certainty that all platforms\n> will have a consistent behaviour? (also, realpath() is POSIX.1-2001)\n> Would that be a better idea?\n\nYeah, I think that is a good idea. Note that 'realpath()' itself is\nnot used anywhere in our codebase either, but there is\n'strbuf_realpath()', that from reading the function documentation does\nexactly what 'realpath()' would do. So using 'strbuf_realpath()'\nwould probably be the right thing to do here.\n"},{"id":"372730","messageId":"20190329201541.GP32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"87sgv69itl.fsf@evledraar.gmail.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-29T20:15:41Z","receivedAt":"2019-03-29T20:15:47Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/29, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Mar 28 2019, Thomas Gummerer wrote:\n> > I notice that we are currently not using 'linkat()' anywhere else in\n> > our codebase. It looks like it has been introduced in POSIX.1-2008,\n> > which sounds fairly recent by git's standards. So I wonder if this is\n> > really supported on all platforms that git is being built on.\n> >\n> > I also wonder what would need to be done on Windows if we were to\n> > introduce this. I see we define the 'link()' function in\n> > 'compat/mingw.c' for that currently, so I guess something similar\n> > would be needed for 'linkat()'. I added Dscho to Cc for Windows\n> > expertise.\n> \n> For better of worse this particular quest started because I pointed out\n> (with some WIP patches) that for understanding this change we should\n> test whatever we did now, to ensure that the refactoring didn't have\n> unintended side-effects.\n> \n> But that's a separate question from whether or not we want to keep the\n> current behavior.\n> \n> I think the current behavior is clearly insane, so I think we should\n> change it with some follow-up patches. In particular options like\n> --dissociate should clearly (in my mind at least) have behavior similar\n> to \"cp -L\", and --local should hardlink to the *target* of the symlink,\n> if anything, at least for objects/{??,pack,info}\n\nRight, I definitely agree with all of that. Adding tests for the\ncurrent behaviour is definitely a good thing if we can do it in a sane\nway. And I also agree that the current behaviour is insane, and\nshould be fixed, but that may not want to be part of this patch\nseries.\n\n> I think that changes the portability story with linkat(), since it's not\n> something we should be planning to keep, just an intermediate step so we\n> don't have a gigantic patch that both adds tests, refactors and changes\n> the behavior.\n\nFair enough, but that also means that this patch series necessarily\nhas to introduce the changes in behaviour as well as switching clone\nto use dir-iterator. Of course we could say that the switch-over to\nusing dir-iterator could be done as a separate patch series, but that\nseems a bit too much of a change in scope of this series.\n\nNow I think Matheus has actually found a nice solution to this issue\nusing 'strbuf_readlink()', which gives us the same behaviour as using\n'linkat()' in this patch would give us, so this might not be that big\nan issue in the end.\n"},{"id":"372759","messageId":"CAHd-oW6HmovFFKEPRCvyD8hfiToV8TNc=YFUKdDhPtzxUeoMdw@mail.gmail.com","threadId":"50591","inReplyTo":"20190329193158.GN32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 1/7] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T02:49:42Z","receivedAt":"2019-03-30T02:57:36Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Mar 29, 2019 at 4:32 PM Thomas Gummerer wrote:\n>\n> On 03/29, Matheus Tavares Bernardino wrote:\n> > On Thu, Mar 28, 2019 at 6:49 PM Thomas Gummerer wrote:\n> > > When sending someone elses patch in a slightly modified version, it\n> > > may also be useful to add which parts you changed, as it was done in\n> > > e8dfcace31 (\"poll: use GetTickCount64() to avoid wrap-around issues\",\n> > > 2018-10-31) for example.\n> >\n> > Thanks, I didn't know about that! I searched the log and didn't see\n> > many of this on patches with 'Helped-by' tags, is there a particular\n> > case to use it or not?\n>\n> Helped-by tags are usually used when you want to give someone credit\n> for help you got on a patch that you originally authored. It's up to\n> you at which point of involvement you actually want to add the tag, I\n> tend to add them whenever someones input significantly\n> changes/improves the patch. I think adding it here might be okay,\n> it's just less common when sending a patch that someone else authored\n> originally.\n>\n\nOk, got it, thanks!\n\n> > > Iirc, the test that is added in this patch does not work on some\n> > > platforms, notably MacOS. That would mean that we would break\n> > > bisectability at this patch on some platforms if we were to introduce\n> > > it here. Therefore I think it would be better to squash this patch\n> > > into the next one which fixes these inconsistencies.\n> > > Note that I can't test this at the moment, so this concern is only\n> > > based on previous discussions that I remember. If that's already\n> > > addressed somehow, all the better!\n> >\n> > Yes, it is already addressed :) The section of these tests that used\n> > to break on some platforms is now moved to the next patch which also\n> > fixes the platform inconsistencies. Now both patches (this and the\n> > next) work on macOS, NetBSD and GNU/Linux.\n>\n> Great!\n>\n> > Also every test and job is\n> > passing at travis-ci, except by the job named \"Documentation\"[1]. But,\n> > it's weird since these patches don't even touch Documentation/... And\n> > master is failing the same job at my fork as well [2]... Any thoughts\n> > on that?\n>\n> Yeah, this error seems to have nothing to do with your patch series.\n> Since the last run of travis on master [*1*] at least the asciidoc\n> package doesn't seem to have changed, so from a first look I don't\n> quite understand what's going on there. In any case, I don't think\n> you need to worry about that for now, as it hasn't been triggered by\n> your changes (I won't discourage you from looking at why it is failing\n> and to try and fix that, but I think your time is probably better\n> spent looking at this patch series and the proposal for GSoC for\n> now).\n>\n\nOk, thanks again.\n\n> *1*: https://travis-ci.org/git/git/builds/508784487\n>\n> > [1] https://travis-ci.org/MatheusBernardino/git/builds/512713775\n> > [2] https://travis-ci.org/MatheusBernardino/git/builds/513028692\n"},{"id":"372760","messageId":"CAHd-oW5aTyr7OmYwETLHYbHbRYgcUPuHpt2eN=Z2FWrqUQTzJQ@mail.gmail.com","threadId":"50591","inReplyTo":"20190329200517.GO32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T05:32:03Z","receivedAt":"2019-03-30T05:32:16Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Mar 29, 2019 at 5:05 PM Thomas Gummerer wrote:\n>\n> On 03/29, Matheus Tavares Bernardino wrote:\n> > On Thu, Mar 28, 2019 at 7:10 PM Thomas Gummerer wrote:\n> > > I notice that we are currently not using 'linkat()' anywhere else in\n> > > our codebase. It looks like it has been introduced in POSIX.1-2008,\n> > > which sounds fairly recent by git's standards. So I wonder if this is\n> > > really supported on all platforms that git is being built on.\n> > >\n> > > I also wonder what would need to be done on Windows if we were to\n> > > introduce this. I see we define the 'link()' function in\n> > > 'compat/mingw.c' for that currently, so I guess something similar\n> > > would be needed for 'linkat()'. I added Dscho to Cc for Windows\n> > > expertise.\n> >\n> > Ok, what if instead of using linkat() we use 'realpath(const char\n> > *path, char *resolved_path)', which will resolve any symlinks at\n> > 'path' and store the canonical path at 'resolved_path'? Then, we can\n> > still keep using link() but now, with the certainty that all platforms\n> > will have a consistent behaviour? (also, realpath() is POSIX.1-2001)\n> > Would that be a better idea?\n>\n> Yeah, I think that is a good idea. Note that 'realpath()' itself is\n> not used anywhere in our codebase either, but there is\n> 'strbuf_realpath()', that from reading the function documentation does\n> exactly what 'realpath()' would do. So using 'strbuf_realpath()'\n> would probably be the right thing to do here.\n\nThanks. While I was looking for realpath() at git codebase (before I\nsaw your email), I got a little confused: Besides strbuf_realpath() I\nalso found real_path(), real_path_if_valid() and real_pathdup(). All\nthese last three use strbuf_realpath() but they also initialize the\nstruct strbuf internally and just return a 'char *', which is much\nconvenient in some cases. What seems weird to me is that, whilst\nreal_pathdup() releases the internally initialized struct strubuf\n(leaving just the returned string to be free'd by the user), the other\ntwo don't. So, if struct strbuf change in the future to have more\ndynamic allocated resources, these functions will also have to be\nmodified. Also, since real_pathdup() can already do what the other two\ndo, do you know if there is a reason to keep all of them?\n\nOne last question: I found some places which don't free the string\nreturned by, for example, real_path() (e.g., find_worktree() at\nworktree.c). Would it be a valid/good patch (or patches) to add free()\ncalls in this places? (I'm currently trying to get more people here at\nUSP to contribute to git, and maybe this could be a nice first\ncontribution for them...)\n"},{"id":"372813","messageId":"20190330192738.GQ32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"CAHd-oW5aTyr7OmYwETLHYbHbRYgcUPuHpt2eN=Z2FWrqUQTzJQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-30T19:27:38Z","receivedAt":"2019-03-30T19:27:44Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/30, Matheus Tavares Bernardino wrote:\n> On Fri, Mar 29, 2019 at 5:05 PM Thomas Gummerer wrote:\n> >\n> > On 03/29, Matheus Tavares Bernardino wrote:\n> > > Ok, what if instead of using linkat() we use 'realpath(const char\n> > > *path, char *resolved_path)', which will resolve any symlinks at\n> > > 'path' and store the canonical path at 'resolved_path'? Then, we can\n> > > still keep using link() but now, with the certainty that all platforms\n> > > will have a consistent behaviour? (also, realpath() is POSIX.1-2001)\n> > > Would that be a better idea?\n> >\n> > Yeah, I think that is a good idea. Note that 'realpath()' itself is\n> > not used anywhere in our codebase either, but there is\n> > 'strbuf_realpath()', that from reading the function documentation does\n> > exactly what 'realpath()' would do. So using 'strbuf_realpath()'\n> > would probably be the right thing to do here.\n> \n> Thanks. While I was looking for realpath() at git codebase (before I\n> saw your email), I got a little confused: Besides strbuf_realpath() I\n> also found real_path(), real_path_if_valid() and real_pathdup(). All\n> these last three use strbuf_realpath() but they also initialize the\n> struct strbuf internally and just return a 'char *', which is much\n> convenient in some cases.\n\nRight, feel free to use whichever is most convenient for you, and\nwhichever works in the context.\n\n> What seems weird to me is that, whilst\n> real_pathdup() releases the internally initialized struct strubuf\n> (leaving just the returned string to be free'd by the user), the other\n> two don't. So, if struct strbuf change in the future to have more\n> dynamic allocated resources, these functions will also have to be\n> modified. Also, since real_pathdup() can already do what the other two\n> do, do you know if there is a reason to keep all of them?\n\nRight, '*dup()' functions usually leave the return value to be free'd\nby the caller. And while 'real_pathdup()' could do what the others do\nalready it also takes more effort to use it. Users don't need to free\nthe return value from 'real_path()' to avoid a memory leak. This\nalone justifies its existence I think.\n\n> One last question: I found some places which don't free the string\n> returned by, for example, real_path() (e.g., find_worktree() at\n> worktree.c). Would it be a valid/good patch (or patches) to add free()\n> calls in this places? (I'm currently trying to get more people here at\n> USP to contribute to git, and maybe this could be a nice first\n> contribution for them...)\n\nTrying to plug memory leaks in the codebase is definitely something\nthat I think is worthy of doing. Sometimes it's not worth actually\nfree'ing the memory, for example just before the program exits, in\nwhich case we can use the UNLEAK annotation. It was introduced in\n0e5bba53af (\"add UNLEAK annotation for reducing leak false positives\",\n2017-09-08) if you want more background.\n\nThat said, the memory from 'real_path()' should actually not be\nfree'd. The strbuf there has a static lifetime, so it is valid until\ngit exits. If we were to free the return value of the function we'd\nactually free an internal buffer of the strbuf, that is still valid.\nSo if someone were to use 'real_path()' after that, the memory that\nstrbuf still thinks it owns would actually have been free'd, which\nwould result in undefined behaviour, and probably would make git\nsegfault.\n"},{"id":"372817","messageId":"20190330224907.3277-1-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190322232237.13293-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 0/7] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:00Z","receivedAt":"2019-03-30T22:49:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This patchset contains:\n- a replacement of explicit recursive dir iteration at\n copy_or_link_directory for the dir-iterator API;\n- some refactoring and behaviour changes at local clone, mainly to\n take care of symlinks and hidden files at .git/objects; and\n- tests for this type of files\n\nChanges since v4:\n- Improved and fixed errors at messages from patches 1, 3, 5, 6 and 7.\n- At first patch:\n - Simplified construction, changing a multi-line cat for an echo.\n - Removed unnecessary subshells.\n - Disabled gc.auto, just to make sure we don't get any undesired\n behaviour for this test\n - Removed the first section of a sed command (\"s!/..\\$!/X!;\")\n that converts SHA-1s to fixed strings. No SHA-1 seemed to\n be changed by this section and neither it seemed to be used\n after the command.\n- At second patch, removed linkat() usage, which is POSIX.1-2008\n and may not be supported in all platforms git is being built.\n Now the same effect is achieved using real_pathdup() + link().\n\nv4: https://public-inbox.org/git/20190322232237.13293-1-matheus.bernardino@usp.br/\n\nMatheus Tavares (6):\n clone: better handle symlinked files at .git/objects/\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n clone: replace strcmp by fspathcmp\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n builtin/clone.c | 75 ++++++++++++---------\n dir-iterator.c | 28 +++++++-\n dir-iterator.h | 39 +++++++++--\n refs/files-backend.c | 2 +-\n t/t5604-clone-reference.sh | 133 +++++++++++++++++++++++++++++++++++++\n 5 files changed, 235 insertions(+), 42 deletions(-)\n\n-- \n2.20.1\n\n"},{"id":"372818","messageId":"20190330224907.3277-2-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 1/7] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:01Z","receivedAt":"2019-03-30T22:49:23Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nAdd tests for what happens when we perform a local clone on a repo\ncontaining odd files at .git/object directory, such as symlinks to other\ndirs, or unknown files.\n\nI'm bending over backwards here to avoid a SHA-1 dependency. See [1]\nfor an earlier and simpler version that hardcoded SHA-1s.\n\nThis behavior has been the same for a *long* time, but hasn't been\ntested for.\n\nThere's a good post-hoc argument to be made for copying over unknown\nthings, e.g. I'd like a git version that doesn't know about the\ncommit-graph to copy it under \"clone --local\" so a newer git version\ncan make use of it.\n\nIn follow-up commits we'll look at changing some of this behavior, but\nfor now, let's just assert it as-is so we'll notice what we'll change\nlater.\n\n1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n[matheus.bernardino: improved and split tests in more than one patch]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n t/t5604-clone-reference.sh | 111 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 111 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..207650cb95 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,115 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success 'setup repo with garbage in objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\n+\t\tcd .git/objects &&\n+\t\t>.some-hidden-file &&\n+\t\t>some-file &&\n+\t\tmkdir .some-hidden-dir &&\n+\t\t>.some-hidden-dir/some-file &&\n+\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\tmkdir some-dir &&\n+\t\t>some-dir/some-file &&\n+\t\t>some-dir/.some-dot-file\n+\t)\n+'\n+\n+test_expect_success 'clone a repo with garbage in objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+\tgit init T &&\n+\t(\n+\t\tcd T &&\n+\t\tgit config gc.auto 0 &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\ttest_commit B &&\n+\n+\t\tcd .git/objects &&\n+\t\tmv pack packs &&\n+\t\tln -s packs pack &&\n+\t\tfind ?? -type d >loose-dirs &&\n+\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\t\tmv $last_loose a-loose-dir &&\n+\t\tln -s a-loose-dir $last_loose &&\n+\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\techo unknown_content> unknown_file\n+\t) &&\n+\tgit -C T fsck &&\n+\tgit -C T rev-list --all --objects >T.objects\n+'\n+\n+\n+test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option T T$option || return 1 &&\n+\t\tgit -C T$option fsck || return 1 &&\n+\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n+\t\ttest_cmp T.objects T$option.objects &&\n+\t\t(\n+\t\t\tcd T$option/.git/objects &&\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t)\n+\tdone &&\n+\n+\tfor raw in $(ls T*.raw)\n+\tdo\n+\t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n+\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./Y/Z\n+\t./info/packs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\t./unknown_file\n+\tEOF\n+\n+\tfor option in --local --dissociate --no-hardlinks\n+\tdo\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\tdone &&\n+\n+\techo ./info/alternates >expected-files &&\n+\ttest_cmp expected-files T--shared.objects-files.raw\n+'\n+\n test_done\n-- \n2.20.1\n\n"},{"id":"372819","messageId":"20190330224907.3277-3-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:02Z","receivedAt":"2019-03-30T22:49:26Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is currently an odd behaviour when locally cloning a repository\nwith symlinks at .git/objects: using --no-hardlinks all symlinks are\ndereferenced but without it, Git will try to hardlink the files with the\nlink() function, which has an OS-specific behaviour on symlinks. On OSX\nand NetBSD, it creates a hardlink to the file pointed by the symlink\nwhilst on GNU/Linux, it creates a hardlink to the symlink itself.\n\nOn Manjaro GNU/Linux:\n $ touch a\n $ ln -s a b\n $ link b c\n $ ls -li a b c\n 155 [...] a\n 156 [...] b -> a\n 156 [...] c -> a\n\nBut on NetBSD:\n $ ls -li a b c\n 2609160 [...] a\n 2609164 [...] b -> a\n 2609160 [...] c\n\nIt's not good to have the result of a local clone to be OS-dependent and\nbesides that, the current behaviour on GNU/Linux may result in broken\nsymlinks. So let's standardize this by making the hardlinks always point\nto dereferenced paths, instead of the symlinks themselves. Also, add\ntests for symlinked files at .git/objects/.\n\nNote: Git won't create symlinks at .git/objects itself, but it's better\nto handle this case and be friendly with users who manually create them.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 5 ++++-\n t/t5604-clone-reference.sh | 27 ++++++++++++++++++++-------\n 2 files changed, 24 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 50bde99618..f975b509f1 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -443,7 +443,10 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (unlink(dest->buf) && errno != ENOENT)\n \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n \t\tif (!option_no_hardlinks) {\n-\t\t\tif (!link(src->buf, dest->buf))\n+\t\t\tchar *resolved_path = real_pathdup(src->buf, 1);\n+\t\t\tint status = link(resolved_path, dest->buf);\n+\t\t\tfree(resolved_path);\n+\t\t\tif (!status)\n \t\t\t\tcontinue;\n \t\t\tif (option_local > 0)\n \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 207650cb95..0800c3853f 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -266,7 +266,7 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'setup repo with manually symlinked or unknown files at objects/' '\n \tgit init T &&\n \t(\n \t\tcd T &&\n@@ -280,10 +280,19 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n \t\tln -s packs pack &&\n \t\tfind ?? -type d >loose-dirs &&\n \t\tlast_loose=$(tail -n 1 loose-dirs) &&\n-\t\trm -f loose-dirs &&\n \t\tmv $last_loose a-loose-dir &&\n \t\tln -s a-loose-dir $last_loose &&\n+\t\tfirst_loose=$(head -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\n+\t\tcd $first_loose &&\n+\t\tobj=$(ls *) &&\n+\t\tmv $obj ../an-object &&\n+\t\tln -s ../an-object $obj &&\n+\n+\t\tcd ../ &&\n \t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\tfind . -type l | sort >../../../T.objects-symlinks.raw &&\n \t\techo unknown_content> unknown_file\n \t) &&\n \tgit -C T fsck &&\n@@ -291,7 +300,7 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n '\n \n \n-test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at objects/' '\n \tfor option in --local --no-hardlinks --shared --dissociate\n \tdo\n \t\tgit clone $option T T$option || return 1 &&\n@@ -300,7 +309,8 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t\ttest_cmp T.objects T$option.objects &&\n \t\t(\n \t\t\tcd T$option/.git/objects &&\n-\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n \t\t)\n \tdone &&\n \n@@ -314,6 +324,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./Y/Z\n \t./Y/Z\n \t./a-loose-dir/Z\n+\t./an-object\n \t./Y/Z\n \t./info/packs\n \t./pack/pack-Z.idx\n@@ -323,13 +334,15 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./unknown_file\n \tEOF\n \n-\tfor option in --local --dissociate --no-hardlinks\n+\tfor option in --local --no-hardlinks --dissociate\n \tdo\n-\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n+\t\ttest_must_be_empty T$option.objects-symlinks.raw.de-sha || return 1\n \tdone &&\n \n \techo ./info/alternates >expected-files &&\n-\ttest_cmp expected-files T--shared.objects-files.raw\n+\ttest_cmp expected-files T--shared.objects-files.raw &&\n+\ttest_must_be_empty T--shared.objects-symlinks.raw\n '\n \n test_done\n-- \n2.20.1\n\n"},{"id":"372820","messageId":"20190330224907.3277-4-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:03Z","receivedAt":"2019-03-30T22:49:29Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are DIR_ITERATOR_PEDANTIC, which makes\ndir_iterator_advance abort immediately in the case of an error while\ntrying to fetch next entry; and DIR_ITERATOR_FOLLOW_SYMLINKS, which\nmakes the iteration follow symlinks to directories and include its\ncontents in the iteration. These new flags will be used in a subsequent\npatch.\n\nAlso adjust refs/files-backend.c to the new dir_iterator_begin\nsignature.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 28 +++++++++++++++++++++++++---\n dir-iterator.h | 39 +++++++++++++++++++++++++++++++++------\n refs/files-backend.c | 2 +-\n 3 files changed, 59 insertions(+), 10 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..17aca8ea41 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -48,12 +48,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned flags;\n };\n \n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n+\tint ret;\n \n \twhile (1) {\n \t\tstruct dir_iterator_level *level =\n@@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n \t\t\t\twarning(\"error opening directory %s: %s\",\n \t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t/* Popping the level is handled below */\n@@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading directory %s: %s\",\n \t\t\t\t\t\titer->base.path.buf, strerror(errno));\n \t\t\t\t} else if (closedir(level->dir))\n@@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\t\tcontinue;\n \n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n+\n+\t\t\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\t\t\tret = stat(iter->base.path.buf, &iter->base.st);\n+\t\t\telse\n+\t\t\t\tret = lstat(iter->base.path.buf, &iter->base.st);\n+\n+\t\t\tif (ret < 0) {\n+\t\t\t\tif (errno != ENOENT) {\n+\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\t\tgoto error_out;\n \t\t\t\t\twarning(\"error reading path '%s': %s\",\n \t\t\t\t\t\titer->base.path.buf,\n \t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t}\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\treturn ITER_OK;\n \t\t}\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -182,7 +203,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -195,6 +216,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n+\titer->flags = flags;\n \titer->levels_nr = 1;\n \titer->levels[0].initialized = 0;\n \ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..93646c3bea 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -19,7 +19,7 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * struct iterator *iter = dir_iterator_begin(path, 0);\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n@@ -40,6 +40,20 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error while trying to fetch the next entry, which is\n+ * to emit a warning and keep going. With this flag, resouces are\n+ * freed and ITER_ERROR is return immediately.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n+ * directories, i.e., iterate over linked directories' contents.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -54,20 +68,28 @@ struct dir_iterator {\n \t/* The current basename: */\n \tconst char *basename;\n \n-\t/* The result of calling lstat() on path: */\n+\t/*\n+\t * The result of calling lstat() on path or stat(), if the\n+\t * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at\n+\t * dir_iterator's initialization.\n+\t */\n \tstruct stat st;\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. Return a dir_iterator that holds the\n+ * internal state of the iteration.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behaviour.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n@@ -76,6 +98,11 @@ struct dir_iterator *dir_iterator_begin(const char *path);\n * dir_iterator and associated resources and return ITER_ERROR. It is\n * a bug to use iterator or call this function again after it has\n * returned ITER_DONE or ITER_ERROR.\n+ *\n+ * Note that whether dir-iterator will return ITER_ERROR when failing\n+ * to fetch the next entry or just emit a warning and try to fetch the\n+ * next is defined by the 'pedantic' option at dir-iterator's\n+ * initialization.\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex ef053f716c..2ce9783097 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,7 +2143,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\titer->dir_iterator = dir_iterator_begin(sb.buf, 0);\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \n-- \n2.20.1\n\n"},{"id":"372821","messageId":"20190330224907.3277-5-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 4/7] clone: copy hidden paths at local clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:04Z","receivedAt":"2019-03-30T22:49:32Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make the copy_or_link_directory function no longer skip hidden\ndirectories. This function, used to copy .git/objects, currently skips\nall hidden directories but not hidden files, which is an odd behaviour.\nThe reason for that could be unintentional: probably the intention was\nto skip '.' and '..' only but it ended up accidentally skipping all\ndirectories starting with '.'. Besides being more natural, the new\nbehaviour is more permissive to the user.\n\nAlso adjust tests to reflect this behaviour change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f975b509f1..81e1a39c61 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -428,7 +428,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 0800c3853f..c3998f2f9e 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \tdone &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.20.1\n\n"},{"id":"372822","messageId":"20190330224907.3277-6-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 5/7] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:05Z","receivedAt":"2019-03-30T22:49:36Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Extract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help to remove\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes the code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 81e1a39c61..f348eb02d4 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -392,6 +392,21 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\tstruct stat st;\n+\n+\tif (!mkdir(pathname, mode))\n+\t\treturn;\n+\n+\tif (errno != EEXIST)\n+\t\tdie_errno(_(\"failed to create directory '%s'\"), pathname);\n+\telse if (stat(pathname, &st))\n+\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\telse if (!S_ISDIR(st.st_mode))\n+\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -404,14 +419,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.20.1\n\n"},{"id":"372823","messageId":"20190330224907.3277-7-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 6/7] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:06Z","receivedAt":"2019-03-30T22:49:39Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoids recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat, inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone could succeed even\nthough the .git/objects copy didn't fully succeed.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 44 ++++++++++++++++++++++----------------------\n 1 file changed, 22 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f348eb02d4..ebe8d83334 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -408,42 +410,36 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tunsigned flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -463,7 +459,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -481,7 +481,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.20.1\n\n"},{"id":"372824","messageId":"20190330224907.3277-8-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v5 7/7] clone: replace strcmp by fspathcmp","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-03-30T22:49:07Z","receivedAt":"2019-03-30T22:49:42Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace the use of strcmp by fspathcmp at copy_or_link_directory, which\nis more permissive/friendly to case-insensitive file systems.\n\nSigned-off-by: Matheus Tavares \nSuggested-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex ebe8d83334..bf56a01638 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -439,7 +439,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n+\t\tif (!fspathcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n-- \n2.20.1\n\n"},{"id":"372834","messageId":"20190331174038.GS32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"20190330224907.3277-3-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v5 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-31T17:40:38Z","receivedAt":"2019-03-31T17:40:44Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/30, Matheus Tavares wrote:\n> There is currently an odd behaviour when locally cloning a repository\n> with symlinks at .git/objects: using --no-hardlinks all symlinks are\n> dereferenced but without it, Git will try to hardlink the files with the\n> link() function, which has an OS-specific behaviour on symlinks. On OSX\n> and NetBSD, it creates a hardlink to the file pointed by the symlink\n> whilst on GNU/Linux, it creates a hardlink to the symlink itself.\n> \n> On Manjaro GNU/Linux:\n> $ touch a\n> $ ln -s a b\n> $ link b c\n> $ ls -li a b c\n> 155 [...] a\n> 156 [...] b -> a\n> 156 [...] c -> a\n> \n> But on NetBSD:\n> $ ls -li a b c\n> 2609160 [...] a\n> 2609164 [...] b -> a\n> 2609160 [...] c\n> \n> It's not good to have the result of a local clone to be OS-dependent and\n> besides that, the current behaviour on GNU/Linux may result in broken\n> symlinks. So let's standardize this by making the hardlinks always point\n> to dereferenced paths, instead of the symlinks themselves. Also, add\n> tests for symlinked files at .git/objects/.\n> \n> Note: Git won't create symlinks at .git/objects itself, but it's better\n> to handle this case and be friendly with users who manually create them.\n> \n> Signed-off-by: Matheus Tavares \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Co-authored-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/clone.c | 5 ++++-\n> t/t5604-clone-reference.sh | 27 ++++++++++++++++++++-------\n> 2 files changed, 24 insertions(+), 8 deletions(-)\n> \n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 50bde99618..f975b509f1 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -443,7 +443,10 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> \t\tif (unlink(dest->buf) && errno != ENOENT)\n> \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n> \t\tif (!option_no_hardlinks) {\n> -\t\t\tif (!link(src->buf, dest->buf))\n> +\t\t\tchar *resolved_path = real_pathdup(src->buf, 1);\n> +\t\t\tint status = link(resolved_path, dest->buf);\n> +\t\t\tfree(resolved_path);\n> +\t\t\tif (!status)\n\nIs there any reason why we can't use 'real_path()' here? As I\nmentioned in [*1*], 'real_path()' doesn't require the callers to free\nany memory, so the above could become much simpler, and could just be\n\n+\t\t\tif (!link(real_path(src->buf), dest->buf))\n\n*1*: <20190330192738.GQ32487@hank.intra.tgummerer.com>\n\n> \t\t\t\tcontinue;\n> \t\t\tif (option_local > 0)\n> \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\n"},{"id":"372836","messageId":"20190331181209.GT32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"20190330224907.3277-4-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-31T18:12:09Z","receivedAt":"2019-03-31T18:12:16Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/30, Matheus Tavares wrote:\n> Add the possibility of giving flags to dir_iterator_begin to initialize\n> a dir-iterator with special options.\n> \n> Currently possible flags are DIR_ITERATOR_PEDANTIC, which makes\n> dir_iterator_advance abort immediately in the case of an error while\n> trying to fetch next entry; and DIR_ITERATOR_FOLLOW_SYMLINKS, which\n> makes the iteration follow symlinks to directories and include its\n> contents in the iteration. These new flags will be used in a subsequent\n> patch.\n> \n> Also adjust refs/files-backend.c to the new dir_iterator_begin\n> signature.\n> \n> Signed-off-by: Matheus Tavares \n> ---\n> dir-iterator.c | 28 +++++++++++++++++++++++++---\n> dir-iterator.h | 39 +++++++++++++++++++++++++++++++++------\n> refs/files-backend.c | 2 +-\n> 3 files changed, 59 insertions(+), 10 deletions(-)\n> \n> diff --git a/dir-iterator.c b/dir-iterator.c\n> index f2dcd82fde..17aca8ea41 100644\n> --- a/dir-iterator.c\n> +++ b/dir-iterator.c\n> @@ -48,12 +48,16 @@ struct dir_iterator_int {\n> \t * that will be included in this iteration.\n> \t */\n> \tstruct dir_iterator_level *levels;\n> +\n> +\t/* Combination of flags for this dir-iterator */\n> +\tunsigned flags;\n> };\n> \n> int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> {\n> \tstruct dir_iterator_int *iter =\n> \t\t(struct dir_iterator_int *)dir_iterator;\n> +\tint ret;\n\nMinor nit: I'd define this variable closer to where it is actually\nused, inside the second 'while(1)' loop in this function. That would\nmake it clearer that it's only used there and not in other places in\nthe function as well, which I had first expected when I read this.\n\n> \twhile (1) {\n> \t\tstruct dir_iterator_level *level =\n> @@ -71,6 +75,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \n> \t\t\tlevel->dir = opendir(iter->base.path.buf);\n> \t\t\tif (!level->dir && errno != ENOENT) {\n> +\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n> +\t\t\t\t\tgoto error_out;\n> \t\t\t\twarning(\"error opening directory %s: %s\",\n> \t\t\t\t\titer->base.path.buf, strerror(errno));\n> \t\t\t\t/* Popping the level is handled below */\n> @@ -122,6 +128,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \t\t\tif (!de) {\n> \t\t\t\t/* This level is exhausted; pop up a level. */\n> \t\t\t\tif (errno) {\n> +\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n> +\t\t\t\t\t\tgoto error_out;\n> \t\t\t\t\twarning(\"error reading directory %s: %s\",\n> \t\t\t\t\t\titer->base.path.buf, strerror(errno));\n> \t\t\t\t} else if (closedir(level->dir))\n> @@ -138,11 +146,20 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \t\t\t\tcontinue;\n> \n> \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n> -\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n> -\t\t\t\tif (errno != ENOENT)\n> +\n> +\t\t\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n> +\t\t\t\tret = stat(iter->base.path.buf, &iter->base.st);\n> +\t\t\telse\n> +\t\t\t\tret = lstat(iter->base.path.buf, &iter->base.st);\n> +\n> +\t\t\tif (ret < 0) {\n> +\t\t\t\tif (errno != ENOENT) {\n> +\t\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n> +\t\t\t\t\t\tgoto error_out;\n> \t\t\t\t\twarning(\"error reading path '%s': %s\",\n> \t\t\t\t\t\titer->base.path.buf,\n> \t\t\t\t\t\tstrerror(errno));\n> +\t\t\t\t}\n> \t\t\t\tcontinue;\n> \t\t\t}\n> \n> @@ -159,6 +176,10 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \t\t\treturn ITER_OK;\n> \t\t}\n> \t}\n> +\n> +error_out:\n> +\tdir_iterator_abort(dir_iterator);\n> +\treturn ITER_ERROR;\n> }\n> \n> int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> @@ -182,7 +203,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> \treturn ITER_DONE;\n> }\n> \n> -struct dir_iterator *dir_iterator_begin(const char *path)\n> +struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags)\n> {\n> \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n> \tstruct dir_iterator *dir_iterator = &iter->base;\n> @@ -195,6 +216,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n> \n> \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n> \n> +\titer->flags = flags;\n> \titer->levels_nr = 1;\n> \titer->levels[0].initialized = 0;\n> \n> diff --git a/dir-iterator.h b/dir-iterator.h\n> index 970793d07a..93646c3bea 100644\n> --- a/dir-iterator.h\n> +++ b/dir-iterator.h\n> @@ -19,7 +19,7 @@\n> * A typical iteration looks like this:\n> *\n> * int ok;\n> - * struct iterator *iter = dir_iterator_begin(path);\n> + * struct iterator *iter = dir_iterator_begin(path, 0);\n\nOutside of this context, we already mentione errorhandling when\n'ok != ITER_DONE' in his example. This still can't happen with the\nway the dir iterator is used here, but it serves as a reminder if\npeople are using the DIR_ITERATOR_PEDANTIC flag. Good.\n\n> *\n> * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n> * if (want_to_stop_iteration()) {\n> @@ -40,6 +40,20 @@\n> * dir_iterator_advance() again.\n> */\n> \n> +/*\n> + * Flags for dir_iterator_begin:\n> + *\n> + * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n> + * in case of an error while trying to fetch the next entry, which is\n> + * to emit a warning and keep going. With this flag, resouces are\n> + * freed and ITER_ERROR is return immediately.\n> + *\n> + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks to\n> + * directories, i.e., iterate over linked directories' contents.\n> + */\n> +#define DIR_ITERATOR_PEDANTIC (1 << 0)\n> +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n> +\n> struct dir_iterator {\n> \t/* The current path: */\n> \tstruct strbuf path;\n> @@ -54,20 +68,28 @@ struct dir_iterator {\n> \t/* The current basename: */\n> \tconst char *basename;\n> \n> -\t/* The result of calling lstat() on path: */\n> +\t/*\n> +\t * The result of calling lstat() on path or stat(), if the\n> +\t * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at\n> +\t * dir_iterator's initialization.\n> +\t */\n> \tstruct stat st;\n> };\n> \n> /*\n> - * Start a directory iteration over path. Return a dir_iterator that\n> - * holds the internal state of the iteration.\n> + * Start a directory iteration over path with the combination of\n> + * options specified by flags. Return a dir_iterator that holds the\n> + * internal state of the iteration.\n> *\n> * The iteration includes all paths under path, not including path\n> * itself and not including \".\" or \"..\" entries.\n> *\n> - * path is the starting directory. An internal copy will be made.\n> + * Parameters are:\n> + * - path is the starting directory. An internal copy will be made.\n> + * - flags is a combination of the possible flags to initialize a\n> + * dir-iterator or 0 for default behaviour.\n> */\n> -struct dir_iterator *dir_iterator_begin(const char *path);\n> +struct dir_iterator *dir_iterator_begin(const char *path, unsigned flags);\n> \n> /*\n> * Advance the iterator to the first or next item and return ITER_OK.\n> @@ -76,6 +98,11 @@ struct dir_iterator *dir_iterator_begin(const char *path);\n> * dir_iterator and associated resources and return ITER_ERROR. It is\n> * a bug to use iterator or call this function again after it has\n> * returned ITER_DONE or ITER_ERROR.\n> + *\n> + * Note that whether dir-iterator will return ITER_ERROR when failing\n> + * to fetch the next entry or just emit a warning and try to fetch the\n> + * next is defined by the 'pedantic' option at dir-iterator's\n> + * initialization.\n\nI feel like at this point we are repeating documentation that already\nexists for the flags. Should we ever find a reason to return\nITER_ERROR without the pedantic flag, this comment is likely to become\nout of date. I think not adding this note is probably better in this\ncase.\n\n> */\n> int dir_iterator_advance(struct dir_iterator *iterator);\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index ef053f716c..2ce9783097 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -2143,7 +2143,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n> \n> \tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n> \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n> -\titer->dir_iterator = dir_iterator_begin(sb.buf);\n> +\titer->dir_iterator = dir_iterator_begin(sb.buf, 0);\n> \titer->ref_store = ref_store;\n> \tstrbuf_release(&sb);\n> \n> -- \n> 2.20.1\n> \n"},{"id":"372837","messageId":"20190331181635.GU32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v5 0/7] clone: dir-iterator refactoring with tests","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-03-31T18:16:35Z","receivedAt":"2019-03-31T18:16:40Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 03/30, Matheus Tavares wrote:\n> This patchset contains:\n> - a replacement of explicit recursive dir iteration at\n> copy_or_link_directory for the dir-iterator API;\n> - some refactoring and behaviour changes at local clone, mainly to\n> take care of symlinks and hidden files at .git/objects; and\n> - tests for this type of files\n\nThanks. I read through the series, and only found a few minor nits.\n\nOne note on the cover letter, as I'm not sure I mentioned this before.\nBut as the series progresses and there are less changes in individual\npatches, it is useful to include a 'range-diff', so reviewers can\nquickly see what changed in the series. This is especially useful if\nthey can still remember the last iteration, so they don't necessarily\nhave to re-read the whole series.\n\nThis can be added using the '--range-diff' option in 'git\nformat-patch'.\n\n> Changes since v4:\n> - Improved and fixed errors at messages from patches 1, 3, 5, 6 and 7.\n> - At first patch:\n> - Simplified construction, changing a multi-line cat for an echo.\n> - Removed unnecessary subshells.\n> - Disabled gc.auto, just to make sure we don't get any undesired\n> behaviour for this test\n> - Removed the first section of a sed command (\"s!/..\\$!/X!;\")\n> that converts SHA-1s to fixed strings. No SHA-1 seemed to\n> be changed by this section and neither it seemed to be used\n> after the command.\n> - At second patch, removed linkat() usage, which is POSIX.1-2008\n> and may not be supported in all platforms git is being built.\n> Now the same effect is achieved using real_pathdup() + link().\n> \n> v4: https://public-inbox.org/git/20190322232237.13293-1-matheus.bernardino@usp.br/\n> \n> Matheus Tavares (6):\n> clone: better handle symlinked files at .git/objects/\n> dir-iterator: add flags parameter to dir_iterator_begin\n> clone: copy hidden paths at local clone\n> clone: extract function from copy_or_link_directory\n> clone: use dir-iterator to avoid explicit dir traversal\n> clone: replace strcmp by fspathcmp\n> \n> Ævar Arnfjörð Bjarmason (1):\n> clone: test for our behavior on odd objects/* content\n> \n> builtin/clone.c | 75 ++++++++++++---------\n> dir-iterator.c | 28 +++++++-\n> dir-iterator.h | 39 +++++++++--\n> refs/files-backend.c | 2 +-\n> t/t5604-clone-reference.sh | 133 +++++++++++++++++++++++++++++++++++++\n> 5 files changed, 235 insertions(+), 42 deletions(-)\n> \n> -- \n> 2.20.1\n> \n"},{"id":"372852","messageId":"CAHd-oW7jJWyww1i6rLG60M91NJa4AV2nbyOaXA6KE=StmmN6_A@mail.gmail.com","threadId":"50591","inReplyTo":"20190330192738.GQ32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v4 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-04-01T03:56:52Z","receivedAt":"2019-04-01T03:57:07Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, Mar 30, 2019 at 4:27 PM Thomas Gummerer wrote:\n>\n> On 03/30, Matheus Tavares Bernardino wrote:\n> > On Fri, Mar 29, 2019 at 5:05 PM Thomas Gummerer wrote:\n> > >\n> > > On 03/29, Matheus Tavares Bernardino wrote:\n> > > > Ok, what if instead of using linkat() we use 'realpath(const char\n> > > > *path, char *resolved_path)', which will resolve any symlinks at\n> > > > 'path' and store the canonical path at 'resolved_path'? Then, we can\n> > > > still keep using link() but now, with the certainty that all platforms\n> > > > will have a consistent behaviour? (also, realpath() is POSIX.1-2001)\n> > > > Would that be a better idea?\n> > >\n> > > Yeah, I think that is a good idea. Note that 'realpath()' itself is\n> > > not used anywhere in our codebase either, but there is\n> > > 'strbuf_realpath()', that from reading the function documentation does\n> > > exactly what 'realpath()' would do. So using 'strbuf_realpath()'\n> > > would probably be the right thing to do here.\n> >\n> > Thanks. While I was looking for realpath() at git codebase (before I\n> > saw your email), I got a little confused: Besides strbuf_realpath() I\n> > also found real_path(), real_path_if_valid() and real_pathdup(). All\n> > these last three use strbuf_realpath() but they also initialize the\n> > struct strbuf internally and just return a 'char *', which is much\n> > convenient in some cases.\n>\n> Right, feel free to use whichever is most convenient for you, and\n> whichever works in the context.\n>\n> > What seems weird to me is that, whilst\n> > real_pathdup() releases the internally initialized struct strubuf\n> > (leaving just the returned string to be free'd by the user), the other\n> > two don't. So, if struct strbuf change in the future to have more\n> > dynamic allocated resources, these functions will also have to be\n> > modified. Also, since real_pathdup() can already do what the other two\n> > do, do you know if there is a reason to keep all of them?\n>\n> Right, '*dup()' functions usually leave the return value to be free'd\n> by the caller. And while 'real_pathdup()' could do what the others do\n> already it also takes more effort to use it. Users don't need to free\n> the return value from 'real_path()' to avoid a memory leak. This\n> alone justifies its existence I think.\n>\n> > One last question: I found some places which don't free the string\n> > returned by, for example, real_path() (e.g., find_worktree() at\n> > worktree.c). Would it be a valid/good patch (or patches) to add free()\n> > calls in this places? (I'm currently trying to get more people here at\n> > USP to contribute to git, and maybe this could be a nice first\n> > contribution for them...)\n>\n> Trying to plug memory leaks in the codebase is definitely something\n> that I think is worthy of doing. Sometimes it's not worth actually\n> free'ing the memory, for example just before the program exits, in\n> which case we can use the UNLEAK annotation. It was introduced in\n> 0e5bba53af (\"add UNLEAK annotation for reducing leak false positives\",\n> 2017-09-08) if you want more background.\n>\n> That said, the memory from 'real_path()' should actually not be\n> free'd. The strbuf there has a static lifetime, so it is valid until\n> git exits. If we were to free the return value of the function we'd\n> actually free an internal buffer of the strbuf, that is still valid.\n> So if someone were to use 'real_path()' after that, the memory that\n> strbuf still thinks it owns would actually have been free'd, which\n> would result in undefined behaviour, and probably would make git\n> segfault.\n>\n\nThanks for the great explanation, Thomas. I hadn't noticed that the\nstrbuf variable inside real_path() is declared as static. I also took\nsome time, now, to better understand how strbuf functions deal with\nthe buf attribute (especially how it's realloc'ed) and now I think I\nunderstand it better. Thanks again for the help!\n\n> --\n> You received this message because you are subscribed to the Google Groups \"Kernel USP\" group.\n> To unsubscribe from this group and stop receiving emails from it, send an email to kernel-usp+unsubscribe@googlegroups.com.\n> To post to this group, send email to kernel-usp@googlegroups.com.\n> To view this discussion on the web visit https://groups.google.com/d/msgid/kernel-usp/20190330192738.GQ32487%40hank.intra.tgummerer.com.\n> For more options, visit https://groups.google.com/d/optout.\n"},{"id":"372853","messageId":"CAHd-oW5+GuSerJMD9XAdCNEFa1qXZFtD=Tov0EyRAn4Hckw9Fw@mail.gmail.com","threadId":"50591","inReplyTo":"20190331174038.GS32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v5 2/7] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-04-01T03:59:36Z","receivedAt":"2019-04-01T03:59:50Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sun, Mar 31, 2019 at 2:40 PM Thomas Gummerer wrote:\n>\n> On 03/30, Matheus Tavares wrote:\n> > There is currently an odd behaviour when locally cloning a repository\n> > with symlinks at .git/objects: using --no-hardlinks all symlinks are\n> > dereferenced but without it, Git will try to hardlink the files with the\n> > link() function, which has an OS-specific behaviour on symlinks. On OSX\n> > and NetBSD, it creates a hardlink to the file pointed by the symlink\n> > whilst on GNU/Linux, it creates a hardlink to the symlink itself.\n> >\n> > On Manjaro GNU/Linux:\n> > $ touch a\n> > $ ln -s a b\n> > $ link b c\n> > $ ls -li a b c\n> > 155 [...] a\n> > 156 [...] b -> a\n> > 156 [...] c -> a\n> >\n> > But on NetBSD:\n> > $ ls -li a b c\n> > 2609160 [...] a\n> > 2609164 [...] b -> a\n> > 2609160 [...] c\n> >\n> > It's not good to have the result of a local clone to be OS-dependent and\n> > besides that, the current behaviour on GNU/Linux may result in broken\n> > symlinks. So let's standardize this by making the hardlinks always point\n> > to dereferenced paths, instead of the symlinks themselves. Also, add\n> > tests for symlinked files at .git/objects/.\n> >\n> > Note: Git won't create symlinks at .git/objects itself, but it's better\n> > to handle this case and be friendly with users who manually create them.\n> >\n> > Signed-off-by: Matheus Tavares \n> > Signed-off-by: Ævar Arnfjörð Bjarmason \n> > Co-authored-by: Ævar Arnfjörð Bjarmason \n> > ---\n> > builtin/clone.c | 5 ++++-\n> > t/t5604-clone-reference.sh | 27 ++++++++++++++++++++-------\n> > 2 files changed, 24 insertions(+), 8 deletions(-)\n> >\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index 50bde99618..f975b509f1 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -443,7 +443,10 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > if (unlink(dest->buf) && errno != ENOENT)\n> > die_errno(_(\"failed to unlink '%s'\"), dest->buf);\n> > if (!option_no_hardlinks) {\n> > - if (!link(src->buf, dest->buf))\n> > + char *resolved_path = real_pathdup(src->buf, 1);\n> > + int status = link(resolved_path, dest->buf);\n> > + free(resolved_path);\n> > + if (!status)\n>\n> Is there any reason why we can't use 'real_path()' here? As I\n> mentioned in [*1*], 'real_path()' doesn't require the callers to free\n> any memory, so the above could become much simpler, and could just be\n>\n> + if (!link(real_path(src->buf), dest->buf))\n>\n\nYes, you are right. I will change this! I sent this v5 before\ncarefully reading your previous email and studding strbuf functions\nand real_path(), now that I did that, I see that real_path() is the\nbest option here. Thanks!\n\n> *1*: <20190330192738.GQ32487@hank.intra.tgummerer.com>\n>\n> > continue;\n> > if (option_local > 0)\n> > die_errno(_(\"failed to create link '%s'\"), dest->buf);\n"},{"id":"372889","messageId":"CAHd-oW63ysav-xgL4txuwMTtu+EPi5W82SOoKFpgfPz9eKz0Nw@mail.gmail.com","threadId":"50591","inReplyTo":"20190331181635.GU32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v5 0/7] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-04-01T13:56:14Z","receivedAt":"2019-04-01T13:56:28Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sun, Mar 31, 2019 at 3:16 PM Thomas Gummerer wrote:\n>\n> On 03/30, Matheus Tavares wrote:\n> > This patchset contains:\n> > - a replacement of explicit recursive dir iteration at\n> > copy_or_link_directory for the dir-iterator API;\n> > - some refactoring and behaviour changes at local clone, mainly to\n> > take care of symlinks and hidden files at .git/objects; and\n> > - tests for this type of files\n>\n> Thanks. I read through the series, and only found a few minor nits.\n>\n> One note on the cover letter, as I'm not sure I mentioned this before.\n> But as the series progresses and there are less changes in individual\n> patches, it is useful to include a 'range-diff', so reviewers can\n> quickly see what changed in the series. This is especially useful if\n> they can still remember the last iteration, so they don't necessarily\n> have to re-read the whole series.\n>\n> This can be added using the '--range-diff' option in 'git\n> format-patch'.\n\nThanks! I think you've said it earlier, but I forgot to use. I will\ninclude it in v6! Thanks for remembering me about it.\n\n> > Changes since v4:\n> > - Improved and fixed errors at messages from patches 1, 3, 5, 6 and 7.\n> > - At first patch:\n> > - Simplified construction, changing a multi-line cat for an echo.\n> > - Removed unnecessary subshells.\n> > - Disabled gc.auto, just to make sure we don't get any undesired\n> > behaviour for this test\n> > - Removed the first section of a sed command (\"s!/..\\$!/X!;\")\n> > that converts SHA-1s to fixed strings. No SHA-1 seemed to\n> > be changed by this section and neither it seemed to be used\n> > after the command.\n> > - At second patch, removed linkat() usage, which is POSIX.1-2008\n> > and may not be supported in all platforms git is being built.\n> > Now the same effect is achieved using real_pathdup() + link().\n> >\n> > v4: https://public-inbox.org/git/20190322232237.13293-1-matheus.bernardino@usp.br/\n> >\n> > Matheus Tavares (6):\n> > clone: better handle symlinked files at .git/objects/\n> > dir-iterator: add flags parameter to dir_iterator_begin\n> > clone: copy hidden paths at local clone\n> > clone: extract function from copy_or_link_directory\n> > clone: use dir-iterator to avoid explicit dir traversal\n> > clone: replace strcmp by fspathcmp\n> >\n> > Ęvar Arnfjörš Bjarmason (1):\n> > clone: test for our behavior on odd objects/* content\n> >\n> > builtin/clone.c | 75 ++++++++++++---------\n> > dir-iterator.c | 28 +++++++-\n> > dir-iterator.h | 39 +++++++++--\n> > refs/files-backend.c | 2 +-\n> > t/t5604-clone-reference.sh | 133 +++++++++++++++++++++++++++++++++++++\n> > 5 files changed, 235 insertions(+), 42 deletions(-)\n> >\n> > --\n> > 2.20.1\n> >\n"},{"id":"373610","messageId":"CAHd-oW7yPj5Ons3kfdEHtFgowc=F=2BD9EvU+fJbHe9=UXhx1g@mail.gmail.com","threadId":"50591","inReplyTo":"20190331181209.GT32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-04-10T20:24:51Z","receivedAt":"2019-04-10T20:25:06Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Thomas\n\nSorry for the late reply, but now that I submitted my GSoC proposal I\ncan finally come back to this series.\n\nOn Sun, Mar 31, 2019 at 3:12 PM Thomas Gummerer wrote:\n>\n> On 03/30, Matheus Tavares wrote:\n> > Add the possibility of giving flags to dir_iterator_begin to initialize\n> > a dir-iterator with special options.\n> >\n> > Currently possible flags are DIR_ITERATOR_PEDANTIC, which makes\n> > dir_iterator_advance abort immediately in the case of an error while\n> > trying to fetch next entry; and DIR_ITERATOR_FOLLOW_SYMLINKS, which\n> > makes the iteration follow symlinks to directories and include its\n> > contents in the iteration. These new flags will be used in a subsequent\n> > patch.\n> >\n> > Also adjust refs/files-backend.c to the new dir_iterator_begin\n> > signature.\n> >\n> > Signed-off-by: Matheus Tavares \n> > ---\n> > dir-iterator.c | 28 +++++++++++++++++++++++++---\n> > dir-iterator.h | 39 +++++++++++++++++++++++++++++++++------\n> > refs/files-backend.c | 2 +-\n> > 3 files changed, 59 insertions(+), 10 deletions(-)\n> >\n> > diff --git a/dir-iterator.c b/dir-iterator.c\n> > index f2dcd82fde..17aca8ea41 100644\n> > --- a/dir-iterator.c\n> > +++ b/dir-iterator.c\n> > @@ -48,12 +48,16 @@ struct dir_iterator_int {\n> > * that will be included in this iteration.\n> > */\n> > struct dir_iterator_level *levels;\n> > +\n> > + /* Combination of flags for this dir-iterator */\n> > + unsigned flags;\n> > };\n> >\n> > int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> > {\n> > struct dir_iterator_int *iter =\n> > (struct dir_iterator_int *)dir_iterator;\n> > + int ret;\n>\n> Minor nit: I'd define this variable closer to where it is actually\n> used, inside the second 'while(1)' loop in this function. That would\n> make it clearer that it's only used there and not in other places in\n> the function as well, which I had first expected when I read this.\n\nRight, thanks.\n\n> > diff --git a/dir-iterator.h b/dir-iterator.h\n> > index 970793d07a..93646c3bea 100644\n> > --- a/dir-iterator.h\n> > +++ b/dir-iterator.h\n> > @@ -19,7 +19,7 @@\n> > * A typical iteration looks like this:\n> > *\n> > * int ok;\n> > - * struct iterator *iter = dir_iterator_begin(path);\n> > + * struct iterator *iter = dir_iterator_begin(path, 0);\n>\n> Outside of this context, we already mentione errorhandling when\n> 'ok != ITER_DONE' in his example. This still can't happen with the\n> way the dir iterator is used here, but it serves as a reminder if\n> people are using the DIR_ITERATOR_PEDANTIC flag. Good.\n\nThis made me think again about the documentation saying that\ndir_iterator_abort() and dir_iterator_advance() may return ITER_ERROR,\nbut the implementation does not containing these possibilities.\n(Besides when the pedantic flag is used). Maybe the idea was to make\nAPI-users implement the check for an ITER_ERROR in case dir-iterator\nneeds to start returning it in the future.\n\nBut do you think such a change in dir-iterator is likely to happen?\nMaybe we could just make dir_iterator_abort() be void and remove this\nsection from documentation. Then, for dir_iterator_advance() users\nwould only need to check for ITER_ERROR if the pedantic flag was given\nat dir-iterator creation...\n\nAlso CC-ed Michael in case he has some input\n"},{"id":"373690","messageId":"20190411210927.GE32487@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"CAHd-oW7yPj5Ons3kfdEHtFgowc=F=2BD9EvU+fJbHe9=UXhx1g@mail.gmail.com","subject":"Re: [GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-04-11T21:09:27Z","receivedAt":"2019-04-11T21:09:37Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 04/10, Matheus Tavares Bernardino wrote:\n> > > diff --git a/dir-iterator.h b/dir-iterator.h\n> > > index 970793d07a..93646c3bea 100644\n> > > --- a/dir-iterator.h\n> > > +++ b/dir-iterator.h\n> > > @@ -19,7 +19,7 @@\n> > > * A typical iteration looks like this:\n> > > *\n> > > * int ok;\n> > > - * struct iterator *iter = dir_iterator_begin(path);\n> > > + * struct iterator *iter = dir_iterator_begin(path, 0);\n> >\n> > Outside of this context, we already mentione errorhandling when\n> > 'ok != ITER_DONE' in his example. This still can't happen with the\n> > way the dir iterator is used here, but it serves as a reminder if\n> > people are using the DIR_ITERATOR_PEDANTIC flag. Good.\n> \n> This made me think again about the documentation saying that\n> dir_iterator_abort() and dir_iterator_advance() may return ITER_ERROR,\n> but the implementation does not containing these possibilities.\n> (Besides when the pedantic flag is used). Maybe the idea was to make\n> API-users implement the check for an ITER_ERROR in case dir-iterator\n> needs to start returning it in the future.\n\nYeah, I think that was the intention.\n\n> But do you think such a change in dir-iterator is likely to happen?\n> Maybe we could just make dir_iterator_abort() be void and remove this\n> section from documentation. Then, for dir_iterator_advance() users\n> would only need to check for ITER_ERROR if the pedantic flag was given\n> at dir-iterator creation...\n\nDunno. In a world where we have the pedantic flag, I think only\nreturning ITER_ERROR if that flag is given might be what we want to\ndo. I can't think of a reason why we would want to return ITER_ERROR\nwithout the pedantic flag in that case.\n\nThough I think I would change the example the other way in that case,\nand pass DIR_ITERATOR_PEDANTIC to 'dir_iterator_begin()', as it would\nbe easy to forget error handling otherwise, even when it is\nnecessary. I'd rather err on the side of showing too much error\nhandling, than having people forget it and having users run into some\nodd edge cases in the wild that the tests don't cover.\n\n> Also CC-ed Michael in case he has some input\n"},{"id":"374320","messageId":"CAHd-oW48+cEqSMDFF3PXuOnzUvptM6q5Ba_Zb8gGhqa1bGNwZQ@mail.gmail.com","threadId":"50591","inReplyTo":"20190411210927.GE32487@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-04-23T17:07:26Z","receivedAt":"2019-04-23T17:07:40Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Apr 11, 2019 at 6:09 PM Thomas Gummerer wrote:\n>\n> On 04/10, Matheus Tavares Bernardino wrote:\n> > > > diff --git a/dir-iterator.h b/dir-iterator.h\n> > > > index 970793d07a..93646c3bea 100644\n> > > > --- a/dir-iterator.h\n> > > > +++ b/dir-iterator.h\n> > > > @@ -19,7 +19,7 @@\n> > > > * A typical iteration looks like this:\n> > > > *\n> > > > * int ok;\n> > > > - * struct iterator *iter = dir_iterator_begin(path);\n> > > > + * struct iterator *iter = dir_iterator_begin(path, 0);\n> > >\n> > > Outside of this context, we already mentione errorhandling when\n> > > 'ok != ITER_DONE' in his example. This still can't happen with the\n> > > way the dir iterator is used here, but it serves as a reminder if\n> > > people are using the DIR_ITERATOR_PEDANTIC flag. Good.\n> >\n> > This made me think again about the documentation saying that\n> > dir_iterator_abort() and dir_iterator_advance() may return ITER_ERROR,\n> > but the implementation does not containing these possibilities.\n> > (Besides when the pedantic flag is used). Maybe the idea was to make\n> > API-users implement the check for an ITER_ERROR in case dir-iterator\n> > needs to start returning it in the future.\n>\n> Yeah, I think that was the intention.\n>\n> > But do you think such a change in dir-iterator is likely to happen?\n> > Maybe we could just make dir_iterator_abort() be void and remove this\n> > section from documentation. Then, for dir_iterator_advance() users\n> > would only need to check for ITER_ERROR if the pedantic flag was given\n> > at dir-iterator creation...\n>\n> Dunno. In a world where we have the pedantic flag, I think only\n> returning ITER_ERROR if that flag is given might be what we want to\n> do. I can't think of a reason why we would want to return ITER_ERROR\n> without the pedantic flag in that case.\n\nOk. I began doing the change, but got stuck in a specific decision.\nWhat I was trying to do is:\n\n1) Make dir_iterator_advance() return ITER_ERROR only when the\npedantic flag is given;\n2) Make dir_iterator_abort() be void.\n\nThe first change is trivial. But the second is not so easy: Since the\n[only] current API user defines other iterators on top of\ndir-iterator, it would require a somehow big surgery on refs/* to make\nthis change. Should I proceed and make the changes at refs/* or should\nI keep dir_iterator_abort() returning int, although it can never fail?\n\nThere's also a third option: The only operation that may fail during\ndir_iterator_abort() is closedir(). But even on\ndir_iterator_advance(), I'm treating this error as \"non-fatal\" in the\nsense that it's not caught by the pedantic flag (although a warning is\nemitted). I did it like this because it doesn't seem like a major\nerror during dir iteration... But I could change this and make\nDIR_ITERATOR_PEDANTIC return ITER_ERROR upon closedir() errors for\nboth dir-iterator advance() and abort() functions. What do you think?\n\n> Though I think I would change the example the other way in that case,\n> and pass DIR_ITERATOR_PEDANTIC to 'dir_iterator_begin()', as it would\n> be easy to forget error handling otherwise, even when it is\n> necessary. I'd rather err on the side of showing too much error\n> handling, than having people forget it and having users run into some\n> odd edge cases in the wild that the tests don't cover.\n\nYes, I agree.\n\n> > Also CC-ed Michael in case he has some input\n"},{"id":"374415","messageId":"20190424183622.GB2068@hank.intra.tgummerer.com","threadId":"50591","inReplyTo":"CAHd-oW48+cEqSMDFF3PXuOnzUvptM6q5Ba_Zb8gGhqa1bGNwZQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2019-04-24T18:36:22Z","receivedAt":"2019-04-24T18:36:28Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 04/23, Matheus Tavares Bernardino wrote:\n> On Thu, Apr 11, 2019 at 6:09 PM Thomas Gummerer wrote:\n> >\n> > On 04/10, Matheus Tavares Bernardino wrote:\n> > > > > diff --git a/dir-iterator.h b/dir-iterator.h\n> > > > > index 970793d07a..93646c3bea 100644\n> > > > > --- a/dir-iterator.h\n> > > > > +++ b/dir-iterator.h\n> > > > > @@ -19,7 +19,7 @@\n> > > > > * A typical iteration looks like this:\n> > > > > *\n> > > > > * int ok;\n> > > > > - * struct iterator *iter = dir_iterator_begin(path);\n> > > > > + * struct iterator *iter = dir_iterator_begin(path, 0);\n> > > >\n> > > > Outside of this context, we already mentione errorhandling when\n> > > > 'ok != ITER_DONE' in his example. This still can't happen with the\n> > > > way the dir iterator is used here, but it serves as a reminder if\n> > > > people are using the DIR_ITERATOR_PEDANTIC flag. Good.\n> > >\n> > > This made me think again about the documentation saying that\n> > > dir_iterator_abort() and dir_iterator_advance() may return ITER_ERROR,\n> > > but the implementation does not containing these possibilities.\n> > > (Besides when the pedantic flag is used). Maybe the idea was to make\n> > > API-users implement the check for an ITER_ERROR in case dir-iterator\n> > > needs to start returning it in the future.\n> >\n> > Yeah, I think that was the intention.\n> >\n> > > But do you think such a change in dir-iterator is likely to happen?\n> > > Maybe we could just make dir_iterator_abort() be void and remove this\n> > > section from documentation. Then, for dir_iterator_advance() users\n> > > would only need to check for ITER_ERROR if the pedantic flag was given\n> > > at dir-iterator creation...\n> >\n> > Dunno. In a world where we have the pedantic flag, I think only\n> > returning ITER_ERROR if that flag is given might be what we want to\n> > do. I can't think of a reason why we would want to return ITER_ERROR\n> > without the pedantic flag in that case.\n> \n> Ok. I began doing the change, but got stuck in a specific decision.\n> What I was trying to do is:\n> \n> 1) Make dir_iterator_advance() return ITER_ERROR only when the\n> pedantic flag is given;\n> 2) Make dir_iterator_abort() be void.\n> \n> The first change is trivial. But the second is not so easy: Since the\n> [only] current API user defines other iterators on top of\n> dir-iterator, it would require a somehow big surgery on refs/* to make\n> this change. Should I proceed and make the changes at refs/* or should\n> I keep dir_iterator_abort() returning int, although it can never fail?\n\nMaybe I'm missing something, but wouldn't this change in refs.c be\nenough? (Other than actually making dir_iterator_abort not return\nanything)\n\n\tdiff --git a/refs/files-backend.c b/refs/files-backend.c\n\tindex 5848f32ef8..81863c3ee0 100644\n\t--- a/refs/files-backend.c\n\t+++ b/refs/files-backend.c\n\t@@ -2125,13 +2125,12 @@ static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n\t {\n\t struct files_reflog_iterator *iter =\n\t (struct files_reflog_iterator *)ref_iterator;\n\t- int ok = ITER_DONE;\n\t \n\t if (iter->dir_iterator)\n\t- ok = dir_iterator_abort(iter->dir_iterator);\n\t+ dir_iterator_abort(iter->dir_iterator);\n\t \n\t base_ref_iterator_free(ref_iterator);\n\t- return ok;\n\t+ return ITER_DONE;\n\t }\n\t \n\t static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n\nCurrently the only thing calling dir_iterator_abort() is\nfiles_reflog_iterator_abort() from what I can see, and\ndir_iterator_abort() always returns ITER_DONE.\n\nThat said, I don't know if this is actually worth pursuing. Having it\nreturn some value and having the caller check that makes it more\nfuture proof, as we won't have to change all the callers in the future\nif we want to start returning anything other than ITER_DONE. Just\nleaving it as it is now doesn't actually hurt anybody I think, but may\nhelp in the future.\n\n> There's also a third option: The only operation that may fail during\n> dir_iterator_abort() is closedir(). But even on\n> dir_iterator_advance(), I'm treating this error as \"non-fatal\" in the\n> sense that it's not caught by the pedantic flag (although a warning is\n> emitted). I did it like this because it doesn't seem like a major\n> error during dir iteration... But I could change this and make\n> DIR_ITERATOR_PEDANTIC return ITER_ERROR upon closedir() errors for\n> both dir-iterator advance() and abort() functions. What do you think?\n\nI think this might be the right way to go. We don't really need an\nerror from closedir, but at the same time if we are being pedantic,\nmaybe it should be an error. I don't have a strong opinion here\neither way, other than I think it should probably keep returning an\nint.\n\n> > Though I think I would change the example the other way in that case,\n> > and pass DIR_ITERATOR_PEDANTIC to 'dir_iterator_begin()', as it would\n> > be easy to forget error handling otherwise, even when it is\n> > necessary. I'd rather err on the side of showing too much error\n> > handling, than having people forget it and having users run into some\n> > odd edge cases in the wild that the tests don't cover.\n> \n> Yes, I agree.\n> \n> > > Also CC-ed Michael in case he has some input\n"},{"id":"374542","messageId":"CAHd-oW4XCtGd=jK+17Ms24n1X7cbdm9Q42orpY=kabrDvYbCHA@mail.gmail.com","threadId":"50591","inReplyTo":"20190424183622.GB2068@hank.intra.tgummerer.com","subject":"Re: [GSoC][PATCH v5 3/7] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-04-26T04:13:50Z","receivedAt":"2019-04-26T04:14:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Apr 24, 2019 at 3:36 PM Thomas Gummerer wrote:\n>\n> On 04/23, Matheus Tavares Bernardino wrote:\n> > On Thu, Apr 11, 2019 at 6:09 PM Thomas Gummerer wrote:\n> > >\n> > > On 04/10, Matheus Tavares Bernardino wrote:\n> > > > > > diff --git a/dir-iterator.h b/dir-iterator.h\n> > > > > > index 970793d07a..93646c3bea 100644\n> > > > > > --- a/dir-iterator.h\n> > > > > > +++ b/dir-iterator.h\n> > > > > > @@ -19,7 +19,7 @@\n> > > > > > * A typical iteration looks like this:\n> > > > > > *\n> > > > > > * int ok;\n> > > > > > - * struct iterator *iter = dir_iterator_begin(path);\n> > > > > > + * struct iterator *iter = dir_iterator_begin(path, 0);\n> > > > >\n> > > > > Outside of this context, we already mentione errorhandling when\n> > > > > 'ok != ITER_DONE' in his example. This still can't happen with the\n> > > > > way the dir iterator is used here, but it serves as a reminder if\n> > > > > people are using the DIR_ITERATOR_PEDANTIC flag. Good.\n> > > >\n> > > > This made me think again about the documentation saying that\n> > > > dir_iterator_abort() and dir_iterator_advance() may return ITER_ERROR,\n> > > > but the implementation does not containing these possibilities.\n> > > > (Besides when the pedantic flag is used). Maybe the idea was to make\n> > > > API-users implement the check for an ITER_ERROR in case dir-iterator\n> > > > needs to start returning it in the future.\n> > >\n> > > Yeah, I think that was the intention.\n> > >\n> > > > But do you think such a change in dir-iterator is likely to happen?\n> > > > Maybe we could just make dir_iterator_abort() be void and remove this\n> > > > section from documentation. Then, for dir_iterator_advance() users\n> > > > would only need to check for ITER_ERROR if the pedantic flag was given\n> > > > at dir-iterator creation...\n> > >\n> > > Dunno. In a world where we have the pedantic flag, I think only\n> > > returning ITER_ERROR if that flag is given might be what we want to\n> > > do. I can't think of a reason why we would want to return ITER_ERROR\n> > > without the pedantic flag in that case.\n> >\n> > Ok. I began doing the change, but got stuck in a specific decision.\n> > What I was trying to do is:\n> >\n> > 1) Make dir_iterator_advance() return ITER_ERROR only when the\n> > pedantic flag is given;\n> > 2) Make dir_iterator_abort() be void.\n> >\n> > The first change is trivial. But the second is not so easy: Since the\n> > [only] current API user defines other iterators on top of\n> > dir-iterator, it would require a somehow big surgery on refs/* to make\n> > this change. Should I proceed and make the changes at refs/* or should\n> > I keep dir_iterator_abort() returning int, although it can never fail?\n>\n> Maybe I'm missing something, but wouldn't this change in refs.c be\n> enough? (Other than actually making dir_iterator_abort not return\n> anything)\n>\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 5848f32ef8..81863c3ee0 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -2125,13 +2125,12 @@ static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n> {\n> struct files_reflog_iterator *iter =\n> (struct files_reflog_iterator *)ref_iterator;\n> - int ok = ITER_DONE;\n>\n> if (iter->dir_iterator)\n> - ok = dir_iterator_abort(iter->dir_iterator);\n> + dir_iterator_abort(iter->dir_iterator);\n>\n> base_ref_iterator_free(ref_iterator);\n> - return ok;\n> + return ITER_DONE;\n> }\n>\n> static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n\nYes, indeed. But I thought that since the reason for making\ndir_iterator_abort() be void is that it always returns ITER_DONE, the\nsame change should be applied to files_reflog_iterator_abort() as it\nwould fall into the same case. And this, in turn, would require\nchanges to ref_iterator_abort() and many other functions at\nrefs/iterator.c and refs/files-backend.c\n\n> Currently the only thing calling dir_iterator_abort() is\n> files_reflog_iterator_abort() from what I can see, and\n> dir_iterator_abort() always returns ITER_DONE.\n>\n> That said, I don't know if this is actually worth pursuing. Having it\n> return some value and having the caller check that makes it more\n> future proof, as we won't have to change all the callers in the future\n> if we want to start returning anything other than ITER_DONE. Just\n> leaving it as it is now doesn't actually hurt anybody I think, but may\n> help in the future.\n\nOk, I understand.\n\n> > There's also a third option: The only operation that may fail during\n> > dir_iterator_abort() is closedir(). But even on\n> > dir_iterator_advance(), I'm treating this error as \"non-fatal\" in the\n> > sense that it's not caught by the pedantic flag (although a warning is\n> > emitted). I did it like this because it doesn't seem like a major\n> > error during dir iteration... But I could change this and make\n> > DIR_ITERATOR_PEDANTIC return ITER_ERROR upon closedir() errors for\n> > both dir-iterator advance() and abort() functions. What do you think?\n>\n> I think this might be the right way to go. We don't really need an\n> error from closedir, but at the same time if we are being pedantic,\n> maybe it should be an error. I don't have a strong opinion here\n> either way, other than I think it should probably keep returning an\n> int.\n\nI know I suggested this option, but searching the code base I saw no\nother place that checks closedir()'s return besides dir-iterator. So\nmaybe the best option would be to keep dir_iterator_abort() always\nreturning ITER_DONE, even upon closedir() errors. Them, I can document\nthat the pedantic flag only affects dir_iterator_advance() behavior\n(but closedir() errors wouldn't be considered here as well).\n\nI got stuck in this for a while, but finally this option seems good to me now...\n\n> > > Though I think I would change the example the other way in that case,\n> > > and pass DIR_ITERATOR_PEDANTIC to 'dir_iterator_begin()', as it would\n> > > be easy to forget error handling otherwise, even when it is\n> > > necessary. I'd rather err on the side of showing too much error\n> > > handling, than having people forget it and having users run into some\n> > > odd edge cases in the wild that the tests don't cover.\n> >\n> > Yes, I agree.\n> >\n> > > > Also CC-ed Michael in case he has some input\n"},{"id":"374820","messageId":"20190502144829.4394-1-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190330224907.3277-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 00/10] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:19Z","receivedAt":"2019-05-02T14:48:45Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This patchset contains:\n- a replacement of explicit recursive dir iteration at\n copy_or_link_directory for the dir-iterator API;\n- some refactoring and behaviour changes at local clone, mainly to\n take care of symlinks and hidden files at .git/objects, together\n with tests for this types of files.\n- dir-iterator refactoring and feature adding with tests.\n\nChanges since v5:\n- Add tests for the dir-iterator API\n- Refactor the dir-iterator state machine model, simplifying its\n mechanics to improve readability.\n- Change warning() to warning_errno() at dir-iterator.c\n- Add a recursive symlinks check for dir_iterator_advance() in order\n to avoid unwanted recursions with DIR_ITERATOR_FOLLOW_SYMLIKS\n- Add tests for the dir-iterator flags feature\n- Make warnings be emitted both when DIR_ITERATOR_PEDANTIC is\n supplied and when it's not. It contains more relevant information\n on the error, so I thought it should be always printed.\n- Make dir_iterator_begin() check if the given argument is a valid\n path to a directory.\n- Adjusted some minor codestyle problems and commit messages\n- Address Thomas comments in v5\n\nv5: https://public-inbox.org/git/20190330224907.3277-1-matheus.bernardino@usp.br/\ntravis build: https://travis-ci.org/MatheusBernardino/git/builds/527176611\n\nNote: I tried to use --range-diff as Thomas suggested but I'm not sure\nthe output is as desired. Please, let me know if I did something wrong\nusing it.\n\nDaniel Ferreira (1):\n dir-iterator: add tests for dir-iterator API\n\nMatheus Tavares (8):\n clone: better handle symlinked files at .git/objects/\n dir-iterator: use warning_errno when possible\n dir-iterator: refactor state machine model\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n clone: replace strcmp by fspathcmp\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n Makefile | 1 +\n builtin/clone.c | 75 +++++----\n dir-iterator.c | 289 +++++++++++++++++++++--------------\n dir-iterator.h | 59 +++++--\n refs/files-backend.c | 17 ++-\n t/helper/test-dir-iterator.c | 58 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0066-dir-iterator.sh | 163 ++++++++++++++++++++\n t/t5604-clone-reference.sh | 133 ++++++++++++++++\n 10 files changed, 634 insertions(+), 163 deletions(-)\n create mode 100644 t/helper/test-dir-iterator.c\n create mode 100755 t/t0066-dir-iterator.sh\n\nRange-diff against v5:\n 1: 3d422dd4de = 1: a630b1a129 clone: test for our behavior on odd objects/* content\n 2: 35819e6ed1 ! 2: 51e06687fc clone: better handle symlinked files at .git/objects/\n @@ -45,10 +45,7 @@\n \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n \t\tif (!option_no_hardlinks) {\n -\t\t\tif (!link(src->buf, dest->buf))\n -+\t\t\tchar *resolved_path = real_pathdup(src->buf, 1);\n -+\t\t\tint status = link(resolved_path, dest->buf);\n -+\t\t\tfree(resolved_path);\n -+\t\t\tif (!status)\n ++\t\t\tif (!link(real_path(src->buf), dest->buf))\n \t\t\t\tcontinue;\n \t\t\tif (option_local > 0)\n \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\n 3: 2afe3208a4 < -: ---------- dir-iterator: add flags parameter to dir_iterator_begin\n -: ---------- > 3: c8a860e3a5 dir-iterator: add tests for dir-iterator API\n -: ---------- > 4: b975351080 dir-iterator: use warning_errno when possible\n -: ---------- > 5: 0fdbd1633e dir-iterator: refactor state machine model\n -: ---------- > 6: 7b2a9ae947 dir-iterator: add flags parameter to dir_iterator_begin\n 4: 71d64e6278 = 7: b9f298cbc6 clone: copy hidden paths at local clone\n 5: 35e36756db = 8: 0e7b1e49e2 clone: extract function from copy_or_link_directory\n 6: 1bfda87879 ! 9: f726ce2733 clone: use dir-iterator to avoid explicit dir traversal\n @@ -8,10 +8,14 @@\n copy_or_link_directory.\n \n This process also makes copy_or_link_directory call die() in case of an\n - error on readdir or stat, inside dir_iterator_advance. Previously it\n + error on readdir or stat inside dir_iterator_advance. Previously it\n would just print a warning for errors on stat and ignore errors on\n readdir, which isn't nice because a local git clone could succeed even\n - though the .git/objects copy didn't fully succeed.\n + though the .git/objects copy didn't fully succeed. Also, with the\n + dir-iterator API, recursive symlinks will be detected and skipped. This\n + is another behavior improvement, since the current version would\n + continue to copy the same content over and over until stat() returned an\n + ELOOP error.\n \n Signed-off-by: Matheus Tavares \n \n @@ -44,12 +48,15 @@\n -\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n +\tstruct dir_iterator *iter;\n +\tint iter_status;\n -+\tunsigned flags;\n ++\tunsigned int flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n +\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n +\titer = dir_iterator_begin(src->buf, flags);\n ++\n ++\tif (!iter)\n ++\t\tdie_errno(_(\"failed to start iterator over '%s'\"), src->buf);\n +\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n 7: 3861b30108 = 10: 6a57bb3887 clone: replace strcmp by fspathcmp\n-- \n2.20.1\n\n"},{"id":"374821","messageId":"20190502144829.4394-2-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 01/10] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:20Z","receivedAt":"2019-05-02T14:48:47Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nAdd tests for what happens when we perform a local clone on a repo\ncontaining odd files at .git/object directory, such as symlinks to other\ndirs, or unknown files.\n\nI'm bending over backwards here to avoid a SHA-1 dependency. See [1]\nfor an earlier and simpler version that hardcoded SHA-1s.\n\nThis behavior has been the same for a *long* time, but hasn't been\ntested for.\n\nThere's a good post-hoc argument to be made for copying over unknown\nthings, e.g. I'd like a git version that doesn't know about the\ncommit-graph to copy it under \"clone --local\" so a newer git version\ncan make use of it.\n\nIn follow-up commits we'll look at changing some of this behavior, but\nfor now, let's just assert it as-is so we'll notice what we'll change\nlater.\n\n1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n[matheus.bernardino: improved and split tests in more than one patch]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n t/t5604-clone-reference.sh | 111 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 111 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..207650cb95 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,115 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success 'setup repo with garbage in objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\n+\t\tcd .git/objects &&\n+\t\t>.some-hidden-file &&\n+\t\t>some-file &&\n+\t\tmkdir .some-hidden-dir &&\n+\t\t>.some-hidden-dir/some-file &&\n+\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\tmkdir some-dir &&\n+\t\t>some-dir/some-file &&\n+\t\t>some-dir/.some-dot-file\n+\t)\n+'\n+\n+test_expect_success 'clone a repo with garbage in objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+\tgit init T &&\n+\t(\n+\t\tcd T &&\n+\t\tgit config gc.auto 0 &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\ttest_commit B &&\n+\n+\t\tcd .git/objects &&\n+\t\tmv pack packs &&\n+\t\tln -s packs pack &&\n+\t\tfind ?? -type d >loose-dirs &&\n+\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\t\tmv $last_loose a-loose-dir &&\n+\t\tln -s a-loose-dir $last_loose &&\n+\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\techo unknown_content> unknown_file\n+\t) &&\n+\tgit -C T fsck &&\n+\tgit -C T rev-list --all --objects >T.objects\n+'\n+\n+\n+test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option T T$option || return 1 &&\n+\t\tgit -C T$option fsck || return 1 &&\n+\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n+\t\ttest_cmp T.objects T$option.objects &&\n+\t\t(\n+\t\t\tcd T$option/.git/objects &&\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t)\n+\tdone &&\n+\n+\tfor raw in $(ls T*.raw)\n+\tdo\n+\t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n+\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./Y/Z\n+\t./info/packs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\t./unknown_file\n+\tEOF\n+\n+\tfor option in --local --dissociate --no-hardlinks\n+\tdo\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\tdone &&\n+\n+\techo ./info/alternates >expected-files &&\n+\ttest_cmp expected-files T--shared.objects-files.raw\n+'\n+\n test_done\n-- \n2.20.1\n\n"},{"id":"374822","messageId":"20190502144829.4394-3-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 02/10] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:21Z","receivedAt":"2019-05-02T14:48:51Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is currently an odd behaviour when locally cloning a repository\nwith symlinks at .git/objects: using --no-hardlinks all symlinks are\ndereferenced but without it, Git will try to hardlink the files with the\nlink() function, which has an OS-specific behaviour on symlinks. On OSX\nand NetBSD, it creates a hardlink to the file pointed by the symlink\nwhilst on GNU/Linux, it creates a hardlink to the symlink itself.\n\nOn Manjaro GNU/Linux:\n $ touch a\n $ ln -s a b\n $ link b c\n $ ls -li a b c\n 155 [...] a\n 156 [...] b -> a\n 156 [...] c -> a\n\nBut on NetBSD:\n $ ls -li a b c\n 2609160 [...] a\n 2609164 [...] b -> a\n 2609160 [...] c\n\nIt's not good to have the result of a local clone to be OS-dependent and\nbesides that, the current behaviour on GNU/Linux may result in broken\nsymlinks. So let's standardize this by making the hardlinks always point\nto dereferenced paths, instead of the symlinks themselves. Also, add\ntests for symlinked files at .git/objects/.\n\nNote: Git won't create symlinks at .git/objects itself, but it's better\nto handle this case and be friendly with users who manually create them.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 27 ++++++++++++++++++++-------\n 2 files changed, 21 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 50bde99618..d1aba3b13f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -443,7 +443,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (unlink(dest->buf) && errno != ENOENT)\n \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n \t\tif (!option_no_hardlinks) {\n-\t\t\tif (!link(src->buf, dest->buf))\n+\t\t\tif (!link(real_path(src->buf), dest->buf))\n \t\t\t\tcontinue;\n \t\t\tif (option_local > 0)\n \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 207650cb95..0800c3853f 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -266,7 +266,7 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'setup repo with manually symlinked or unknown files at objects/' '\n \tgit init T &&\n \t(\n \t\tcd T &&\n@@ -280,10 +280,19 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n \t\tln -s packs pack &&\n \t\tfind ?? -type d >loose-dirs &&\n \t\tlast_loose=$(tail -n 1 loose-dirs) &&\n-\t\trm -f loose-dirs &&\n \t\tmv $last_loose a-loose-dir &&\n \t\tln -s a-loose-dir $last_loose &&\n+\t\tfirst_loose=$(head -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\n+\t\tcd $first_loose &&\n+\t\tobj=$(ls *) &&\n+\t\tmv $obj ../an-object &&\n+\t\tln -s ../an-object $obj &&\n+\n+\t\tcd ../ &&\n \t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\tfind . -type l | sort >../../../T.objects-symlinks.raw &&\n \t\techo unknown_content> unknown_file\n \t) &&\n \tgit -C T fsck &&\n@@ -291,7 +300,7 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n '\n \n \n-test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at objects/' '\n \tfor option in --local --no-hardlinks --shared --dissociate\n \tdo\n \t\tgit clone $option T T$option || return 1 &&\n@@ -300,7 +309,8 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t\ttest_cmp T.objects T$option.objects &&\n \t\t(\n \t\t\tcd T$option/.git/objects &&\n-\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n \t\t)\n \tdone &&\n \n@@ -314,6 +324,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./Y/Z\n \t./Y/Z\n \t./a-loose-dir/Z\n+\t./an-object\n \t./Y/Z\n \t./info/packs\n \t./pack/pack-Z.idx\n@@ -323,13 +334,15 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./unknown_file\n \tEOF\n \n-\tfor option in --local --dissociate --no-hardlinks\n+\tfor option in --local --no-hardlinks --dissociate\n \tdo\n-\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n+\t\ttest_must_be_empty T$option.objects-symlinks.raw.de-sha || return 1\n \tdone &&\n \n \techo ./info/alternates >expected-files &&\n-\ttest_cmp expected-files T--shared.objects-files.raw\n+\ttest_cmp expected-files T--shared.objects-files.raw &&\n+\ttest_must_be_empty T--shared.objects-symlinks.raw\n '\n \n test_done\n-- \n2.20.1\n\n"},{"id":"374823","messageId":"20190502144829.4394-4-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 03/10] dir-iterator: add tests for dir-iterator API","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:22Z","receivedAt":"2019-05-02T14:48:54Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Daniel Ferreira \n\nCreate t/helper/test-dir-iterator.c, which prints relevant information\nabout a directory tree iterated over with dir-iterator.\n\nCreate t/t0066-dir-iterator.sh, which tests that dir-iterator does\niterate through a whole directory tree as expected.\n\nSigned-off-by: Daniel Ferreira \n[matheus.bernardino: update to use test-tool and some minor aesthetics]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n t/helper/test-dir-iterator.c | 33 ++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0066-dir-iterator.sh | 55 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 91 insertions(+)\n create mode 100644 t/helper/test-dir-iterator.c\n create mode 100755 t/t0066-dir-iterator.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9f1b6e8926..61da7e4f35 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -713,6 +713,7 @@ TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\n+TEST_BUILTINS_OBJS += test-dir-iterator.o\n TEST_BUILTINS_OBJS += test-drop-caches.o\n TEST_BUILTINS_OBJS += test-dump-cache-tree.o\n TEST_BUILTINS_OBJS += test-dump-fsmonitor.o\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nnew file mode 100644\nindex 0000000000..84f50bed8c\n--- /dev/null\n+++ b/t/helper/test-dir-iterator.c\n@@ -0,0 +1,33 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+\n+/* Argument is a directory path to iterate over */\n+int cmd__dir_iterator(int argc, const char **argv)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct dir_iterator *diter;\n+\n+\tif (argc < 2)\n+\t\tdie(\"BUG: test-dir-iterator needs one argument\");\n+\n+\tstrbuf_add(&path, argv[1], strlen(argv[1]));\n+\n+\tditer = dir_iterator_begin(path.buf);\n+\n+\twhile (dir_iterator_advance(diter) == ITER_OK) {\n+\t\tif (S_ISDIR(diter->st.st_mode))\n+\t\t\tprintf(\"[d] \");\n+\t\telse if (S_ISREG(diter->st.st_mode))\n+\t\t\tprintf(\"[f] \");\n+\t\telse\n+\t\t\tprintf(\"[?] \");\n+\n+\t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n+\t\t diter->path.buf);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 53c06932c4..89b3bfcad8 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -13,6 +13,7 @@ static struct test_cmd cmds[] = {\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\n+\t{ \"dir-iterator\", cmd__dir_iterator },\n \t{ \"drop-caches\", cmd__drop_caches },\n \t{ \"dump-cache-tree\", cmd__dump_cache_tree },\n \t{ \"dump-fsmonitor\", cmd__dump_fsmonitor },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ffab4d19d7..0a831c839c 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -9,6 +9,7 @@ int cmd__config(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\n+int cmd__dir_iterator(int argc, const char **argv);\n int cmd__drop_caches(int argc, const char **argv);\n int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nnew file mode 100755\nindex 0000000000..6e06dc038d\n--- /dev/null\n+++ b/t/t0066-dir-iterator.sh\n@@ -0,0 +1,55 @@\n+#!/bin/sh\n+\n+test_description='Test the dir-iterator functionality'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tmkdir -p dir &&\n+\tmkdir -p dir/a/b/c/ &&\n+\t>dir/b &&\n+\t>dir/c &&\n+\tmkdir -p dir/d/e/d/ &&\n+\t>dir/a/b/c/d &&\n+\t>dir/a/e &&\n+\t>dir/d/e/d/a &&\n+\n+\tmkdir -p dir2/a/b/c/ &&\n+\t>dir2/a/b/c/d\n+'\n+\n+test_expect_success 'dir-iterator should iterate through all files' '\n+\tcat >expected-iteration-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir/a\n+\t[d] (a/b) [b] ./dir/a/b\n+\t[d] (a/b/c) [c] ./dir/a/b/c\n+\t[d] (d) [d] ./dir/d\n+\t[d] (d/e) [e] ./dir/d/e\n+\t[d] (d/e/d) [d] ./dir/d/e/d\n+\t[f] (a/b/c/d) [d] ./dir/a/b/c/d\n+\t[f] (a/e) [e] ./dir/a/e\n+\t[f] (b) [b] ./dir/b\n+\t[f] (c) [c] ./dir/c\n+\t[f] (d/e/d/a) [a] ./dir/d/e/d/a\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir >out &&\n+\tsort ./actual-iteration-sorted-output &&\n+\n+\ttest_cmp expected-iteration-sorted-output actual-iteration-sorted-output\n+'\n+\n+test_expect_success 'dir-iterator should list files in the correct order' '\n+\tcat >expected-pre-order-output <<-EOF &&\n+\t[d] (a) [a] ./dir2/a\n+\t[d] (a/b) [b] ./dir2/a/b\n+\t[d] (a/b/c) [c] ./dir2/a/b/c\n+\t[f] (a/b/c/d) [d] ./dir2/a/b/c/d\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir2 >actual-pre-order-output &&\n+\n+\ttest_cmp expected-pre-order-output actual-pre-order-output\n+'\n+\n+test_done\n-- \n2.20.1\n\n"},{"id":"374824","messageId":"20190502144829.4394-5-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 04/10] dir-iterator: use warning_errno when possible","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:23Z","receivedAt":"2019-05-02T14:48:57Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Change warning(..., strerror(errno)) by warning_errno(...). This helps\nto unify warning display besides simplifying a bit the code. Also,\nimprove warning messages by surrounding paths with quotation marks and\nusing more meaningful statements.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 23 ++++++++++++-----------\n 1 file changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..0c8880868a 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -71,8 +71,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n-\t\t\t\twarning(\"error opening directory %s: %s\",\n-\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\t\t\t\t iter->base.path.buf);\n \t\t\t\t/* Popping the level is handled below */\n \t\t\t}\n \n@@ -122,11 +122,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n-\t\t\t\t\twarning(\"error reading directory %s: %s\",\n-\t\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\t\twarning_errno(\"error reading directory '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \t\t\t\t} else if (closedir(level->dir))\n-\t\t\t\t\twarning(\"error closing directory %s: %s\",\n-\t\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \n \t\t\t\tlevel->dir = NULL;\n \t\t\t\tif (--iter->levels_nr == 0)\n@@ -140,9 +140,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n \t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n \t\t\t\tif (errno != ENOENT)\n-\t\t\t\t\twarning(\"error reading path '%s': %s\",\n-\t\t\t\t\t\titer->base.path.buf,\n-\t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t\twarning_errno(\"failed to stat '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -170,9 +169,11 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \t\t\t&iter->levels[iter->levels_nr - 1];\n \n \t\tif (level->dir && closedir(level->dir)) {\n+\t\t\tint saved_errno = errno;\n \t\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n-\t\t\twarning(\"error closing directory %s: %s\",\n-\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\terrno = saved_errno;\n+\t\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t\t iter->base.path.buf);\n \t\t}\n \t}\n \n-- \n2.20.1\n\n"},{"id":"374825","messageId":"20190502144829.4394-6-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 05/10] dir-iterator: refactor state machine model","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:24Z","receivedAt":"2019-05-02T14:49:02Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"dir_iterator_advance() is a large function with two nested loops. Let's\nimprove its readability factoring out three functions and simplifying\nits mechanics. The refactored model will no longer depend on\nlevel.initialized and level.dir_state to keep track of the iteration\nstate and will perform on a single loop.\n\nAlso, dir_iterator_begin() currently does not check if the given string\nrepresents a valid directory path. Since the refactored model will have\nto stat() the given path at initialization, let's also check for this\nkind of error and make dir_iterator_begin() return NULL, on failures,\nwith errno appropriately set. And add tests for this new behavior.\n\nImprove documentation at dir-iteration.h and code comments at\ndir-iterator.c to reflect the changes and eliminate possible\nambiguities.\n\nFinally, adjust refs/files-backend.c to check for now possible\ndir_iterator_begin() failures.\n\nOriginal-patch-by: Daniel Ferreira \nSigned-off-by: Matheus Tavares \n---\n\nWhen dir_iterator_begin() fails at refs/files-backend.c, I used the same\nidea Daniel proposed, which is to initialize an empty iterator with\nempty_ref_iterator_begin(). Still, I'm not sure wether we shouldn't\nabort execution there instead of returning an empty iterator.\n\ndir_iterator_begin() will fail if the give argument is an empty string,\nNULL, an invalid path, a non directory path or on some other stat()\nerrors. Maybe, on these kind of errors, we don't want the users\nof refs/files-backend.c, and therefore should call die() right there?\n\n(Also, NULL and empty string arguments were considered a bug in the\nprevious version of dir_iterator_begin)\n\n dir-iterator.c | 234 ++++++++++++++++++-----------------\n dir-iterator.h | 15 ++-\n refs/files-backend.c | 17 ++-\n t/helper/test-dir-iterator.c | 5 +\n t/t0066-dir-iterator.sh | 13 ++\n 5 files changed, 163 insertions(+), 121 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex 0c8880868a..594fe4d67b 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -4,8 +4,6 @@\n #include \"dir-iterator.h\"\n \n struct dir_iterator_level {\n-\tint initialized;\n-\n \tDIR *dir;\n \n \t/*\n@@ -13,16 +11,6 @@ struct dir_iterator_level {\n \t * (including a trailing '/'):\n \t */\n \tsize_t prefix_len;\n-\n-\t/*\n-\t * The last action that has been taken with the current entry\n-\t * (needed for directories, which have to be included in the\n-\t * iteration and also iterated into):\n-\t */\n-\tenum {\n-\t\tDIR_STATE_ITER,\n-\t\tDIR_STATE_RECURSE\n-\t} dir_state;\n };\n \n /*\n@@ -34,9 +22,11 @@ struct dir_iterator_int {\n \tstruct dir_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack. After the first\n+\t * call to dir_iterator_begin(), if it succeeds to open the\n+\t * first level's dir, this will always be at least 1. Then,\n+\t * when it comes to zero the iteration is ended and this\n+\t * struct is freed.\n \t */\n \tsize_t levels_nr;\n \n@@ -50,113 +40,118 @@ struct dir_iterator_int {\n \tstruct dir_iterator_level *levels;\n };\n \n+/*\n+ * Push a level in the iter stack and initialize it with information from\n+ * the directory pointed by iter->base->path. It is assumed that this\n+ * strbuf points to a valid directory path. Return 0 on success and -1\n+ * otherwise, leaving the stack unchanged.\n+ */\n+static int push_level(struct dir_iterator_int *iter)\n+{\n+\tstruct dir_iterator_level *level;\n+\n+\tALLOC_GROW(iter->levels, iter->levels_nr + 1, iter->levels_alloc);\n+\tlevel = &iter->levels[iter->levels_nr++];\n+\n+\tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n+\t\tstrbuf_addch(&iter->base.path, '/');\n+\tlevel->prefix_len = iter->base.path.len;\n+\n+\tlevel->dir = opendir(iter->base.path.buf);\n+\tif (!level->dir) {\n+\t\tif (errno != ENOENT) {\n+\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\t\t\t iter->base.path.buf);\n+\t\t}\n+\t\titer->levels_nr--;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Pop the top level on the iter stack, releasing any resources associated\n+ * with it. Return the new value of iter->levels_nr.\n+ */\n+static int pop_level(struct dir_iterator_int *iter)\n+{\n+\tstruct dir_iterator_level *level =\n+\t\t&iter->levels[iter->levels_nr - 1];\n+\n+\tif (level->dir && closedir(level->dir))\n+\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t iter->base.path.buf);\n+\tlevel->dir = NULL;\n+\n+\treturn --iter->levels_nr;\n+}\n+\n+/*\n+ * Populate iter->base with the necessary information on the next iteration\n+ * entry, represented by the given dirent de. Return 0 on success and -1\n+ * otherwise.\n+ */\n+static int prepare_next_entry_data(struct dir_iterator_int *iter,\n+\t\t\t\t struct dirent *de)\n+{\n+\tstrbuf_addstr(&iter->base.path, de->d_name);\n+\t/*\n+\t * We have to reset these because the path strbuf might have\n+\t * been realloc()ed at the previous strbuf_addstr().\n+\t */\n+\titer->base.relative_path = iter->base.path.buf +\n+\t\t\t\t iter->levels[0].prefix_len;\n+\titer->base.basename = iter->base.path.buf +\n+\t\t\t iter->levels[iter->levels_nr - 1].prefix_len;\n+\n+\tif (lstat(iter->base.path.buf, &iter->base.st)) {\n+\t\tif (errno != ENOENT)\n+\t\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n \n+\tif (S_ISDIR(iter->base.st.st_mode)) {\n+\t\tif (push_level(iter) && iter->levels_nr == 0) {\n+\t\t\t/* Pushing the first level failed */\n+\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t}\n+\t}\n+\n+\t/* Loop until we find an entry that we can give back to the caller. */\n \twhile (1) {\n+\t\tstruct dirent *de;\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n-\t\tstruct dirent *de;\n \n-\t\tif (!level->initialized) {\n-\t\t\t/*\n-\t\t\t * Note: dir_iterator_begin() ensures that\n-\t\t\t * path is not the empty string.\n-\t\t\t */\n-\t\t\tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n-\t\t\t\tstrbuf_addch(&iter->base.path, '/');\n-\t\t\tlevel->prefix_len = iter->base.path.len;\n-\n-\t\t\tlevel->dir = opendir(iter->base.path.buf);\n-\t\t\tif (!level->dir && errno != ENOENT) {\n-\t\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n+\t\terrno = 0;\n+\t\tde = readdir(level->dir);\n+\n+\t\tif (!de) {\n+\t\t\tif (errno)\n+\t\t\t\twarning_errno(\"error reading directory '%s'\",\n \t\t\t\t\t iter->base.path.buf);\n-\t\t\t\t/* Popping the level is handled below */\n-\t\t\t}\n-\n-\t\t\tlevel->initialized = 1;\n-\t\t} else if (S_ISDIR(iter->base.st.st_mode)) {\n-\t\t\tif (level->dir_state == DIR_STATE_ITER) {\n-\t\t\t\t/*\n-\t\t\t\t * The directory was just iterated\n-\t\t\t\t * over; now prepare to iterate into\n-\t\t\t\t * it.\n-\t\t\t\t */\n-\t\t\t\tlevel->dir_state = DIR_STATE_RECURSE;\n-\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n-\t\t\t\t\t iter->levels_alloc);\n-\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n-\t\t\t\tlevel->initialized = 0;\n-\t\t\t\tcontinue;\n-\t\t\t} else {\n-\t\t\t\t/*\n-\t\t\t\t * The directory has already been\n-\t\t\t\t * iterated over and iterated into;\n-\t\t\t\t * we're done with it.\n-\t\t\t\t */\n-\t\t\t}\n+\t\t\telse if (pop_level(iter) == 0)\n+\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\tcontinue;\n \t\t}\n \n-\t\tif (!level->dir) {\n-\t\t\t/*\n-\t\t\t * This level is exhausted (or wasn't opened\n-\t\t\t * successfully); pop up a level.\n-\t\t\t */\n-\t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n \n+\t\tif (prepare_next_entry_data(iter, de))\n \t\t\tcontinue;\n-\t\t}\n \n-\t\t/*\n-\t\t * Loop until we find an entry that we can give back\n-\t\t * to the caller:\n-\t\t */\n-\t\twhile (1) {\n-\t\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n-\t\t\terrno = 0;\n-\t\t\tde = readdir(level->dir);\n-\n-\t\t\tif (!de) {\n-\t\t\t\t/* This level is exhausted; pop up a level. */\n-\t\t\t\tif (errno) {\n-\t\t\t\t\twarning_errno(\"error reading directory '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\t\t\t\t} else if (closedir(level->dir))\n-\t\t\t\t\twarning_errno(\"error closing directory '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\n-\t\t\t\tlevel->dir = NULL;\n-\t\t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n-\t\t\t\tbreak;\n-\t\t\t}\n-\n-\t\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n-\t\t\t\t\twarning_errno(\"failed to stat '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\n-\t\t\t/*\n-\t\t\t * We have to set these each time because\n-\t\t\t * the path strbuf might have been realloc()ed.\n-\t\t\t */\n-\t\t\titer->base.relative_path =\n-\t\t\t\titer->base.path.buf + iter->levels[0].prefix_len;\n-\t\t\titer->base.basename =\n-\t\t\t\titer->base.path.buf + level->prefix_len;\n-\t\t\tlevel->dir_state = DIR_STATE_ITER;\n-\n-\t\t\treturn ITER_OK;\n-\t\t}\n+\t\treturn ITER_OK;\n \t}\n }\n \n@@ -187,17 +182,32 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n-\n-\tif (!path || !*path)\n-\t\tBUG(\"empty path passed to dir_iterator_begin()\");\n+\tint saved_errno;\n \n \tstrbuf_init(&iter->base.path, PATH_MAX);\n \tstrbuf_addstr(&iter->base.path, path);\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n+\titer->levels_nr = 0;\n \n-\titer->levels_nr = 1;\n-\titer->levels[0].initialized = 0;\n+\t/*\n+\t * Note: stat already checks for NULL or empty strings and\n+\t * inexistent paths.\n+\t */\n+\tif (stat(iter->base.path.buf, &iter->base.st) < 0) {\n+\t\tsaved_errno = errno;\n+\t\tgoto error_out;\n+\t}\n+\n+\tif (!S_ISDIR(iter->base.st.st_mode)) {\n+\t\tsaved_errno = ENOTDIR;\n+\t\tgoto error_out;\n+\t}\n \n \treturn dir_iterator;\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\terrno = saved_errno;\n+\treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..0822821e56 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -8,19 +8,23 @@\n *\n * Iterate over a directory tree, recursively, including paths of all\n * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n- * symlinks except for the original path.\n+ * symlinks except for the original path. Note that the original path\n+ * is not included in the iteration.\n *\n * Every time dir_iterator_advance() is called, update the members of\n * the dir_iterator structure to reflect the next path in the\n * iteration. The order that paths are iterated over within a\n- * directory is undefined, but directory paths are always iterated\n- * over before the subdirectory contents.\n+ * directory is undefined, directory paths are always given before\n+ * their contents.\n *\n * A typical iteration looks like this:\n *\n * int ok;\n * struct iterator *iter = dir_iterator_begin(path);\n *\n+ * if (!iter)\n+ * goto error_handler;\n+ *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n * ok = dir_iterator_abort(iter);\n@@ -59,8 +63,9 @@ struct dir_iterator {\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path. On success, return a\n+ * dir_iterator that holds the internal state of the iteration.\n+ * In case of failure, return NULL and set errno accordingly.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 63e55e6773..97a54532e3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,13 +2143,22 @@ static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \t\t\t\t\t\t const char *gitdir)\n {\n-\tstruct files_reflog_iterator *iter = xcalloc(1, sizeof(*iter));\n-\tstruct ref_iterator *ref_iterator = &iter->base;\n+\tstruct dir_iterator *diter;\n+\tstruct files_reflog_iterator *iter;\n+\tstruct ref_iterator *ref_iterator;\n \tstruct strbuf sb = STRBUF_INIT;\n \n-\tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\n+\tditer = dir_iterator_begin(sb.buf);\n+\tif (!diter)\n+\t\treturn empty_ref_iterator_begin();\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tref_iterator = &iter->base;\n+\n+\tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n+\titer->dir_iterator = diter;\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 84f50bed8c..fab1ff6237 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -17,6 +17,11 @@ int cmd__dir_iterator(int argc, const char **argv)\n \n \tditer = dir_iterator_begin(path.buf);\n \n+\tif (!diter) {\n+\t\tprintf(\"dir_iterator_begin failure: %d\\n\", errno);\n+\t\texit(EXIT_FAILURE);\n+\t}\n+\n \twhile (dir_iterator_advance(diter) == ITER_OK) {\n \t\tif (S_ISDIR(diter->st.st_mode))\n \t\t\tprintf(\"[d] \");\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex 6e06dc038d..c739ed7911 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -52,4 +52,17 @@ test_expect_success 'dir-iterator should list files in the correct order' '\n \ttest_cmp expected-pre-order-output actual-pre-order-output\n '\n \n+test_expect_success 'begin should fail upon inexistent paths' '\n+\ttest_must_fail test-tool dir-iterator ./inexistent-path \\\n+\t\t>actual-inexistent-path-output &&\n+\techo \"dir_iterator_begin failure: 2\" >expected-inexistent-path-output &&\n+\ttest_cmp expected-inexistent-path-output actual-inexistent-path-output\n+'\n+\n+test_expect_success 'begin should fail upon non directory paths' '\n+\ttest_must_fail test-tool dir-iterator ./dir/b >actual-non-dir-output &&\n+\techo \"dir_iterator_begin failure: 20\" >expected-non-dir-output &&\n+\ttest_cmp expected-non-dir-output actual-non-dir-output\n+'\n+\n test_done\n-- \n2.20.1\n\n"},{"id":"374826","messageId":"20190502144829.4394-7-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:25Z","receivedAt":"2019-05-02T14:49:07Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are:\n- DIR_ITERATOR_PEDANTIC, which makes dir_iterator_advance abort\nimmediately in the case of an error, instead of keep looking for the\nnext valid entry;\n- DIR_ITERATOR_FOLLOW_SYMLINKS, which makes the iterator follow\nsymlinks and include linked directories' contents in the iteration.\n\nThese new flags will be used in a subsequent patch.\n\nAlso add tests for the flags' usage and adjust refs/files-backend.c to\nthe new dir_iterator_begin signature.\n\nSigned-off-by: Matheus Tavares \n---\n\nrefs/files_backend.c is currently using no flags at the place it calls\ndir_iterator_begin(), to keep the same behavior it previously had. But\nas ITER_ERROR will now only be possibly returned by\ndir_iterator_avance() when DIR_ITERATOR_PEDANTIC is used and as\nrefs/files_backend.c already checks for ITER_ERRORs, should we, perhaps,\nuse this flag when initializing an iterator here?\n\nAnother uncertainty I had is why we ignore ENOENT at dir-iterator. Is it\nso that files may be remove during iteration? If not, maybe we should\nconsider to start looking for them, as, for example, broken symlinks\nwill simply be ignored in this current version as an ENOENT will be\nreturned when trying to dereference them.\n\n dir-iterator.c | 82 +++++++++++++++++++++++++------\n dir-iterator.h | 50 ++++++++++++++-----\n refs/files-backend.c | 2 +-\n t/helper/test-dir-iterator.c | 34 ++++++++++---\n t/t0066-dir-iterator.sh | 95 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 228 insertions(+), 35 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex 594fe4d67b..52db87bdc9 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -6,6 +6,9 @@\n struct dir_iterator_level {\n \tDIR *dir;\n \n+\t/* The inode number of this level's directory. */\n+\tino_t ino;\n+\n \t/*\n \t * The length of the directory part of path at this level\n \t * (including a trailing '/'):\n@@ -38,13 +41,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned int flags;\n };\n \n /*\n * Push a level in the iter stack and initialize it with information from\n * the directory pointed by iter->base->path. It is assumed that this\n * strbuf points to a valid directory path. Return 0 on success and -1\n- * otherwise, leaving the stack unchanged.\n+ * otherwise, setting errno accordingly and leaving the stack unchanged.\n */\n static int push_level(struct dir_iterator_int *iter)\n {\n@@ -56,14 +62,17 @@ static int push_level(struct dir_iterator_int *iter)\n \tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n \t\tstrbuf_addch(&iter->base.path, '/');\n \tlevel->prefix_len = iter->base.path.len;\n+\tlevel->ino = iter->base.st.st_ino;\n \n \tlevel->dir = opendir(iter->base.path.buf);\n \tif (!level->dir) {\n+\t\tint saved_errno = errno;\n \t\tif (errno != ENOENT) {\n \t\t\twarning_errno(\"error opening directory '%s'\",\n \t\t\t\t iter->base.path.buf);\n \t\t}\n \t\titer->levels_nr--;\n+\t\terrno = saved_errno;\n \t\treturn -1;\n \t}\n \n@@ -90,11 +99,13 @@ static int pop_level(struct dir_iterator_int *iter)\n /*\n * Populate iter->base with the necessary information on the next iteration\n * entry, represented by the given dirent de. Return 0 on success and -1\n- * otherwise.\n+ * otherwise, setting errno accordingly.\n */\n static int prepare_next_entry_data(struct dir_iterator_int *iter,\n \t\t\t\t struct dirent *de)\n {\n+\tint err, saved_errno;\n+\n \tstrbuf_addstr(&iter->base.path, de->d_name);\n \t/*\n \t * We have to reset these because the path strbuf might have\n@@ -105,12 +116,34 @@ static int prepare_next_entry_data(struct dir_iterator_int *iter,\n \titer->base.basename = iter->base.path.buf +\n \t\t\t iter->levels[iter->levels_nr - 1].prefix_len;\n \n-\tif (lstat(iter->base.path.buf, &iter->base.st)) {\n-\t\tif (errno != ENOENT)\n-\t\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n-\t\treturn -1;\n-\t}\n+\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\terr = stat(iter->base.path.buf, &iter->base.st);\n+\telse\n+\t\terr = lstat(iter->base.path.buf, &iter->base.st);\n+\n+\tsaved_errno = errno;\n+\tif (err && errno != ENOENT)\n+\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n+\n+\terrno = saved_errno;\n+\treturn err;\n+}\n+\n+/*\n+ * Look for a recursive symlink at iter->base.path pointing to any directory on\n+ * the previous stack levels. If it is found, return 1. If not, return 0.\n+ */\n+static int find_recursive_symlinks(struct dir_iterator_int *iter)\n+{\n+\tint i;\n+\n+\tif (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n+\t !S_ISDIR(iter->base.st.st_mode))\n+\t\treturn 0;\n \n+\tfor (i = 0; i < iter->levels_nr; ++i)\n+\t\tif (iter->base.st.st_ino == iter->levels[i].ino)\n+\t\t\treturn 1;\n \treturn 0;\n }\n \n@@ -119,11 +152,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n \n-\tif (S_ISDIR(iter->base.st.st_mode)) {\n-\t\tif (push_level(iter) && iter->levels_nr == 0) {\n-\t\t\t/* Pushing the first level failed */\n-\t\t\treturn dir_iterator_abort(dir_iterator);\n-\t\t}\n+\tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n+\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\tgoto error_out;\n+\t\tif (iter->levels_nr == 0)\n+\t\t\tgoto error_out;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -137,22 +170,38 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\tde = readdir(level->dir);\n \n \t\tif (!de) {\n-\t\t\tif (errno)\n+\t\t\tif (errno) {\n \t\t\t\twarning_errno(\"error reading directory '%s'\",\n \t\t\t\t\t iter->base.path.buf);\n-\t\t\telse if (pop_level(iter) == 0)\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n+\t\t\t} else if (pop_level(iter) == 0) {\n \t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t}\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n-\t\tif (prepare_next_entry_data(iter, de))\n+\t\tif (prepare_next_entry_data(iter, de)) {\n+\t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\tgoto error_out;\n \t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (find_recursive_symlinks(iter)) {\n+\t\t\twarning(\"ignoring recursive symlink at '%s'\",\n+\t\t\t\titer->base.path.buf);\n+\t\t\tcontinue;\n+\t\t}\n \n \t\treturn ITER_OK;\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -178,7 +227,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -189,6 +238,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \titer->levels_nr = 0;\n+\titer->flags = flags;\n \n \t/*\n \t * Note: stat already checks for NULL or empty strings and\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 0822821e56..28e50dabdb 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -20,7 +20,8 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * unsigned int flags = DIR_ITERATOR_PEDANTIC;\n+ * struct iterator *iter = dir_iterator_begin(path, flags);\n *\n * if (!iter)\n * goto error_handler;\n@@ -44,6 +45,24 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error at dir_iterator_advance(), which is to keep\n+ * looking for a next valid entry. With this flag, resources are freed\n+ * and ITER_ERROR is returned immediately. In both cases, a meaningful\n+ * warning is emitted.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks.\n+ * i.e., linked directories' contents will be iterated over and\n+ * iter->base.st will contain information on the referred files,\n+ * not the symlinks themselves, which is the default behavior.\n+ * Recursive symlinks are skipped.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -58,29 +77,38 @@ struct dir_iterator {\n \t/* The current basename: */\n \tconst char *basename;\n \n-\t/* The result of calling lstat() on path: */\n+\t/*\n+\t * The result of calling lstat() on path; or stat(), if the\n+\t * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at\n+\t * dir_iterator's initialization.\n+\t */\n \tstruct stat st;\n };\n \n /*\n- * Start a directory iteration over path. On success, return a\n- * dir_iterator that holds the internal state of the iteration.\n- * In case of failure, return NULL and set errno accordingly.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. On success, return a dir_iterator\n+ * that holds the internal state of the iteration. In case of\n+ * failure, return NULL and set errno accordingly.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behavior.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n * If the iteration is exhausted, free the dir_iterator and any\n- * resources associated with it and return ITER_DONE. On error, free\n- * dir_iterator and associated resources and return ITER_ERROR. It is\n- * a bug to use iterator or call this function again after it has\n- * returned ITER_DONE or ITER_ERROR.\n+ * resources associated with it and return ITER_DONE.\n+ *\n+ * It is a bug to use iterator or call this function again after it\n+ * has returned ITER_DONE or ITER_ERROR (which may be returned iff\n+ * the DIR_ITERATOR_PEDANTIC flag was set).\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 97a54532e3..ce78656823 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2150,7 +2150,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n \n-\tditer = dir_iterator_begin(sb.buf);\n+\tditer = dir_iterator_begin(sb.buf, 0);\n \tif (!diter)\n \t\treturn empty_ref_iterator_begin();\n \ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex fab1ff6237..a5b96cb0dc 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -4,29 +4,44 @@\n #include \"iterator.h\"\n #include \"dir-iterator.h\"\n \n-/* Argument is a directory path to iterate over */\n+/*\n+ * usage:\n+ * tool-test dir-iterator [--follow-symlinks] [--pedantic] directory_path\n+ */\n int cmd__dir_iterator(int argc, const char **argv)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct dir_iterator *diter;\n+\tunsigned int flags = 0;\n+\tint iter_status;\n+\n+\tfor (++argv, --argc; *argv && starts_with(*argv, \"--\"); ++argv, --argc) {\n+\t\tif (strcmp(*argv, \"--follow-symlinks\") == 0)\n+\t\t\tflags |= DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\t\telse if (strcmp(*argv, \"--pedantic\") == 0)\n+\t\t\tflags |= DIR_ITERATOR_PEDANTIC;\n+\t\telse\n+\t\t\tdie(\"invalid option '%s'\", *argv);\n+\t}\n \n-\tif (argc < 2)\n-\t\tdie(\"BUG: test-dir-iterator needs one argument\");\n-\n-\tstrbuf_add(&path, argv[1], strlen(argv[1]));\n+\tif (!*argv || argc != 1)\n+\t\tdie(\"dir-iterator needs exactly one non-option argument\");\n \n-\tditer = dir_iterator_begin(path.buf);\n+\tstrbuf_add(&path, *argv, strlen(*argv));\n+\tditer = dir_iterator_begin(path.buf, flags);\n \n \tif (!diter) {\n \t\tprintf(\"dir_iterator_begin failure: %d\\n\", errno);\n \t\texit(EXIT_FAILURE);\n \t}\n \n-\twhile (dir_iterator_advance(diter) == ITER_OK) {\n+\twhile ((iter_status = dir_iterator_advance(diter)) == ITER_OK) {\n \t\tif (S_ISDIR(diter->st.st_mode))\n \t\t\tprintf(\"[d] \");\n \t\telse if (S_ISREG(diter->st.st_mode))\n \t\t\tprintf(\"[f] \");\n+\t\telse if (S_ISLNK(diter->st.st_mode))\n+\t\t\tprintf(\"[s] \");\n \t\telse\n \t\t\tprintf(\"[?] \");\n \n@@ -34,5 +49,10 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\t diter->path.buf);\n \t}\n \n+\tif (iter_status != ITER_DONE) {\n+\t\tprintf(\"dir_iterator_advance failure\\n\");\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex c739ed7911..8f996a31fa 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -65,4 +65,99 @@ test_expect_success 'begin should fail upon non directory paths' '\n \ttest_cmp expected-non-dir-output actual-non-dir-output\n '\n \n+test_expect_success POSIXPERM,SANITY 'advance should not fail on errors by default' '\n+\tcat >expected-no-permissions-output <<-EOF &&\n+\t[d] (a) [a] ./dir3/a\n+\tEOF\n+\n+\tmkdir -p dir3/a &&\n+\t> dir3/a/b &&\n+\tchmod 0 dir3/a &&\n+\n+\ttest-tool dir-iterator ./dir3 >actual-no-permissions-output &&\n+\ttest_cmp expected-no-permissions-output actual-no-permissions-output &&\n+\tchmod 755 dir3/a &&\n+\trm -rf dir3\n+'\n+\n+test_expect_success POSIXPERM,SANITY 'advance should fail on errors, w/ pedantic flag' '\n+\tcat >expected-no-permissions-pedantic-output <<-EOF &&\n+\t[d] (a) [a] ./dir3/a\n+\tdir_iterator_advance failure\n+\tEOF\n+\n+\tmkdir -p dir3/a &&\n+\t> dir3/a/b &&\n+\tchmod 0 dir3/a &&\n+\n+\ttest_must_fail test-tool dir-iterator --pedantic ./dir3 \\\n+\t\t>actual-no-permissions-pedantic-output &&\n+\ttest_cmp expected-no-permissions-pedantic-output \\\n+\t\tactual-no-permissions-pedantic-output &&\n+\tchmod 755 dir3/a &&\n+\trm -rf dir3\n+'\n+\n+test_expect_success SYMLINKS 'setup dirs with symlinks' '\n+\tmkdir -p dir4/a &&\n+\tmkdir -p dir4/b/c &&\n+\t>dir4/a/d &&\n+\tln -s d dir4/a/e &&\n+\tln -s ../b dir4/a/f &&\n+\n+\tmkdir -p dir5/a/b &&\n+\tmkdir -p dir5/a/c &&\n+\tln -s ../c dir5/a/b/d &&\n+\tln -s ../ dir5/a/b/e &&\n+\tln -s ../../ dir5/a/b/f\n+'\n+\n+test_expect_success SYMLINKS 'dir-iterator should not follow symlinks by default' '\n+\tcat >expected-no-follow-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir4/a\n+\t[d] (b) [b] ./dir4/b\n+\t[d] (b/c) [c] ./dir4/b/c\n+\t[f] (a/d) [d] ./dir4/a/d\n+\t[s] (a/e) [e] ./dir4/a/e\n+\t[s] (a/f) [f] ./dir4/a/f\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir4 >out &&\n+\tsort actual-no-follow-sorted-output &&\n+\n+\ttest_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output\n+'\n+\n+test_expect_success SYMLINKS 'dir-iterator should follow symlinks w/ follow flag' '\n+\tcat >expected-follow-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir4/a\n+\t[d] (a/f) [f] ./dir4/a/f\n+\t[d] (a/f/c) [c] ./dir4/a/f/c\n+\t[d] (b) [b] ./dir4/b\n+\t[d] (b/c) [c] ./dir4/b/c\n+\t[f] (a/d) [d] ./dir4/a/d\n+\t[f] (a/e) [e] ./dir4/a/e\n+\tEOF\n+\n+\ttest-tool dir-iterator --follow-symlinks ./dir4 >out &&\n+\tsort actual-follow-sorted-output &&\n+\n+\ttest_cmp expected-follow-sorted-output actual-follow-sorted-output\n+'\n+\n+\n+test_expect_success SYMLINKS 'dir-iterator should ignore recursive symlinks w/ follow flag' '\n+\tcat >expected-rec-symlinks-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir5/a\n+\t[d] (a/b) [b] ./dir5/a/b\n+\t[d] (a/b/d) [d] ./dir5/a/b/d\n+\t[d] (a/c) [c] ./dir5/a/c\n+\tEOF\n+\n+\ttest-tool dir-iterator --follow-symlinks ./dir5 >out &&\n+\tsort actual-rec-symlinks-sorted-output &&\n+\n+\ttest_cmp expected-rec-symlinks-sorted-output actual-rec-symlinks-sorted-output\n+'\n+\n test_done\n-- \n2.20.1\n\n"},{"id":"374827","messageId":"20190502144829.4394-8-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 07/10] clone: copy hidden paths at local clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:26Z","receivedAt":"2019-05-02T14:49:10Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make the copy_or_link_directory function no longer skip hidden\ndirectories. This function, used to copy .git/objects, currently skips\nall hidden directories but not hidden files, which is an odd behaviour.\nThe reason for that could be unintentional: probably the intention was\nto skip '.' and '..' only but it ended up accidentally skipping all\ndirectories starting with '.'. Besides being more natural, the new\nbehaviour is more permissive to the user.\n\nAlso adjust tests to reflect this behaviour change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex d1aba3b13f..f117a6b206 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -428,7 +428,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 0800c3853f..c3998f2f9e 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \tdone &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.20.1\n\n"},{"id":"374828","messageId":"20190502144829.4394-9-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 08/10] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:27Z","receivedAt":"2019-05-02T14:49:13Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Extract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help to remove\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes the code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f117a6b206..1ee6d6050e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -392,6 +392,21 @@ static void copy_alternates(struct strbuf *src, struct strbuf *dst,\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\tstruct stat st;\n+\n+\tif (!mkdir(pathname, mode))\n+\t\treturn;\n+\n+\tif (errno != EEXIST)\n+\t\tdie_errno(_(\"failed to create directory '%s'\"), pathname);\n+\telse if (stat(pathname, &st))\n+\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\telse if (!S_ISDIR(st.st_mode))\n+\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -404,14 +419,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.20.1\n\n"},{"id":"374829","messageId":"20190502144829.4394-10-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 09/10] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:28Z","receivedAt":"2019-05-02T14:49:15Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoids recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone could succeed even\nthough the .git/objects copy didn't fully succeed. Also, with the\ndir-iterator API, recursive symlinks will be detected and skipped. This\nis another behavior improvement, since the current version would\ncontinue to copy the same content over and over until stat() returned an\nELOOP error.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 47 +++++++++++++++++++++++++----------------------\n 1 file changed, 25 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 1ee6d6050e..f99acd878f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -408,42 +410,39 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tunsigned int flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n+\tif (!iter)\n+\t\tdie_errno(_(\"failed to start iterator over '%s'\"), src->buf);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -460,7 +459,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -478,7 +481,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.20.1\n\n"},{"id":"374830","messageId":"20190502144829.4394-11-matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v6 10/10] clone: replace strcmp by fspathcmp","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-05-02T14:48:29Z","receivedAt":"2019-05-02T14:49:18Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace the use of strcmp by fspathcmp at copy_or_link_directory, which\nis more permissive/friendly to case-insensitive file systems.\n\nSigned-off-by: Matheus Tavares \nSuggested-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f99acd878f..6e0f194c3b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -442,7 +442,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n+\t\tif (!fspathcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, dest, src_repo);\n \t\t\tcontinue;\n \t\t}\n-- \n2.20.1\n\n"},{"id":"377467","messageId":"cover.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"20190502144829.4394-1-matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 00/10] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:37Z","receivedAt":"2019-06-18T23:28:25Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This patchset contains:\n- tests to the dir-iterator API;\n- dir-iterator refactoring to make its state machine simpler\n and feature adding with tests;\n- a replacement of explicit recursive dir iteration at\n copy_or_link_directory for the dir-iterator API;\n- some refactoring and behavior changes at local clone, mainly to\n take care of symlinks and hidden files at .git/objects, together\n with tests for these types of files.\n\nChanges since v6:\n- Rebased with master;\n- Added to dir-iterator documentation that ENOENT errors and hence broken\n symlinks are both ignored.\n\nWith the changes brought by this patchset, dir_iterator_begin() may now\nreturn NULL (setting errno) when it finds an error. Also, it's possible\nto pass a pedantic flag to it so that dir_iterator_advance() return\nimmediately on errors. But at refs/files-backend.c, the only user of\nthe API so far, the flag wasn't used and an empty iterator is\nreturned in case of errors at dir_iterator_begin(). These actions were\ntaken in order to keep the files-backend's behavior as close as\npossible to the one it previously had. But since it already has guards\nfor possible errors at dir_iterator_advance(), I'm wondering whether I\nshould send a follow-up patch making it use the pedantic flag.\n\nAlso, should I perhaps call die_errno() on dir_iterator_begin() errors\nat files-backend? I mean, we should continue returning an empty\niterator on ENOENT errors since \".git/logs\", the dir it iterates over,\nmay not be present. But we could possibly abort on other errors, just\nto be sure...\n\nAny comments on this possible follow-up patch will be highly appreciated.\n\nv6: https://public-inbox.org/git/20190502144829.4394-1-matheus.bernardino@usp.br/\ntravis build: https://travis-ci.org/matheustavares/git/builds/547451528\n\nDaniel Ferreira (1):\n dir-iterator: add tests for dir-iterator API\n\nMatheus Tavares (8):\n clone: better handle symlinked files at .git/objects/\n dir-iterator: use warning_errno when possible\n dir-iterator: refactor state machine model\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n clone: replace strcmp by fspathcmp\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n Makefile | 1 +\n builtin/clone.c | 75 +++++----\n dir-iterator.c | 289 +++++++++++++++++++++--------------\n dir-iterator.h | 60 ++++++--\n refs/files-backend.c | 17 ++-\n t/helper/test-dir-iterator.c | 58 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0066-dir-iterator.sh | 163 ++++++++++++++++++++\n t/t5604-clone-reference.sh | 133 ++++++++++++++++\n 10 files changed, 635 insertions(+), 163 deletions(-)\n create mode 100644 t/helper/test-dir-iterator.c\n create mode 100755 t/t0066-dir-iterator.sh\n\n-- \n2.22.0\n\n"},{"id":"377468","messageId":"17c0ba24062a61eb7b3a03dc4d5630a64db1e9f9.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 01/10] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:38Z","receivedAt":"2019-06-18T23:28:48Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nAdd tests for what happens when we perform a local clone on a repo\ncontaining odd files at .git/object directory, such as symlinks to other\ndirs, or unknown files.\n\nI'm bending over backwards here to avoid a SHA-1 dependency. See [1]\nfor an earlier and simpler version that hardcoded SHA-1s.\n\nThis behavior has been the same for a *long* time, but hasn't been\ntested for.\n\nThere's a good post-hoc argument to be made for copying over unknown\nthings, e.g. I'd like a git version that doesn't know about the\ncommit-graph to copy it under \"clone --local\" so a newer git version\ncan make use of it.\n\nIn follow-up commits we'll look at changing some of this behavior, but\nfor now, let's just assert it as-is so we'll notice what we'll change\nlater.\n\n1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n[matheus.bernardino: improved and split tests in more than one patch]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n t/t5604-clone-reference.sh | 111 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 111 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..207650cb95 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,115 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success 'setup repo with garbage in objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\n+\t\tcd .git/objects &&\n+\t\t>.some-hidden-file &&\n+\t\t>some-file &&\n+\t\tmkdir .some-hidden-dir &&\n+\t\t>.some-hidden-dir/some-file &&\n+\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\tmkdir some-dir &&\n+\t\t>some-dir/some-file &&\n+\t\t>some-dir/.some-dot-file\n+\t)\n+'\n+\n+test_expect_success 'clone a repo with garbage in objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+\tgit init T &&\n+\t(\n+\t\tcd T &&\n+\t\tgit config gc.auto 0 &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\ttest_commit B &&\n+\n+\t\tcd .git/objects &&\n+\t\tmv pack packs &&\n+\t\tln -s packs pack &&\n+\t\tfind ?? -type d >loose-dirs &&\n+\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\t\tmv $last_loose a-loose-dir &&\n+\t\tln -s a-loose-dir $last_loose &&\n+\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\techo unknown_content> unknown_file\n+\t) &&\n+\tgit -C T fsck &&\n+\tgit -C T rev-list --all --objects >T.objects\n+'\n+\n+\n+test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option T T$option || return 1 &&\n+\t\tgit -C T$option fsck || return 1 &&\n+\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n+\t\ttest_cmp T.objects T$option.objects &&\n+\t\t(\n+\t\t\tcd T$option/.git/objects &&\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t)\n+\tdone &&\n+\n+\tfor raw in $(ls T*.raw)\n+\tdo\n+\t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n+\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./Y/Z\n+\t./info/packs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\t./unknown_file\n+\tEOF\n+\n+\tfor option in --local --dissociate --no-hardlinks\n+\tdo\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\tdone &&\n+\n+\techo ./info/alternates >expected-files &&\n+\ttest_cmp expected-files T--shared.objects-files.raw\n+'\n+\n test_done\n-- \n2.22.0\n\n"},{"id":"377469","messageId":"722350fe00de73d28df1a9c8667c84171aa8c1f5.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 02/10] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:39Z","receivedAt":"2019-06-18T23:28:57Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is currently an odd behaviour when locally cloning a repository\nwith symlinks at .git/objects: using --no-hardlinks all symlinks are\ndereferenced but without it, Git will try to hardlink the files with the\nlink() function, which has an OS-specific behaviour on symlinks. On OSX\nand NetBSD, it creates a hardlink to the file pointed by the symlink\nwhilst on GNU/Linux, it creates a hardlink to the symlink itself.\n\nOn Manjaro GNU/Linux:\n $ touch a\n $ ln -s a b\n $ link b c\n $ ls -li a b c\n 155 [...] a\n 156 [...] b -> a\n 156 [...] c -> a\n\nBut on NetBSD:\n $ ls -li a b c\n 2609160 [...] a\n 2609164 [...] b -> a\n 2609160 [...] c\n\nIt's not good to have the result of a local clone to be OS-dependent and\nbesides that, the current behaviour on GNU/Linux may result in broken\nsymlinks. So let's standardize this by making the hardlinks always point\nto dereferenced paths, instead of the symlinks themselves. Also, add\ntests for symlinked files at .git/objects/.\n\nNote: Git won't create symlinks at .git/objects itself, but it's better\nto handle this case and be friendly with users who manually create them.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 27 ++++++++++++++++++++-------\n 2 files changed, 21 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5b9ebe9947..4a0a2455a7 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -445,7 +445,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (unlink(dest->buf) && errno != ENOENT)\n \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n \t\tif (!option_no_hardlinks) {\n-\t\t\tif (!link(src->buf, dest->buf))\n+\t\t\tif (!link(real_path(src->buf), dest->buf))\n \t\t\t\tcontinue;\n \t\t\tif (option_local > 0)\n \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 207650cb95..0800c3853f 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -266,7 +266,7 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'setup repo with manually symlinked or unknown files at objects/' '\n \tgit init T &&\n \t(\n \t\tcd T &&\n@@ -280,10 +280,19 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n \t\tln -s packs pack &&\n \t\tfind ?? -type d >loose-dirs &&\n \t\tlast_loose=$(tail -n 1 loose-dirs) &&\n-\t\trm -f loose-dirs &&\n \t\tmv $last_loose a-loose-dir &&\n \t\tln -s a-loose-dir $last_loose &&\n+\t\tfirst_loose=$(head -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\n+\t\tcd $first_loose &&\n+\t\tobj=$(ls *) &&\n+\t\tmv $obj ../an-object &&\n+\t\tln -s ../an-object $obj &&\n+\n+\t\tcd ../ &&\n \t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\tfind . -type l | sort >../../../T.objects-symlinks.raw &&\n \t\techo unknown_content> unknown_file\n \t) &&\n \tgit -C T fsck &&\n@@ -291,7 +300,7 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n '\n \n \n-test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at objects/' '\n \tfor option in --local --no-hardlinks --shared --dissociate\n \tdo\n \t\tgit clone $option T T$option || return 1 &&\n@@ -300,7 +309,8 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t\ttest_cmp T.objects T$option.objects &&\n \t\t(\n \t\t\tcd T$option/.git/objects &&\n-\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n \t\t)\n \tdone &&\n \n@@ -314,6 +324,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./Y/Z\n \t./Y/Z\n \t./a-loose-dir/Z\n+\t./an-object\n \t./Y/Z\n \t./info/packs\n \t./pack/pack-Z.idx\n@@ -323,13 +334,15 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./unknown_file\n \tEOF\n \n-\tfor option in --local --dissociate --no-hardlinks\n+\tfor option in --local --no-hardlinks --dissociate\n \tdo\n-\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n+\t\ttest_must_be_empty T$option.objects-symlinks.raw.de-sha || return 1\n \tdone &&\n \n \techo ./info/alternates >expected-files &&\n-\ttest_cmp expected-files T--shared.objects-files.raw\n+\ttest_cmp expected-files T--shared.objects-files.raw &&\n+\ttest_must_be_empty T--shared.objects-symlinks.raw\n '\n \n test_done\n-- \n2.22.0\n\n"},{"id":"377470","messageId":"60c5fc6ad28a01c767befeb06601eaf6db4ca42a.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 03/10] dir-iterator: add tests for dir-iterator API","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:40Z","receivedAt":"2019-06-18T23:29:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Daniel Ferreira \n\nCreate t/helper/test-dir-iterator.c, which prints relevant information\nabout a directory tree iterated over with dir-iterator.\n\nCreate t/t0066-dir-iterator.sh, which tests that dir-iterator does\niterate through a whole directory tree as expected.\n\nSigned-off-by: Daniel Ferreira \n[matheus.bernardino: update to use test-tool and some minor aesthetics]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n t/helper/test-dir-iterator.c | 33 ++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0066-dir-iterator.sh | 55 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 91 insertions(+)\n create mode 100644 t/helper/test-dir-iterator.c\n create mode 100755 t/t0066-dir-iterator.sh\n\ndiff --git a/Makefile b/Makefile\nindex f58bf14c7b..7e2a44cccc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -704,6 +704,7 @@ TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\n+TEST_BUILTINS_OBJS += test-dir-iterator.o\n TEST_BUILTINS_OBJS += test-drop-caches.o\n TEST_BUILTINS_OBJS += test-dump-cache-tree.o\n TEST_BUILTINS_OBJS += test-dump-fsmonitor.o\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nnew file mode 100644\nindex 0000000000..84f50bed8c\n--- /dev/null\n+++ b/t/helper/test-dir-iterator.c\n@@ -0,0 +1,33 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+\n+/* Argument is a directory path to iterate over */\n+int cmd__dir_iterator(int argc, const char **argv)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct dir_iterator *diter;\n+\n+\tif (argc < 2)\n+\t\tdie(\"BUG: test-dir-iterator needs one argument\");\n+\n+\tstrbuf_add(&path, argv[1], strlen(argv[1]));\n+\n+\tditer = dir_iterator_begin(path.buf);\n+\n+\twhile (dir_iterator_advance(diter) == ITER_OK) {\n+\t\tif (S_ISDIR(diter->st.st_mode))\n+\t\t\tprintf(\"[d] \");\n+\t\telse if (S_ISREG(diter->st.st_mode))\n+\t\t\tprintf(\"[f] \");\n+\t\telse\n+\t\t\tprintf(\"[?] \");\n+\n+\t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n+\t\t diter->path.buf);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 087a8c0cc9..7bc9bb231e 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -19,6 +19,7 @@ static struct test_cmd cmds[] = {\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\n+\t{ \"dir-iterator\", cmd__dir_iterator },\n \t{ \"drop-caches\", cmd__drop_caches },\n \t{ \"dump-cache-tree\", cmd__dump_cache_tree },\n \t{ \"dump-fsmonitor\", cmd__dump_fsmonitor },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 7e703f3038..ec0ffbd0cb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -9,6 +9,7 @@ int cmd__config(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\n+int cmd__dir_iterator(int argc, const char **argv);\n int cmd__drop_caches(int argc, const char **argv);\n int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nnew file mode 100755\nindex 0000000000..6e06dc038d\n--- /dev/null\n+++ b/t/t0066-dir-iterator.sh\n@@ -0,0 +1,55 @@\n+#!/bin/sh\n+\n+test_description='Test the dir-iterator functionality'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tmkdir -p dir &&\n+\tmkdir -p dir/a/b/c/ &&\n+\t>dir/b &&\n+\t>dir/c &&\n+\tmkdir -p dir/d/e/d/ &&\n+\t>dir/a/b/c/d &&\n+\t>dir/a/e &&\n+\t>dir/d/e/d/a &&\n+\n+\tmkdir -p dir2/a/b/c/ &&\n+\t>dir2/a/b/c/d\n+'\n+\n+test_expect_success 'dir-iterator should iterate through all files' '\n+\tcat >expected-iteration-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir/a\n+\t[d] (a/b) [b] ./dir/a/b\n+\t[d] (a/b/c) [c] ./dir/a/b/c\n+\t[d] (d) [d] ./dir/d\n+\t[d] (d/e) [e] ./dir/d/e\n+\t[d] (d/e/d) [d] ./dir/d/e/d\n+\t[f] (a/b/c/d) [d] ./dir/a/b/c/d\n+\t[f] (a/e) [e] ./dir/a/e\n+\t[f] (b) [b] ./dir/b\n+\t[f] (c) [c] ./dir/c\n+\t[f] (d/e/d/a) [a] ./dir/d/e/d/a\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir >out &&\n+\tsort ./actual-iteration-sorted-output &&\n+\n+\ttest_cmp expected-iteration-sorted-output actual-iteration-sorted-output\n+'\n+\n+test_expect_success 'dir-iterator should list files in the correct order' '\n+\tcat >expected-pre-order-output <<-EOF &&\n+\t[d] (a) [a] ./dir2/a\n+\t[d] (a/b) [b] ./dir2/a/b\n+\t[d] (a/b/c) [c] ./dir2/a/b/c\n+\t[f] (a/b/c/d) [d] ./dir2/a/b/c/d\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir2 >actual-pre-order-output &&\n+\n+\ttest_cmp expected-pre-order-output actual-pre-order-output\n+'\n+\n+test_done\n-- \n2.22.0\n\n"},{"id":"377471","messageId":"8b060b3bf4389d337d84d53cebcc8b02aa1a7a74.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 04/10] dir-iterator: use warning_errno when possible","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:41Z","receivedAt":"2019-06-18T23:29:10Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Change warning(..., strerror(errno)) by warning_errno(...). This helps\nto unify warning display besides simplifying a bit the code. Also,\nimprove warning messages by surrounding paths with quotation marks and\nusing more meaningful statements.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 23 ++++++++++++-----------\n 1 file changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..0c8880868a 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -71,8 +71,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n-\t\t\t\twarning(\"error opening directory %s: %s\",\n-\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\t\t\t\t iter->base.path.buf);\n \t\t\t\t/* Popping the level is handled below */\n \t\t\t}\n \n@@ -122,11 +122,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n-\t\t\t\t\twarning(\"error reading directory %s: %s\",\n-\t\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\t\twarning_errno(\"error reading directory '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \t\t\t\t} else if (closedir(level->dir))\n-\t\t\t\t\twarning(\"error closing directory %s: %s\",\n-\t\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \n \t\t\t\tlevel->dir = NULL;\n \t\t\t\tif (--iter->levels_nr == 0)\n@@ -140,9 +140,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n \t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n \t\t\t\tif (errno != ENOENT)\n-\t\t\t\t\twarning(\"error reading path '%s': %s\",\n-\t\t\t\t\t\titer->base.path.buf,\n-\t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t\twarning_errno(\"failed to stat '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -170,9 +169,11 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \t\t\t&iter->levels[iter->levels_nr - 1];\n \n \t\tif (level->dir && closedir(level->dir)) {\n+\t\t\tint saved_errno = errno;\n \t\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n-\t\t\twarning(\"error closing directory %s: %s\",\n-\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\terrno = saved_errno;\n+\t\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t\t iter->base.path.buf);\n \t\t}\n \t}\n \n-- \n2.22.0\n\n"},{"id":"377472","messageId":"c2466d5ad9f0680045fa16db6d8a6852c28f5066.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 05/10] dir-iterator: refactor state machine model","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:42Z","receivedAt":"2019-06-18T23:29:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"dir_iterator_advance() is a large function with two nested loops. Let's\nimprove its readability factoring out three functions and simplifying\nits mechanics. The refactored model will no longer depend on\nlevel.initialized and level.dir_state to keep track of the iteration\nstate and will perform on a single loop.\n\nAlso, dir_iterator_begin() currently does not check if the given string\nrepresents a valid directory path. Since the refactored model will have\nto stat() the given path at initialization, let's also check for this\nkind of error and make dir_iterator_begin() return NULL, on failures,\nwith errno appropriately set. And add tests for this new behavior.\n\nImprove documentation at dir-iteration.h and code comments at\ndir-iterator.c to reflect the changes and eliminate possible\nambiguities.\n\nFinally, adjust refs/files-backend.c to check for now possible\ndir_iterator_begin() failures.\n\nOriginal-patch-by: Daniel Ferreira \nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 234 ++++++++++++++++++-----------------\n dir-iterator.h | 15 ++-\n refs/files-backend.c | 17 ++-\n t/helper/test-dir-iterator.c | 5 +\n t/t0066-dir-iterator.sh | 13 ++\n 5 files changed, 163 insertions(+), 121 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex 0c8880868a..594fe4d67b 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -4,8 +4,6 @@\n #include \"dir-iterator.h\"\n \n struct dir_iterator_level {\n-\tint initialized;\n-\n \tDIR *dir;\n \n \t/*\n@@ -13,16 +11,6 @@ struct dir_iterator_level {\n \t * (including a trailing '/'):\n \t */\n \tsize_t prefix_len;\n-\n-\t/*\n-\t * The last action that has been taken with the current entry\n-\t * (needed for directories, which have to be included in the\n-\t * iteration and also iterated into):\n-\t */\n-\tenum {\n-\t\tDIR_STATE_ITER,\n-\t\tDIR_STATE_RECURSE\n-\t} dir_state;\n };\n \n /*\n@@ -34,9 +22,11 @@ struct dir_iterator_int {\n \tstruct dir_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack. After the first\n+\t * call to dir_iterator_begin(), if it succeeds to open the\n+\t * first level's dir, this will always be at least 1. Then,\n+\t * when it comes to zero the iteration is ended and this\n+\t * struct is freed.\n \t */\n \tsize_t levels_nr;\n \n@@ -50,113 +40,118 @@ struct dir_iterator_int {\n \tstruct dir_iterator_level *levels;\n };\n \n+/*\n+ * Push a level in the iter stack and initialize it with information from\n+ * the directory pointed by iter->base->path. It is assumed that this\n+ * strbuf points to a valid directory path. Return 0 on success and -1\n+ * otherwise, leaving the stack unchanged.\n+ */\n+static int push_level(struct dir_iterator_int *iter)\n+{\n+\tstruct dir_iterator_level *level;\n+\n+\tALLOC_GROW(iter->levels, iter->levels_nr + 1, iter->levels_alloc);\n+\tlevel = &iter->levels[iter->levels_nr++];\n+\n+\tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n+\t\tstrbuf_addch(&iter->base.path, '/');\n+\tlevel->prefix_len = iter->base.path.len;\n+\n+\tlevel->dir = opendir(iter->base.path.buf);\n+\tif (!level->dir) {\n+\t\tif (errno != ENOENT) {\n+\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\t\t\t iter->base.path.buf);\n+\t\t}\n+\t\titer->levels_nr--;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Pop the top level on the iter stack, releasing any resources associated\n+ * with it. Return the new value of iter->levels_nr.\n+ */\n+static int pop_level(struct dir_iterator_int *iter)\n+{\n+\tstruct dir_iterator_level *level =\n+\t\t&iter->levels[iter->levels_nr - 1];\n+\n+\tif (level->dir && closedir(level->dir))\n+\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t iter->base.path.buf);\n+\tlevel->dir = NULL;\n+\n+\treturn --iter->levels_nr;\n+}\n+\n+/*\n+ * Populate iter->base with the necessary information on the next iteration\n+ * entry, represented by the given dirent de. Return 0 on success and -1\n+ * otherwise.\n+ */\n+static int prepare_next_entry_data(struct dir_iterator_int *iter,\n+\t\t\t\t struct dirent *de)\n+{\n+\tstrbuf_addstr(&iter->base.path, de->d_name);\n+\t/*\n+\t * We have to reset these because the path strbuf might have\n+\t * been realloc()ed at the previous strbuf_addstr().\n+\t */\n+\titer->base.relative_path = iter->base.path.buf +\n+\t\t\t\t iter->levels[0].prefix_len;\n+\titer->base.basename = iter->base.path.buf +\n+\t\t\t iter->levels[iter->levels_nr - 1].prefix_len;\n+\n+\tif (lstat(iter->base.path.buf, &iter->base.st)) {\n+\t\tif (errno != ENOENT)\n+\t\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n \n+\tif (S_ISDIR(iter->base.st.st_mode)) {\n+\t\tif (push_level(iter) && iter->levels_nr == 0) {\n+\t\t\t/* Pushing the first level failed */\n+\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t}\n+\t}\n+\n+\t/* Loop until we find an entry that we can give back to the caller. */\n \twhile (1) {\n+\t\tstruct dirent *de;\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n-\t\tstruct dirent *de;\n \n-\t\tif (!level->initialized) {\n-\t\t\t/*\n-\t\t\t * Note: dir_iterator_begin() ensures that\n-\t\t\t * path is not the empty string.\n-\t\t\t */\n-\t\t\tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n-\t\t\t\tstrbuf_addch(&iter->base.path, '/');\n-\t\t\tlevel->prefix_len = iter->base.path.len;\n-\n-\t\t\tlevel->dir = opendir(iter->base.path.buf);\n-\t\t\tif (!level->dir && errno != ENOENT) {\n-\t\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n+\t\terrno = 0;\n+\t\tde = readdir(level->dir);\n+\n+\t\tif (!de) {\n+\t\t\tif (errno)\n+\t\t\t\twarning_errno(\"error reading directory '%s'\",\n \t\t\t\t\t iter->base.path.buf);\n-\t\t\t\t/* Popping the level is handled below */\n-\t\t\t}\n-\n-\t\t\tlevel->initialized = 1;\n-\t\t} else if (S_ISDIR(iter->base.st.st_mode)) {\n-\t\t\tif (level->dir_state == DIR_STATE_ITER) {\n-\t\t\t\t/*\n-\t\t\t\t * The directory was just iterated\n-\t\t\t\t * over; now prepare to iterate into\n-\t\t\t\t * it.\n-\t\t\t\t */\n-\t\t\t\tlevel->dir_state = DIR_STATE_RECURSE;\n-\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n-\t\t\t\t\t iter->levels_alloc);\n-\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n-\t\t\t\tlevel->initialized = 0;\n-\t\t\t\tcontinue;\n-\t\t\t} else {\n-\t\t\t\t/*\n-\t\t\t\t * The directory has already been\n-\t\t\t\t * iterated over and iterated into;\n-\t\t\t\t * we're done with it.\n-\t\t\t\t */\n-\t\t\t}\n+\t\t\telse if (pop_level(iter) == 0)\n+\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\tcontinue;\n \t\t}\n \n-\t\tif (!level->dir) {\n-\t\t\t/*\n-\t\t\t * This level is exhausted (or wasn't opened\n-\t\t\t * successfully); pop up a level.\n-\t\t\t */\n-\t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n \n+\t\tif (prepare_next_entry_data(iter, de))\n \t\t\tcontinue;\n-\t\t}\n \n-\t\t/*\n-\t\t * Loop until we find an entry that we can give back\n-\t\t * to the caller:\n-\t\t */\n-\t\twhile (1) {\n-\t\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n-\t\t\terrno = 0;\n-\t\t\tde = readdir(level->dir);\n-\n-\t\t\tif (!de) {\n-\t\t\t\t/* This level is exhausted; pop up a level. */\n-\t\t\t\tif (errno) {\n-\t\t\t\t\twarning_errno(\"error reading directory '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\t\t\t\t} else if (closedir(level->dir))\n-\t\t\t\t\twarning_errno(\"error closing directory '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\n-\t\t\t\tlevel->dir = NULL;\n-\t\t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n-\t\t\t\tbreak;\n-\t\t\t}\n-\n-\t\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n-\t\t\t\t\twarning_errno(\"failed to stat '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\n-\t\t\t/*\n-\t\t\t * We have to set these each time because\n-\t\t\t * the path strbuf might have been realloc()ed.\n-\t\t\t */\n-\t\t\titer->base.relative_path =\n-\t\t\t\titer->base.path.buf + iter->levels[0].prefix_len;\n-\t\t\titer->base.basename =\n-\t\t\t\titer->base.path.buf + level->prefix_len;\n-\t\t\tlevel->dir_state = DIR_STATE_ITER;\n-\n-\t\t\treturn ITER_OK;\n-\t\t}\n+\t\treturn ITER_OK;\n \t}\n }\n \n@@ -187,17 +182,32 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n-\n-\tif (!path || !*path)\n-\t\tBUG(\"empty path passed to dir_iterator_begin()\");\n+\tint saved_errno;\n \n \tstrbuf_init(&iter->base.path, PATH_MAX);\n \tstrbuf_addstr(&iter->base.path, path);\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n+\titer->levels_nr = 0;\n \n-\titer->levels_nr = 1;\n-\titer->levels[0].initialized = 0;\n+\t/*\n+\t * Note: stat already checks for NULL or empty strings and\n+\t * inexistent paths.\n+\t */\n+\tif (stat(iter->base.path.buf, &iter->base.st) < 0) {\n+\t\tsaved_errno = errno;\n+\t\tgoto error_out;\n+\t}\n+\n+\tif (!S_ISDIR(iter->base.st.st_mode)) {\n+\t\tsaved_errno = ENOTDIR;\n+\t\tgoto error_out;\n+\t}\n \n \treturn dir_iterator;\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\terrno = saved_errno;\n+\treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..0822821e56 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -8,19 +8,23 @@\n *\n * Iterate over a directory tree, recursively, including paths of all\n * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n- * symlinks except for the original path.\n+ * symlinks except for the original path. Note that the original path\n+ * is not included in the iteration.\n *\n * Every time dir_iterator_advance() is called, update the members of\n * the dir_iterator structure to reflect the next path in the\n * iteration. The order that paths are iterated over within a\n- * directory is undefined, but directory paths are always iterated\n- * over before the subdirectory contents.\n+ * directory is undefined, directory paths are always given before\n+ * their contents.\n *\n * A typical iteration looks like this:\n *\n * int ok;\n * struct iterator *iter = dir_iterator_begin(path);\n *\n+ * if (!iter)\n+ * goto error_handler;\n+ *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n * ok = dir_iterator_abort(iter);\n@@ -59,8 +63,9 @@ struct dir_iterator {\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path. On success, return a\n+ * dir_iterator that holds the internal state of the iteration.\n+ * In case of failure, return NULL and set errno accordingly.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 63e55e6773..7ed81046d4 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,13 +2143,22 @@ static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \t\t\t\t\t\t const char *gitdir)\n {\n-\tstruct files_reflog_iterator *iter = xcalloc(1, sizeof(*iter));\n-\tstruct ref_iterator *ref_iterator = &iter->base;\n+\tstruct dir_iterator *diter;\n+\tstruct files_reflog_iterator *iter;\n+\tstruct ref_iterator *ref_iterator;\n \tstruct strbuf sb = STRBUF_INIT;\n \n-\tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\n+\tditer = dir_iterator_begin(sb.buf);\n+\tif(!diter)\n+\t\treturn empty_ref_iterator_begin();\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tref_iterator = &iter->base;\n+\n+\tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n+\titer->dir_iterator = diter;\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 84f50bed8c..fab1ff6237 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -17,6 +17,11 @@ int cmd__dir_iterator(int argc, const char **argv)\n \n \tditer = dir_iterator_begin(path.buf);\n \n+\tif (!diter) {\n+\t\tprintf(\"dir_iterator_begin failure: %d\\n\", errno);\n+\t\texit(EXIT_FAILURE);\n+\t}\n+\n \twhile (dir_iterator_advance(diter) == ITER_OK) {\n \t\tif (S_ISDIR(diter->st.st_mode))\n \t\t\tprintf(\"[d] \");\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex 6e06dc038d..c739ed7911 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -52,4 +52,17 @@ test_expect_success 'dir-iterator should list files in the correct order' '\n \ttest_cmp expected-pre-order-output actual-pre-order-output\n '\n \n+test_expect_success 'begin should fail upon inexistent paths' '\n+\ttest_must_fail test-tool dir-iterator ./inexistent-path \\\n+\t\t>actual-inexistent-path-output &&\n+\techo \"dir_iterator_begin failure: 2\" >expected-inexistent-path-output &&\n+\ttest_cmp expected-inexistent-path-output actual-inexistent-path-output\n+'\n+\n+test_expect_success 'begin should fail upon non directory paths' '\n+\ttest_must_fail test-tool dir-iterator ./dir/b >actual-non-dir-output &&\n+\techo \"dir_iterator_begin failure: 20\" >expected-non-dir-output &&\n+\ttest_cmp expected-non-dir-output actual-non-dir-output\n+'\n+\n test_done\n-- \n2.22.0\n\n"},{"id":"377473","messageId":"5a678ee74de42f1373deeed718fa24d368347d13.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:43Z","receivedAt":"2019-06-18T23:29:27Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are:\n- DIR_ITERATOR_PEDANTIC, which makes dir_iterator_advance abort\nimmediately in the case of an error, instead of keep looking for the\nnext valid entry;\n- DIR_ITERATOR_FOLLOW_SYMLINKS, which makes the iterator follow\nsymlinks and include linked directories' contents in the iteration.\n\nThese new flags will be used in a subsequent patch.\n\nAlso add tests for the flags' usage and adjust refs/files-backend.c to\nthe new dir_iterator_begin signature.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 82 +++++++++++++++++++++++++------\n dir-iterator.h | 51 ++++++++++++++-----\n refs/files-backend.c | 2 +-\n t/helper/test-dir-iterator.c | 34 ++++++++++---\n t/t0066-dir-iterator.sh | 95 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 229 insertions(+), 35 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex 594fe4d67b..52db87bdc9 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -6,6 +6,9 @@\n struct dir_iterator_level {\n \tDIR *dir;\n \n+\t/* The inode number of this level's directory. */\n+\tino_t ino;\n+\n \t/*\n \t * The length of the directory part of path at this level\n \t * (including a trailing '/'):\n@@ -38,13 +41,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned int flags;\n };\n \n /*\n * Push a level in the iter stack and initialize it with information from\n * the directory pointed by iter->base->path. It is assumed that this\n * strbuf points to a valid directory path. Return 0 on success and -1\n- * otherwise, leaving the stack unchanged.\n+ * otherwise, setting errno accordingly and leaving the stack unchanged.\n */\n static int push_level(struct dir_iterator_int *iter)\n {\n@@ -56,14 +62,17 @@ static int push_level(struct dir_iterator_int *iter)\n \tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n \t\tstrbuf_addch(&iter->base.path, '/');\n \tlevel->prefix_len = iter->base.path.len;\n+\tlevel->ino = iter->base.st.st_ino;\n \n \tlevel->dir = opendir(iter->base.path.buf);\n \tif (!level->dir) {\n+\t\tint saved_errno = errno;\n \t\tif (errno != ENOENT) {\n \t\t\twarning_errno(\"error opening directory '%s'\",\n \t\t\t\t iter->base.path.buf);\n \t\t}\n \t\titer->levels_nr--;\n+\t\terrno = saved_errno;\n \t\treturn -1;\n \t}\n \n@@ -90,11 +99,13 @@ static int pop_level(struct dir_iterator_int *iter)\n /*\n * Populate iter->base with the necessary information on the next iteration\n * entry, represented by the given dirent de. Return 0 on success and -1\n- * otherwise.\n+ * otherwise, setting errno accordingly.\n */\n static int prepare_next_entry_data(struct dir_iterator_int *iter,\n \t\t\t\t struct dirent *de)\n {\n+\tint err, saved_errno;\n+\n \tstrbuf_addstr(&iter->base.path, de->d_name);\n \t/*\n \t * We have to reset these because the path strbuf might have\n@@ -105,12 +116,34 @@ static int prepare_next_entry_data(struct dir_iterator_int *iter,\n \titer->base.basename = iter->base.path.buf +\n \t\t\t iter->levels[iter->levels_nr - 1].prefix_len;\n \n-\tif (lstat(iter->base.path.buf, &iter->base.st)) {\n-\t\tif (errno != ENOENT)\n-\t\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n-\t\treturn -1;\n-\t}\n+\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\terr = stat(iter->base.path.buf, &iter->base.st);\n+\telse\n+\t\terr = lstat(iter->base.path.buf, &iter->base.st);\n+\n+\tsaved_errno = errno;\n+\tif (err && errno != ENOENT)\n+\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n+\n+\terrno = saved_errno;\n+\treturn err;\n+}\n+\n+/*\n+ * Look for a recursive symlink at iter->base.path pointing to any directory on\n+ * the previous stack levels. If it is found, return 1. If not, return 0.\n+ */\n+static int find_recursive_symlinks(struct dir_iterator_int *iter)\n+{\n+\tint i;\n+\n+\tif (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n+\t !S_ISDIR(iter->base.st.st_mode))\n+\t\treturn 0;\n \n+\tfor (i = 0; i < iter->levels_nr; ++i)\n+\t\tif (iter->base.st.st_ino == iter->levels[i].ino)\n+\t\t\treturn 1;\n \treturn 0;\n }\n \n@@ -119,11 +152,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n \n-\tif (S_ISDIR(iter->base.st.st_mode)) {\n-\t\tif (push_level(iter) && iter->levels_nr == 0) {\n-\t\t\t/* Pushing the first level failed */\n-\t\t\treturn dir_iterator_abort(dir_iterator);\n-\t\t}\n+\tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n+\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\tgoto error_out;\n+\t\tif (iter->levels_nr == 0)\n+\t\t\tgoto error_out;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -137,22 +170,38 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\tde = readdir(level->dir);\n \n \t\tif (!de) {\n-\t\t\tif (errno)\n+\t\t\tif (errno) {\n \t\t\t\twarning_errno(\"error reading directory '%s'\",\n \t\t\t\t\t iter->base.path.buf);\n-\t\t\telse if (pop_level(iter) == 0)\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n+\t\t\t} else if (pop_level(iter) == 0) {\n \t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t}\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n-\t\tif (prepare_next_entry_data(iter, de))\n+\t\tif (prepare_next_entry_data(iter, de)) {\n+\t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\tgoto error_out;\n \t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (find_recursive_symlinks(iter)) {\n+\t\t\twarning(\"ignoring recursive symlink at '%s'\",\n+\t\t\t\titer->base.path.buf);\n+\t\t\tcontinue;\n+\t\t}\n \n \t\treturn ITER_OK;\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -178,7 +227,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -189,6 +238,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \titer->levels_nr = 0;\n+\titer->flags = flags;\n \n \t/*\n \t * Note: stat already checks for NULL or empty strings and\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 0822821e56..42751091a5 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -20,7 +20,8 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * unsigned int flags = DIR_ITERATOR_PEDANTIC;\n+ * struct dir_iterator *iter = dir_iterator_begin(path, flags);\n *\n * if (!iter)\n * goto error_handler;\n@@ -44,6 +45,25 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error at dir_iterator_advance(), which is to keep\n+ * looking for a next valid entry. With this flag, resources are freed\n+ * and ITER_ERROR is returned immediately. In both cases, a meaningful\n+ * warning is emitted. Note: ENOENT errors are always ignored so that\n+ * the API users may remove files during iteration.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks.\n+ * i.e., linked directories' contents will be iterated over and\n+ * iter->base.st will contain information on the referred files,\n+ * not the symlinks themselves, which is the default behavior.\n+ * Recursive symlinks are skipped with a warning and broken symlinks\n+ * are ignored.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -58,29 +78,38 @@ struct dir_iterator {\n \t/* The current basename: */\n \tconst char *basename;\n \n-\t/* The result of calling lstat() on path: */\n+\t/*\n+\t * The result of calling lstat() on path; or stat(), if the\n+\t * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at\n+\t * dir_iterator's initialization.\n+\t */\n \tstruct stat st;\n };\n \n /*\n- * Start a directory iteration over path. On success, return a\n- * dir_iterator that holds the internal state of the iteration.\n- * In case of failure, return NULL and set errno accordingly.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. On success, return a dir_iterator\n+ * that holds the internal state of the iteration. In case of\n+ * failure, return NULL and set errno accordingly.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behavior.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n * If the iteration is exhausted, free the dir_iterator and any\n- * resources associated with it and return ITER_DONE. On error, free\n- * dir_iterator and associated resources and return ITER_ERROR. It is\n- * a bug to use iterator or call this function again after it has\n- * returned ITER_DONE or ITER_ERROR.\n+ * resources associated with it and return ITER_DONE.\n+ *\n+ * It is a bug to use iterator or call this function again after it\n+ * has returned ITER_DONE or ITER_ERROR (which may be returned iff\n+ * the DIR_ITERATOR_PEDANTIC flag was set).\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 7ed81046d4..b1f8f53a09 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2150,7 +2150,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n \n-\tditer = dir_iterator_begin(sb.buf);\n+\tditer = dir_iterator_begin(sb.buf, 0);\n \tif(!diter)\n \t\treturn empty_ref_iterator_begin();\n \ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex fab1ff6237..a5b96cb0dc 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -4,29 +4,44 @@\n #include \"iterator.h\"\n #include \"dir-iterator.h\"\n \n-/* Argument is a directory path to iterate over */\n+/*\n+ * usage:\n+ * tool-test dir-iterator [--follow-symlinks] [--pedantic] directory_path\n+ */\n int cmd__dir_iterator(int argc, const char **argv)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct dir_iterator *diter;\n+\tunsigned int flags = 0;\n+\tint iter_status;\n+\n+\tfor (++argv, --argc; *argv && starts_with(*argv, \"--\"); ++argv, --argc) {\n+\t\tif (strcmp(*argv, \"--follow-symlinks\") == 0)\n+\t\t\tflags |= DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\t\telse if (strcmp(*argv, \"--pedantic\") == 0)\n+\t\t\tflags |= DIR_ITERATOR_PEDANTIC;\n+\t\telse\n+\t\t\tdie(\"invalid option '%s'\", *argv);\n+\t}\n \n-\tif (argc < 2)\n-\t\tdie(\"BUG: test-dir-iterator needs one argument\");\n-\n-\tstrbuf_add(&path, argv[1], strlen(argv[1]));\n+\tif (!*argv || argc != 1)\n+\t\tdie(\"dir-iterator needs exactly one non-option argument\");\n \n-\tditer = dir_iterator_begin(path.buf);\n+\tstrbuf_add(&path, *argv, strlen(*argv));\n+\tditer = dir_iterator_begin(path.buf, flags);\n \n \tif (!diter) {\n \t\tprintf(\"dir_iterator_begin failure: %d\\n\", errno);\n \t\texit(EXIT_FAILURE);\n \t}\n \n-\twhile (dir_iterator_advance(diter) == ITER_OK) {\n+\twhile ((iter_status = dir_iterator_advance(diter)) == ITER_OK) {\n \t\tif (S_ISDIR(diter->st.st_mode))\n \t\t\tprintf(\"[d] \");\n \t\telse if (S_ISREG(diter->st.st_mode))\n \t\t\tprintf(\"[f] \");\n+\t\telse if (S_ISLNK(diter->st.st_mode))\n+\t\t\tprintf(\"[s] \");\n \t\telse\n \t\t\tprintf(\"[?] \");\n \n@@ -34,5 +49,10 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\t diter->path.buf);\n \t}\n \n+\tif (iter_status != ITER_DONE) {\n+\t\tprintf(\"dir_iterator_advance failure\\n\");\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex c739ed7911..8f996a31fa 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -65,4 +65,99 @@ test_expect_success 'begin should fail upon non directory paths' '\n \ttest_cmp expected-non-dir-output actual-non-dir-output\n '\n \n+test_expect_success POSIXPERM,SANITY 'advance should not fail on errors by default' '\n+\tcat >expected-no-permissions-output <<-EOF &&\n+\t[d] (a) [a] ./dir3/a\n+\tEOF\n+\n+\tmkdir -p dir3/a &&\n+\t> dir3/a/b &&\n+\tchmod 0 dir3/a &&\n+\n+\ttest-tool dir-iterator ./dir3 >actual-no-permissions-output &&\n+\ttest_cmp expected-no-permissions-output actual-no-permissions-output &&\n+\tchmod 755 dir3/a &&\n+\trm -rf dir3\n+'\n+\n+test_expect_success POSIXPERM,SANITY 'advance should fail on errors, w/ pedantic flag' '\n+\tcat >expected-no-permissions-pedantic-output <<-EOF &&\n+\t[d] (a) [a] ./dir3/a\n+\tdir_iterator_advance failure\n+\tEOF\n+\n+\tmkdir -p dir3/a &&\n+\t> dir3/a/b &&\n+\tchmod 0 dir3/a &&\n+\n+\ttest_must_fail test-tool dir-iterator --pedantic ./dir3 \\\n+\t\t>actual-no-permissions-pedantic-output &&\n+\ttest_cmp expected-no-permissions-pedantic-output \\\n+\t\tactual-no-permissions-pedantic-output &&\n+\tchmod 755 dir3/a &&\n+\trm -rf dir3\n+'\n+\n+test_expect_success SYMLINKS 'setup dirs with symlinks' '\n+\tmkdir -p dir4/a &&\n+\tmkdir -p dir4/b/c &&\n+\t>dir4/a/d &&\n+\tln -s d dir4/a/e &&\n+\tln -s ../b dir4/a/f &&\n+\n+\tmkdir -p dir5/a/b &&\n+\tmkdir -p dir5/a/c &&\n+\tln -s ../c dir5/a/b/d &&\n+\tln -s ../ dir5/a/b/e &&\n+\tln -s ../../ dir5/a/b/f\n+'\n+\n+test_expect_success SYMLINKS 'dir-iterator should not follow symlinks by default' '\n+\tcat >expected-no-follow-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir4/a\n+\t[d] (b) [b] ./dir4/b\n+\t[d] (b/c) [c] ./dir4/b/c\n+\t[f] (a/d) [d] ./dir4/a/d\n+\t[s] (a/e) [e] ./dir4/a/e\n+\t[s] (a/f) [f] ./dir4/a/f\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir4 >out &&\n+\tsort actual-no-follow-sorted-output &&\n+\n+\ttest_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output\n+'\n+\n+test_expect_success SYMLINKS 'dir-iterator should follow symlinks w/ follow flag' '\n+\tcat >expected-follow-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir4/a\n+\t[d] (a/f) [f] ./dir4/a/f\n+\t[d] (a/f/c) [c] ./dir4/a/f/c\n+\t[d] (b) [b] ./dir4/b\n+\t[d] (b/c) [c] ./dir4/b/c\n+\t[f] (a/d) [d] ./dir4/a/d\n+\t[f] (a/e) [e] ./dir4/a/e\n+\tEOF\n+\n+\ttest-tool dir-iterator --follow-symlinks ./dir4 >out &&\n+\tsort actual-follow-sorted-output &&\n+\n+\ttest_cmp expected-follow-sorted-output actual-follow-sorted-output\n+'\n+\n+\n+test_expect_success SYMLINKS 'dir-iterator should ignore recursive symlinks w/ follow flag' '\n+\tcat >expected-rec-symlinks-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir5/a\n+\t[d] (a/b) [b] ./dir5/a/b\n+\t[d] (a/b/d) [d] ./dir5/a/b/d\n+\t[d] (a/c) [c] ./dir5/a/c\n+\tEOF\n+\n+\ttest-tool dir-iterator --follow-symlinks ./dir5 >out &&\n+\tsort actual-rec-symlinks-sorted-output &&\n+\n+\ttest_cmp expected-rec-symlinks-sorted-output actual-rec-symlinks-sorted-output\n+'\n+\n test_done\n-- \n2.22.0\n\n"},{"id":"377474","messageId":"5dc1cb8f840dcb01692d54183356b5c9c8ae169f.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 07/10] clone: copy hidden paths at local clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:44Z","receivedAt":"2019-06-18T23:29:32Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make the copy_or_link_directory function no longer skip hidden\ndirectories. This function, used to copy .git/objects, currently skips\nall hidden directories but not hidden files, which is an odd behaviour.\nThe reason for that could be unintentional: probably the intention was\nto skip '.' and '..' only but it ended up accidentally skipping all\ndirectories starting with '.'. Besides being more natural, the new\nbehaviour is more permissive to the user.\n\nAlso adjust tests to reflect this behaviour change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4a0a2455a7..9dd083e34d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -430,7 +430,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 0800c3853f..c3998f2f9e 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \tdone &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.22.0\n\n"},{"id":"377475","messageId":"8a3f7103366cef36d1c505251d89d64cdf274889.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 08/10] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:45Z","receivedAt":"2019-06-18T23:29:39Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Extract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help to remove\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes the code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9dd083e34d..96566c1bab 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -394,6 +394,21 @@ static void copy_alternates(struct strbuf *src, const char *src_repo)\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\tstruct stat st;\n+\n+\tif (!mkdir(pathname, mode))\n+\t\treturn;\n+\n+\tif (errno != EEXIST)\n+\t\tdie_errno(_(\"failed to create directory '%s'\"), pathname);\n+\telse if (stat(pathname, &st))\n+\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\telse if (!S_ISDIR(st.st_mode))\n+\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -406,14 +421,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.22.0\n\n"},{"id":"377476","messageId":"47ccb3be1a92a30cbe084730e4a139f049da1351.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 09/10] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:46Z","receivedAt":"2019-06-18T23:29:44Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoids recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone could succeed even\nthough the .git/objects copy didn't fully succeed. Also, with the\ndir-iterator API, recursive symlinks will be detected and skipped. This\nis another behavior improvement, since the current version would\ncontinue to copy the same content over and over until stat() returned an\nELOOP error.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 47 +++++++++++++++++++++++++----------------------\n 1 file changed, 25 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 96566c1bab..47cb4a2a8e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -410,42 +412,39 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tunsigned int flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n+\tif (!iter)\n+\t\tdie_errno(_(\"failed to start iterator over '%s'\"), src->buf);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -462,7 +461,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -480,7 +483,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.22.0\n\n"},{"id":"377477","messageId":"1ba1e9c395237aa34e698d5e5d2d0bbd875f6117.1560898723.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v7 10/10] clone: replace strcmp by fspathcmp","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-18T23:27:47Z","receivedAt":"2019-06-18T23:29:49Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace the use of strcmp by fspathcmp at copy_or_link_directory, which\nis more permissive/friendly to case-insensitive file systems.\n\nSuggested-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 47cb4a2a8e..8da696ef30 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -444,7 +444,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n+\t\tif (!fspathcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, src_repo);\n \t\t\tcontinue;\n \t\t}\n-- \n2.22.0\n\n"},{"id":"377484","messageId":"CAHd-oW65UE6prjSv-wBe5rON2aMtNDZFsuyJuZMef8KzKKdC0A@mail.gmail.com","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v7 00/10] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-19T04:36:40Z","receivedAt":"2019-06-19T04:36:54Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Jun 18, 2019 at 8:28 PM Matheus Tavares\n wrote:\n>\n> This patchset contains:\n> - tests to the dir-iterator API;\n> - dir-iterator refactoring to make its state machine simpler\n> and feature adding with tests;\n> - a replacement of explicit recursive dir iteration at\n> copy_or_link_directory for the dir-iterator API;\n> - some refactoring and behavior changes at local clone, mainly to\n> take care of symlinks and hidden files at .git/objects, together\n> with tests for these types of files.\n>\n> Changes since v6:\n> - Rebased with master;\n> - Added to dir-iterator documentation that ENOENT errors and hence broken\n> symlinks are both ignored.\n>\n> With the changes brought by this patchset, dir_iterator_begin() may now\n> return NULL (setting errno) when it finds an error. Also, it's possible\n> to pass a pedantic flag to it so that dir_iterator_advance() return\n> immediately on errors. But at refs/files-backend.c, the only user of\n> the API so far, the flag wasn't used and an empty iterator is\n> returned in case of errors at dir_iterator_begin(). These actions were\n> taken in order to keep the files-backend's behavior as close as\n> possible to the one it previously had. But since it already has guards\n> for possible errors at dir_iterator_advance(), I'm wondering whether I\n> should send a follow-up patch making it use the pedantic flag.\n>\n> Also, should I perhaps call die_errno() on dir_iterator_begin() errors\n> at files-backend? I mean, we should continue returning an empty\n> iterator on ENOENT errors since \".git/logs\", the dir it iterates over,\n> may not be present. But we could possibly abort on other errors, just\n> to be sure...\n\nI got ahead of myself in this last paragraph. \".git/logs\" is one of the dirs\nthat files-backend.c is used to iterate over, but it doesn't mean it's the only\none. This dir, in particular, is iterated when we run 'git rev-list\n--reflog', for\nexample. And upon ENOENTs, the iteration is expected to end\nsuccessfully but with no entries.\n\n(also adding Michael and Daniel to CC, in case they have some input on\nthese ideas)\n"},{"id":"377697","messageId":"xmqqef3om2nr.fsf@gitster-ct.c.googlers.com","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v7 00/10] clone: dir-iterator refactoring with tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-20T20:18:16Z","receivedAt":"2019-06-20T20:18:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n> Daniel Ferreira (1):\n> dir-iterator: add tests for dir-iterator API\n>\n> Matheus Tavares (8):\n> clone: better handle symlinked files at .git/objects/\n> dir-iterator: use warning_errno when possible\n> dir-iterator: refactor state machine model\n> dir-iterator: add flags parameter to dir_iterator_begin\n> clone: copy hidden paths at local clone\n> clone: extract function from copy_or_link_directory\n> clone: use dir-iterator to avoid explicit dir traversal\n> clone: replace strcmp by fspathcmp\n>\n> Ævar Arnfjörð Bjarmason (1):\n> clone: test for our behavior on odd objects/* content\n>\n> Makefile | 1 +\n> builtin/clone.c | 75 +++++----\n> dir-iterator.c | 289 +++++++++++++++++++++--------------\n> dir-iterator.h | 60 ++++++--\n> refs/files-backend.c | 17 ++-\n> t/helper/test-dir-iterator.c | 58 +++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t0066-dir-iterator.sh | 163 ++++++++++++++++++++\n> t/t5604-clone-reference.sh | 133 ++++++++++++++++\n> 10 files changed, 635 insertions(+), 163 deletions(-)\n> create mode 100644 t/helper/test-dir-iterator.c\n> create mode 100755 t/t0066-dir-iterator.sh\n\nA higher level question is what's the benefit of using dir-iterator\nAPI in the first place. After subtracting 356 added lines to t/,\nit still adds 279 lines while removing only 163 lines, so it is not\nlike \"we have a perfect dir-iterator API that can be applied as-is\nbut an older code that predates dir-iterator API was still using an\nold way, so let's make the latter use the former.\"\n\n\n"},{"id":"377743","messageId":"CAHd-oW5y0_2YVtKiMoSAy7f58Yj99baLsrnn=NrH_Fom+hRsVg@mail.gmail.com","threadId":"50591","inReplyTo":"xmqqef3om2nr.fsf@gitster-ct.c.googlers.com","subject":"Re: [GSoC][PATCH v7 00/10] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-21T13:41:32Z","receivedAt":"2019-06-21T13:41:45Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Jun 20, 2019 at 5:18 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > Daniel Ferreira (1):\n> > dir-iterator: add tests for dir-iterator API\n> >\n> > Matheus Tavares (8):\n> > clone: better handle symlinked files at .git/objects/\n> > dir-iterator: use warning_errno when possible\n> > dir-iterator: refactor state machine model\n> > dir-iterator: add flags parameter to dir_iterator_begin\n> > clone: copy hidden paths at local clone\n> > clone: extract function from copy_or_link_directory\n> > clone: use dir-iterator to avoid explicit dir traversal\n> > clone: replace strcmp by fspathcmp\n> >\n> > Ævar Arnfjörð Bjarmason (1):\n> > clone: test for our behavior on odd objects/* content\n> >\n> > Makefile | 1 +\n> > builtin/clone.c | 75 +++++----\n> > dir-iterator.c | 289 +++++++++++++++++++++--------------\n> > dir-iterator.h | 60 ++++++--\n> > refs/files-backend.c | 17 ++-\n> > t/helper/test-dir-iterator.c | 58 +++++++\n> > t/helper/test-tool.c | 1 +\n> > t/helper/test-tool.h | 1 +\n> > t/t0066-dir-iterator.sh | 163 ++++++++++++++++++++\n> > t/t5604-clone-reference.sh | 133 ++++++++++++++++\n> > 10 files changed, 635 insertions(+), 163 deletions(-)\n> > create mode 100644 t/helper/test-dir-iterator.c\n> > create mode 100755 t/t0066-dir-iterator.sh\n>\n> A higher level question is what's the benefit of using dir-iterator\n> API in the first place. After subtracting 356 added lines to t/,\n> it still adds 279 lines while removing only 163 lines, so it is not\n> like \"we have a perfect dir-iterator API that can be applied as-is\n> but an older code that predates dir-iterator API was still using an\n> old way, so let's make the latter use the former.\"\n>\n\nYes, indeed the dir-iterator API didn't nicely fit in clone without\nsome tweaking. Yet I think most of those line additions were not only\nto adjust the API, but also trying to improve both dir-iterator and\nlocal clone (I should have maybe split those changes into other\npatchsets, though). For example, these changes make local clone better\nhandle possible symlinks and hidden files at git dir. And the API\nchanges should make it easier to apply it as-is in other sections of\nthe codebase from now on.\n\nAs for the benefit of using the API here, I think it mainly resides in\nthe security it brings, avoiding recursive iteration (even though it\nshould be shallow in local clone) and more carefully handling\nsymlinks.\n"},{"id":"378032","messageId":"xmqqimstfsui.fsf@gitster-ct.c.googlers.com","threadId":"50591","inReplyTo":"5a678ee74de42f1373deeed718fa24d368347d13.1560898723.git.matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-25T18:00:21Z","receivedAt":"2019-06-25T18:00:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\nThis hunk, which claims to have 25 lines in the postimage ...\n\n> @@ -44,6 +45,25 @@\n> * dir_iterator_advance() again.\n> */\n> \n> +/*\n> + * Flags for dir_iterator_begin:\n> + *\n> + * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n> + * in case of an error at dir_iterator_advance(), which is to keep\n> + * looking for a next valid entry. With this flag, resources are freed\n> + * and ITER_ERROR is returned immediately. In both cases, a meaningful\n> + * warning is emitted. Note: ENOENT errors are always ignored so that\n> + * the API users may remove files during iteration.\n> + *\n> + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks.\n> + * i.e., linked directories' contents will be iterated over and\n> + * iter->base.st will contain information on the referred files,\n> + * not the symlinks themselves, which is the default behavior.\n> + * Recursive symlinks are skipped with a warning and broken symlinks\n> + * are ignored.\n> + */\n> +#define DIR_ITERATOR_PEDANTIC (1 << 0)\n> +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n> +\n> struct dir_iterator {\n> \t/* The current path: */\n> \tstruct strbuf path;\n> @@ -58,29 +78,38 @@ struct dir_iterator {\n\n... adds 20 lines, making the postimage 26 lines long.\n\nDid you hand edit your patch? It is OK to do so, as long as you\nknow what you are doing ;-). Adjust the length of the postimage on\nthe @@ ... @@ line to make it consistent with the patch text, and\nalso make sure a tweak you do here won't make later patches not\napply.\n"},{"id":"378034","messageId":"CAHd-oW5UCqnXDoLcx5=BRBwobHF52h-wJUBJh2B3-sy-p1nS5Q@mail.gmail.com","threadId":"50591","inReplyTo":"xmqqimstfsui.fsf@gitster-ct.c.googlers.com","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-25T18:11:37Z","receivedAt":"2019-06-25T18:11:50Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Jun 25, 2019 at 3:00 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> This hunk, which claims to have 25 lines in the postimage ...\n>\n> > @@ -44,6 +45,25 @@\n> > * dir_iterator_advance() again.\n> > */\n> >\n> > +/*\n> > + * Flags for dir_iterator_begin:\n> > + *\n> > + * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n> > + * in case of an error at dir_iterator_advance(), which is to keep\n> > + * looking for a next valid entry. With this flag, resources are freed\n> > + * and ITER_ERROR is returned immediately. In both cases, a meaningful\n> > + * warning is emitted. Note: ENOENT errors are always ignored so that\n> > + * the API users may remove files during iteration.\n> > + *\n> > + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks.\n> > + * i.e., linked directories' contents will be iterated over and\n> > + * iter->base.st will contain information on the referred files,\n> > + * not the symlinks themselves, which is the default behavior.\n> > + * Recursive symlinks are skipped with a warning and broken symlinks\n> > + * are ignored.\n> > + */\n> > +#define DIR_ITERATOR_PEDANTIC (1 << 0)\n> > +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n> > +\n> > struct dir_iterator {\n> > /* The current path: */\n> > struct strbuf path;\n> > @@ -58,29 +78,38 @@ struct dir_iterator {\n>\n> ... adds 20 lines, making the postimage 26 lines long.\n>\n> Did you hand edit your patch? It is OK to do so, as long as you\n> know what you are doing ;-). Adjust the length of the postimage on\n> the @@ ... @@ line to make it consistent with the patch text, and\n> also make sure a tweak you do here won't make later patches not\n> apply.\n\nOh, I'm sorry for that, I'll be more careful with hand editing next\ntime. Thanks for the advice. I think for this time it won't affect the\nlater patches as it was a minor addition at one comment, but should I\nperhaps re-send it?\n"},{"id":"378062","messageId":"nycvar.QRO.7.76.6.1906261528360.44@tvgsbejvaqbjf.bet","threadId":"50591","inReplyTo":"5a678ee74de42f1373deeed718fa24d368347d13.1560898723.git.matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-26T13:34:52Z","receivedAt":"2019-06-26T13:34:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Tue, 18 Jun 2019, Matheus Tavares wrote:\n\n>[...]\n> +/*\n> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> + */\n> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> +{\n> +\tint i;\n> +\n> +\tif (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> +\t !S_ISDIR(iter->base.st.st_mode))\n> +\t\treturn 0;\n>\n> +\tfor (i = 0; i < iter->levels_nr; ++i)\n> +\t\tif (iter->base.st.st_ino == iter->levels[i].ino)\n\nThis does not work on Windows. Remember, Git relies on (too) many areas\nwhere Linux is strong, and the `lstat()` call is one of them. Therefore,\nGit overuses that call.\n\nIn the Git for Windows project, we struggled a bit to emulate it in the\nbest way.\n\nIt is pretty expensive, for example, to find out the number of hard\nlinks, the device ID, an equivalent of the inode, etc. Many `lstat()`\ncalls are really only interested in the `mtime`, though, meaning that we\nwould waste a ton of time if we tried to be more faithful in our `lstat()`\nemulation.\n\nTherefore, we simply assign `0` as inode.\n\nSure, this violates the POSIX standard, but imagine this: the FAT\nfilesystem (which is still in use!) does not have _anything_ resembling\ninodes.\n\nI fear, therefore, that we will require at least a workaround for the\nsituation where `st_ino` is always zero.\n\nCiao,\nJohannes\n"},{"id":"378072","messageId":"xmqqwoh8cjeg.fsf@gitster-ct.c.googlers.com","threadId":"50591","inReplyTo":"nycvar.QRO.7.76.6.1906261528360.44@tvgsbejvaqbjf.bet","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-26T18:04:55Z","receivedAt":"2019-06-26T18:05:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Hi Matheus,\n>\n> On Tue, 18 Jun 2019, Matheus Tavares wrote:\n>\n>>[...]\n>> +/*\n>> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n>> + * the previous stack levels. If it is found, return 1. If not, return 0.\n>> + */\n>> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n>> +{\n>> +\tint i;\n>> +\n>> +\tif (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n>> +\t !S_ISDIR(iter->base.st.st_mode))\n>> +\t\treturn 0;\n>>\n>> +\tfor (i = 0; i < iter->levels_nr; ++i)\n>> +\t\tif (iter->base.st.st_ino == iter->levels[i].ino)\n>\n> This does not work on Windows. [[ Windows port does not have\n> usable st_ino field ]]]\n\nAnd if you cross mountpoint, st_ino alone does not guarantee\nuniqueness; you'd need to combine it with st_dev, I would think,\neven on POSIX systems.\n\n"},{"id":"378134","messageId":"CACsJy8BHWvbL_rPuRN-=Q4sDL+wkJUNAoPhdjcrs+VqY5-Kr5w@mail.gmail.com","threadId":"50591","inReplyTo":"xmqqwoh8cjeg.fsf@gitster-ct.c.googlers.com","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-06-27T09:20:51Z","receivedAt":"2019-06-27T09:21:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 27, 2019 at 1:04 AM Junio C Hamano wrote:\n>\n> Johannes Schindelin writes:\n>\n> > Hi Matheus,\n> >\n> > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> >\n> >>[...]\n> >> +/*\n> >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> >> + */\n> >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> >> +{\n> >> + int i;\n> >> +\n> >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> >> + !S_ISDIR(iter->base.st.st_mode))\n> >> + return 0;\n> >>\n> >> + for (i = 0; i < iter->levels_nr; ++i)\n> >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> >\n> > This does not work on Windows. [[ Windows port does not have\n> > usable st_ino field ]]]\n>\n> And if you cross mountpoint, st_ino alone does not guarantee\n> uniqueness; you'd need to combine it with st_dev, I would think,\n> even on POSIX systems.\n\nwhich should be protected by USE_STDEV. There's another code that\nignore st_ino on Windows in entry.c. Maybe it's time to define\nUSE_STINO instead of spreading \"#if GIT_WINDOWS_NATIVE\" more.\n-- \nDuy\n"},{"id":"378175","messageId":"CAHd-oW5zea9wzobAQG4FzN-KSS5BsRoM5vf_x_F83=yqGRfUQw@mail.gmail.com","threadId":"50591","inReplyTo":"xmqqwoh8cjeg.fsf@gitster-ct.c.googlers.com","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-27T17:23:19Z","receivedAt":"2019-06-27T17:23:33Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Jun 26, 2019 at 3:04 PM Junio C Hamano wrote:\n>\n> Johannes Schindelin writes:\n>\n> > Hi Matheus,\n> >\n> > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> >\n> >>[...]\n> >> +/*\n> >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> >> + */\n> >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> >> +{\n> >> + int i;\n> >> +\n> >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> >> + !S_ISDIR(iter->base.st.st_mode))\n> >> + return 0;\n> >>\n> >> + for (i = 0; i < iter->levels_nr; ++i)\n> >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> >\n> > This does not work on Windows. [[ Windows port does not have\n> > usable st_ino field ]]]\n>\n> And if you cross mountpoint, st_ino alone does not guarantee\n> uniqueness; you'd need to combine it with st_dev, I would think,\n> even on POSIX systems.\n\nOk, thanks for letting me know. I'm trying to think of another\napproach to test for recursive symlinks that does not rely on inode:\nGiven any symlink, we could get its real_path() and compare it with\nthe path of the directory current being iterated. If the first is a\nprefix of the second, than we mark it as a recursive symlink.\n\nWhat do you think of this idea?\n"},{"id":"378187","messageId":"nycvar.QRO.7.76.6.1906272046180.44@tvgsbejvaqbjf.bet","threadId":"50591","inReplyTo":"CAHd-oW5zea9wzobAQG4FzN-KSS5BsRoM5vf_x_F83=yqGRfUQw@mail.gmail.com","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-27T18:48:02Z","receivedAt":"2019-06-27T18:48:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n\n> On Wed, Jun 26, 2019 at 3:04 PM Junio C Hamano wrote:\n> >\n> > Johannes Schindelin writes:\n> >\n> > > Hi Matheus,\n> > >\n> > > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> > >\n> > >>[...]\n> > >> +/*\n> > >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> > >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> > >> + */\n> > >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> > >> +{\n> > >> + int i;\n> > >> +\n> > >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> > >> + !S_ISDIR(iter->base.st.st_mode))\n> > >> + return 0;\n> > >>\n> > >> + for (i = 0; i < iter->levels_nr; ++i)\n> > >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> > >\n> > > This does not work on Windows. [[ Windows port does not have\n> > > usable st_ino field ]]]\n> >\n> > And if you cross mountpoint, st_ino alone does not guarantee\n> > uniqueness; you'd need to combine it with st_dev, I would think,\n> > even on POSIX systems.\n>\n> Ok, thanks for letting me know. I'm trying to think of another\n> approach to test for recursive symlinks that does not rely on inode:\n> Given any symlink, we could get its real_path() and compare it with\n> the path of the directory current being iterated. If the first is a\n> prefix of the second, than we mark it as a recursive symlink.\n>\n> What do you think of this idea?\n\nI think this would be pretty expensive. Too expensive.\n\nA better method might be to rely on st_ino/st_dev when we can, and just\nnot bother looking for recursive symlinks when we cannot, like I did in\nhttps://github.com/git-for-windows/git/commit/979b00ccf44ec31cff4686e24adf27474923c33a\n\nCiao,\nJohannes\n"},{"id":"378193","messageId":"CAHd-oW6c7EoY+t-ymkcBmZM20YcGb1SWhM7XMO35Fofj-UNmYQ@mail.gmail.com","threadId":"50591","inReplyTo":"nycvar.QRO.7.76.6.1906272046180.44@tvgsbejvaqbjf.bet","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-27T19:33:58Z","receivedAt":"2019-06-27T19:34:12Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Jun 27, 2019 at 3:47 PM Johannes Schindelin\n wrote:\n>\n> Hi Matheus,\n>\n> On Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n>\n> > On Wed, Jun 26, 2019 at 3:04 PM Junio C Hamano wrote:\n> > >\n> > > Johannes Schindelin writes:\n> > >\n> > > > Hi Matheus,\n> > > >\n> > > > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> > > >\n> > > >>[...]\n> > > >> +/*\n> > > >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> > > >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> > > >> + */\n> > > >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> > > >> +{\n> > > >> + int i;\n> > > >> +\n> > > >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> > > >> + !S_ISDIR(iter->base.st.st_mode))\n> > > >> + return 0;\n> > > >>\n> > > >> + for (i = 0; i < iter->levels_nr; ++i)\n> > > >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> > > >\n> > > > This does not work on Windows. [[ Windows port does not have\n> > > > usable st_ino field ]]]\n> > >\n> > > And if you cross mountpoint, st_ino alone does not guarantee\n> > > uniqueness; you'd need to combine it with st_dev, I would think,\n> > > even on POSIX systems.\n> >\n> > Ok, thanks for letting me know. I'm trying to think of another\n> > approach to test for recursive symlinks that does not rely on inode:\n> > Given any symlink, we could get its real_path() and compare it with\n> > the path of the directory current being iterated. If the first is a\n> > prefix of the second, than we mark it as a recursive symlink.\n> >\n> > What do you think of this idea?\n>\n> I think this would be pretty expensive. Too expensive.\n\nHmm, yes unfortunately :(\n\n> A better method might be to rely on st_ino/st_dev when we can, and just\n> not bother looking for recursive symlinks when we cannot,\n\nWhat if we fallback on the path prefix strategy when st_ino is not\navailable? I mean, if we don't look for recursive symlinks, they would\nbe iterated over and over until we get an ELOOP error. So I think\nusing real_path() should be less expensive in this case. (But just as\na fallback to st_ino, off course)\n\n> like I did in\n> https://github.com/git-for-windows/git/commit/979b00ccf44ec31cff4686e24adf27474923c33a\n\nNice! At dir-iterator.h the documentation says that recursive symlinks\nwill be ignored. If we don't implement any fallback, should we add\nthat this is not available on Windows, perhaps?\n\n> Ciao,\n> Johannes\n"},{"id":"378265","messageId":"nycvar.QRO.7.76.6.1906281449250.44@tvgsbejvaqbjf.bet","threadId":"50591","inReplyTo":"CAHd-oW6c7EoY+t-ymkcBmZM20YcGb1SWhM7XMO35Fofj-UNmYQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-28T12:51:07Z","receivedAt":"2019-06-28T12:51:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n\n> On Thu, Jun 27, 2019 at 3:47 PM Johannes Schindelin\n> wrote:\n> >\n> > On Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n> >\n> > > On Wed, Jun 26, 2019 at 3:04 PM Junio C Hamano wrote:\n> > > >\n> > > > Johannes Schindelin writes:\n> > > >\n> > > > > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> > > > >\n> > > > >>[...]\n> > > > >> +/*\n> > > > >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> > > > >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> > > > >> + */\n> > > > >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> > > > >> +{\n> > > > >> + int i;\n> > > > >> +\n> > > > >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> > > > >> + !S_ISDIR(iter->base.st.st_mode))\n> > > > >> + return 0;\n> > > > >>\n> > > > >> + for (i = 0; i < iter->levels_nr; ++i)\n> > > > >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> > > > >\n> > > > > This does not work on Windows. [[ Windows port does not have\n> > > > > usable st_ino field ]]]\n> > > >\n> > > > And if you cross mountpoint, st_ino alone does not guarantee\n> > > > uniqueness; you'd need to combine it with st_dev, I would think,\n> > > > even on POSIX systems.\n> > >\n> > > Ok, thanks for letting me know. I'm trying to think of another\n> > > approach to test for recursive symlinks that does not rely on inode:\n> > > Given any symlink, we could get its real_path() and compare it with\n> > > the path of the directory current being iterated. If the first is a\n> > > prefix of the second, than we mark it as a recursive symlink.\n> > >\n> > > What do you think of this idea?\n> >\n> > I think this would be pretty expensive. Too expensive.\n>\n> Hmm, yes unfortunately :(\n>\n> > A better method might be to rely on st_ino/st_dev when we can, and just\n> > not bother looking for recursive symlinks when we cannot,\n>\n> What if we fallback on the path prefix strategy when st_ino is not\n> available? I mean, if we don't look for recursive symlinks, they would\n> be iterated over and over until we get an ELOOP error. So I think\n> using real_path() should be less expensive in this case. (But just as\n> a fallback to st_ino, off course)\n>\n> > like I did in\n> > https://github.com/git-for-windows/git/commit/979b00ccf44ec31cff4686e24adf27474923c33a\n>\n> Nice! At dir-iterator.h the documentation says that recursive symlinks\n> will be ignored. If we don't implement any fallback, should we add\n> that this is not available on Windows, perhaps?\n\nI do not really care, unless it breaks things on Windows that were not\nbroken before.\n\nYou might also want to guard this behind `USE_STDEV` as Duy suggested (and\nmaybe use the opportunity to correct that constant to `USE_ST_DEV`; I\nlooked for it and did not find it because of that naming mistake).\n\nCiao,\nDscho\n"},{"id":"378270","messageId":"CAHd-oW45RAov-5paURQ7j5dg0nZktF2J0bME7Wo1pF8YCd2cww@mail.gmail.com","threadId":"50591","inReplyTo":"nycvar.QRO.7.76.6.1906281449250.44@tvgsbejvaqbjf.bet","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-06-28T14:16:38Z","receivedAt":"2019-06-28T14:16:52Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Jun 28, 2019 at 9:50 AM Johannes Schindelin\n wrote:\n>\n> Hi Matheus,\n>\n> On Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n>\n> > On Thu, Jun 27, 2019 at 3:47 PM Johannes Schindelin\n> > wrote:\n> > >\n> > > On Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n> > >\n> > > > On Wed, Jun 26, 2019 at 3:04 PM Junio C Hamano wrote:\n> > > > >\n> > > > > Johannes Schindelin writes:\n> > > > >\n> > > > > > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> > > > > >\n> > > > > >>[...]\n> > > > > >> +/*\n> > > > > >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> > > > > >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> > > > > >> + */\n> > > > > >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> > > > > >> +{\n> > > > > >> + int i;\n> > > > > >> +\n> > > > > >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> > > > > >> + !S_ISDIR(iter->base.st.st_mode))\n> > > > > >> + return 0;\n> > > > > >>\n> > > > > >> + for (i = 0; i < iter->levels_nr; ++i)\n> > > > > >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> > > > > >\n> > > > > > This does not work on Windows. [[ Windows port does not have\n> > > > > > usable st_ino field ]]]\n> > > > >\n> > > > > And if you cross mountpoint, st_ino alone does not guarantee\n> > > > > uniqueness; you'd need to combine it with st_dev, I would think,\n> > > > > even on POSIX systems.\n> > > >\n> > > > Ok, thanks for letting me know. I'm trying to think of another\n> > > > approach to test for recursive symlinks that does not rely on inode:\n> > > > Given any symlink, we could get its real_path() and compare it with\n> > > > the path of the directory current being iterated. If the first is a\n> > > > prefix of the second, than we mark it as a recursive symlink.\n> > > >\n> > > > What do you think of this idea?\n> > >\n> > > I think this would be pretty expensive. Too expensive.\n> >\n> > Hmm, yes unfortunately :(\n> >\n> > > A better method might be to rely on st_ino/st_dev when we can, and just\n> > > not bother looking for recursive symlinks when we cannot,\n> >\n> > What if we fallback on the path prefix strategy when st_ino is not\n> > available? I mean, if we don't look for recursive symlinks, they would\n> > be iterated over and over until we get an ELOOP error. So I think\n> > using real_path() should be less expensive in this case. (But just as\n> > a fallback to st_ino, off course)\n> >\n> > > like I did in\n> > > https://github.com/git-for-windows/git/commit/979b00ccf44ec31cff4686e24adf27474923c33a\n> >\n> > Nice! At dir-iterator.h the documentation says that recursive symlinks\n> > will be ignored. If we don't implement any fallback, should we add\n> > that this is not available on Windows, perhaps?\n>\n> I do not really care, unless it breaks things on Windows that were not\n> broken before.\n>\n> You might also want to guard this behind `USE_STDEV` as Duy suggested (and\n> maybe use the opportunity to correct that constant to `USE_ST_DEV`; I\n> looked for it and did not find it because of that naming mistake).\n\nOk, just to confirm, what I should do is send your fixup patch with\nthe USE_STDEV guard addition, right? Also, USE_STDEV docs says it is\nused \"from the update-index perspective\", should I make it more\ngeneric as we're using it for other purposes or is it OK like this?\n\nThanks,\nMatheus\n"},{"id":"378368","messageId":"nycvar.QRO.7.76.6.1907011413530.44@tvgsbejvaqbjf.bet","threadId":"50591","inReplyTo":"CAHd-oW45RAov-5paURQ7j5dg0nZktF2J0bME7Wo1pF8YCd2cww@mail.gmail.com","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-07-01T12:15:35Z","receivedAt":"2019-07-01T12:15:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Fri, 28 Jun 2019, Matheus Tavares Bernardino wrote:\n\n> On Fri, Jun 28, 2019 at 9:50 AM Johannes Schindelin\n> wrote:\n> >\n> > Hi Matheus,\n> >\n> > On Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n> >\n> > > On Thu, Jun 27, 2019 at 3:47 PM Johannes Schindelin\n> > > wrote:\n> > > >\n> > > > On Thu, 27 Jun 2019, Matheus Tavares Bernardino wrote:\n> > > >\n> > > > > On Wed, Jun 26, 2019 at 3:04 PM Junio C Hamano wrote:\n> > > > > >\n> > > > > > Johannes Schindelin writes:\n> > > > > >\n> > > > > > > On Tue, 18 Jun 2019, Matheus Tavares wrote:\n> > > > > > >\n> > > > > > >>[...]\n> > > > > > >> +/*\n> > > > > > >> + * Look for a recursive symlink at iter->base.path pointing to any directory on\n> > > > > > >> + * the previous stack levels. If it is found, return 1. If not, return 0.\n> > > > > > >> + */\n> > > > > > >> +static int find_recursive_symlinks(struct dir_iterator_int *iter)\n> > > > > > >> +{\n> > > > > > >> + int i;\n> > > > > > >> +\n> > > > > > >> + if (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n> > > > > > >> + !S_ISDIR(iter->base.st.st_mode))\n> > > > > > >> + return 0;\n> > > > > > >>\n> > > > > > >> + for (i = 0; i < iter->levels_nr; ++i)\n> > > > > > >> + if (iter->base.st.st_ino == iter->levels[i].ino)\n> > > > > > >\n> > > > > > > This does not work on Windows. [[ Windows port does not have\n> > > > > > > usable st_ino field ]]]\n> > > > > >\n> > > > > > And if you cross mountpoint, st_ino alone does not guarantee\n> > > > > > uniqueness; you'd need to combine it with st_dev, I would think,\n> > > > > > even on POSIX systems.\n> > > > >\n> > > > > Ok, thanks for letting me know. I'm trying to think of another\n> > > > > approach to test for recursive symlinks that does not rely on inode:\n> > > > > Given any symlink, we could get its real_path() and compare it with\n> > > > > the path of the directory current being iterated. If the first is a\n> > > > > prefix of the second, than we mark it as a recursive symlink.\n> > > > >\n> > > > > What do you think of this idea?\n> > > >\n> > > > I think this would be pretty expensive. Too expensive.\n> > >\n> > > Hmm, yes unfortunately :(\n> > >\n> > > > A better method might be to rely on st_ino/st_dev when we can, and just\n> > > > not bother looking for recursive symlinks when we cannot,\n> > >\n> > > What if we fallback on the path prefix strategy when st_ino is not\n> > > available? I mean, if we don't look for recursive symlinks, they would\n> > > be iterated over and over until we get an ELOOP error. So I think\n> > > using real_path() should be less expensive in this case. (But just as\n> > > a fallback to st_ino, off course)\n> > >\n> > > > like I did in\n> > > > https://github.com/git-for-windows/git/commit/979b00ccf44ec31cff4686e24adf27474923c33a\n> > >\n> > > Nice! At dir-iterator.h the documentation says that recursive symlinks\n> > > will be ignored. If we don't implement any fallback, should we add\n> > > that this is not available on Windows, perhaps?\n> >\n> > I do not really care, unless it breaks things on Windows that were not\n> > broken before.\n> >\n> > You might also want to guard this behind `USE_STDEV` as Duy suggested (and\n> > maybe use the opportunity to correct that constant to `USE_ST_DEV`; I\n> > looked for it and did not find it because of that naming mistake).\n>\n> Ok, just to confirm, what I should do is send your fixup patch with\n> the USE_STDEV guard addition, right? Also, USE_STDEV docs says it is\n> used \"from the update-index perspective\", should I make it more\n> generic as we're using it for other purposes or is it OK like this?\n\nI thought Duy had verified that `USE_STDEV` would make sense in this\ninstance, but I agree with you that the idea of that compile time flag was\nnot to guard against a missing `st_dev` field, but about trusting it in\nthe presence of network filesystems.\n\nSo no, I revert my vote for using `USE_STDEV`.\n\nThanks for the sanity check.\n\nCiao,\nDscho\n"},{"id":"378529","messageId":"20190703085726.GN21574@szeder.dev","threadId":"50591","inReplyTo":"5a678ee74de42f1373deeed718fa24d368347d13.1560898723.git.matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-07-03T08:57:26Z","receivedAt":"2019-07-03T08:57:33Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> diff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\n> index c739ed7911..8f996a31fa 100755\n> --- a/t/t0066-dir-iterator.sh\n> +++ b/t/t0066-dir-iterator.sh\n> @@ -65,4 +65,99 @@ test_expect_success 'begin should fail upon non directory paths' '\n> \ttest_cmp expected-non-dir-output actual-non-dir-output\n> '\n> \n> +test_expect_success POSIXPERM,SANITY 'advance should not fail on errors by default' '\n> +\tcat >expected-no-permissions-output <<-EOF &&\n> +\t[d] (a) [a] ./dir3/a\n> +\tEOF\n> +\n> +\tmkdir -p dir3/a &&\n> +\t> dir3/a/b &&\n\nStyle nit: space between redirection op and pathname.\n\n> +\tchmod 0 dir3/a &&\n> +\n> +\ttest-tool dir-iterator ./dir3 >actual-no-permissions-output &&\n> +\ttest_cmp expected-no-permissions-output actual-no-permissions-output &&\n> +\tchmod 755 dir3/a &&\n> +\trm -rf dir3\n> +'\n> +\n> +test_expect_success POSIXPERM,SANITY 'advance should fail on errors, w/ pedantic flag' '\n> +\tcat >expected-no-permissions-pedantic-output <<-EOF &&\n> +\t[d] (a) [a] ./dir3/a\n> +\tdir_iterator_advance failure\n> +\tEOF\n> +\n> +\tmkdir -p dir3/a &&\n> +\t> dir3/a/b &&\n\nLikewise.\n\n> +\tchmod 0 dir3/a &&\n> +\n> +\ttest_must_fail test-tool dir-iterator --pedantic ./dir3 \\\n> +\t\t>actual-no-permissions-pedantic-output &&\n> +\ttest_cmp expected-no-permissions-pedantic-output \\\n> +\t\tactual-no-permissions-pedantic-output &&\n> +\tchmod 755 dir3/a &&\n> +\trm -rf dir3\n> +'\n> +\n> +test_expect_success SYMLINKS 'setup dirs with symlinks' '\n> +\tmkdir -p dir4/a &&\n> +\tmkdir -p dir4/b/c &&\n> +\t>dir4/a/d &&\n> +\tln -s d dir4/a/e &&\n> +\tln -s ../b dir4/a/f &&\n> +\n> +\tmkdir -p dir5/a/b &&\n> +\tmkdir -p dir5/a/c &&\n> +\tln -s ../c dir5/a/b/d &&\n> +\tln -s ../ dir5/a/b/e &&\n> +\tln -s ../../ dir5/a/b/f\n> +'\n> +\n> +test_expect_success SYMLINKS 'dir-iterator should not follow symlinks by default' '\n> +\tcat >expected-no-follow-sorted-output <<-EOF &&\n> +\t[d] (a) [a] ./dir4/a\n> +\t[d] (b) [b] ./dir4/b\n> +\t[d] (b/c) [c] ./dir4/b/c\n> +\t[f] (a/d) [d] ./dir4/a/d\n> +\t[s] (a/e) [e] ./dir4/a/e\n> +\t[s] (a/f) [f] ./dir4/a/f\n> +\tEOF\n> +\n> +\ttest-tool dir-iterator ./dir4 >out &&\n> +\tsort actual-no-follow-sorted-output &&\n\nUnnecessary redirection, 'sort' is capable to open the file on its\nown.\n\n> +\n> +\ttest_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output\n> +'\n> +\n> +test_expect_success SYMLINKS 'dir-iterator should follow symlinks w/ follow flag' '\n> +\tcat >expected-follow-sorted-output <<-EOF &&\n> +\t[d] (a) [a] ./dir4/a\n> +\t[d] (a/f) [f] ./dir4/a/f\n> +\t[d] (a/f/c) [c] ./dir4/a/f/c\n> +\t[d] (b) [b] ./dir4/b\n> +\t[d] (b/c) [c] ./dir4/b/c\n> +\t[f] (a/d) [d] ./dir4/a/d\n> +\t[f] (a/e) [e] ./dir4/a/e\n> +\tEOF\n> +\n> +\ttest-tool dir-iterator --follow-symlinks ./dir4 >out &&\n> +\tsort actual-follow-sorted-output &&\n\nLikewise.\n\n> +\ttest_cmp expected-follow-sorted-output actual-follow-sorted-output\n> +'\n> +\n> +\n> +test_expect_success SYMLINKS 'dir-iterator should ignore recursive symlinks w/ follow flag' '\n> +\tcat >expected-rec-symlinks-sorted-output <<-EOF &&\n> +\t[d] (a) [a] ./dir5/a\n> +\t[d] (a/b) [b] ./dir5/a/b\n> +\t[d] (a/b/d) [d] ./dir5/a/b/d\n> +\t[d] (a/c) [c] ./dir5/a/c\n> +\tEOF\n> +\n> +\ttest-tool dir-iterator --follow-symlinks ./dir5 >out &&\n> +\tsort actual-rec-symlinks-sorted-output &&\n\nLikewise.\n\n> +\ttest_cmp expected-rec-symlinks-sorted-output actual-rec-symlinks-sorted-output\n> +'\n> +\n> test_done\n> -- \n> 2.22.0\n> \n"},{"id":"378713","messageId":"CAHd-oW4kFtCaogk17r4r1+b3Z_j75+_vcMfunRr5M1XE0VKU8Q@mail.gmail.com","threadId":"50591","inReplyTo":"20190703085726.GN21574@szeder.dev","subject":"Re: [GSoC][PATCH v7 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-08T22:21:42Z","receivedAt":"2019-07-08T22:21:56Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Thanks for the review. I'll address those issues in v8.\n\nBest,\nMatheus\n\n\nOn Wed, Jul 3, 2019 at 5:57 AM SZEDER Gábor wrote:\n>\n> > diff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\n> > index c739ed7911..8f996a31fa 100755\n> > --- a/t/t0066-dir-iterator.sh\n> > +++ b/t/t0066-dir-iterator.sh\n> > @@ -65,4 +65,99 @@ test_expect_success 'begin should fail upon non directory paths' '\n> > test_cmp expected-non-dir-output actual-non-dir-output\n> > '\n> >\n> > +test_expect_success POSIXPERM,SANITY 'advance should not fail on errors by default' '\n> > + cat >expected-no-permissions-output <<-EOF &&\n> > + [d] (a) [a] ./dir3/a\n> > + EOF\n> > +\n> > + mkdir -p dir3/a &&\n> > + > dir3/a/b &&\n>\n> Style nit: space between redirection op and pathname.\n>\n> > + chmod 0 dir3/a &&\n> > +\n> > + test-tool dir-iterator ./dir3 >actual-no-permissions-output &&\n> > + test_cmp expected-no-permissions-output actual-no-permissions-output &&\n> > + chmod 755 dir3/a &&\n> > + rm -rf dir3\n> > +'\n> > +\n> > +test_expect_success POSIXPERM,SANITY 'advance should fail on errors, w/ pedantic flag' '\n> > + cat >expected-no-permissions-pedantic-output <<-EOF &&\n> > + [d] (a) [a] ./dir3/a\n> > + dir_iterator_advance failure\n> > + EOF\n> > +\n> > + mkdir -p dir3/a &&\n> > + > dir3/a/b &&\n>\n> Likewise.\n>\n> > + chmod 0 dir3/a &&\n> > +\n> > + test_must_fail test-tool dir-iterator --pedantic ./dir3 \\\n> > + >actual-no-permissions-pedantic-output &&\n> > + test_cmp expected-no-permissions-pedantic-output \\\n> > + actual-no-permissions-pedantic-output &&\n> > + chmod 755 dir3/a &&\n> > + rm -rf dir3\n> > +'\n> > +\n> > +test_expect_success SYMLINKS 'setup dirs with symlinks' '\n> > + mkdir -p dir4/a &&\n> > + mkdir -p dir4/b/c &&\n> > + >dir4/a/d &&\n> > + ln -s d dir4/a/e &&\n> > + ln -s ../b dir4/a/f &&\n> > +\n> > + mkdir -p dir5/a/b &&\n> > + mkdir -p dir5/a/c &&\n> > + ln -s ../c dir5/a/b/d &&\n> > + ln -s ../ dir5/a/b/e &&\n> > + ln -s ../../ dir5/a/b/f\n> > +'\n> > +\n> > +test_expect_success SYMLINKS 'dir-iterator should not follow symlinks by default' '\n> > + cat >expected-no-follow-sorted-output <<-EOF &&\n> > + [d] (a) [a] ./dir4/a\n> > + [d] (b) [b] ./dir4/b\n> > + [d] (b/c) [c] ./dir4/b/c\n> > + [f] (a/d) [d] ./dir4/a/d\n> > + [s] (a/e) [e] ./dir4/a/e\n> > + [s] (a/f) [f] ./dir4/a/f\n> > + EOF\n> > +\n> > + test-tool dir-iterator ./dir4 >out &&\n> > + sort actual-no-follow-sorted-output &&\n>\n> Unnecessary redirection, 'sort' is capable to open the file on its\n> own.\n>\n> > +\n> > + test_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output\n> > +'\n> > +\n> > +test_expect_success SYMLINKS 'dir-iterator should follow symlinks w/ follow flag' '\n> > + cat >expected-follow-sorted-output <<-EOF &&\n> > + [d] (a) [a] ./dir4/a\n> > + [d] (a/f) [f] ./dir4/a/f\n> > + [d] (a/f/c) [c] ./dir4/a/f/c\n> > + [d] (b) [b] ./dir4/b\n> > + [d] (b/c) [c] ./dir4/b/c\n> > + [f] (a/d) [d] ./dir4/a/d\n> > + [f] (a/e) [e] ./dir4/a/e\n> > + EOF\n> > +\n> > + test-tool dir-iterator --follow-symlinks ./dir4 >out &&\n> > + sort actual-follow-sorted-output &&\n>\n> Likewise.\n>\n> > + test_cmp expected-follow-sorted-output actual-follow-sorted-output\n> > +'\n> > +\n> > +\n> > +test_expect_success SYMLINKS 'dir-iterator should ignore recursive symlinks w/ follow flag' '\n> > + cat >expected-rec-symlinks-sorted-output <<-EOF &&\n> > + [d] (a) [a] ./dir5/a\n> > + [d] (a/b) [b] ./dir5/a/b\n> > + [d] (a/b/d) [d] ./dir5/a/b/d\n> > + [d] (a/c) [c] ./dir5/a/c\n> > + EOF\n> > +\n> > + test-tool dir-iterator --follow-symlinks ./dir5 >out &&\n> > + sort actual-rec-symlinks-sorted-output &&\n>\n> Likewise.\n>\n> > + test_cmp expected-rec-symlinks-sorted-output actual-rec-symlinks-sorted-output\n> > +'\n> > +\n> > test_done\n> > --\n> > 2.22.0\n> >\n"},{"id":"378782","messageId":"cover.1562801254.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1560898723.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 00/10] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:58:54Z","receivedAt":"2019-07-10T23:59:24Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This patchset contains:\n- tests to the dir-iterator API;\n- dir-iterator refactoring to make its state machine simpler\n and feature adding with tests;\n- a replacement of explicit recursive dir iteration at\n copy_or_link_directory for the dir-iterator API;\n- some refactoring and behavior changes at local clone, mainly to\n take care of symlinks and hidden files at .git/objects, together\n with tests for these types of files.\n\nChanges since v7[1]:\n- Applied some style fixes at tests, as suggested by SZEDER\n- Removed the code to find circular symlinks as suggested in this[2]\nthread. The way it was previously implemented wouldn't work on Windows.\nSo Dscho suggested me to remove this section until we come up with a\nmore portable implementation.\n\n[1]: https://public-inbox.org/git/cover.1560898723.git.matheus.bernardino@usp.br/\n[2]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1907041136530.44@tvgsbejvaqbjf.bet/\ntravis build: https://travis-ci.org/matheustavares/git/builds/557047597\n\nDaniel Ferreira (1):\n dir-iterator: add tests for dir-iterator API\n\nMatheus Tavares (8):\n clone: better handle symlinked files at .git/objects/\n dir-iterator: use warning_errno when possible\n dir-iterator: refactor state machine model\n dir-iterator: add flags parameter to dir_iterator_begin\n clone: copy hidden paths at local clone\n clone: extract function from copy_or_link_directory\n clone: use dir-iterator to avoid explicit dir traversal\n clone: replace strcmp by fspathcmp\n\nÆvar Arnfjörð Bjarmason (1):\n clone: test for our behavior on odd objects/* content\n\n Makefile | 1 +\n builtin/clone.c | 75 +++++-----\n dir-iterator.c | 263 ++++++++++++++++++++---------------\n dir-iterator.h | 64 +++++++--\n refs/files-backend.c | 17 ++-\n t/helper/test-dir-iterator.c | 58 ++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0066-dir-iterator.sh | 148 ++++++++++++++++++++\n t/t5604-clone-reference.sh | 133 ++++++++++++++++++\n 10 files changed, 597 insertions(+), 164 deletions(-)\n create mode 100644 t/helper/test-dir-iterator.c\n create mode 100755 t/t0066-dir-iterator.sh\n\nRange-diff against v7:\n 1: 437b1eb1c7 ! 1: a2016d9d3b clone: test for our behavior on odd objects/* content\n @@ -98,7 +98,7 @@\n +\t\tmv $last_loose a-loose-dir &&\n +\t\tln -s a-loose-dir $last_loose &&\n +\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n -+\t\techo unknown_content> unknown_file\n ++\t\techo unknown_content >unknown_file\n +\t) &&\n +\tgit -C T fsck &&\n +\tgit -C T rev-list --all --objects >T.objects\n 2: 108bea2652 ! 2: 47a4f9b31c clone: better handle symlinked files at .git/objects/\n @@ -80,7 +80,7 @@\n +\t\tcd ../ &&\n \t\tfind . -type f | sort >../../../T.objects-files.raw &&\n +\t\tfind . -type l | sort >../../../T.objects-symlinks.raw &&\n - \t\techo unknown_content> unknown_file\n + \t\techo unknown_content >unknown_file\n \t) &&\n \tgit -C T fsck &&\n @@\n 3: 2c0232be6c ! 3: bbce6a601b dir-iterator: add tests for dir-iterator API\n @@ -129,7 +129,7 @@\n +\tEOF\n +\n +\ttest-tool dir-iterator ./dir >out &&\n -+\tsort ./actual-iteration-sorted-output &&\n ++\tsort out >./actual-iteration-sorted-output &&\n +\n +\ttest_cmp expected-iteration-sorted-output actual-iteration-sorted-output\n +'\n 4: 0b76044165 = 4: 0cc5f1f0b4 dir-iterator: use warning_errno when possible\n 5: 44c47d579c ! 5: f871b5d3f4 dir-iterator: refactor state machine model\n @@ -340,14 +340,14 @@\n * A typical iteration looks like this:\n *\n * int ok;\n - * struct iterator *iter = dir_iterator_begin(path);\n - *\n +- * struct iterator *iter = dir_iterator_begin(path);\n ++ * struct dir_iterator *iter = dir_iterator_begin(path);\n ++ *\n + * if (!iter)\n + * goto error_handler;\n -+ *\n + *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n - * ok = dir_iterator_abort(iter);\n @@\n };\n \n 6: 86fc04ad0e ! 6: fe838d7eb4 dir-iterator: add flags parameter to dir_iterator_begin\n @@ -22,16 +22,6 @@\n diff --git a/dir-iterator.c b/dir-iterator.c\n --- a/dir-iterator.c\n +++ b/dir-iterator.c\n -@@\n - struct dir_iterator_level {\n - \tDIR *dir;\n - \n -+\t/* The inode number of this level's directory. */\n -+\tino_t ino;\n -+\n - \t/*\n - \t * The length of the directory part of path at this level\n - \t * (including a trailing '/'):\n @@\n \t * that will be included in this iteration.\n \t */\n @@ -51,10 +41,6 @@\n static int push_level(struct dir_iterator_int *iter)\n {\n @@\n - \tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n - \t\tstrbuf_addch(&iter->base.path, '/');\n - \tlevel->prefix_len = iter->base.path.len;\n -+\tlevel->ino = iter->base.st.st_ino;\n \n \tlevel->dir = opendir(iter->base.path.buf);\n \tif (!level->dir) {\n @@ -96,33 +82,17 @@\n +\t\terr = stat(iter->base.path.buf, &iter->base.st);\n +\telse\n +\t\terr = lstat(iter->base.path.buf, &iter->base.st);\n -+\n + \n +-\treturn 0;\n +\tsaved_errno = errno;\n +\tif (err && errno != ENOENT)\n +\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n +\n +\terrno = saved_errno;\n +\treturn err;\n -+}\n -+\n -+/*\n -+ * Look for a recursive symlink at iter->base.path pointing to any directory on\n -+ * the previous stack levels. If it is found, return 1. If not, return 0.\n -+ */\n -+static int find_recursive_symlinks(struct dir_iterator_int *iter)\n -+{\n -+\tint i;\n -+\n -+\tif (!(iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) ||\n -+\t !S_ISDIR(iter->base.st.st_mode))\n -+\t\treturn 0;\n - \n -+\tfor (i = 0; i < iter->levels_nr; ++i)\n -+\t\tif (iter->base.st.st_ino == iter->levels[i].ino)\n -+\t\t\treturn 1;\n - \treturn 0;\n }\n \n + int dir_iterator_advance(struct dir_iterator *dir_iterator)\n @@\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n @@ -165,12 +135,6 @@\n +\t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n +\t\t\t\tgoto error_out;\n \t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\tif (find_recursive_symlinks(iter)) {\n -+\t\t\twarning(\"ignoring recursive symlink at '%s'\",\n -+\t\t\t\titer->base.path.buf);\n -+\t\t\tcontinue;\n +\t\t}\n \n \t\treturn ITER_OK;\n @@ -207,7 +171,7 @@\n * A typical iteration looks like this:\n *\n * int ok;\n -- * struct iterator *iter = dir_iterator_begin(path);\n +- * struct dir_iterator *iter = dir_iterator_begin(path);\n + * unsigned int flags = DIR_ITERATOR_PEDANTIC;\n + * struct dir_iterator *iter = dir_iterator_begin(path, flags);\n *\n @@ -230,9 +194,12 @@\n + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks.\n + * i.e., linked directories' contents will be iterated over and\n + * iter->base.st will contain information on the referred files,\n -+ * not the symlinks themselves, which is the default behavior.\n -+ * Recursive symlinks are skipped with a warning and broken symlinks\n -+ * are ignored.\n ++ * not the symlinks themselves, which is the default behavior. Broken\n ++ * symlinks are ignored.\n ++ *\n ++ * Warning: circular symlinks are also followed when\n ++ * DIR_ITERATOR_FOLLOW_SYMLINKS is set. The iteration may end up with\n ++ * an ELOOP if they happen and DIR_ITERATOR_PEDANTIC is set.\n + */\n +#define DIR_ITERATOR_PEDANTIC (1 << 0)\n +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n @@ -383,7 +350,7 @@\n +\tEOF\n +\n +\tmkdir -p dir3/a &&\n -+\t> dir3/a/b &&\n ++\t>dir3/a/b &&\n +\tchmod 0 dir3/a &&\n +\n +\ttest-tool dir-iterator ./dir3 >actual-no-permissions-output &&\n @@ -399,7 +366,7 @@\n +\tEOF\n +\n +\tmkdir -p dir3/a &&\n -+\t> dir3/a/b &&\n ++\t>dir3/a/b &&\n +\tchmod 0 dir3/a &&\n +\n +\ttest_must_fail test-tool dir-iterator --pedantic ./dir3 \\\n @@ -435,7 +402,7 @@\n +\tEOF\n +\n +\ttest-tool dir-iterator ./dir4 >out &&\n -+\tsort actual-no-follow-sorted-output &&\n ++\tsort out >actual-no-follow-sorted-output &&\n +\n +\ttest_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output\n +'\n @@ -452,24 +419,9 @@\n +\tEOF\n +\n +\ttest-tool dir-iterator --follow-symlinks ./dir4 >out &&\n -+\tsort actual-follow-sorted-output &&\n ++\tsort out >actual-follow-sorted-output &&\n +\n +\ttest_cmp expected-follow-sorted-output actual-follow-sorted-output\n +'\n -+\n -+\n -+test_expect_success SYMLINKS 'dir-iterator should ignore recursive symlinks w/ follow flag' '\n -+\tcat >expected-rec-symlinks-sorted-output <<-EOF &&\n -+\t[d] (a) [a] ./dir5/a\n -+\t[d] (a/b) [b] ./dir5/a/b\n -+\t[d] (a/b/d) [d] ./dir5/a/b/d\n -+\t[d] (a/c) [c] ./dir5/a/c\n -+\tEOF\n -+\n -+\ttest-tool dir-iterator --follow-symlinks ./dir5 >out &&\n -+\tsort actual-rec-symlinks-sorted-output &&\n -+\n -+\ttest_cmp expected-rec-symlinks-sorted-output actual-rec-symlinks-sorted-output\n -+'\n +\n test_done\n 7: 17685057cd = 7: 3da6408e04 clone: copy hidden paths at local clone\n 8: c7f3a8640e = 8: af7430eb2c clone: extract function from copy_or_link_directory\n 9: 7934036d30 ! 9: e8308c7408 clone: use dir-iterator to avoid explicit dir traversal\n @@ -11,11 +11,7 @@\n error on readdir or stat inside dir_iterator_advance. Previously it\n would just print a warning for errors on stat and ignore errors on\n readdir, which isn't nice because a local git clone could succeed even\n - though the .git/objects copy didn't fully succeed. Also, with the\n - dir-iterator API, recursive symlinks will be detected and skipped. This\n - is another behavior improvement, since the current version would\n - continue to copy the same content over and over until stat() returned an\n - ELOOP error.\n + though the .git/objects copy didn't fully succeed.\n \n Signed-off-by: Matheus Tavares \n \n10: 2e25c03c07 = 10: 782ca07eed clone: replace strcmp by fspathcmp\n-- \n2.22.0\n\n"},{"id":"378783","messageId":"a2016d9d3b8e54ff9b9e6dfbd3ab4ce4a1bf7e4d.1562801254.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 01/10] clone: test for our behavior on odd objects/* content","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:58:55Z","receivedAt":"2019-07-10T23:59:37Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nAdd tests for what happens when we perform a local clone on a repo\ncontaining odd files at .git/object directory, such as symlinks to other\ndirs, or unknown files.\n\nI'm bending over backwards here to avoid a SHA-1 dependency. See [1]\nfor an earlier and simpler version that hardcoded SHA-1s.\n\nThis behavior has been the same for a *long* time, but hasn't been\ntested for.\n\nThere's a good post-hoc argument to be made for copying over unknown\nthings, e.g. I'd like a git version that doesn't know about the\ncommit-graph to copy it under \"clone --local\" so a newer git version\ncan make use of it.\n\nIn follow-up commits we'll look at changing some of this behavior, but\nfor now, let's just assert it as-is so we'll notice what we'll change\nlater.\n\n1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n[matheus.bernardino: improved and split tests in more than one patch]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n t/t5604-clone-reference.sh | 111 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 111 insertions(+)\n\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 4320082b1b..11250cab40 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -221,4 +221,115 @@ test_expect_success 'clone, dissociate from alternates' '\n \t( cd C && git fsck )\n '\n \n+test_expect_success 'setup repo with garbage in objects/*' '\n+\tgit init S &&\n+\t(\n+\t\tcd S &&\n+\t\ttest_commit A &&\n+\n+\t\tcd .git/objects &&\n+\t\t>.some-hidden-file &&\n+\t\t>some-file &&\n+\t\tmkdir .some-hidden-dir &&\n+\t\t>.some-hidden-dir/some-file &&\n+\t\t>.some-hidden-dir/.some-dot-file &&\n+\t\tmkdir some-dir &&\n+\t\t>some-dir/some-file &&\n+\t\t>some-dir/.some-dot-file\n+\t)\n+'\n+\n+test_expect_success 'clone a repo with garbage in objects/*' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option S S$option || return 1 &&\n+\t\tgit -C S$option fsck || return 1\n+\tdone &&\n+\tfind S-* -name \"*some*\" | sort >actual &&\n+\tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-file\n+\tS--dissociate/.git/objects/some-dir\n+\tS--dissociate/.git/objects/some-dir/.some-dot-file\n+\tS--dissociate/.git/objects/some-dir/some-file\n+\tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-file\n+\tS--local/.git/objects/some-dir\n+\tS--local/.git/objects/some-dir/.some-dot-file\n+\tS--local/.git/objects/some-dir/some-file\n+\tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-file\n+\tS--no-hardlinks/.git/objects/some-dir\n+\tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/some-dir/some-file\n+\tS--no-hardlinks/.git/objects/some-file\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+\tgit init T &&\n+\t(\n+\t\tcd T &&\n+\t\tgit config gc.auto 0 &&\n+\t\ttest_commit A &&\n+\t\tgit gc &&\n+\t\ttest_commit B &&\n+\n+\t\tcd .git/objects &&\n+\t\tmv pack packs &&\n+\t\tln -s packs pack &&\n+\t\tfind ?? -type d >loose-dirs &&\n+\t\tlast_loose=$(tail -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\t\tmv $last_loose a-loose-dir &&\n+\t\tln -s a-loose-dir $last_loose &&\n+\t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\techo unknown_content >unknown_file\n+\t) &&\n+\tgit -C T fsck &&\n+\tgit -C T rev-list --all --objects >T.objects\n+'\n+\n+\n+test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+\tfor option in --local --no-hardlinks --shared --dissociate\n+\tdo\n+\t\tgit clone $option T T$option || return 1 &&\n+\t\tgit -C T$option fsck || return 1 &&\n+\t\tgit -C T$option rev-list --all --objects >T$option.objects &&\n+\t\ttest_cmp T.objects T$option.objects &&\n+\t\t(\n+\t\t\tcd T$option/.git/objects &&\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t)\n+\tdone &&\n+\n+\tfor raw in $(ls T*.raw)\n+\tdo\n+\t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n+\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha || return 1\n+\tdone &&\n+\n+\tcat >expected-files <<-EOF &&\n+\t./Y/Z\n+\t./Y/Z\n+\t./a-loose-dir/Z\n+\t./Y/Z\n+\t./info/packs\n+\t./pack/pack-Z.idx\n+\t./pack/pack-Z.pack\n+\t./packs/pack-Z.idx\n+\t./packs/pack-Z.pack\n+\t./unknown_file\n+\tEOF\n+\n+\tfor option in --local --dissociate --no-hardlinks\n+\tdo\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\tdone &&\n+\n+\techo ./info/alternates >expected-files &&\n+\ttest_cmp expected-files T--shared.objects-files.raw\n+'\n+\n test_done\n-- \n2.22.0\n\n"},{"id":"378784","messageId":"47a4f9b31c03499bc1317b9a0fccb11c2f5b4d34.1562801254.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 02/10] clone: better handle symlinked files at .git/objects/","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:58:56Z","receivedAt":"2019-07-10T23:59:49Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is currently an odd behaviour when locally cloning a repository\nwith symlinks at .git/objects: using --no-hardlinks all symlinks are\ndereferenced but without it, Git will try to hardlink the files with the\nlink() function, which has an OS-specific behaviour on symlinks. On OSX\nand NetBSD, it creates a hardlink to the file pointed by the symlink\nwhilst on GNU/Linux, it creates a hardlink to the symlink itself.\n\nOn Manjaro GNU/Linux:\n $ touch a\n $ ln -s a b\n $ link b c\n $ ls -li a b c\n 155 [...] a\n 156 [...] b -> a\n 156 [...] c -> a\n\nBut on NetBSD:\n $ ls -li a b c\n 2609160 [...] a\n 2609164 [...] b -> a\n 2609160 [...] c\n\nIt's not good to have the result of a local clone to be OS-dependent and\nbesides that, the current behaviour on GNU/Linux may result in broken\nsymlinks. So let's standardize this by making the hardlinks always point\nto dereferenced paths, instead of the symlinks themselves. Also, add\ntests for symlinked files at .git/objects/.\n\nNote: Git won't create symlinks at .git/objects itself, but it's better\nto handle this case and be friendly with users who manually create them.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 27 ++++++++++++++++++++-------\n 2 files changed, 21 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5b9ebe9947..4a0a2455a7 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -445,7 +445,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (unlink(dest->buf) && errno != ENOENT)\n \t\t\tdie_errno(_(\"failed to unlink '%s'\"), dest->buf);\n \t\tif (!option_no_hardlinks) {\n-\t\t\tif (!link(src->buf, dest->buf))\n+\t\t\tif (!link(real_path(src->buf), dest->buf))\n \t\t\t\tcontinue;\n \t\t\tif (option_local > 0)\n \t\t\t\tdie_errno(_(\"failed to create link '%s'\"), dest->buf);\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 11250cab40..459ad8a20b 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -266,7 +266,7 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'setup repo with manually symlinked or unknown files at objects/' '\n \tgit init T &&\n \t(\n \t\tcd T &&\n@@ -280,10 +280,19 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n \t\tln -s packs pack &&\n \t\tfind ?? -type d >loose-dirs &&\n \t\tlast_loose=$(tail -n 1 loose-dirs) &&\n-\t\trm -f loose-dirs &&\n \t\tmv $last_loose a-loose-dir &&\n \t\tln -s a-loose-dir $last_loose &&\n+\t\tfirst_loose=$(head -n 1 loose-dirs) &&\n+\t\trm -f loose-dirs &&\n+\n+\t\tcd $first_loose &&\n+\t\tobj=$(ls *) &&\n+\t\tmv $obj ../an-object &&\n+\t\tln -s ../an-object $obj &&\n+\n+\t\tcd ../ &&\n \t\tfind . -type f | sort >../../../T.objects-files.raw &&\n+\t\tfind . -type l | sort >../../../T.objects-symlinks.raw &&\n \t\techo unknown_content >unknown_file\n \t) &&\n \tgit -C T fsck &&\n@@ -291,7 +300,7 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow\n '\n \n \n-test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' '\n+test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at objects/' '\n \tfor option in --local --no-hardlinks --shared --dissociate\n \tdo\n \t\tgit clone $option T T$option || return 1 &&\n@@ -300,7 +309,8 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t\ttest_cmp T.objects T$option.objects &&\n \t\t(\n \t\t\tcd T$option/.git/objects &&\n-\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw\n+\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n \t\t)\n \tdone &&\n \n@@ -314,6 +324,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./Y/Z\n \t./Y/Z\n \t./a-loose-dir/Z\n+\t./an-object\n \t./Y/Z\n \t./info/packs\n \t./pack/pack-Z.idx\n@@ -323,13 +334,15 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a\n \t./unknown_file\n \tEOF\n \n-\tfor option in --local --dissociate --no-hardlinks\n+\tfor option in --local --no-hardlinks --dissociate\n \tdo\n-\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1\n+\t\ttest_cmp expected-files T$option.objects-files.raw.de-sha || return 1 &&\n+\t\ttest_must_be_empty T$option.objects-symlinks.raw.de-sha || return 1\n \tdone &&\n \n \techo ./info/alternates >expected-files &&\n-\ttest_cmp expected-files T--shared.objects-files.raw\n+\ttest_cmp expected-files T--shared.objects-files.raw &&\n+\ttest_must_be_empty T--shared.objects-symlinks.raw\n '\n \n test_done\n-- \n2.22.0\n\n"},{"id":"378785","messageId":"bbce6a601b9dfe018fb482298ab9e4e79968cd05.1562801254.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 03/10] dir-iterator: add tests for dir-iterator API","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:58:57Z","receivedAt":"2019-07-10T23:59:56Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Daniel Ferreira \n\nCreate t/helper/test-dir-iterator.c, which prints relevant information\nabout a directory tree iterated over with dir-iterator.\n\nCreate t/t0066-dir-iterator.sh, which tests that dir-iterator does\niterate through a whole directory tree as expected.\n\nSigned-off-by: Daniel Ferreira \n[matheus.bernardino: update to use test-tool and some minor aesthetics]\nHelped-by: Matheus Tavares \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n t/helper/test-dir-iterator.c | 33 ++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0066-dir-iterator.sh | 55 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 91 insertions(+)\n create mode 100644 t/helper/test-dir-iterator.c\n create mode 100755 t/t0066-dir-iterator.sh\n\ndiff --git a/Makefile b/Makefile\nindex f58bf14c7b..7e2a44cccc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -704,6 +704,7 @@ TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\n+TEST_BUILTINS_OBJS += test-dir-iterator.o\n TEST_BUILTINS_OBJS += test-drop-caches.o\n TEST_BUILTINS_OBJS += test-dump-cache-tree.o\n TEST_BUILTINS_OBJS += test-dump-fsmonitor.o\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nnew file mode 100644\nindex 0000000000..84f50bed8c\n--- /dev/null\n+++ b/t/helper/test-dir-iterator.c\n@@ -0,0 +1,33 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+\n+/* Argument is a directory path to iterate over */\n+int cmd__dir_iterator(int argc, const char **argv)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct dir_iterator *diter;\n+\n+\tif (argc < 2)\n+\t\tdie(\"BUG: test-dir-iterator needs one argument\");\n+\n+\tstrbuf_add(&path, argv[1], strlen(argv[1]));\n+\n+\tditer = dir_iterator_begin(path.buf);\n+\n+\twhile (dir_iterator_advance(diter) == ITER_OK) {\n+\t\tif (S_ISDIR(diter->st.st_mode))\n+\t\t\tprintf(\"[d] \");\n+\t\telse if (S_ISREG(diter->st.st_mode))\n+\t\t\tprintf(\"[f] \");\n+\t\telse\n+\t\t\tprintf(\"[?] \");\n+\n+\t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n+\t\t diter->path.buf);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 087a8c0cc9..7bc9bb231e 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -19,6 +19,7 @@ static struct test_cmd cmds[] = {\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\n+\t{ \"dir-iterator\", cmd__dir_iterator },\n \t{ \"drop-caches\", cmd__drop_caches },\n \t{ \"dump-cache-tree\", cmd__dump_cache_tree },\n \t{ \"dump-fsmonitor\", cmd__dump_fsmonitor },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 7e703f3038..ec0ffbd0cb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -9,6 +9,7 @@ int cmd__config(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\n+int cmd__dir_iterator(int argc, const char **argv);\n int cmd__drop_caches(int argc, const char **argv);\n int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nnew file mode 100755\nindex 0000000000..59bce868f4\n--- /dev/null\n+++ b/t/t0066-dir-iterator.sh\n@@ -0,0 +1,55 @@\n+#!/bin/sh\n+\n+test_description='Test the dir-iterator functionality'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tmkdir -p dir &&\n+\tmkdir -p dir/a/b/c/ &&\n+\t>dir/b &&\n+\t>dir/c &&\n+\tmkdir -p dir/d/e/d/ &&\n+\t>dir/a/b/c/d &&\n+\t>dir/a/e &&\n+\t>dir/d/e/d/a &&\n+\n+\tmkdir -p dir2/a/b/c/ &&\n+\t>dir2/a/b/c/d\n+'\n+\n+test_expect_success 'dir-iterator should iterate through all files' '\n+\tcat >expected-iteration-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir/a\n+\t[d] (a/b) [b] ./dir/a/b\n+\t[d] (a/b/c) [c] ./dir/a/b/c\n+\t[d] (d) [d] ./dir/d\n+\t[d] (d/e) [e] ./dir/d/e\n+\t[d] (d/e/d) [d] ./dir/d/e/d\n+\t[f] (a/b/c/d) [d] ./dir/a/b/c/d\n+\t[f] (a/e) [e] ./dir/a/e\n+\t[f] (b) [b] ./dir/b\n+\t[f] (c) [c] ./dir/c\n+\t[f] (d/e/d/a) [a] ./dir/d/e/d/a\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir >out &&\n+\tsort out >./actual-iteration-sorted-output &&\n+\n+\ttest_cmp expected-iteration-sorted-output actual-iteration-sorted-output\n+'\n+\n+test_expect_success 'dir-iterator should list files in the correct order' '\n+\tcat >expected-pre-order-output <<-EOF &&\n+\t[d] (a) [a] ./dir2/a\n+\t[d] (a/b) [b] ./dir2/a/b\n+\t[d] (a/b/c) [c] ./dir2/a/b/c\n+\t[f] (a/b/c/d) [d] ./dir2/a/b/c/d\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir2 >actual-pre-order-output &&\n+\n+\ttest_cmp expected-pre-order-output actual-pre-order-output\n+'\n+\n+test_done\n-- \n2.22.0\n\n"},{"id":"378786","messageId":"0cc5f1f0b4ea7de4e0508316e861ace50f39de1f.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 04/10] dir-iterator: use warning_errno when possible","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:58:58Z","receivedAt":"2019-07-11T00:00:03Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Change warning(..., strerror(errno)) by warning_errno(...). This helps\nto unify warning display besides simplifying a bit the code. Also,\nimprove warning messages by surrounding paths with quotation marks and\nusing more meaningful statements.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 23 ++++++++++++-----------\n 1 file changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex f2dcd82fde..0c8880868a 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -71,8 +71,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\t\tlevel->dir = opendir(iter->base.path.buf);\n \t\t\tif (!level->dir && errno != ENOENT) {\n-\t\t\t\twarning(\"error opening directory %s: %s\",\n-\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\t\t\t\t iter->base.path.buf);\n \t\t\t\t/* Popping the level is handled below */\n \t\t\t}\n \n@@ -122,11 +122,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tif (!de) {\n \t\t\t\t/* This level is exhausted; pop up a level. */\n \t\t\t\tif (errno) {\n-\t\t\t\t\twarning(\"error reading directory %s: %s\",\n-\t\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\t\twarning_errno(\"error reading directory '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \t\t\t\t} else if (closedir(level->dir))\n-\t\t\t\t\twarning(\"error closing directory %s: %s\",\n-\t\t\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\t\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \n \t\t\t\tlevel->dir = NULL;\n \t\t\t\tif (--iter->levels_nr == 0)\n@@ -140,9 +140,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n \t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n \t\t\t\tif (errno != ENOENT)\n-\t\t\t\t\twarning(\"error reading path '%s': %s\",\n-\t\t\t\t\t\titer->base.path.buf,\n-\t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t\twarning_errno(\"failed to stat '%s'\",\n+\t\t\t\t\t\t iter->base.path.buf);\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -170,9 +169,11 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \t\t\t&iter->levels[iter->levels_nr - 1];\n \n \t\tif (level->dir && closedir(level->dir)) {\n+\t\t\tint saved_errno = errno;\n \t\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n-\t\t\twarning(\"error closing directory %s: %s\",\n-\t\t\t\titer->base.path.buf, strerror(errno));\n+\t\t\terrno = saved_errno;\n+\t\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t\t iter->base.path.buf);\n \t\t}\n \t}\n \n-- \n2.22.0\n\n"},{"id":"378787","messageId":"f871b5d3f4c916599265d34bbb0f7aeb021392c8.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 05/10] dir-iterator: refactor state machine model","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:58:59Z","receivedAt":"2019-07-11T00:00:11Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"dir_iterator_advance() is a large function with two nested loops. Let's\nimprove its readability factoring out three functions and simplifying\nits mechanics. The refactored model will no longer depend on\nlevel.initialized and level.dir_state to keep track of the iteration\nstate and will perform on a single loop.\n\nAlso, dir_iterator_begin() currently does not check if the given string\nrepresents a valid directory path. Since the refactored model will have\nto stat() the given path at initialization, let's also check for this\nkind of error and make dir_iterator_begin() return NULL, on failures,\nwith errno appropriately set. And add tests for this new behavior.\n\nImprove documentation at dir-iteration.h and code comments at\ndir-iterator.c to reflect the changes and eliminate possible\nambiguities.\n\nFinally, adjust refs/files-backend.c to check for now possible\ndir_iterator_begin() failures.\n\nOriginal-patch-by: Daniel Ferreira \nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 234 ++++++++++++++++++-----------------\n dir-iterator.h | 17 ++-\n refs/files-backend.c | 17 ++-\n t/helper/test-dir-iterator.c | 5 +\n t/t0066-dir-iterator.sh | 13 ++\n 5 files changed, 164 insertions(+), 122 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex 0c8880868a..594fe4d67b 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -4,8 +4,6 @@\n #include \"dir-iterator.h\"\n \n struct dir_iterator_level {\n-\tint initialized;\n-\n \tDIR *dir;\n \n \t/*\n@@ -13,16 +11,6 @@ struct dir_iterator_level {\n \t * (including a trailing '/'):\n \t */\n \tsize_t prefix_len;\n-\n-\t/*\n-\t * The last action that has been taken with the current entry\n-\t * (needed for directories, which have to be included in the\n-\t * iteration and also iterated into):\n-\t */\n-\tenum {\n-\t\tDIR_STATE_ITER,\n-\t\tDIR_STATE_RECURSE\n-\t} dir_state;\n };\n \n /*\n@@ -34,9 +22,11 @@ struct dir_iterator_int {\n \tstruct dir_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack. After the first\n+\t * call to dir_iterator_begin(), if it succeeds to open the\n+\t * first level's dir, this will always be at least 1. Then,\n+\t * when it comes to zero the iteration is ended and this\n+\t * struct is freed.\n \t */\n \tsize_t levels_nr;\n \n@@ -50,113 +40,118 @@ struct dir_iterator_int {\n \tstruct dir_iterator_level *levels;\n };\n \n+/*\n+ * Push a level in the iter stack and initialize it with information from\n+ * the directory pointed by iter->base->path. It is assumed that this\n+ * strbuf points to a valid directory path. Return 0 on success and -1\n+ * otherwise, leaving the stack unchanged.\n+ */\n+static int push_level(struct dir_iterator_int *iter)\n+{\n+\tstruct dir_iterator_level *level;\n+\n+\tALLOC_GROW(iter->levels, iter->levels_nr + 1, iter->levels_alloc);\n+\tlevel = &iter->levels[iter->levels_nr++];\n+\n+\tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n+\t\tstrbuf_addch(&iter->base.path, '/');\n+\tlevel->prefix_len = iter->base.path.len;\n+\n+\tlevel->dir = opendir(iter->base.path.buf);\n+\tif (!level->dir) {\n+\t\tif (errno != ENOENT) {\n+\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\t\t\t iter->base.path.buf);\n+\t\t}\n+\t\titer->levels_nr--;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Pop the top level on the iter stack, releasing any resources associated\n+ * with it. Return the new value of iter->levels_nr.\n+ */\n+static int pop_level(struct dir_iterator_int *iter)\n+{\n+\tstruct dir_iterator_level *level =\n+\t\t&iter->levels[iter->levels_nr - 1];\n+\n+\tif (level->dir && closedir(level->dir))\n+\t\twarning_errno(\"error closing directory '%s'\",\n+\t\t\t iter->base.path.buf);\n+\tlevel->dir = NULL;\n+\n+\treturn --iter->levels_nr;\n+}\n+\n+/*\n+ * Populate iter->base with the necessary information on the next iteration\n+ * entry, represented by the given dirent de. Return 0 on success and -1\n+ * otherwise.\n+ */\n+static int prepare_next_entry_data(struct dir_iterator_int *iter,\n+\t\t\t\t struct dirent *de)\n+{\n+\tstrbuf_addstr(&iter->base.path, de->d_name);\n+\t/*\n+\t * We have to reset these because the path strbuf might have\n+\t * been realloc()ed at the previous strbuf_addstr().\n+\t */\n+\titer->base.relative_path = iter->base.path.buf +\n+\t\t\t\t iter->levels[0].prefix_len;\n+\titer->base.basename = iter->base.path.buf +\n+\t\t\t iter->levels[iter->levels_nr - 1].prefix_len;\n+\n+\tif (lstat(iter->base.path.buf, &iter->base.st)) {\n+\t\tif (errno != ENOENT)\n+\t\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n \n+\tif (S_ISDIR(iter->base.st.st_mode)) {\n+\t\tif (push_level(iter) && iter->levels_nr == 0) {\n+\t\t\t/* Pushing the first level failed */\n+\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t}\n+\t}\n+\n+\t/* Loop until we find an entry that we can give back to the caller. */\n \twhile (1) {\n+\t\tstruct dirent *de;\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n-\t\tstruct dirent *de;\n \n-\t\tif (!level->initialized) {\n-\t\t\t/*\n-\t\t\t * Note: dir_iterator_begin() ensures that\n-\t\t\t * path is not the empty string.\n-\t\t\t */\n-\t\t\tif (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1]))\n-\t\t\t\tstrbuf_addch(&iter->base.path, '/');\n-\t\t\tlevel->prefix_len = iter->base.path.len;\n-\n-\t\t\tlevel->dir = opendir(iter->base.path.buf);\n-\t\t\tif (!level->dir && errno != ENOENT) {\n-\t\t\t\twarning_errno(\"error opening directory '%s'\",\n+\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n+\t\terrno = 0;\n+\t\tde = readdir(level->dir);\n+\n+\t\tif (!de) {\n+\t\t\tif (errno)\n+\t\t\t\twarning_errno(\"error reading directory '%s'\",\n \t\t\t\t\t iter->base.path.buf);\n-\t\t\t\t/* Popping the level is handled below */\n-\t\t\t}\n-\n-\t\t\tlevel->initialized = 1;\n-\t\t} else if (S_ISDIR(iter->base.st.st_mode)) {\n-\t\t\tif (level->dir_state == DIR_STATE_ITER) {\n-\t\t\t\t/*\n-\t\t\t\t * The directory was just iterated\n-\t\t\t\t * over; now prepare to iterate into\n-\t\t\t\t * it.\n-\t\t\t\t */\n-\t\t\t\tlevel->dir_state = DIR_STATE_RECURSE;\n-\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n-\t\t\t\t\t iter->levels_alloc);\n-\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n-\t\t\t\tlevel->initialized = 0;\n-\t\t\t\tcontinue;\n-\t\t\t} else {\n-\t\t\t\t/*\n-\t\t\t\t * The directory has already been\n-\t\t\t\t * iterated over and iterated into;\n-\t\t\t\t * we're done with it.\n-\t\t\t\t */\n-\t\t\t}\n+\t\t\telse if (pop_level(iter) == 0)\n+\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\tcontinue;\n \t\t}\n \n-\t\tif (!level->dir) {\n-\t\t\t/*\n-\t\t\t * This level is exhausted (or wasn't opened\n-\t\t\t * successfully); pop up a level.\n-\t\t\t */\n-\t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n \n+\t\tif (prepare_next_entry_data(iter, de))\n \t\t\tcontinue;\n-\t\t}\n \n-\t\t/*\n-\t\t * Loop until we find an entry that we can give back\n-\t\t * to the caller:\n-\t\t */\n-\t\twhile (1) {\n-\t\t\tstrbuf_setlen(&iter->base.path, level->prefix_len);\n-\t\t\terrno = 0;\n-\t\t\tde = readdir(level->dir);\n-\n-\t\t\tif (!de) {\n-\t\t\t\t/* This level is exhausted; pop up a level. */\n-\t\t\t\tif (errno) {\n-\t\t\t\t\twarning_errno(\"error reading directory '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\t\t\t\t} else if (closedir(level->dir))\n-\t\t\t\t\twarning_errno(\"error closing directory '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\n-\t\t\t\tlevel->dir = NULL;\n-\t\t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n-\t\t\t\tbreak;\n-\t\t\t}\n-\n-\t\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addstr(&iter->base.path, de->d_name);\n-\t\t\tif (lstat(iter->base.path.buf, &iter->base.st) < 0) {\n-\t\t\t\tif (errno != ENOENT)\n-\t\t\t\t\twarning_errno(\"failed to stat '%s'\",\n-\t\t\t\t\t\t iter->base.path.buf);\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\n-\t\t\t/*\n-\t\t\t * We have to set these each time because\n-\t\t\t * the path strbuf might have been realloc()ed.\n-\t\t\t */\n-\t\t\titer->base.relative_path =\n-\t\t\t\titer->base.path.buf + iter->levels[0].prefix_len;\n-\t\t\titer->base.basename =\n-\t\t\t\titer->base.path.buf + level->prefix_len;\n-\t\t\tlevel->dir_state = DIR_STATE_ITER;\n-\n-\t\t\treturn ITER_OK;\n-\t\t}\n+\t\treturn ITER_OK;\n \t}\n }\n \n@@ -187,17 +182,32 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n-\n-\tif (!path || !*path)\n-\t\tBUG(\"empty path passed to dir_iterator_begin()\");\n+\tint saved_errno;\n \n \tstrbuf_init(&iter->base.path, PATH_MAX);\n \tstrbuf_addstr(&iter->base.path, path);\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n+\titer->levels_nr = 0;\n \n-\titer->levels_nr = 1;\n-\titer->levels[0].initialized = 0;\n+\t/*\n+\t * Note: stat already checks for NULL or empty strings and\n+\t * inexistent paths.\n+\t */\n+\tif (stat(iter->base.path.buf, &iter->base.st) < 0) {\n+\t\tsaved_errno = errno;\n+\t\tgoto error_out;\n+\t}\n+\n+\tif (!S_ISDIR(iter->base.st.st_mode)) {\n+\t\tsaved_errno = ENOTDIR;\n+\t\tgoto error_out;\n+\t}\n \n \treturn dir_iterator;\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\terrno = saved_errno;\n+\treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 970793d07a..9b4cb7acd2 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -8,18 +8,22 @@\n *\n * Iterate over a directory tree, recursively, including paths of all\n * types and hidden paths. Skip \".\" and \"..\" entries and don't follow\n- * symlinks except for the original path.\n+ * symlinks except for the original path. Note that the original path\n+ * is not included in the iteration.\n *\n * Every time dir_iterator_advance() is called, update the members of\n * the dir_iterator structure to reflect the next path in the\n * iteration. The order that paths are iterated over within a\n- * directory is undefined, but directory paths are always iterated\n- * over before the subdirectory contents.\n+ * directory is undefined, directory paths are always given before\n+ * their contents.\n *\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct iterator *iter = dir_iterator_begin(path);\n+ * struct dir_iterator *iter = dir_iterator_begin(path);\n+ *\n+ * if (!iter)\n+ * goto error_handler;\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n@@ -59,8 +63,9 @@ struct dir_iterator {\n };\n \n /*\n- * Start a directory iteration over path. Return a dir_iterator that\n- * holds the internal state of the iteration.\n+ * Start a directory iteration over path. On success, return a\n+ * dir_iterator that holds the internal state of the iteration.\n+ * In case of failure, return NULL and set errno accordingly.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 63e55e6773..7ed81046d4 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2143,13 +2143,22 @@ static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \t\t\t\t\t\t const char *gitdir)\n {\n-\tstruct files_reflog_iterator *iter = xcalloc(1, sizeof(*iter));\n-\tstruct ref_iterator *ref_iterator = &iter->base;\n+\tstruct dir_iterator *diter;\n+\tstruct files_reflog_iterator *iter;\n+\tstruct ref_iterator *ref_iterator;\n \tstruct strbuf sb = STRBUF_INIT;\n \n-\tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n-\titer->dir_iterator = dir_iterator_begin(sb.buf);\n+\n+\tditer = dir_iterator_begin(sb.buf);\n+\tif(!diter)\n+\t\treturn empty_ref_iterator_begin();\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tref_iterator = &iter->base;\n+\n+\tbase_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0);\n+\titer->dir_iterator = diter;\n \titer->ref_store = ref_store;\n \tstrbuf_release(&sb);\n \ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 84f50bed8c..fab1ff6237 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -17,6 +17,11 @@ int cmd__dir_iterator(int argc, const char **argv)\n \n \tditer = dir_iterator_begin(path.buf);\n \n+\tif (!diter) {\n+\t\tprintf(\"dir_iterator_begin failure: %d\\n\", errno);\n+\t\texit(EXIT_FAILURE);\n+\t}\n+\n \twhile (dir_iterator_advance(diter) == ITER_OK) {\n \t\tif (S_ISDIR(diter->st.st_mode))\n \t\t\tprintf(\"[d] \");\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex 59bce868f4..cc4b19c34c 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -52,4 +52,17 @@ test_expect_success 'dir-iterator should list files in the correct order' '\n \ttest_cmp expected-pre-order-output actual-pre-order-output\n '\n \n+test_expect_success 'begin should fail upon inexistent paths' '\n+\ttest_must_fail test-tool dir-iterator ./inexistent-path \\\n+\t\t>actual-inexistent-path-output &&\n+\techo \"dir_iterator_begin failure: 2\" >expected-inexistent-path-output &&\n+\ttest_cmp expected-inexistent-path-output actual-inexistent-path-output\n+'\n+\n+test_expect_success 'begin should fail upon non directory paths' '\n+\ttest_must_fail test-tool dir-iterator ./dir/b >actual-non-dir-output &&\n+\techo \"dir_iterator_begin failure: 20\" >expected-non-dir-output &&\n+\ttest_cmp expected-non-dir-output actual-non-dir-output\n+'\n+\n test_done\n-- \n2.22.0\n\n"},{"id":"378788","messageId":"fe838d7eb4a3f9affca32478397abf8aca9b0230.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 06/10] dir-iterator: add flags parameter to dir_iterator_begin","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:59:00Z","receivedAt":"2019-07-11T00:00:19Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the possibility of giving flags to dir_iterator_begin to initialize\na dir-iterator with special options.\n\nCurrently possible flags are:\n- DIR_ITERATOR_PEDANTIC, which makes dir_iterator_advance abort\nimmediately in the case of an error, instead of keep looking for the\nnext valid entry;\n- DIR_ITERATOR_FOLLOW_SYMLINKS, which makes the iterator follow\nsymlinks and include linked directories' contents in the iteration.\n\nThese new flags will be used in a subsequent patch.\n\nAlso add tests for the flags' usage and adjust refs/files-backend.c to\nthe new dir_iterator_begin signature.\n\nSigned-off-by: Matheus Tavares \n---\n dir-iterator.c | 56 +++++++++++++++++--------\n dir-iterator.h | 55 ++++++++++++++++++++-----\n refs/files-backend.c | 2 +-\n t/helper/test-dir-iterator.c | 34 +++++++++++----\n t/t0066-dir-iterator.sh | 80 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 191 insertions(+), 36 deletions(-)\n\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex 594fe4d67b..b17e9f970a 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -38,13 +38,16 @@ struct dir_iterator_int {\n \t * that will be included in this iteration.\n \t */\n \tstruct dir_iterator_level *levels;\n+\n+\t/* Combination of flags for this dir-iterator */\n+\tunsigned int flags;\n };\n \n /*\n * Push a level in the iter stack and initialize it with information from\n * the directory pointed by iter->base->path. It is assumed that this\n * strbuf points to a valid directory path. Return 0 on success and -1\n- * otherwise, leaving the stack unchanged.\n+ * otherwise, setting errno accordingly and leaving the stack unchanged.\n */\n static int push_level(struct dir_iterator_int *iter)\n {\n@@ -59,11 +62,13 @@ static int push_level(struct dir_iterator_int *iter)\n \n \tlevel->dir = opendir(iter->base.path.buf);\n \tif (!level->dir) {\n+\t\tint saved_errno = errno;\n \t\tif (errno != ENOENT) {\n \t\t\twarning_errno(\"error opening directory '%s'\",\n \t\t\t\t iter->base.path.buf);\n \t\t}\n \t\titer->levels_nr--;\n+\t\terrno = saved_errno;\n \t\treturn -1;\n \t}\n \n@@ -90,11 +95,13 @@ static int pop_level(struct dir_iterator_int *iter)\n /*\n * Populate iter->base with the necessary information on the next iteration\n * entry, represented by the given dirent de. Return 0 on success and -1\n- * otherwise.\n+ * otherwise, setting errno accordingly.\n */\n static int prepare_next_entry_data(struct dir_iterator_int *iter,\n \t\t\t\t struct dirent *de)\n {\n+\tint err, saved_errno;\n+\n \tstrbuf_addstr(&iter->base.path, de->d_name);\n \t/*\n \t * We have to reset these because the path strbuf might have\n@@ -105,13 +112,17 @@ static int prepare_next_entry_data(struct dir_iterator_int *iter,\n \titer->base.basename = iter->base.path.buf +\n \t\t\t iter->levels[iter->levels_nr - 1].prefix_len;\n \n-\tif (lstat(iter->base.path.buf, &iter->base.st)) {\n-\t\tif (errno != ENOENT)\n-\t\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n-\t\treturn -1;\n-\t}\n+\tif (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS)\n+\t\terr = stat(iter->base.path.buf, &iter->base.st);\n+\telse\n+\t\terr = lstat(iter->base.path.buf, &iter->base.st);\n \n-\treturn 0;\n+\tsaved_errno = errno;\n+\tif (err && errno != ENOENT)\n+\t\twarning_errno(\"failed to stat '%s'\", iter->base.path.buf);\n+\n+\terrno = saved_errno;\n+\treturn err;\n }\n \n int dir_iterator_advance(struct dir_iterator *dir_iterator)\n@@ -119,11 +130,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \tstruct dir_iterator_int *iter =\n \t\t(struct dir_iterator_int *)dir_iterator;\n \n-\tif (S_ISDIR(iter->base.st.st_mode)) {\n-\t\tif (push_level(iter) && iter->levels_nr == 0) {\n-\t\t\t/* Pushing the first level failed */\n-\t\t\treturn dir_iterator_abort(dir_iterator);\n-\t\t}\n+\tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n+\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\tgoto error_out;\n+\t\tif (iter->levels_nr == 0)\n+\t\t\tgoto error_out;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -137,22 +148,32 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\tde = readdir(level->dir);\n \n \t\tif (!de) {\n-\t\t\tif (errno)\n+\t\t\tif (errno) {\n \t\t\t\twarning_errno(\"error reading directory '%s'\",\n \t\t\t\t\t iter->base.path.buf);\n-\t\t\telse if (pop_level(iter) == 0)\n+\t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\t\tgoto error_out;\n+\t\t\t} else if (pop_level(iter) == 0) {\n \t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t}\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n-\t\tif (prepare_next_entry_data(iter, de))\n+\t\tif (prepare_next_entry_data(iter, de)) {\n+\t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n+\t\t\t\tgoto error_out;\n \t\t\tcontinue;\n+\t\t}\n \n \t\treturn ITER_OK;\n \t}\n+\n+error_out:\n+\tdir_iterator_abort(dir_iterator);\n+\treturn ITER_ERROR;\n }\n \n int dir_iterator_abort(struct dir_iterator *dir_iterator)\n@@ -178,7 +199,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \treturn ITER_DONE;\n }\n \n-struct dir_iterator *dir_iterator_begin(const char *path)\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n {\n \tstruct dir_iterator_int *iter = xcalloc(1, sizeof(*iter));\n \tstruct dir_iterator *dir_iterator = &iter->base;\n@@ -189,6 +210,7 @@ struct dir_iterator *dir_iterator_begin(const char *path)\n \n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \titer->levels_nr = 0;\n+\titer->flags = flags;\n \n \t/*\n \t * Note: stat already checks for NULL or empty strings and\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 9b4cb7acd2..08229157c6 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -20,7 +20,8 @@\n * A typical iteration looks like this:\n *\n * int ok;\n- * struct dir_iterator *iter = dir_iterator_begin(path);\n+ * unsigned int flags = DIR_ITERATOR_PEDANTIC;\n+ * struct dir_iterator *iter = dir_iterator_begin(path, flags);\n *\n * if (!iter)\n * goto error_handler;\n@@ -44,6 +45,29 @@\n * dir_iterator_advance() again.\n */\n \n+/*\n+ * Flags for dir_iterator_begin:\n+ *\n+ * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior\n+ * in case of an error at dir_iterator_advance(), which is to keep\n+ * looking for a next valid entry. With this flag, resources are freed\n+ * and ITER_ERROR is returned immediately. In both cases, a meaningful\n+ * warning is emitted. Note: ENOENT errors are always ignored so that\n+ * the API users may remove files during iteration.\n+ *\n+ * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks.\n+ * i.e., linked directories' contents will be iterated over and\n+ * iter->base.st will contain information on the referred files,\n+ * not the symlinks themselves, which is the default behavior. Broken\n+ * symlinks are ignored.\n+ *\n+ * Warning: circular symlinks are also followed when\n+ * DIR_ITERATOR_FOLLOW_SYMLINKS is set. The iteration may end up with\n+ * an ELOOP if they happen and DIR_ITERATOR_PEDANTIC is set.\n+ */\n+#define DIR_ITERATOR_PEDANTIC (1 << 0)\n+#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1)\n+\n struct dir_iterator {\n \t/* The current path: */\n \tstruct strbuf path;\n@@ -58,29 +82,38 @@ struct dir_iterator {\n \t/* The current basename: */\n \tconst char *basename;\n \n-\t/* The result of calling lstat() on path: */\n+\t/*\n+\t * The result of calling lstat() on path; or stat(), if the\n+\t * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at\n+\t * dir_iterator's initialization.\n+\t */\n \tstruct stat st;\n };\n \n /*\n- * Start a directory iteration over path. On success, return a\n- * dir_iterator that holds the internal state of the iteration.\n- * In case of failure, return NULL and set errno accordingly.\n+ * Start a directory iteration over path with the combination of\n+ * options specified by flags. On success, return a dir_iterator\n+ * that holds the internal state of the iteration. In case of\n+ * failure, return NULL and set errno accordingly.\n *\n * The iteration includes all paths under path, not including path\n * itself and not including \".\" or \"..\" entries.\n *\n- * path is the starting directory. An internal copy will be made.\n+ * Parameters are:\n+ * - path is the starting directory. An internal copy will be made.\n+ * - flags is a combination of the possible flags to initialize a\n+ * dir-iterator or 0 for default behavior.\n */\n-struct dir_iterator *dir_iterator_begin(const char *path);\n+struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n * If the iteration is exhausted, free the dir_iterator and any\n- * resources associated with it and return ITER_DONE. On error, free\n- * dir_iterator and associated resources and return ITER_ERROR. It is\n- * a bug to use iterator or call this function again after it has\n- * returned ITER_DONE or ITER_ERROR.\n+ * resources associated with it and return ITER_DONE.\n+ *\n+ * It is a bug to use iterator or call this function again after it\n+ * has returned ITER_DONE or ITER_ERROR (which may be returned iff\n+ * the DIR_ITERATOR_PEDANTIC flag was set).\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 7ed81046d4..b1f8f53a09 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2150,7 +2150,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n \n \tstrbuf_addf(&sb, \"%s/logs\", gitdir);\n \n-\tditer = dir_iterator_begin(sb.buf);\n+\tditer = dir_iterator_begin(sb.buf, 0);\n \tif(!diter)\n \t\treturn empty_ref_iterator_begin();\n \ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex fab1ff6237..a5b96cb0dc 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -4,29 +4,44 @@\n #include \"iterator.h\"\n #include \"dir-iterator.h\"\n \n-/* Argument is a directory path to iterate over */\n+/*\n+ * usage:\n+ * tool-test dir-iterator [--follow-symlinks] [--pedantic] directory_path\n+ */\n int cmd__dir_iterator(int argc, const char **argv)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct dir_iterator *diter;\n+\tunsigned int flags = 0;\n+\tint iter_status;\n+\n+\tfor (++argv, --argc; *argv && starts_with(*argv, \"--\"); ++argv, --argc) {\n+\t\tif (strcmp(*argv, \"--follow-symlinks\") == 0)\n+\t\t\tflags |= DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\t\telse if (strcmp(*argv, \"--pedantic\") == 0)\n+\t\t\tflags |= DIR_ITERATOR_PEDANTIC;\n+\t\telse\n+\t\t\tdie(\"invalid option '%s'\", *argv);\n+\t}\n \n-\tif (argc < 2)\n-\t\tdie(\"BUG: test-dir-iterator needs one argument\");\n-\n-\tstrbuf_add(&path, argv[1], strlen(argv[1]));\n+\tif (!*argv || argc != 1)\n+\t\tdie(\"dir-iterator needs exactly one non-option argument\");\n \n-\tditer = dir_iterator_begin(path.buf);\n+\tstrbuf_add(&path, *argv, strlen(*argv));\n+\tditer = dir_iterator_begin(path.buf, flags);\n \n \tif (!diter) {\n \t\tprintf(\"dir_iterator_begin failure: %d\\n\", errno);\n \t\texit(EXIT_FAILURE);\n \t}\n \n-\twhile (dir_iterator_advance(diter) == ITER_OK) {\n+\twhile ((iter_status = dir_iterator_advance(diter)) == ITER_OK) {\n \t\tif (S_ISDIR(diter->st.st_mode))\n \t\t\tprintf(\"[d] \");\n \t\telse if (S_ISREG(diter->st.st_mode))\n \t\t\tprintf(\"[f] \");\n+\t\telse if (S_ISLNK(diter->st.st_mode))\n+\t\t\tprintf(\"[s] \");\n \t\telse\n \t\t\tprintf(\"[?] \");\n \n@@ -34,5 +49,10 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\t diter->path.buf);\n \t}\n \n+\tif (iter_status != ITER_DONE) {\n+\t\tprintf(\"dir_iterator_advance failure\\n\");\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\ndiff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh\nindex cc4b19c34c..9354d3f1ed 100755\n--- a/t/t0066-dir-iterator.sh\n+++ b/t/t0066-dir-iterator.sh\n@@ -65,4 +65,84 @@ test_expect_success 'begin should fail upon non directory paths' '\n \ttest_cmp expected-non-dir-output actual-non-dir-output\n '\n \n+test_expect_success POSIXPERM,SANITY 'advance should not fail on errors by default' '\n+\tcat >expected-no-permissions-output <<-EOF &&\n+\t[d] (a) [a] ./dir3/a\n+\tEOF\n+\n+\tmkdir -p dir3/a &&\n+\t>dir3/a/b &&\n+\tchmod 0 dir3/a &&\n+\n+\ttest-tool dir-iterator ./dir3 >actual-no-permissions-output &&\n+\ttest_cmp expected-no-permissions-output actual-no-permissions-output &&\n+\tchmod 755 dir3/a &&\n+\trm -rf dir3\n+'\n+\n+test_expect_success POSIXPERM,SANITY 'advance should fail on errors, w/ pedantic flag' '\n+\tcat >expected-no-permissions-pedantic-output <<-EOF &&\n+\t[d] (a) [a] ./dir3/a\n+\tdir_iterator_advance failure\n+\tEOF\n+\n+\tmkdir -p dir3/a &&\n+\t>dir3/a/b &&\n+\tchmod 0 dir3/a &&\n+\n+\ttest_must_fail test-tool dir-iterator --pedantic ./dir3 \\\n+\t\t>actual-no-permissions-pedantic-output &&\n+\ttest_cmp expected-no-permissions-pedantic-output \\\n+\t\tactual-no-permissions-pedantic-output &&\n+\tchmod 755 dir3/a &&\n+\trm -rf dir3\n+'\n+\n+test_expect_success SYMLINKS 'setup dirs with symlinks' '\n+\tmkdir -p dir4/a &&\n+\tmkdir -p dir4/b/c &&\n+\t>dir4/a/d &&\n+\tln -s d dir4/a/e &&\n+\tln -s ../b dir4/a/f &&\n+\n+\tmkdir -p dir5/a/b &&\n+\tmkdir -p dir5/a/c &&\n+\tln -s ../c dir5/a/b/d &&\n+\tln -s ../ dir5/a/b/e &&\n+\tln -s ../../ dir5/a/b/f\n+'\n+\n+test_expect_success SYMLINKS 'dir-iterator should not follow symlinks by default' '\n+\tcat >expected-no-follow-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir4/a\n+\t[d] (b) [b] ./dir4/b\n+\t[d] (b/c) [c] ./dir4/b/c\n+\t[f] (a/d) [d] ./dir4/a/d\n+\t[s] (a/e) [e] ./dir4/a/e\n+\t[s] (a/f) [f] ./dir4/a/f\n+\tEOF\n+\n+\ttest-tool dir-iterator ./dir4 >out &&\n+\tsort out >actual-no-follow-sorted-output &&\n+\n+\ttest_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output\n+'\n+\n+test_expect_success SYMLINKS 'dir-iterator should follow symlinks w/ follow flag' '\n+\tcat >expected-follow-sorted-output <<-EOF &&\n+\t[d] (a) [a] ./dir4/a\n+\t[d] (a/f) [f] ./dir4/a/f\n+\t[d] (a/f/c) [c] ./dir4/a/f/c\n+\t[d] (b) [b] ./dir4/b\n+\t[d] (b/c) [c] ./dir4/b/c\n+\t[f] (a/d) [d] ./dir4/a/d\n+\t[f] (a/e) [e] ./dir4/a/e\n+\tEOF\n+\n+\ttest-tool dir-iterator --follow-symlinks ./dir4 >out &&\n+\tsort out >actual-follow-sorted-output &&\n+\n+\ttest_cmp expected-follow-sorted-output actual-follow-sorted-output\n+'\n+\n test_done\n-- \n2.22.0\n\n"},{"id":"378789","messageId":"3da6408e045de6e39166227a60472bd1952664ad.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 07/10] clone: copy hidden paths at local clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:59:01Z","receivedAt":"2019-07-11T00:00:23Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make the copy_or_link_directory function no longer skip hidden\ndirectories. This function, used to copy .git/objects, currently skips\nall hidden directories but not hidden files, which is an odd behaviour.\nThe reason for that could be unintentional: probably the intention was\nto skip '.' and '..' only but it ended up accidentally skipping all\ndirectories starting with '.'. Besides being more natural, the new\nbehaviour is more permissive to the user.\n\nAlso adjust tests to reflect this behaviour change.\n\nSigned-off-by: Matheus Tavares \nSigned-off-by: Ævar Arnfjörð Bjarmason \nCo-authored-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 2 +-\n t/t5604-clone-reference.sh | 9 +++++++++\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4a0a2455a7..9dd083e34d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -430,7 +430,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\tcontinue;\n \t\t}\n \t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (de->d_name[0] != '.')\n+\t\t\tif (!is_dot_or_dotdot(de->d_name))\n \t\t\t\tcopy_or_link_directory(src, dest,\n \t\t\t\t\t\t src_repo, src_baselen);\n \t\t\tcontinue;\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 459ad8a20b..4894237ab8 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' '\n \tdone &&\n \tfind S-* -name \"*some*\" | sort >actual &&\n \tcat >expected <<-EOF &&\n+\tS--dissociate/.git/objects/.some-hidden-dir\n+\tS--dissociate/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--dissociate/.git/objects/.some-hidden-dir/some-file\n \tS--dissociate/.git/objects/.some-hidden-file\n \tS--dissociate/.git/objects/some-dir\n \tS--dissociate/.git/objects/some-dir/.some-dot-file\n \tS--dissociate/.git/objects/some-dir/some-file\n \tS--dissociate/.git/objects/some-file\n+\tS--local/.git/objects/.some-hidden-dir\n+\tS--local/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--local/.git/objects/.some-hidden-dir/some-file\n \tS--local/.git/objects/.some-hidden-file\n \tS--local/.git/objects/some-dir\n \tS--local/.git/objects/some-dir/.some-dot-file\n \tS--local/.git/objects/some-dir/some-file\n \tS--local/.git/objects/some-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file\n+\tS--no-hardlinks/.git/objects/.some-hidden-dir/some-file\n \tS--no-hardlinks/.git/objects/.some-hidden-file\n \tS--no-hardlinks/.git/objects/some-dir\n \tS--no-hardlinks/.git/objects/some-dir/.some-dot-file\n-- \n2.22.0\n\n"},{"id":"378790","messageId":"af7430eb2c29ce35691e15d68e1c59d48d6e9144.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 08/10] clone: extract function from copy_or_link_directory","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:59:02Z","receivedAt":"2019-07-11T00:00:28Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Extract dir creation code snippet from copy_or_link_directory to its own\nfunction named mkdir_if_missing. This change will help to remove\ncopy_or_link_directory's explicit recursion, which will be done in a\nfollowing patch. Also makes the code more readable.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9dd083e34d..96566c1bab 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -394,6 +394,21 @@ static void copy_alternates(struct strbuf *src, const char *src_repo)\n \tfclose(in);\n }\n \n+static void mkdir_if_missing(const char *pathname, mode_t mode)\n+{\n+\tstruct stat st;\n+\n+\tif (!mkdir(pathname, mode))\n+\t\treturn;\n+\n+\tif (errno != EEXIST)\n+\t\tdie_errno(_(\"failed to create directory '%s'\"), pathname);\n+\telse if (stat(pathname, &st))\n+\t\tdie_errno(_(\"failed to stat '%s'\"), pathname);\n+\telse if (!S_ISDIR(st.st_mode))\n+\t\tdie(_(\"%s exists and is not a directory\"), pathname);\n+}\n+\n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t\t\t const char *src_repo, int src_baselen)\n {\n@@ -406,14 +421,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \tif (!dir)\n \t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n \n-\tif (mkdir(dest->buf, 0777)) {\n-\t\tif (errno != EEXIST)\n-\t\t\tdie_errno(_(\"failed to create directory '%s'\"), dest->buf);\n-\t\telse if (stat(dest->buf, &buf))\n-\t\t\tdie_errno(_(\"failed to stat '%s'\"), dest->buf);\n-\t\telse if (!S_ISDIR(buf.st_mode))\n-\t\t\tdie(_(\"%s exists and is not a directory\"), dest->buf);\n-\t}\n+\tmkdir_if_missing(dest->buf, 0777);\n \n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n-- \n2.22.0\n\n"},{"id":"378791","messageId":"e8308c74085689876e25cc88e5628cfd68fc1606.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 09/10] clone: use dir-iterator to avoid explicit dir traversal","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:59:03Z","receivedAt":"2019-07-11T00:00:34Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace usage of opendir/readdir/closedir API to traverse directories\nrecursively, at copy_or_link_directory function, by the dir-iterator\nAPI. This simplifies the code and avoids recursive calls to\ncopy_or_link_directory.\n\nThis process also makes copy_or_link_directory call die() in case of an\nerror on readdir or stat inside dir_iterator_advance. Previously it\nwould just print a warning for errors on stat and ignore errors on\nreaddir, which isn't nice because a local git clone could succeed even\nthough the .git/objects copy didn't fully succeed.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 47 +++++++++++++++++++++++++----------------------\n 1 file changed, 25 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 96566c1bab..47cb4a2a8e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,8 @@\n #include \"transport.h\"\n #include \"strbuf.h\"\n #include \"dir.h\"\n+#include \"dir-iterator.h\"\n+#include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n #include \"remote.h\"\n@@ -410,42 +412,39 @@ static void mkdir_if_missing(const char *pathname, mode_t mode)\n }\n \n static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n-\t\t\t\t const char *src_repo, int src_baselen)\n+\t\t\t\t const char *src_repo)\n {\n-\tstruct dirent *de;\n-\tstruct stat buf;\n \tint src_len, dest_len;\n-\tDIR *dir;\n-\n-\tdir = opendir(src->buf);\n-\tif (!dir)\n-\t\tdie_errno(_(\"failed to open '%s'\"), src->buf);\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tunsigned int flags;\n \n \tmkdir_if_missing(dest->buf, 0777);\n \n+\tflags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;\n+\titer = dir_iterator_begin(src->buf, flags);\n+\n+\tif (!iter)\n+\t\tdie_errno(_(\"failed to start iterator over '%s'\"), src->buf);\n+\n \tstrbuf_addch(src, '/');\n \tsrc_len = src->len;\n \tstrbuf_addch(dest, '/');\n \tdest_len = dest->len;\n \n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tstrbuf_setlen(src, src_len);\n-\t\tstrbuf_addstr(src, de->d_name);\n+\t\tstrbuf_addstr(src, iter->relative_path);\n \t\tstrbuf_setlen(dest, dest_len);\n-\t\tstrbuf_addstr(dest, de->d_name);\n-\t\tif (stat(src->buf, &buf)) {\n-\t\t\twarning (_(\"failed to stat %s\\n\"), src->buf);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (S_ISDIR(buf.st_mode)) {\n-\t\t\tif (!is_dot_or_dotdot(de->d_name))\n-\t\t\t\tcopy_or_link_directory(src, dest,\n-\t\t\t\t\t\t src_repo, src_baselen);\n+\t\tstrbuf_addstr(dest, iter->relative_path);\n+\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tmkdir_if_missing(dest->buf, 0777);\n \t\t\tcontinue;\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(src->buf + src_baselen, \"/info/alternates\")) {\n+\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, src_repo);\n \t\t\tcontinue;\n \t\t}\n@@ -462,7 +461,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tif (copy_file_with_time(dest->buf, src->buf, 0666))\n \t\t\tdie_errno(_(\"failed to copy file to '%s'\"), dest->buf);\n \t}\n-\tclosedir(dir);\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tstrbuf_setlen(src, src_len);\n+\t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n+\t}\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\n@@ -480,7 +483,7 @@ static void clone_local(const char *src_repo, const char *dest_repo)\n \t\tget_common_dir(&dest, dest_repo);\n \t\tstrbuf_addstr(&src, \"/objects\");\n \t\tstrbuf_addstr(&dest, \"/objects\");\n-\t\tcopy_or_link_directory(&src, &dest, src_repo, src.len);\n+\t\tcopy_or_link_directory(&src, &dest, src_repo);\n \t\tstrbuf_release(&src);\n \t\tstrbuf_release(&dest);\n \t}\n-- \n2.22.0\n\n"},{"id":"378792","messageId":"782ca07eed2c9bac4378e5128ff996b25ed86a43.1562801255.git.matheus.bernardino@usp.br","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"[GSoC][PATCH v8 10/10] clone: replace strcmp by fspathcmp","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-10T23:59:04Z","receivedAt":"2019-07-11T00:00:39Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Replace the use of strcmp by fspathcmp at copy_or_link_directory, which\nis more permissive/friendly to case-insensitive file systems.\n\nSuggested-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n builtin/clone.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 47cb4a2a8e..8da696ef30 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -444,7 +444,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\t}\n \n \t\t/* Files that cannot be copied bit-for-bit... */\n-\t\tif (!strcmp(iter->relative_path, \"info/alternates\")) {\n+\t\tif (!fspathcmp(iter->relative_path, \"info/alternates\")) {\n \t\t\tcopy_alternates(src, src_repo);\n \t\t\tcontinue;\n \t\t}\n-- \n2.22.0\n\n"},{"id":"378802","messageId":"nycvar.QRO.7.76.6.1907111355360.47@tvgsbejvaqbjf.bet","threadId":"50591","inReplyTo":"cover.1562801254.git.matheus.bernardino@usp.br","subject":"Re: [GSoC][PATCH v8 00/10] clone: dir-iterator refactoring with tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-07-11T11:56:36Z","receivedAt":"2019-07-11T11:56:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Wed, 10 Jul 2019, Matheus Tavares wrote:\n\n> - a replacement of explicit recursive dir iteration at\n> copy_or_link_directory for the dir-iterator API;\n\nAs far as I can see, it was not replaced, but just dropped. Which is\ngood, as it will most likely address the CI failures.\n\nThanks,\nDscho\n"},{"id":"378808","messageId":"CAHd-oW7jwkHXjpExZOsP3OfJN4HP2ZmLR5h+NphovYoShC2OVQ@mail.gmail.com","threadId":"50591","inReplyTo":"nycvar.QRO.7.76.6.1907111355360.47@tvgsbejvaqbjf.bet","subject":"Re: [GSoC][PATCH v8 00/10] clone: dir-iterator refactoring with tests","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2019-07-11T15:24:46Z","receivedAt":"2019-07-11T15:25:00Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Jul 11, 2019 at 8:56 AM Johannes Schindelin\n wrote:\n>\n> Hi Matheus,\n>\n> On Wed, 10 Jul 2019, Matheus Tavares wrote:\n>\n> > - a replacement of explicit recursive dir iteration at\n> > copy_or_link_directory for the dir-iterator API;\n>\n> As far as I can see, it was not replaced, but just dropped. Which is\n> good, as it will most likely address the CI failures.\n\nYou mean the circular symlink checker, right? Yes, it was dropped. At\nthis item I was referring to a dir iteration code at builtin/clone.c\n(using opendir/readdir) which was replaced by the dir-iterator API.\n\n> Thanks,\n> Dscho\n"}]}