[GSoC][PATCH v6 00/10] clone: dir-iterator refactoring with tests

This patchset contains:
- a replacement of explicit recursive dir iteration at
  copy_or_link_directory for the dir-iterator API;
- some refactoring and behaviour changes at local clone, mainly to
  take care of symlinks and hidden files at .git/objects, together
  with tests for this types of files.
- dir-iterator refactoring and feature adding with tests.

Changes since v5:
- Add tests for the dir-iterator API
- Refactor the dir-iterator state machine model, simplifying its
  mechanics to improve readability.
- Change warning() to warning_errno() at dir-iterator.c
- Add a recursive symlinks check for dir_iterator_advance() in order
  to avoid unwanted recursions with DIR_ITERATOR_FOLLOW_SYMLIKS
- Add tests for the dir-iterator flags feature
- Make warnings be emitted both when DIR_ITERATOR_PEDANTIC is
  supplied and when it's not. It contains more relevant information
  on the error, so I thought it should be always printed.
- Make dir_iterator_begin() check if the given argument is a valid
  path to a directory.
- Adjusted some minor codestyle problems and commit messages
- Address Thomas comments in v5

v5: https://public-inbox.org/git/20190330224907.3277-1-matheus.bernardino@usp.br/ travis build: https://travis-ci.org/MatheusBernardino/git/builds/527176611

Note: I tried to use --range-diff as Thomas suggested but I'm not sure
the output is as desired. Please, let me know if I did something wrong
using it.

Daniel Ferreira (1):
  dir-iterator: add tests for dir-iterator API

Matheus Tavares (8):
  clone: better handle symlinked files at .git/objects/
  dir-iterator: use warning_errno when possible
  dir-iterator: refactor state machine model
  dir-iterator: add flags parameter to dir_iterator_begin
  clone: copy hidden paths at local clone
  clone: extract function from copy_or_link_directory
  clone: use dir-iterator to avoid explicit dir traversal
  clone: replace strcmp by fspathcmp

Ævar Arnfjörð Bjarmason (1):
  clone: test for our behavior on odd objects/* content

 Makefile                     |   1 +
 builtin/clone.c              |  75 +++++----
 dir-iterator.c               | 289 +++++++++++++++++++++--------------
 dir-iterator.h               |  59 +++++--
 refs/files-backend.c         |  17 ++-
 t/helper/test-dir-iterator.c |  58 +++++++
 t/helper/test-tool.c         |   1 +
 t/helper/test-tool.h         |   1 +
 t/t0066-dir-iterator.sh      | 163 ++++++++++++++++++++
 t/t5604-clone-reference.sh   | 133 ++++++++++++++++
 10 files changed, 634 insertions(+), 163 deletions(-)
 create mode 100644 t/helper/test-dir-iterator.c
 create mode 100755 t/t0066-dir-iterator.sh

Range-diff against v5:
 1:  3d422dd4de =  1:  a630b1a129 clone: test for our behavior on odd objects/* content
 2:  35819e6ed1 !  2:  51e06687fc clone: better handle symlinked files at .git/objects/
    @@ -45,10 +45,7 @@
      			die_errno(_("failed to unlink '%s'"), dest->buf);
      		if (!option_no_hardlinks) {
     -			if (!link(src->buf, dest->buf))
    -+			char *resolved_path = real_pathdup(src->buf, 1);
    -+			int status = link(resolved_path, dest->buf);
    -+			free(resolved_path);
    -+			if (!status)
    ++			if (!link(real_path(src->buf), dest->buf))
      				continue;
      			if (option_local > 0)
      				die_errno(_("failed to create link '%s'"), dest->buf);
 3:  2afe3208a4 <  -:  ---------- dir-iterator: add flags parameter to dir_iterator_begin
 -:  ---------- >  3:  c8a860e3a5 dir-iterator: add tests for dir-iterator API
 -:  ---------- >  4:  b975351080 dir-iterator: use warning_errno when possible
 -:  ---------- >  5:  0fdbd1633e dir-iterator: refactor state machine model
 -:  ---------- >  6:  7b2a9ae947 dir-iterator: add flags parameter to dir_iterator_begin
 4:  71d64e6278 =  7:  b9f298cbc6 clone: copy hidden paths at local clone
 5:  35e36756db =  8:  0e7b1e49e2 clone: extract function from copy_or_link_directory
 6:  1bfda87879 !  9:  f726ce2733 clone: use dir-iterator to avoid explicit dir traversal
    @@ -8,10 +8,14 @@
         copy_or_link_directory.
     
         This process also makes copy_or_link_directory call die() in case of an
    -    error on readdir or stat, inside dir_iterator_advance. Previously it
    +    error on readdir or stat inside dir_iterator_advance. Previously it
         would just print a warning for errors on stat and ignore errors on
         readdir, which isn't nice because a local git clone could succeed even
    -    though the .git/objects copy didn't fully succeed.
    +    though the .git/objects copy didn't fully succeed. Also, with the
    +    dir-iterator API, recursive symlinks will be detected and skipped. This
    +    is another behavior improvement, since the current version would
    +    continue to copy the same content over and over until stat() returned an
    +    ELOOP error.
     
         Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
     
    @@ -44,12 +48,15 @@
     -		die_errno(_("failed to open '%s'"), src->buf);
     +	struct dir_iterator *iter;
     +	int iter_status;
    -+	unsigned flags;
    ++	unsigned int flags;
      
      	mkdir_if_missing(dest->buf, 0777);
      
     +	flags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS;
     +	iter = dir_iterator_begin(src->buf, flags);
    ++
    ++	if (!iter)
    ++		die_errno(_("failed to start iterator over '%s'"), src->buf);
     +
      	strbuf_addch(src, '/');
      	src_len = src->len;
 7:  3861b30108 = 10:  6a57bb3887 clone: replace strcmp by fspathcmp