{"thread":{"id":"47510","subject":"[PATCH] git-archive: accept --owner and --group like GNU tar","startedAt":"2017-12-29T14:05:55Z","lastAt":"2018-01-05T19:10:25Z","messageCount":13,"participants":["suzuki toshiya","René Scharfe","Perry Hutchison","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"335531","messageId":"20171229140535.10746-1-mpsuzuki@hiroshima-u.ac.jp","threadId":"47510","inReplyTo":null,"subject":"[PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"suzuki toshiya","fromEmail":"mpsuzuki@hiroshima-u.ac.jp","sentAt":"2017-12-29T14:05:35Z","receivedAt":"2017-12-29T14:05:55Z","isPatch":true,"sender":{"key":"mpsuzuki@hiroshima-u.ac.jp","avatar":null},"body":"The ownership of files created by git-archive is always\nroot:root. Add --owner and --group options which work\nlike the GNU tar equivalent to allow overriding these\ndefaults.\n\nSigned-off-by: suzuki toshiya <mpsuzuki@hiroshima-u.ac.jp>\n---\n Documentation/git-archive.txt |  13 +++\n archive-tar.c                 |   8 +-\n archive.c                     | 224 ++++++++++++++++++++++++++++++++++++++++++\n archive.h                     |   4 +\n t/t5005-archive-uid-gid.sh    | 140 ++++++++++++++++++++++++++\n t/t5005/parse-tar-file.py     |  60 +++++++++++\n tar.h                         |   2 +\n 7 files changed, 447 insertions(+), 4 deletions(-)\n create mode 100755 t/t5005-archive-uid-gid.sh\n create mode 100755 t/t5005/parse-tar-file.py\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex cfa1e4ebe..0d156f6c1 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -11,6 +11,7 @@ SYNOPSIS\n [verse]\n 'git archive' [--format=<fmt>] [--list] [--prefix=<prefix>/] [<extra>]\n \t      [-o <file> | --output=<file>] [--worktree-attributes]\n+\t      [--owner [username[:uid]] [--group [groupname[:gid]]\n \t      [--remote=<repo> [--exec=<git-upload-archive>]] <tree-ish>\n \t      [<path>...]\n \n@@ -63,6 +64,18 @@ OPTIONS\n \tThis can be any options that the archiver backend understands.\n \tSee next section.\n \n+--owner=<name>[:<uid>]::\n+\tForce <name> as owner and <uid> as uid for the files in the tar\n+\tarchive.  If <uid> is not supplied, <name> can be either a user\n+\tname or numeric UID.  In this case the missing part (UID or\n+\tname) will be inferred from the current host's user database.\n+\n+--group=<name>[:<gid>]::\n+\tForce <name> as group and <gid> as gid for the files in the tar\n+\tarchive.  If <gid> is not supplied, <name> can be either a group\n+\tname or numeric GID.  In this case the missing part (GID or\n+\tname) will be inferred from the current host's group database.\n+\n --remote=<repo>::\n \tInstead of making a tar archive from the local repository,\n \tretrieve a tar archive from a remote repository. Note that the\ndiff --git a/archive-tar.c b/archive-tar.c\nindex c6ed96ee7..ca6471870 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -204,10 +204,10 @@ static void prepare_header(struct archiver_args *args,\n \txsnprintf(header->size, sizeof(header->size), \"%011lo\", S_ISREG(mode) ? size : 0);\n \txsnprintf(header->mtime, sizeof(header->mtime), \"%011lo\", (unsigned long) args->time);\n \n-\txsnprintf(header->uid, sizeof(header->uid), \"%07o\", 0);\n-\txsnprintf(header->gid, sizeof(header->gid), \"%07o\", 0);\n-\tstrlcpy(header->uname, \"root\", sizeof(header->uname));\n-\tstrlcpy(header->gname, \"root\", sizeof(header->gname));\n+\txsnprintf(header->uid, sizeof(header->uid), \"%07lo\", args->uid);\n+\txsnprintf(header->gid, sizeof(header->gid), \"%07lo\", args->gid);\n+\tstrlcpy(header->uname, args->uname, sizeof(header->uname));\n+\tstrlcpy(header->gname, args->gname, sizeof(header->gname));\n \txsnprintf(header->devmajor, sizeof(header->devmajor), \"%07o\", 0);\n \txsnprintf(header->devminor, sizeof(header->devminor), \"%07o\", 0);\n \ndiff --git a/archive.c b/archive.c\nindex 0b7b62af0..aa4b16b75 100644\n--- a/archive.c\n+++ b/archive.c\n@@ -8,6 +8,7 @@\n #include \"parse-options.h\"\n #include \"unpack-trees.h\"\n #include \"dir.h\"\n+#include \"tar.h\"\n \n static char const * const archive_usage[] = {\n \tN_(\"git archive [<options>] <tree-ish> [<path>...]\"),\n@@ -417,6 +418,223 @@ static void parse_treeish_arg(const char **argv,\n \t{ OPTION_SET_INT, (s), NULL, (v), NULL, \"\", \\\n \t  PARSE_OPT_NOARG | PARSE_OPT_NONEG | PARSE_OPT_HIDDEN, NULL, (p) }\n \n+/*\n+ * GNU tar --owner, --group options reject hexdigit, signed int values.\n+ * strtol(), atoi() are too permissive to simulate the behaviour.\n+ */\n+#define STR_IS_DIGIT_OK 0\n+#define STR_IS_NOT_DIGIT -1\n+#define STR_IS_DIGIT_TOO_LARGE -2\n+\n+static int try_as_simple_digit(const char *s, unsigned long *dst)\n+{\n+\tunsigned long ul;\n+\tchar *endptr;\n+\n+\tif (strlen(s) != strspn(s, \"0123456789\"))\n+\t\treturn STR_IS_NOT_DIGIT;\n+\n+\terrno = 0;\n+\tul = strtoul(s, &endptr, 10);\n+\n+\t/* catch ERANGE */\n+\tif (errno) {\n+\t\terrno = 0;\n+\t\treturn STR_IS_DIGIT_TOO_LARGE;\n+\t}\n+\n+#if ULONG_MAX > 0xFFFFFFFFUL\n+\t/*\n+\t * --owner, --group rejects uid/gid greater than 32-bit\n+\t * limits, even on 64-bit platforms.\n+\t */\n+\tif (ul > 0xFFFFFFFFUL)\n+\t\treturn STR_IS_DIGIT_TOO_LARGE;\n+#endif\n+\n+\tif (dst)\n+\t\t*dst = ul;\n+\treturn STR_IS_DIGIT_OK;\n+}\n+\n+static const char *skip_leading_colon(const char *s)\n+{\n+\tconst char *col_pos;\n+\n+\tcol_pos = strchr(s, ':');\n+\tif (!col_pos)\n+\t\treturn s;\n+\n+\treturn (col_pos + 1);\n+}\n+\n+#define STR_IS_NAME_COLON_DIGIT 0\n+#define STR_HAS_NO_COLON -1\n+#define STR_HAS_DIGIT_BROKEN -2\n+#define STR_HAS_DIGIT_TOO_LARGE -3\n+\n+static int try_as_name_colon_digit(const char *s, const char **dst_s,\n+\t\tunsigned long *dst_ul)\n+{\n+\tint r;\n+\tconst char *s2;\n+\n+\ts2 = skip_leading_colon(s);\n+\tif (s2 == s)\n+\t\treturn STR_HAS_NO_COLON;\n+\n+\tr = try_as_simple_digit(s2, dst_ul);\n+\tswitch (r) {\n+\tcase STR_IS_DIGIT_OK:\n+\t\t*dst_s = xstrndup(s, s2 - s - 1);\n+\t\treturn STR_IS_NAME_COLON_DIGIT;\n+\tcase STR_IS_DIGIT_TOO_LARGE:\n+\t\treturn STR_HAS_DIGIT_TOO_LARGE;\n+\tdefault:\n+\t\treturn STR_HAS_DIGIT_BROKEN;\n+\t}\n+}\n+\n+#define NAME_ID_BOTH_GIVEN 0\n+#define NAME_ID_ID_GUESSED 1\n+#define NAME_ID_ID_UNTOUCHED 2\n+#define NAME_ID_NAME_GUESSED 3\n+#define NAME_ID_NAME_EMPTY 4\n+#define NAME_ID_ERR_ID_TOO_LARGE -126\n+#define NAME_ID_ERR_SYNTAX -127\n+#define NAME_ID_ERR_PARAMS -128\n+\n+static int set_args_uname_uid(struct archiver_args *args,\n+\t\tconst char *tar_owner)\n+{\n+\tint r;\n+\tstruct passwd *pw = NULL;\n+\n+\tif (!args || !tar_owner)\n+\t\treturn NAME_ID_ERR_PARAMS;\n+\n+\tr = try_as_name_colon_digit(tar_owner, &(args->uname),\n+\t\t\t\t    &(args->uid));\n+\tswitch (r) {\n+\tcase STR_IS_NAME_COLON_DIGIT:\n+\t\treturn NAME_ID_BOTH_GIVEN;\n+\tcase STR_HAS_DIGIT_TOO_LARGE:\n+\t\treturn NAME_ID_ERR_ID_TOO_LARGE;\n+\tcase STR_HAS_DIGIT_BROKEN:\n+\t\treturn NAME_ID_ERR_SYNTAX;\n+\t}\n+\n+\t/* the operand is known to be single token */\n+\n+\tr = try_as_simple_digit(tar_owner, &(args->uid));\n+\tswitch (r) {\n+\tcase STR_IS_DIGIT_TOO_LARGE:\n+\t\treturn NAME_ID_ERR_ID_TOO_LARGE;\n+\tcase STR_IS_DIGIT_OK:\n+\t\tpw = getpwuid(args->uid);\n+\t\tif (!pw) {\n+\t\t\targs->uname = xstrdup(\"\");\n+\t\t\treturn NAME_ID_NAME_EMPTY;\n+\t\t}\n+\t\targs->uname = xstrdup(pw->pw_name);\n+\t\treturn NAME_ID_NAME_GUESSED;\n+\t}\n+\n+\t/* the operand is known to be non-digit */\n+\n+\targs->uname = xstrdup(tar_owner);\n+\tpw = getpwnam(tar_owner);\n+\tif (!pw)\n+\t\treturn NAME_ID_ID_UNTOUCHED;\n+\targs->uid = pw->pw_uid;\n+\treturn NAME_ID_ID_GUESSED;\n+}\n+\n+static int set_args_gname_gid(struct archiver_args *args,\n+\t\tconst char *tar_group)\n+{\n+\tint r;\n+\tstruct group *gr = NULL;\n+\n+\tif (!args || !tar_group)\n+\t\treturn NAME_ID_ERR_PARAMS;\n+\n+\tr = try_as_name_colon_digit(tar_group, &(args->gname),\n+\t\t\t\t    &(args->gid));\n+\tswitch (r) {\n+\tcase STR_IS_NAME_COLON_DIGIT:\n+\t\treturn NAME_ID_BOTH_GIVEN;\n+\tcase STR_HAS_DIGIT_TOO_LARGE:\n+\t\treturn NAME_ID_ERR_ID_TOO_LARGE;\n+\tcase STR_HAS_DIGIT_BROKEN:\n+\t\treturn NAME_ID_ERR_SYNTAX;\n+\t}\n+\n+\t/* the operand is known to be single token */\n+\n+\tr = try_as_simple_digit(tar_group, &(args->gid));\n+\tswitch (r) {\n+\tcase STR_IS_DIGIT_TOO_LARGE:\n+\t\treturn NAME_ID_ERR_ID_TOO_LARGE;\n+\tcase STR_IS_DIGIT_OK:\n+\t\tgr = getgrgid(args->gid);\n+\t\tif (!gr) {\n+\t\t\targs->gname = xstrdup(\"\");\n+\t\t\treturn NAME_ID_NAME_EMPTY;\n+\t\t}\n+\t\targs->gname = xstrdup(gr->gr_name);\n+\t\treturn NAME_ID_NAME_GUESSED;\n+\t}\n+\n+\t/* the operand is known to be non-digit */\n+\n+\targs->gname = xstrdup(tar_group);\n+\tgr = getgrnam(tar_group);\n+\tif (!gr)\n+\t\treturn NAME_ID_ID_UNTOUCHED;\n+\targs->gid = gr->gr_gid;\n+\treturn NAME_ID_ID_GUESSED;\n+}\n+\n+static void set_args_tar_owner_group(struct archiver_args *args,\n+\t\tconst char *tar_owner, const char *tar_group)\n+{\n+\tint r;\n+\n+\t/* initialize by default values */\n+\targs->uname = xstrdup(\"root\");\n+\targs->gname = xstrdup(\"root\");\n+\targs->uid = 0;\n+\targs->gid = 0;\n+\n+\t/*\n+\t * GNU tar --format=ustar checks if uid is in 0..209751.\n+\t * Too long digit string could not be dealt as numeric,\n+\t * it is rejected as a syntax error before range check.\n+\t */\n+\tr = set_args_uname_uid(args, tar_owner);\n+\tswitch (r) {\n+\tcase NAME_ID_ERR_ID_TOO_LARGE:\n+\tcase NAME_ID_ERR_SYNTAX:\n+\t\tdie(\"'%s': Invalid owner ID\",\n+\t\t    skip_leading_colon(tar_owner));\n+\t}\n+\tif (args->uid > MAX_ID_IN_TAR_US)\n+\t\tdie(\"value %ld out of uid_t range 0..%ld\", args->uid,\n+\t\t     MAX_ID_IN_TAR_US);\n+\n+\tr = set_args_gname_gid(args, tar_group);\n+\tswitch (r) {\n+\tcase NAME_ID_ERR_ID_TOO_LARGE:\n+\tcase NAME_ID_ERR_SYNTAX:\n+\t\tdie(\"'%s': Invalid group ID\",\n+\t\t    skip_leading_colon(tar_group));\n+\t}\n+\tif (args->gid > MAX_ID_IN_TAR_US)\n+\t\tdie(\"value %ld out of gid_t range 0..%ld\", args->gid,\n+\t\t    MAX_ID_IN_TAR_US);\n+}\n+\n static int parse_archive_args(int argc, const char **argv,\n \t\tconst struct archiver **ar, struct archiver_args *args,\n \t\tconst char *name_hint, int is_remote)\n@@ -431,6 +649,8 @@ static int parse_archive_args(int argc, const char **argv,\n \tint i;\n \tint list = 0;\n \tint worktree_attributes = 0;\n+\tconst char *tar_owner = NULL;\n+\tconst char *tar_group = NULL;\n \tstruct option opts[] = {\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_STRING(0, \"format\", &format, N_(\"fmt\"), N_(\"archive format\")),\n@@ -459,6 +679,8 @@ static int parse_archive_args(int argc, const char **argv,\n \t\t\tN_(\"retrieve the archive from remote repository <repo>\")),\n \t\tOPT_STRING(0, \"exec\", &exec, N_(\"command\"),\n \t\t\tN_(\"path to the remote git-upload-archive command\")),\n+\t\tOPT_STRING(0, \"owner\", &tar_owner, N_(\"owner\"), N_(\"<name[:uid]> in tar\")),\n+\t\tOPT_STRING(0, \"group\", &tar_group, N_(\"group\"), N_(\"<name[:gid]> in tar\")),\n \t\tOPT_END()\n \t};\n \n@@ -507,6 +729,8 @@ static int parse_archive_args(int argc, const char **argv,\n \targs->baselen = strlen(base);\n \targs->worktree_attributes = worktree_attributes;\n \n+\tset_args_tar_owner_group(args, tar_owner, tar_group);\n+\n \treturn argc;\n }\n \ndiff --git a/archive.h b/archive.h\nindex 62d1d82c1..b2cfb1e4e 100644\n--- a/archive.h\n+++ b/archive.h\n@@ -15,6 +15,10 @@ struct archiver_args {\n \tunsigned int worktree_attributes : 1;\n \tunsigned int convert : 1;\n \tint compression_level;\n+\tunsigned long uid;\n+\tunsigned long gid;\n+\tconst char *uname;\n+\tconst char *gname;\n };\n \n #define ARCHIVER_WANT_COMPRESSION_LEVELS 1\ndiff --git a/t/t5005-archive-uid-gid.sh b/t/t5005-archive-uid-gid.sh\nnew file mode 100755\nindex 000000000..c5e08d890\n--- /dev/null\n+++ b/t/t5005-archive-uid-gid.sh\n@@ -0,0 +1,140 @@\n+#!/bin/sh\n+\n+test_description='test --owner --group options for git-archive'\n+. ./test-lib.sh\n+\n+test_expect_success 'create commit with a few empty files' '\n+\tgit init . 1>/dev/null 2>/dev/null &&\n+\ttouch uid-gid-test.001 &&\n+\tmkdir uid-gid-test.002 &&\n+\tmkdir uid-gid-test.002/uid-gid-test.003 &&\n+\tgit add uid-gid-test.001 &&\n+\tgit add uid-gid-test.002 &&\n+\tgit add uid-gid-test.002/uid-gid-test.003 &&\n+\tgit commit -m \"uid-gid-test\" 2>/dev/null 1>/dev/null\n+'\n+\n+check_uid_gid_uname_gname_in_tar() {\n+\t# $1 tar pathname\n+\t# $2 uid (digit in string)\n+\t# $3 gid (digit in string)\n+\t# $4 uname (string)\n+\t# $5 gname (string)\n+\tuid=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=uid --fail-if-multi $1`\n+\tif test $? != 0 -o x\"${uid}\" != \"x\"$2\n+\tthen\n+\t\techo \"(some) uid differs from the specified value\"\n+\t\treturn $?\n+\tfi\n+\n+\tgid=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=gid --fail-if-multi $1`\n+\tif test $? != 0 -o x\"${gid}\" != \"x\"$3\n+\tthen\n+\t\techo \"(some) gid differs from the specified value\"\n+\t\treturn $?\n+\tfi\n+\n+\tuname=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=uname --fail-if-multi $1`\n+\tif test $? != 0 -o x\"${uname}\" != \"x\"$4\n+\tthen\n+\t\techo \"(some) uname differs from the specified value\"\n+\t\treturn $?\n+\tfi\n+\n+\tgname=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=gname --fail-if-multi $1`\n+\tif test $? != 0 -o x\"${gname}\" != \"x\"$5\n+\tthen\n+\t\techo \"(some) gname differs from the specified value\"\n+\t\treturn $?\n+\tfi\n+\n+\treturn 0\n+}\n+\n+test_expect_success 'test a case with explicitly specified name/id, owner=nobody:1234 group=nogroup:5678' '\n+\tgit archive --format=tar --owner nobody:1234 --group nogroup:5678 HEAD > uid-gid-test1.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test1.tar 1234 5678 nobody nogroup &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with only string is given, owner=(current my name) group=(current my group)' '\n+\tmy_uid=`id -u` &&\n+\tmy_gid=`id -g` &&\n+\tmy_uname=`id -u -n` &&\n+\tmy_gname=`id -g -n` &&\n+\tgit archive --format=tar --owner ${my_uname} --group ${my_gname} HEAD > uid-gid-test2.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test2.tar ${my_uid} ${my_gid} ${my_uname} ${my_gname} &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with only number is given, owner=(current my uid) group=(current my gid)' '\n+\tmy_uid=`id -u` &&\n+\tmy_gid=`id -g` &&\n+\tmy_uname=`id -u -n` &&\n+\tmy_gname=`id -g -n` &&\n+\tgit archive --format=tar --owner ${my_uid} --group ${my_gid} HEAD > uid-gid-test3.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test3.tar ${my_uid} ${my_gid} ${my_uname} ${my_gname} &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with only uid is given, owner=(current my uid)' '\n+\tmy_uid=`id -u` &&\n+\tmy_gid=`id -g` &&\n+\tmy_uname=`id -u -n` &&\n+\tmy_gname=`id -g -n` &&\n+\tgit archive --format=tar --owner ${my_uid} HEAD > uid-gid-test4.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test4.tar ${my_uid} ${my_gid} ${my_uname} ${my_gname} &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with no owner/group are given' '\n+\tgit archive --format=tar HEAD > uid-gid-test5.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test5.tar 0 0 root root &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with max uid for ustar' '\n+\tgit archive --format=tar --owner nobody:209751 --group nogroup:1234 HEAD > uid-gid-test6.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test6.tar 209751 1234 nobody nogroup &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with max gid for ustar' '\n+\tgit archive --format=tar --group nogroup:209751 --owner nobody:1234 HEAD > uid-gid-test7.tar &&\n+\tcheck_uid_gid_uname_gname_in_tar uid-gid-test7.tar 1234 209751 nobody nogroup &&\n+\treturn $?\n+'\n+\n+test_expect_success 'test a case with uid greater than 32-bit (must fail)' '\n+\ttest_must_fail git archive --format=tar --owner 4294967296 --group 1234 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with gid greater than 32-bit (must fail)' '\n+\ttest_must_fail git archive --format=tar --group 4294967296 --owner 1234 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with uid greater than ustar limit (must fail)' '\n+\ttest_must_fail git archive --format=tar --owner 2097152 --group 1234 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with gid greater than ustar limit (must fail)' '\n+\ttest_must_fail git archive --format=tar --group 2097152 --owner 1234 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with valid username plus uid greater than 32-bit (must fail)' '\n+\ttest_must_fail git archive --format=tar --owner nobody:4294967296 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with valid groupname plus gid greater than 32-bit (must fail)' '\n+\ttest_must_fail git archive --format=tar --group nogroup:4294967296 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with valid username plus uid greater than ustar limit (must fail)' '\n+\ttest_must_fail git archive --format=tar --owner nobody:2097152 HEAD >/dev/null\n+'\n+\n+test_expect_success 'test a case with valid groupname plus gid greater than ustar limit (must fail)' '\n+\ttest_must_fail git archive --format=tar --group nogroup:2097152 HEAD >/dev/null\n+'\n+\n+test_done\ndiff --git a/t/t5005/parse-tar-file.py b/t/t5005/parse-tar-file.py\nnew file mode 100755\nindex 000000000..7d03b55d4\n--- /dev/null\n+++ b/t/t5005/parse-tar-file.py\n@@ -0,0 +1,60 @@\n+#!/usr/bin/env python\n+\n+import sys\n+import getopt\n+import tarfile\n+\n+optlist, args = getopt.getopt( sys.argv[1:], \"\", [\n+                               \"print=\", \"show=\",\n+                               \"uniq\",\n+                               \"fail-if-multi\",\n+                ] )\n+\n+infos_to_print = []\n+uniq = False\n+fail_if_multi = False\n+\n+for opt in optlist:\n+    if opt[0] == \"--print\":\n+        infos_to_print.append(opt[1])\n+    elif opt[0] == \"--show\":\n+        infos_to_print.append(opt[1])\n+    elif opt[0] == \"--uniq\":\n+        uniq = True\n+    elif opt[0] == \"--fail-if-multi\":\n+        uniq = True\n+        fail_if_multi = True\n+\n+if len(infos_to_print) == 0:\n+    infos_to_print = [\"uid\", \"gid\", \"uname\", \"gname\", \"name\"]\n+\n+if len(args) > 0:\n+    tar = tarfile.open( path=args[0], mode=\"r|\" )\n+else:\n+    tar = tarfile.open( mode=\"r|\", fileobj=sys.stdin )\n+\n+out_lines = []\n+for tarinfo in tar:\n+    infos = []\n+    for info_tag in infos_to_print:\n+        if info_tag == \"uid\":\n+            infos.append( str(tarinfo.uid) )\n+        elif info_tag == \"gid\":\n+            infos.append( str(tarinfo.gid) )\n+        elif info_tag == \"uname\" or info_tag == \"owner\":\n+            infos.append( tarinfo.uname )\n+        elif info_tag == \"gname\" or info_tag == \"group\":\n+            infos.append( tarinfo.gname )\n+        elif info_tag == \"name\" or info_tag == \"pathname\":\n+            infos.append( tarinfo.name )\n+        out_lines.append( \"\\t\".join(infos) )\n+tar.close()\n+\n+if uniq:\n+    out_lines = list(set(out_lines))\n+    if fail_if_multi and (len(out_lines) > 1):\n+        sys.stderr.write(\"*** not unique value, \" + str(len(out_lines)) + \" values found\\n\")\n+        sys.exit(len(out_lines))\n+\n+for line in out_lines:\n+    print line\ndiff --git a/tar.h b/tar.h\nindex 3467705e9..1e6b57470 100644\n--- a/tar.h\n+++ b/tar.h\n@@ -5,6 +5,8 @@\n #define TYPEFLAG_GLOBAL_HEADER\t'g'\n #define TYPEFLAG_EXT_HEADER\t'x'\n \n+#define MAX_ID_IN_TAR_US\t0x1FFFFFUL\n+\n struct ustar_header {\n \tchar name[100];\t\t/*   0 */\n \tchar mode[8];\t\t/* 100 */\n-- \n2.11.0\n\n\n"},{"id":"335592","messageId":"81b882a5-0c35-f3c4-78e2-d3e36290fec1@web.de","threadId":"47510","inReplyTo":"20171229140535.10746-1-mpsuzuki@hiroshima-u.ac.jp","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-01-01T18:29:52Z","receivedAt":"2018-01-01T18:33:17Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n> The ownership of files created by git-archive is always\n> root:root. Add --owner and --group options which work\n> like the GNU tar equivalent to allow overriding these\n> defaults.\n\nIn which situations do you use the new options?\n\n(The sender would need to know the names and/or IDs on the receiving\nend.  And the receiver would need to be root to set both IDs, or be a\ngroup member to set the group ID; I guess the latter is more common.)\n\n> Signed-off-by: suzuki toshiya <mpsuzuki@hiroshima-u.ac.jp>\n> ---\n>   Documentation/git-archive.txt |  13 +++\n>   archive-tar.c                 |   8 +-\n>   archive.c                     | 224 ++++++++++++++++++++++++++++++++++++++++++\n>   archive.h                     |   4 +\n>   t/t5005-archive-uid-gid.sh    | 140 ++++++++++++++++++++++++++\n>   t/t5005/parse-tar-file.py     |  60 +++++++++++\n>   tar.h                         |   2 +\n>   7 files changed, 447 insertions(+), 4 deletions(-)\n>   create mode 100755 t/t5005-archive-uid-gid.sh\n>   create mode 100755 t/t5005/parse-tar-file.py\n\nWould it make sense to support the new options for ZIP files as well?\n\n> diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\n> index cfa1e4ebe..0d156f6c1 100644\n> --- a/Documentation/git-archive.txt\n> +++ b/Documentation/git-archive.txt\n> @@ -11,6 +11,7 @@ SYNOPSIS\n>   [verse]\n>   'git archive' [--format=<fmt>] [--list] [--prefix=<prefix>/] [<extra>]\n>   \t      [-o <file> | --output=<file>] [--worktree-attributes]\n> +\t      [--owner [username[:uid]] [--group [groupname[:gid]]\n>   \t      [--remote=<repo> [--exec=<git-upload-archive>]] <tree-ish>\n>   \t      [<path>...]\n>   \n> @@ -63,6 +64,18 @@ OPTIONS\n>   \tThis can be any options that the archiver backend understands.\n>   \tSee next section.\n>   \n> +--owner=<name>[:<uid>]::\n> +\tForce <name> as owner and <uid> as uid for the files in the tar\n> +\tarchive.  If <uid> is not supplied, <name> can be either a user\n> +\tname or numeric UID.  In this case the missing part (UID or\n> +\tname) will be inferred from the current host's user database.\n> +\n> +--group=<name>[:<gid>]::\n> +\tForce <name> as group and <gid> as gid for the files in the tar\n> +\tarchive.  If <gid> is not supplied, <name> can be either a group\n> +\tname or numeric GID.  In this case the missing part (GID or\n> +\tname) will be inferred from the current host's group database.\n> +\n\nIIUC the default behavior is kept, i.e. without these options the\narchive entries appear to be owned by root:root.  I think it's a good\nidea to mention this here.\n\nbsdtar has --uname, --uid, --gname, and -gid, which seem simpler.  At\nleast you could use OPT_STRING and OPT_INTEGER with them (plus a range\ncheck).  And they should be easier to explain.\n\n> diff --git a/archive.c b/archive.c\n> index 0b7b62af0..aa4b16b75 100644\n> --- a/archive.c\n> +++ b/archive.c\n> @@ -8,6 +8,7 @@\n>   #include \"parse-options.h\"\n>   #include \"unpack-trees.h\"\n>   #include \"dir.h\"\n> +#include \"tar.h\"\n>   \n>   static char const * const archive_usage[] = {\n>   \tN_(\"git archive [<options>] <tree-ish> [<path>...]\"),\n> @@ -417,6 +418,223 @@ static void parse_treeish_arg(const char **argv,\n>   \t{ OPTION_SET_INT, (s), NULL, (v), NULL, \"\", \\\n>   \t  PARSE_OPT_NOARG | PARSE_OPT_NONEG | PARSE_OPT_HIDDEN, NULL, (p) }\n>   \n> +/*\n> + * GNU tar --owner, --group options reject hexdigit, signed int values.\n> + * strtol(), atoi() are too permissive to simulate the behaviour.\n> + */\n> +#define STR_IS_DIGIT_OK 0\n> +#define STR_IS_NOT_DIGIT -1\n> +#define STR_IS_DIGIT_TOO_LARGE -2\n> +\n> +static int try_as_simple_digit(const char *s, unsigned long *dst)\n> +{\n> +\tunsigned long ul;\n> +\tchar *endptr;\n> +\n> +\tif (strlen(s) != strspn(s, \"0123456789\"))\n> +\t\treturn STR_IS_NOT_DIGIT;\n> +\n> +\terrno = 0;\n> +\tul = strtoul(s, &endptr, 10);\n> +\n> +\t/* catch ERANGE */\n> +\tif (errno) {\n> +\t\terrno = 0;\n> +\t\treturn STR_IS_DIGIT_TOO_LARGE;\n> +\t}\n> +\n> +#if ULONG_MAX > 0xFFFFFFFFUL\n> +\t/*\n> +\t * --owner, --group rejects uid/gid greater than 32-bit\n> +\t * limits, even on 64-bit platforms.\n> +\t */\n> +\tif (ul > 0xFFFFFFFFUL)\n> +\t\treturn STR_IS_DIGIT_TOO_LARGE;\n> +#endif\n\nThe #if is not really necessary, is it?  Compilers should be able to\noptimize the conditional out on 32-bit platforms.\n\n> +static int set_args_uname_uid(struct archiver_args *args,\n> +\t\tconst char *tar_owner)\n> +{\n> +\tint r;\n> +\tstruct passwd *pw = NULL;\n> +\n> +\tif (!args || !tar_owner)\n> +\t\treturn NAME_ID_ERR_PARAMS;\n> +\n> +\tr = try_as_name_colon_digit(tar_owner, &(args->uname),\n> +\t\t\t\t    &(args->uid));\n> +\tswitch (r) {\n> +\tcase STR_IS_NAME_COLON_DIGIT:\n> +\t\treturn NAME_ID_BOTH_GIVEN;\n> +\tcase STR_HAS_DIGIT_TOO_LARGE:\n> +\t\treturn NAME_ID_ERR_ID_TOO_LARGE;\n> +\tcase STR_HAS_DIGIT_BROKEN:\n> +\t\treturn NAME_ID_ERR_SYNTAX;\n> +\t}\n> +\n> +\t/* the operand is known to be single token */\n> +\n> +\tr = try_as_simple_digit(tar_owner, &(args->uid));\n> +\tswitch (r) {\n> +\tcase STR_IS_DIGIT_TOO_LARGE:\n> +\t\treturn NAME_ID_ERR_ID_TOO_LARGE;\n> +\tcase STR_IS_DIGIT_OK:\n> +\t\tpw = getpwuid(args->uid);\n> +\t\tif (!pw) {\n> +\t\t\targs->uname = xstrdup(\"\");\n> +\t\t\treturn NAME_ID_NAME_EMPTY;\n> +\t\t}\n> +\t\targs->uname = xstrdup(pw->pw_name);\n> +\t\treturn NAME_ID_NAME_GUESSED;\n> +\t}\n> +\n> +\t/* the operand is known to be non-digit */\n> +\n> +\targs->uname = xstrdup(tar_owner);\n> +\tpw = getpwnam(tar_owner);\n> +\tif (!pw)\n> +\t\treturn NAME_ID_ID_UNTOUCHED;\n> +\targs->uid = pw->pw_uid;\n> +\treturn NAME_ID_ID_GUESSED;\n> +}\n\nHow well does this work on Windows?  In daemon.c we avoid calling\ngetpwnam(3), getgrnam(3) etc. if NO_POSIX_GOODIES is not defined.\n\n> diff --git a/t/t5005-archive-uid-gid.sh b/t/t5005-archive-uid-gid.sh\n> new file mode 100755\n> index 000000000..c5e08d890\n> --- /dev/null\n> +++ b/t/t5005-archive-uid-gid.sh\n> @@ -0,0 +1,140 @@\n> +#!/bin/sh\n> +\n> +test_description='test --owner --group options for git-archive'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'create commit with a few empty files' '\n> +\tgit init . 1>/dev/null 2>/dev/null &&\n> +\ttouch uid-gid-test.001 &&\n> +\tmkdir uid-gid-test.002 &&\n> +\tmkdir uid-gid-test.002/uid-gid-test.003 &&\n> +\tgit add uid-gid-test.001 &&\n> +\tgit add uid-gid-test.002 &&\n> +\tgit add uid-gid-test.002/uid-gid-test.003 &&\n> +\tgit commit -m \"uid-gid-test\" 2>/dev/null 1>/dev/null\n> +'\n> +\n> +check_uid_gid_uname_gname_in_tar() {\n> +\t# $1 tar pathname\n> +\t# $2 uid (digit in string)\n> +\t# $3 gid (digit in string)\n> +\t# $4 uname (string)\n> +\t# $5 gname (string)\n> +\tuid=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=uid --fail-if-multi $1`\n> +\tif test $? != 0 -o x\"${uid}\" != \"x\"$2\n> +\tthen\n> +\t\techo \"(some) uid differs from the specified value\"\n> +\t\treturn $?\n> +\tfi\n> +\n> +\tgid=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=gid --fail-if-multi $1`\n> +\tif test $? != 0 -o x\"${gid}\" != \"x\"$3\n> +\tthen\n> +\t\techo \"(some) gid differs from the specified value\"\n> +\t\treturn $?\n> +\tfi\n> +\n> +\tuname=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=uname --fail-if-multi $1`\n> +\tif test $? != 0 -o x\"${uname}\" != \"x\"$4\n> +\tthen\n> +\t\techo \"(some) uname differs from the specified value\"\n> +\t\treturn $?\n> +\tfi\n> +\n> +\tgname=`python \"$TEST_DIRECTORY\"/t5005/parse-tar-file.py --print=gname --fail-if-multi $1`\n> +\tif test $? != 0 -o x\"${gname}\" != \"x\"$5\n> +\tthen\n> +\t\techo \"(some) gname differs from the specified value\"\n> +\t\treturn $?\n> +\tfi\n> +\n> +\treturn 0\n> +}\n\nGNU tar and bsdtar show the names of owner and group with -t -v at\nleast, albeit in slightly different formats.  Can this help avoid\nparsing the archive on our own?\n\nBut getting a short program like zipdetails for tar would be nice as\nwell of course. :)\n\nRené\n"},{"id":"335605","messageId":"5a4ad314.C6wihAr5rEC8jRkB%perryh@pluto.rain.com","threadId":"47510","inReplyTo":"81b882a5-0c35-f3c4-78e2-d3e36290fec1@web.de","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"Perry Hutchison","fromEmail":"perryh@pluto.rain.com","sentAt":"2018-01-02T00:32:20Z","receivedAt":"2018-01-02T01:26:29Z","isPatch":true,"sender":{"key":"perryh@pluto.rain.com","avatar":null},"body":"Ren?? Scharfe <l.s.r@web.de> wrote:\n> Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n> > The ownership of files created by git-archive is always\n> > root:root. Add --owner and --group options which work\n> > like the GNU tar equivalent to allow overriding these\n> > defaults.\n> ... the receiver would need to be root to set both IDs, or be a\n> group member to set the group ID; I guess the latter is more common.\n\nIf the received files are owned by root:root as stated, I guess the\nreceiver must be running as root, no?\n"},{"id":"335607","messageId":"5A4B2DA5.907@hiroshima-u.ac.jp","threadId":"47510","inReplyTo":"df39f62558314cf6a9d9df3e23f31dd8@OS2PR01MB1147.jpnprd01.prod.outlook.com","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"suzuki toshiya","fromEmail":"mpsuzuki@hiroshima-u.ac.jp","sentAt":"2018-01-02T06:58:45Z","receivedAt":"2018-01-02T06:58:54Z","isPatch":true,"sender":{"key":"mpsuzuki@hiroshima-u.ac.jp","avatar":null},"body":"Dear René ,\n\nRené Scharfe wrote:\n> Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n>> The ownership of files created by git-archive is always\n>> root:root. Add --owner and --group options which work\n>> like the GNU tar equivalent to allow overriding these\n>> defaults.\n> \n> In which situations do you use the new options?\n> \n> (The sender would need to know the names and/or IDs on the receiving\n> end.  And the receiver would need to be root to set both IDs, or be a\n> group member to set the group ID; I guess the latter is more common.)\n\nThank you for asking the background.\n\nIn the case that additional contents are appended to the tar file\ngenerated by git-archive, the part by git-archive and the part\nappended by common tar would have different UID/GID, because common\ntar preserves the UID/GID of the original files.\n\nOf cource, both of GNU tar and bsdtar have the options to set\nUID/GID manually, but their syntax are different.\n\nIn the recent source package of poppler (poppler.freedesktop.org),\nthere are 2 sets of UID/GIDs are found:\nhttps://poppler.freedesktop.org/poppler-0.62.0.tar.xz\n\nI've discussed with the maintainers of poppler, and there was a\nsuggestion to propose a feature to git.\nhttps://lists.freedesktop.org/archives/poppler/2017-December/012739.html\n\nSo now I'm trying.\n\n> Would it make sense to support the new options for ZIP files as well?\n\nI was not aware of the availability of UID/GID in pkzip file format...\nOh, checking APPNOTE.TXT ( \nhttps://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT ),\nthere is a storage! (see 4.5.7-Unix Extra Field). But it seems\nthat current git-archive emits pkzip without the field.\n\nThe background why I propose the options for tar format was described\nin above. Similar things are hoped by pkzip users? If it's required,\nI will try.\n\n>> +--owner=<name>[:<uid>]::\n>> +\tForce <name> as owner and <uid> as uid for the files in the tar\n>> +\tarchive.  If <uid> is not supplied, <name> can be either a user\n>> +\tname or numeric UID.  In this case the missing part (UID or\n>> +\tname) will be inferred from the current host's user database.\n>> +\n>> +--group=<name>[:<gid>]::\n>> +\tForce <name> as group and <gid> as gid for the files in the tar\n>> +\tarchive.  If <gid> is not supplied, <name> can be either a group\n>> +\tname or numeric GID.  In this case the missing part (GID or\n>> +\tname) will be inferred from the current host's group database.\n>> +\n> \n> IIUC the default behavior is kept, i.e. without these options the\n> archive entries appear to be owned by root:root.  I think it's a good\n> idea to mention this here.\n\nIndeed. The default behaviour of git-archive without these options\n(root:root) would be different from that of (common) tar (preserving\nuid/gid of the files to be archived), it should be clarified.\n\n> bsdtar has --uname, --uid, --gname, and -gid, which seem simpler.  At\n> least you could use OPT_STRING and OPT_INTEGER with them (plus a range\n> check).  And they should be easier to explain.\n\nThank you very much for proposing good alternative. Indeed, such well-\nseparated options make the code simple & stable. However, according\nto the manual search systems of FreeBSD ( https://www.freebsd.org/cgi/man.cgi ),\nthe options for such functionalities are not always same.\n\nFreeBSD 8.2 and earlier: --uname, --gname, --uid, --gid are unavailable.\nit seems that using \"mtree\" was the preferred way to specify such).\n\nFreeBSD 8.3 and later: --uname, --gname, --uid, --gid are available.\nthe manual says follows:\n\n      --uid id\n\t     Use the provided user id number and ignore\tthe user name from the\n\t     archive.  On create, if --uname is\tnot also specified, the\tuser\n\t     name will be set to match the user\tid.\n\n      --uname name\n\t     Use the provided user name.  On extract, this overrides the user\n\t     name in the archive; if the provided user name does not exist on\n\t     the system, it will be ignored and\tthe user id (from the archive\n\t     or\tfrom the --uid option) will be used instead.  On create, this\n\t     sets the user name\tthat will be stored in the archive; the\tname\n\t     is\tnot verified against the system\tuser database.\n\nThus, to emulate (post 2012-) bsdtar perfectly, getpwnam(), getpwuid() etc\nwould be still needed to implement \"--uid\" X-(.\n\nTracking the history of bsdtar, maybe I should track the history of GNU\ntar. According to ChangeLog, even --owner --group are rather newer option\nsince 1.13.18 (released on 2000-10-29). The original syntax was like this.\n\n`--owner=USER'\n      Specifies that `tar' should use USER as the owner of members when\n      creating archives, instead of the user associated with the source\n      file.  USER is first decoded as a user symbolic name, but if this\n      interpretation fails, it has to be a decimal numeric user ID.\n\n      There is no value indicating a missing number, and `0' usually\n      means `root'.  Some people like to force `0' as the value to offer\n      in their distributions for the owner of files, because the `root'\n      user is anonymous anyway, so that might as well be the owner of\n      anonymous archives.\n\n      This option does not affect extraction from archives.\n\nOh, there is no colon separated syntax! According to ChangeLog, the\nintroduction of colon separated syntax was on 2011-08-13 and\nreleased as GNU tar-1.27 (2013-10-06).\n\n`--owner=USER'\n      Specifies that `tar' should use USER as the owner of members when\n      creating archives, instead of the user associated with the source\n      file.  USER can specify a symbolic name, or a numeric ID, or both\n      as NAME:ID.  *Note override::.\n\n      This option does not affect extraction from archives.\n\nComparing the original --owner and current --owner description, a\nstrange point is that the original description says \"USER is first\ndecoded as a user symbolic name, but if this interpretation fails,\nit has to be a decimal numeric user ID.\" What? It seems that\n\"checking whether the specified username is known by the host system\nand its numerical uid is resolvable - if unresolvable, try to\nparse as decimal value - if failed, take it as fatal error\". Here\nI quote the related part.\n\ntar-1.14/src/names.c\n\n     119 /* Given UNAME, set the corresponding UID and return 1, or else, return \n0.  */\n     120 int\n     121 uname_to_uid (char const *uname, uid_t *uidp)\n     122 {\n     123   struct passwd *passwd;\n     124\n     125   if (cached_no_such_uname\n     126       && strcmp (uname, cached_no_such_uname) == 0)\n     127     return 0;\n     128\n     129   if (!cached_uname\n     130       || uname[0] != cached_uname[0]\n     131       || strcmp (uname, cached_uname) != 0)\n     132     {\n     133       passwd = getpwnam (uname);\n     134       if (passwd)\n     135         {\n     136           cached_uid = passwd->pw_uid;\n     137           assign_string (&cached_uname, passwd->pw_name);\n     138         }\n     139       else\n     140         {\n     141           assign_string (&cached_no_such_uname, uname);\n     142           return 0;\n     143         }\n     144     }\n     145   *uidp = cached_uid;\n     146   return 1;\n     147 }   1087       case OWNER_OPTION:\n\ntar-1.14/src/tar.c\n    1088         if (! (strlen (optarg) < UNAME_FIELD_SIZE\n    1089                && uname_to_uid (optarg, &owner_option)))\n    1090           {\n    1091             uintmax_t u;\n    1092             if (xstrtoumax (optarg, 0, 10, &u, \"\") == LONGINT_OK\n    1093                 && u == (uid_t) u)\n    1094               owner_option = u;\n    1095             else\n    1096               FATAL_ERROR ((0, 0, \"%s: %s\", quotearg_colon (optarg),\n    1097                             _(\"Invalid owner\")));\n    1098           }\n    1099         break;\n\nIn summary, there are following types.\n\na) older GNU tar\n--owner must match with the host database, no option to set\nuname & uid separately.\n\nb) newer GNU tar\n--owner accepts unknown username and/or uid.\nif only one part is given and known by the host system,\nthe missing part is deduced by it.\nif only one part is given and unknown by the host system,\nthe missing part is unchanged from the file to be archived.\n\nc) newer bsd tar\n--uname/--uid accept unknown username and/or uid.\nusername is just used to override uname entry of the archive,\nbut uid is used to override both of uid and uname entries,\nif uid is known and username is not specified.\nIf uid is unknown, uid is overriden, but the username entry\nis unchanged from the file to be archived.\n\nwhich behaviour is to be simulated? I want to propose\nyet another one, similar to c) but incompatble.\n\nd) --uname, --uid, --gname, --gid check only the syntax\n(to kick the username starting with digit, non-digit\nuid, etc) and no check for known/unknown.\n\n>> +#if ULONG_MAX > 0xFFFFFFFFUL\n>> +\t/*\n>> +\t * --owner, --group rejects uid/gid greater than 32-bit\n>> +\t * limits, even on 64-bit platforms.\n>> +\t */\n>> +\tif (ul > 0xFFFFFFFFUL)\n>> +\t\treturn STR_IS_DIGIT_TOO_LARGE;\n>> +#endif\n> \n> The #if is not really necessary, is it?  Compilers should be able to\n> optimize the conditional out on 32-bit platforms.\n\nThanks for finding this, I'm glad to have a chance to ask a\nquestion; git is not needed to care for 16-bit platforms?\n\n>> +\t/* the operand is known to be non-digit */\n>> +\n>> +\targs->uname = xstrdup(tar_owner);\n>> +\tpw = getpwnam(tar_owner);\n> \n> How well does this work on Windows?  In daemon.c we avoid calling\n> getpwnam(3), getgrnam(3) etc. if NO_POSIX_GOODIES is not defined.\n\nOK, I can enclose them by ifdefs of NO_POSIX_GOODIES. But,\nmaybe the design the options would be discussed for first.\n\nBoth of latest GNU and BSD tar call getpwnam() or getpwuid(),\nbut designing as all of --uname --uid --gname --gid as \"only syntax\nis checked (non-digit UID/GID should be refused), but known/unknown\nis not checked\" would be the most portable.\n\n> GNU tar and bsdtar show the names of owner and group with -t -v at\n> least, albeit in slightly different formats.  Can this help avoid\n> parsing the archive on our own?\n\nYeah, writing yet another tar archive parser in C, to avoid the additional\ndependency to Python or newer Perl (Archive::Tar since perl-5.10), is\npainful, I feel (not only for me but also for the maintainers).\nIf tar command itself works well, it would be the best.\n\nBut, I'm not sure whether the format of \"tar tv\" output is stably\nstandardized. It's the reason why I wrote Python tool. If I execute\ngit-archive with sufficently long randomized username & uid in\nseveral times, it would be good test?\n\n> But getting a short program like zipdetails for tar would be nice as\n> well of course. :)\n\nI wrote something in C:\nhttps://github.com/mpsuzuki/git/blob/pullreq-20171227-c/t/helper/test-parse-tar-file.c\n\nbut if somebody wants the support of other tar variants,\nhe/she would have some headache :-)\n\nRegards,\nmpsuzuki\n\n"},{"id":"335636","messageId":"e45b654e-4fa5-2ad0-4a38-cf853ab2f0df@web.de","threadId":"47510","inReplyTo":"5A4B2DA5.907@hiroshima-u.ac.jp","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-01-02T21:36:27Z","receivedAt":"2018-01-02T21:36:42Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 02.01.2018 um 07:58 schrieb suzuki toshiya:\n> Dear René ,\n> \n> René Scharfe wrote:\n>> Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n>>> The ownership of files created by git-archive is always\n>>> root:root. Add --owner and --group options which work\n>>> like the GNU tar equivalent to allow overriding these\n>>> defaults.\n>>\n>> In which situations do you use the new options?\n>>\n>> (The sender would need to know the names and/or IDs on the receiving\n>> end.  And the receiver would need to be root to set both IDs, or be a\n>> group member to set the group ID; I guess the latter is more common.)\n> \n> Thank you for asking the background.\n> \n> In the case that additional contents are appended to the tar file\n> generated by git-archive, the part by git-archive and the part\n> appended by common tar would have different UID/GID, because common\n> tar preserves the UID/GID of the original files.\n> \n> Of cource, both of GNU tar and bsdtar have the options to set\n> UID/GID manually, but their syntax are different.\n> \n> In the recent source package of poppler (poppler.freedesktop.org),\n> there are 2 sets of UID/GIDs are found:\n> https://poppler.freedesktop.org/poppler-0.62.0.tar.xz\n> \n> I've discussed with the maintainers of poppler, and there was a\n> suggestion to propose a feature to git.\n> https://lists.freedesktop.org/archives/poppler/2017-December/012739.html\n\nSo this is in the context of generating release tarballs that contain\nuntracked files as well.  That's done in Git's own Makefile, too:\n\n  dist: git-archive$(X) configure\n          ./git-archive --format=tar \\\n                  --prefix=$(GIT_TARNAME)/ HEAD^{tree} > $(GIT_TARNAME).tar\n          @mkdir -p $(GIT_TARNAME)\n          @cp configure $(GIT_TARNAME)\n          @echo $(GIT_VERSION) > $(GIT_TARNAME)/version\n          @$(MAKE) -C git-gui TARDIR=../$(GIT_TARNAME)/git-gui dist-version\n          $(TAR) rf $(GIT_TARNAME).tar \\\n                  $(GIT_TARNAME)/configure \\\n                  $(GIT_TARNAME)/version \\\n                  $(GIT_TARNAME)/git-gui/version\n          @$(RM) -r $(GIT_TARNAME)\n          gzip -f -9 $(GIT_TARNAME).tar\n\nHaving files with different owners and groups is a non-issue when\nextracting with --no-same-owner, which is the default for regular users.\nI assume this covers most use cases in the wild.\n\nThe generated archive leaks the IDs of the user preparing the archive in\nthe appended entries for untracked files.  I think that's more of a\nconcern.  Publishing a valid non-root username on your build system may\ninvite attackers.\n\nChanging the build procedure to set owner and group to root as well as\nUID and GID to zero seems like a better idea.  This is complicated by\nthe inconsistent command line options for GNU tar and bsdtar, as you\nmentioned.\n\nSo how about making it possible to append untracked files using git\narchive?  This could simplify the dist target for Git as well.  It's\northogonal to adding the ability to explicitly specify owner and group,\nbut might suffice in most (all?) cases.\n\nNot sure what kind of file name transformation abilities would be\nneeded and how to package them nicely.  The --transform option of GNU\ntar with its sed replace expressions seems quite heavy for me.  With\npoppler it's only used to add the --prefix string; I'd expect that to\nbe done for all appended files anyway.\n\nPerhaps something like --append-file=<FILE> with no transformation\nfeature is already enough for most cases?\n\n>> Would it make sense to support the new options for ZIP files as well?\n> \n> I was not aware of the availability of UID/GID in pkzip file format...\n> Oh, checking APPNOTE.TXT ( https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT ),\n> there is a storage! (see 4.5.7-Unix Extra Field). But it seems\n> that current git-archive emits pkzip without the field.\n\nIndeed.  Git doesn't track owners and groups, so it doesn't make\nsense to emit that kind of information so far.  If git archive grows\noptions to specify such meta-information then it should be supported\nby all archive formats (or documented to be tar-specific).\n\nHowever, the UNIX Extra Field in ZIP archives only allows to store\nUID and GID (no names), which is useless unless the sender knows the\nID range of the receiver -- which is unlikely when distributing\nsoftware on the Internet.  And even then it won't work with Windows,\nwhich has long Security Identifiers (SIDs) instead.\n\nSo these are more advantages for letting git archive append untracked\nfiles: It's format-agnostic and more portable.\n\n\n[snipped interesting history of security-related tar options]\n\nBtw. I like how bsdtar has --insecure as a synonym for -p (preserve\nfile permissions when extracting).  It's a bit sad that this is still\nthe default for root, though.  OpenBSD cut that behavior out of their\ntar almost 20 years ago.  (An evil tar archive could be used to fill\nthe quota of unsuspecting users, or add setuid executables.)\n\n>>> +#if ULONG_MAX > 0xFFFFFFFFUL\n>>> +    /*\n>>> +     * --owner, --group rejects uid/gid greater than 32-bit\n>>> +     * limits, even on 64-bit platforms.\n>>> +     */\n>>> +    if (ul > 0xFFFFFFFFUL)\n>>> +        return STR_IS_DIGIT_TOO_LARGE;\n>>> +#endif\n>>\n>> The #if is not really necessary, is it?  Compilers should be able to\n>> optimize the conditional out on 32-bit platforms.\n> \n> Thanks for finding this, I'm glad to have a chance to ask a\n> question; git is not needed to care for 16-bit platforms?\n\nI'm not aware of attempts to compile Git on 16-bit systems, but I\ndoubt it would be much fun to use the result anyway.  It couldn't\nhandle files bigger than 64KB for a start.\n\n>> GNU tar and bsdtar show the names of owner and group with -t -v at\n>> least, albeit in slightly different formats.  Can this help avoid\n>> parsing the archive on our own?\n> \n> Yeah, writing yet another tar archive parser in C, to avoid the additional\n> dependency to Python or newer Perl (Archive::Tar since perl-5.10), is\n> painful, I feel (not only for me but also for the maintainers).\n> If tar command itself works well, it would be the best.\n> \n> But, I'm not sure whether the format of \"tar tv\" output is stably\n> standardized. It's the reason why I wrote Python tool. If I execute\n> git-archive with sufficently long randomized username & uid in\n> several times, it would be good test?\n\nThe append-untracked feature would avoid that issue as well.\n\nA unique username doesn't have to be long or random if you control all\nother strings that \"tar tv\" would show (in particular file names).  You\ncould e.g. use \"UserName\" and then simply grep for it.\n\nChecking that git archive doesn't add unwanted characters is harder if\nthe format isn't strictly known -- how to reliably recognize it sneaking\nin an extra space or slash?  Not sure if such a test is really needed,\nthough.\n\n>> But getting a short program like zipdetails for tar would be nice as\n>> well of course. :)\n> \n> I wrote something in C:\n> https://github.com/mpsuzuki/git/blob/pullreq-20171227-c/t/helper/test-parse-tar-file.c\n\nOK.  (But the append-untracked feature could be tested without such a\nhelper. :)\n\nJust some loose comments instead of a real review: You could get the\nline count down by including Git's tar.h and parse-options.h\n(Documentation/technical/api-parse-options.txt).\n\n> but if somebody wants the support of other tar variants,\n> he/she would have some headache :-)\n\nYeah, the history of tar and its various platform-specific formats\nfills volumes, no doubt.\n\nRené\n"},{"id":"335809","messageId":"24162f60-d6bf-8ab4-ee69-9834292fadae@web.de","threadId":"47510","inReplyTo":"5a4ad314.C6wihAr5rEC8jRkB%perryh@pluto.rain.com","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-01-04T00:43:25Z","receivedAt":"2018-01-04T00:43:40Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"[replying only to the list because emails to perryh@pluto.rain.com\n are rejected by my mail server with the following error message:\n \"Requested action not taken: mailbox unavailable\n  invalid DNS MX or A/AAAA resource record.\"]\n\nAm 02.01.2018 um 01:32 schrieb Perry Hutchison:\n> Ren?? Scharfe <l.s.r@web.de> wrote:\n>> Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n>>> The ownership of files created by git-archive is always\n>>> root:root. Add --owner and --group options which work\n>>> like the GNU tar equivalent to allow overriding these\n>>> defaults.\n>> ... the receiver would need to be root to set both IDs, or be a\n>> group member to set the group ID; I guess the latter is more common.\n> \n> If the received files are owned by root:root as stated, I guess the\n> receiver must be running as root, no?\n\nThat depends on what you mean with \"must\".  Users who want the files\nthey extract to be owned by root need root permissions on Unix and\nLinux.  If they are OK with owning the files themselves then regular\nuser accounts suffice.  I assume the latter is much more common.\n\nRené\n"},{"id":"335813","messageId":"5A4D836D.2010409@hiroshima-u.ac.jp","threadId":"47510","inReplyTo":"59a1fc058278463996ed68c970a5e08a@OS2PR01MB1147.jpnprd01.prod.outlook.com","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"suzuki toshiya","fromEmail":"mpsuzuki@hiroshima-u.ac.jp","sentAt":"2018-01-04T01:29:17Z","receivedAt":"2018-01-04T01:29:31Z","isPatch":true,"sender":{"key":"mpsuzuki@hiroshima-u.ac.jp","avatar":null},"body":"Dear René ,\n\nBy overlooking your response, I was writing a patch to add\nuid/gid into zip archive X-D (not finished yet)\nhttps://github.com/mpsuzuki/git/tree/add-zip-uid-gid\nHowever, I found that most unix platforms use infozip's\nextension to store uid/gid instead of pkzip's extension...\n\n> So this is in the context of generating release tarballs that contain\n> untracked files as well.  That's done in Git's own Makefile, too:\n\nOh, I should check other software's tarball :-)\n\n> The generated archive leaks the IDs of the user preparing the archive in\n> the appended entries for untracked files.  I think that's more of a\n> concern.  Publishing a valid non-root username on your build system may\n> invite attackers.\n\nHmm, I was not aware of such security concern about the\ntarball including the developers username.\n\n> So how about making it possible to append untracked files using git\n> archive?  This could simplify the dist target for Git as well.  It's\n> orthogonal to adding the ability to explicitly specify owner and group,\n> but might suffice in most (all?) cases.\n\nHmm, it could be reasonable to assume that --append-file\nwould serve more cases than --uid --gid options. There\nmight be many people who don't care multiple UID/GID in\nthe source tarball, but want to append some files to the\narchive generated by git-archive. I would take a look how\nto do that. A point I'm afraid is that some people may\nrequest to pass the file listing the pathnames instead of\ngiving many --append-file options (and a few people could\nwant to have a built-in default list specified by GNU\nconvention :-)).\n\nI want to hear other experts' comment; no need for me to\nwork \"--uid\" \"--gid\" anymore, and should I switch to\n\"--append-file\" options?\n\nRegards,\nmpsuzuki\n\nRené Scharfe wrote:\n> Am 02.01.2018 um 07:58 schrieb suzuki toshiya:\n>> Dear René ,\n>>\n>> René Scharfe wrote:\n>>> Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n>>>> The ownership of files created by git-archive is always\n>>>> root:root. Add --owner and --group options which work\n>>>> like the GNU tar equivalent to allow overriding these\n>>>> defaults.\n>>> In which situations do you use the new options?\n>>>\n>>> (The sender would need to know the names and/or IDs on the receiving\n>>> end.  And the receiver would need to be root to set both IDs, or be a\n>>> group member to set the group ID; I guess the latter is more common.)\n>> Thank you for asking the background.\n>>\n>> In the case that additional contents are appended to the tar file\n>> generated by git-archive, the part by git-archive and the part\n>> appended by common tar would have different UID/GID, because common\n>> tar preserves the UID/GID of the original files.\n>>\n>> Of cource, both of GNU tar and bsdtar have the options to set\n>> UID/GID manually, but their syntax are different.\n>>\n>> In the recent source package of poppler (poppler.freedesktop.org),\n>> there are 2 sets of UID/GIDs are found:\n>> https://poppler.freedesktop.org/poppler-0.62.0.tar.xz\n>>\n>> I've discussed with the maintainers of poppler, and there was a\n>> suggestion to propose a feature to git.\n>> https://lists.freedesktop.org/archives/poppler/2017-December/012739.html\n> \n> So this is in the context of generating release tarballs that contain\n> untracked files as well.  That's done in Git's own Makefile, too:\n> \n>   dist: git-archive$(X) configure\n>           ./git-archive --format=tar \\\n>                   --prefix=$(GIT_TARNAME)/ HEAD^{tree} > $(GIT_TARNAME).tar\n>           @mkdir -p $(GIT_TARNAME)\n>           @cp configure $(GIT_TARNAME)\n>           @echo $(GIT_VERSION) > $(GIT_TARNAME)/version\n>           @$(MAKE) -C git-gui TARDIR=../$(GIT_TARNAME)/git-gui dist-version\n>           $(TAR) rf $(GIT_TARNAME).tar \\\n>                   $(GIT_TARNAME)/configure \\\n>                   $(GIT_TARNAME)/version \\\n>                   $(GIT_TARNAME)/git-gui/version\n>           @$(RM) -r $(GIT_TARNAME)\n>           gzip -f -9 $(GIT_TARNAME).tar\n> \n> Having files with different owners and groups is a non-issue when\n> extracting with --no-same-owner, which is the default for regular users.\n> I assume this covers most use cases in the wild.\n> \n> The generated archive leaks the IDs of the user preparing the archive in\n> the appended entries for untracked files.  I think that's more of a\n> concern.  Publishing a valid non-root username on your build system may\n> invite attackers.\n> \n> Changing the build procedure to set owner and group to root as well as\n> UID and GID to zero seems like a better idea.  This is complicated by\n> the inconsistent command line options for GNU tar and bsdtar, as you\n> mentioned.\n> \n> So how about making it possible to append untracked files using git\n> archive?  This could simplify the dist target for Git as well.  It's\n> orthogonal to adding the ability to explicitly specify owner and group,\n> but might suffice in most (all?) cases.\n> \n> Not sure what kind of file name transformation abilities would be\n> needed and how to package them nicely.  The --transform option of GNU\n> tar with its sed replace expressions seems quite heavy for me.  With\n> poppler it's only used to add the --prefix string; I'd expect that to\n> be done for all appended files anyway.\n> \n> Perhaps something like --append-file=<FILE> with no transformation\n> feature is already enough for most cases?\n> \n>>> Would it make sense to support the new options for ZIP files as well?\n>> I was not aware of the availability of UID/GID in pkzip file format...\n>> Oh, checking APPNOTE.TXT ( https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT ),\n>> there is a storage! (see 4.5.7-Unix Extra Field). But it seems\n>> that current git-archive emits pkzip without the field.\n> \n> Indeed.  Git doesn't track owners and groups, so it doesn't make\n> sense to emit that kind of information so far.  If git archive grows\n> options to specify such meta-information then it should be supported\n> by all archive formats (or documented to be tar-specific).\n> \n> However, the UNIX Extra Field in ZIP archives only allows to store\n> UID and GID (no names), which is useless unless the sender knows the\n> ID range of the receiver -- which is unlikely when distributing\n> software on the Internet.  And even then it won't work with Windows,\n> which has long Security Identifiers (SIDs) instead.\n> \n> So these are more advantages for letting git archive append untracked\n> files: It's format-agnostic and more portable.\n> \n> \n> [snipped interesting history of security-related tar options]\n> \n> Btw. I like how bsdtar has --insecure as a synonym for -p (preserve\n> file permissions when extracting).  It's a bit sad that this is still\n> the default for root, though.  OpenBSD cut that behavior out of their\n> tar almost 20 years ago.  (An evil tar archive could be used to fill\n> the quota of unsuspecting users, or add setuid executables.)\n> \n>>>> +#if ULONG_MAX > 0xFFFFFFFFUL\n>>>> +    /*\n>>>> +     * --owner, --group rejects uid/gid greater than 32-bit\n>>>> +     * limits, even on 64-bit platforms.\n>>>> +     */\n>>>> +    if (ul > 0xFFFFFFFFUL)\n>>>> +        return STR_IS_DIGIT_TOO_LARGE;\n>>>> +#endif\n>>> The #if is not really necessary, is it?  Compilers should be able to\n>>> optimize the conditional out on 32-bit platforms.\n>> Thanks for finding this, I'm glad to have a chance to ask a\n>> question; git is not needed to care for 16-bit platforms?\n> \n> I'm not aware of attempts to compile Git on 16-bit systems, but I\n> doubt it would be much fun to use the result anyway.  It couldn't\n> handle files bigger than 64KB for a start.\n> \n>>> GNU tar and bsdtar show the names of owner and group with -t -v at\n>>> least, albeit in slightly different formats.  Can this help avoid\n>>> parsing the archive on our own?\n>> Yeah, writing yet another tar archive parser in C, to avoid the additional\n>> dependency to Python or newer Perl (Archive::Tar since perl-5.10), is\n>> painful, I feel (not only for me but also for the maintainers).\n>> If tar command itself works well, it would be the best.\n>>\n>> But, I'm not sure whether the format of \"tar tv\" output is stably\n>> standardized. It's the reason why I wrote Python tool. If I execute\n>> git-archive with sufficently long randomized username & uid in\n>> several times, it would be good test?\n> \n> The append-untracked feature would avoid that issue as well.\n> \n> A unique username doesn't have to be long or random if you control all\n> other strings that \"tar tv\" would show (in particular file names).  You\n> could e.g. use \"UserName\" and then simply grep for it.\n> \n> Checking that git archive doesn't add unwanted characters is harder if\n> the format isn't strictly known -- how to reliably recognize it sneaking\n> in an extra space or slash?  Not sure if such a test is really needed,\n> though.\n> \n>>> But getting a short program like zipdetails for tar would be nice as\n>>> well of course. :)\n>> I wrote something in C:\n>> https://github.com/mpsuzuki/git/blob/pullreq-20171227-c/t/helper/test-parse-tar-file.c\n> \n> OK.  (But the append-untracked feature could be tested without such a\n> helper. :)\n> \n> Just some loose comments instead of a real review: You could get the\n> line count down by including Git's tar.h and parse-options.h\n> (Documentation/technical/api-parse-options.txt).\n> \n>> but if somebody wants the support of other tar variants,\n>> he/she would have some headache :-)\n> \n> Yeah, the history of tar and its various platform-specific formats\n> fills volumes, no doubt.\n> \n> René\n> \n\n"},{"id":"335814","messageId":"5A4D9089.3050209@hiroshima-u.ac.jp","threadId":"47510","inReplyTo":"955dae095d504b00b3e1c8a956ba852a@OS2PR01MB1147.jpnprd01.prod.outlook.com","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"suzuki toshiya","fromEmail":"mpsuzuki@hiroshima-u.ac.jp","sentAt":"2018-01-04T02:25:13Z","receivedAt":"2018-01-04T02:25:22Z","isPatch":true,"sender":{"key":"mpsuzuki@hiroshima-u.ac.jp","avatar":null},"body":"Hi,\n\n> Hmm, it could be reasonable to assume that --append-file\n> would serve more cases than --uid --gid options. There\n> might be many people who don't care multiple UID/GID in\n> the source tarball, but want to append some files to the\n> archive generated by git-archive. I would take a look how\n> to do that. A point I'm afraid is that some people may\n> request to pass the file listing the pathnames instead of\n> giving many --append-file options (and a few people could\n> want to have a built-in default list specified by GNU\n> convention :-)).\n\nTaking a glance on parse-options.h, I could not find the\nexisting class collecting the operands as an array (or\nlinked list) from multiple \"--xxx=yyy\" options. Similar\nthings might be the collecting the pathnames to pathspec\nstructure. Should I write something with OPTION_CALLBACK?\n\nRegards,\nmpsuzuki\n\nsuzuki toshiya wrote:\n> Dear René ,\n> \n> By overlooking your response, I was writing a patch to add\n> uid/gid into zip archive X-D (not finished yet)\n> https://github.com/mpsuzuki/git/tree/add-zip-uid-gid\n> However, I found that most unix platforms use infozip's\n> extension to store uid/gid instead of pkzip's extension...\n> \n>> So this is in the context of generating release tarballs that contain\n>> untracked files as well.  That's done in Git's own Makefile, too:\n> \n> Oh, I should check other software's tarball :-)\n> \n>> The generated archive leaks the IDs of the user preparing the archive in\n>> the appended entries for untracked files.  I think that's more of a\n>> concern.  Publishing a valid non-root username on your build system may\n>> invite attackers.\n> \n> Hmm, I was not aware of such security concern about the\n> tarball including the developers username.\n> \n>> So how about making it possible to append untracked files using git\n>> archive?  This could simplify the dist target for Git as well.  It's\n>> orthogonal to adding the ability to explicitly specify owner and group,\n>> but might suffice in most (all?) cases.\n> \n> Hmm, it could be reasonable to assume that --append-file\n> would serve more cases than --uid --gid options. There\n> might be many people who don't care multiple UID/GID in\n> the source tarball, but want to append some files to the\n> archive generated by git-archive. I would take a look how\n> to do that. A point I'm afraid is that some people may\n> request to pass the file listing the pathnames instead of\n> giving many --append-file options (and a few people could\n> want to have a built-in default list specified by GNU\n> convention :-)).\n> \n> I want to hear other experts' comment; no need for me to\n> work \"--uid\" \"--gid\" anymore, and should I switch to\n> \"--append-file\" options?\n> \n> Regards,\n> mpsuzuki\n> \n> René Scharfe wrote:\n>> Am 02.01.2018 um 07:58 schrieb suzuki toshiya:\n>>> Dear René ,\n>>>\n>>> René Scharfe wrote:\n>>>> Am 29.12.2017 um 15:05 schrieb suzuki toshiya:\n>>>>> The ownership of files created by git-archive is always\n>>>>> root:root. Add --owner and --group options which work\n>>>>> like the GNU tar equivalent to allow overriding these\n>>>>> defaults.\n>>>> In which situations do you use the new options?\n>>>>\n>>>> (The sender would need to know the names and/or IDs on the receiving\n>>>> end.  And the receiver would need to be root to set both IDs, or be a\n>>>> group member to set the group ID; I guess the latter is more common.)\n>>> Thank you for asking the background.\n>>>\n>>> In the case that additional contents are appended to the tar file\n>>> generated by git-archive, the part by git-archive and the part\n>>> appended by common tar would have different UID/GID, because common\n>>> tar preserves the UID/GID of the original files.\n>>>\n>>> Of cource, both of GNU tar and bsdtar have the options to set\n>>> UID/GID manually, but their syntax are different.\n>>>\n>>> In the recent source package of poppler (poppler.freedesktop.org),\n>>> there are 2 sets of UID/GIDs are found:\n>>> https://poppler.freedesktop.org/poppler-0.62.0.tar.xz\n>>>\n>>> I've discussed with the maintainers of poppler, and there was a\n>>> suggestion to propose a feature to git.\n>>> https://lists.freedesktop.org/archives/poppler/2017-December/012739.html\n>> So this is in the context of generating release tarballs that contain\n>> untracked files as well.  That's done in Git's own Makefile, too:\n>>\n>>   dist: git-archive$(X) configure\n>>           ./git-archive --format=tar \\\n>>                   --prefix=$(GIT_TARNAME)/ HEAD^{tree} > $(GIT_TARNAME).tar\n>>           @mkdir -p $(GIT_TARNAME)\n>>           @cp configure $(GIT_TARNAME)\n>>           @echo $(GIT_VERSION) > $(GIT_TARNAME)/version\n>>           @$(MAKE) -C git-gui TARDIR=../$(GIT_TARNAME)/git-gui dist-version\n>>           $(TAR) rf $(GIT_TARNAME).tar \\\n>>                   $(GIT_TARNAME)/configure \\\n>>                   $(GIT_TARNAME)/version \\\n>>                   $(GIT_TARNAME)/git-gui/version\n>>           @$(RM) -r $(GIT_TARNAME)\n>>           gzip -f -9 $(GIT_TARNAME).tar\n>>\n>> Having files with different owners and groups is a non-issue when\n>> extracting with --no-same-owner, which is the default for regular users.\n>> I assume this covers most use cases in the wild.\n>>\n>> The generated archive leaks the IDs of the user preparing the archive in\n>> the appended entries for untracked files.  I think that's more of a\n>> concern.  Publishing a valid non-root username on your build system may\n>> invite attackers.\n>>\n>> Changing the build procedure to set owner and group to root as well as\n>> UID and GID to zero seems like a better idea.  This is complicated by\n>> the inconsistent command line options for GNU tar and bsdtar, as you\n>> mentioned.\n>>\n>> So how about making it possible to append untracked files using git\n>> archive?  This could simplify the dist target for Git as well.  It's\n>> orthogonal to adding the ability to explicitly specify owner and group,\n>> but might suffice in most (all?) cases.\n>>\n>> Not sure what kind of file name transformation abilities would be\n>> needed and how to package them nicely.  The --transform option of GNU\n>> tar with its sed replace expressions seems quite heavy for me.  With\n>> poppler it's only used to add the --prefix string; I'd expect that to\n>> be done for all appended files anyway.\n>>\n>> Perhaps something like --append-file=<FILE> with no transformation\n>> feature is already enough for most cases?\n>>\n>>>> Would it make sense to support the new options for ZIP files as well?\n>>> I was not aware of the availability of UID/GID in pkzip file format...\n>>> Oh, checking APPNOTE.TXT ( https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT ),\n>>> there is a storage! (see 4.5.7-Unix Extra Field). But it seems\n>>> that current git-archive emits pkzip without the field.\n>> Indeed.  Git doesn't track owners and groups, so it doesn't make\n>> sense to emit that kind of information so far.  If git archive grows\n>> options to specify such meta-information then it should be supported\n>> by all archive formats (or documented to be tar-specific).\n>>\n>> However, the UNIX Extra Field in ZIP archives only allows to store\n>> UID and GID (no names), which is useless unless the sender knows the\n>> ID range of the receiver -- which is unlikely when distributing\n>> software on the Internet.  And even then it won't work with Windows,\n>> which has long Security Identifiers (SIDs) instead.\n>>\n>> So these are more advantages for letting git archive append untracked\n>> files: It's format-agnostic and more portable.\n>>\n>>\n>> [snipped interesting history of security-related tar options]\n>>\n>> Btw. I like how bsdtar has --insecure as a synonym for -p (preserve\n>> file permissions when extracting).  It's a bit sad that this is still\n>> the default for root, though.  OpenBSD cut that behavior out of their\n>> tar almost 20 years ago.  (An evil tar archive could be used to fill\n>> the quota of unsuspecting users, or add setuid executables.)\n>>\n>>>>> +#if ULONG_MAX > 0xFFFFFFFFUL\n>>>>> +    /*\n>>>>> +     * --owner, --group rejects uid/gid greater than 32-bit\n>>>>> +     * limits, even on 64-bit platforms.\n>>>>> +     */\n>>>>> +    if (ul > 0xFFFFFFFFUL)\n>>>>> +        return STR_IS_DIGIT_TOO_LARGE;\n>>>>> +#endif\n>>>> The #if is not really necessary, is it?  Compilers should be able to\n>>>> optimize the conditional out on 32-bit platforms.\n>>> Thanks for finding this, I'm glad to have a chance to ask a\n>>> question; git is not needed to care for 16-bit platforms?\n>> I'm not aware of attempts to compile Git on 16-bit systems, but I\n>> doubt it would be much fun to use the result anyway.  It couldn't\n>> handle files bigger than 64KB for a start.\n>>\n>>>> GNU tar and bsdtar show the names of owner and group with -t -v at\n>>>> least, albeit in slightly different formats.  Can this help avoid\n>>>> parsing the archive on our own?\n>>> Yeah, writing yet another tar archive parser in C, to avoid the additional\n>>> dependency to Python or newer Perl (Archive::Tar since perl-5.10), is\n>>> painful, I feel (not only for me but also for the maintainers).\n>>> If tar command itself works well, it would be the best.\n>>>\n>>> But, I'm not sure whether the format of \"tar tv\" output is stably\n>>> standardized. It's the reason why I wrote Python tool. If I execute\n>>> git-archive with sufficently long randomized username & uid in\n>>> several times, it would be good test?\n>> The append-untracked feature would avoid that issue as well.\n>>\n>> A unique username doesn't have to be long or random if you control all\n>> other strings that \"tar tv\" would show (in particular file names).  You\n>> could e.g. use \"UserName\" and then simply grep for it.\n>>\n>> Checking that git archive doesn't add unwanted characters is harder if\n>> the format isn't strictly known -- how to reliably recognize it sneaking\n>> in an extra space or slash?  Not sure if such a test is really needed,\n>> though.\n>>\n>>>> But getting a short program like zipdetails for tar would be nice as\n>>>> well of course. :)\n>>> I wrote something in C:\n>>> https://github.com/mpsuzuki/git/blob/pullreq-20171227-c/t/helper/test-parse-tar-file.c\n>> OK.  (But the append-untracked feature could be tested without such a\n>> helper. :)\n>>\n>> Just some loose comments instead of a real review: You could get the\n>> line count down by including Git's tar.h and parse-options.h\n>> (Documentation/technical/api-parse-options.txt).\n>>\n>>> but if somebody wants the support of other tar variants,\n>>> he/she would have some headache :-)\n>> Yeah, the history of tar and its various platform-specific formats\n>> fills volumes, no doubt.\n>>\n>> René\n>>\n> \n> \n\n"},{"id":"335824","messageId":"f7654cd9-2cd0-0775-3b10-8e3dc1a66dae@web.de","threadId":"47510","inReplyTo":"5A4D9089.3050209@hiroshima-u.ac.jp","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-01-04T16:59:50Z","receivedAt":"2018-01-04T17:00:06Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.01.2018 um 03:25 schrieb suzuki toshiya:\n> Taking a glance on parse-options.h, I could not find the\n> existing class collecting the operands as an array (or\n> linked list) from multiple \"--xxx=yyy\" options. Similar\n> things might be the collecting the pathnames to pathspec\n> structure. Should I write something with OPTION_CALLBACK?\n\nThere is OPT_STRING_LIST; Documentation/technical/api-parse-options.txt\nsays:\n\n  `OPT_STRING_LIST(short, long, &struct string_list, arg_str, description)`::\n      Introduce an option with string argument.\n      The string argument is stored as an element in `string_list`.\n      Use of `--no-option` will clear the list of preceding values.\n\nI don't know if it's a good idea, but perhaps we don't even need a new\noption.  We could change how pathspecs of untracked files are handled:\nInstead of aborting we could include them in the archive.  (Sounds like\nthe simplest possible interface, but may have practical problems.)\n\nRené\n"},{"id":"335831","messageId":"xmqqmv1tfpqh.fsf@gitster.mtv.corp.google.com","threadId":"47510","inReplyTo":"f7654cd9-2cd0-0775-3b10-8e3dc1a66dae@web.de","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-04T18:22:46Z","receivedAt":"2018-01-04T18:22:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> I don't know if it's a good idea, but perhaps we don't even need a new\n> option.  We could change how pathspecs of untracked files are handled:\n> Instead of aborting we could include them in the archive.  (Sounds like\n> the simplest possible interface, but may have practical problems.)\n\nOne practical problem is that users who do this\n\n    $ git archive HEAD Documentation/ | tar tf -\n\nwould be expecting (at least) two different things, depending on the\nsituation they are in.\n\nSo at least you'd need an \"--include-untracked\" option, I guess.\n"},{"id":"335904","messageId":"5A4EFDAD.3050105@hiroshima-u.ac.jp","threadId":"47510","inReplyTo":"2112f9c245d64f4e89361df7e9de9374@OS2PR01MB1147.jpnprd01.prod.outlook.com","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"suzuki toshiya","fromEmail":"mpsuzuki@hiroshima-u.ac.jp","sentAt":"2018-01-05T04:23:09Z","receivedAt":"2018-01-05T04:23:18Z","isPatch":true,"sender":{"key":"mpsuzuki@hiroshima-u.ac.jp","avatar":null},"body":"Dear Junio,\n\nCould you tell me your thought about the way for me to go?\nDo you agree with his suggestion; \"--uid etc is not the right\nsolution, --include-untracked is better and generic\" ? Or,\nshould I work \"--uid etc\" further?\n\nRegards,\nmpsuzuki\n\nJunio C Hamano wrote:\n> René Scharfe <l.s.r@web.de> writes:\n> \n>> I don't know if it's a good idea, but perhaps we don't even need a new\n>> option.  We could change how pathspecs of untracked files are handled:\n>> Instead of aborting we could include them in the archive.  (Sounds like\n>> the simplest possible interface, but may have practical problems.)\n> \n> One practical problem is that users who do this\n> \n>     $ git archive HEAD Documentation/ | tar tf -\n> \n> would be expecting (at least) two different things, depending on the\n> situation they are in.\n> \n> So at least you'd need an \"--include-untracked\" option, I guess.\n> \n\n"},{"id":"335927","messageId":"ee76246b-579c-fe01-eb9c-d400061b47b6@web.de","threadId":"47510","inReplyTo":"xmqqmv1tfpqh.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-01-05T13:54:33Z","receivedAt":"2018-01-05T13:54:54Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.01.2018 um 19:22 schrieb Junio C Hamano:\n> René Scharfe <l.s.r@web.de> writes:\n> \n>> I don't know if it's a good idea, but perhaps we don't even need a new\n>> option.  We could change how pathspecs of untracked files are handled:\n>> Instead of aborting we could include them in the archive.  (Sounds like\n>> the simplest possible interface, but may have practical problems.)\n> \n> One practical problem is that users who do this\n> \n>      $ git archive HEAD Documentation/ | tar tf -\n> \n> would be expecting (at least) two different things, depending on the\n> situation they are in.\n> \n> So at least you'd need an \"--include-untracked\" option, I guess.\n\nRight, this breaks down with directories -- most build artifacts (e.g.\n.o files) are probably not meant to end up in archives.  We could still\ndo it for regular files and symlinks.  Perhaps that's too confusing,\nthough, and an --add-untracked-file parameter (or whatever we want to\ncall it) is the way to go.\n\nRené\n"},{"id":"335944","messageId":"xmqqtvw0ceat.fsf@gitster.mtv.corp.google.com","threadId":"47510","inReplyTo":"ee76246b-579c-fe01-eb9c-d400061b47b6@web.de","subject":"Re: [PATCH] git-archive: accept --owner and --group like GNU tar","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-05T19:10:18Z","receivedAt":"2018-01-05T19:10:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n>> One practical problem is that users who do this\n>> \n>>      $ git archive HEAD Documentation/ | tar tf -\n>> \n>> would be expecting (at least) two different things, depending on the\n>> situation they are in.\n>> \n>> So at least you'd need an \"--include-untracked\" option, I guess.\n>\n> Right, this breaks down with directories -- most build artifacts (e.g.\n> .o files) are probably not meant to end up in archives.\n\nI agree that it is unwise to overload the pathspec for this purpose.\nPerhaps bulk of the documentation of a project is in javadoc in its\nsource code and extracted into some directory, where the user would\nwant to include untracked things as well as tracked ones, while\nuntracked contents of other directories are all not meant to be\npackaged.  As \"git archive\" is primarily about freezing the contents\nof a set of paths in a single revision into an archive, and\nincluding untracked things is secondary, perhaps the right way to do\nso would be to:\n\n (1) leave pathspec as-is---they mean \"only this area of the named\n     revision goes into the resulting archive\", and \n\n (2) introduce a new \"--add-untracked=<wildmatch>\" option, that can\n     be multiply given, is cumulative, and is used to specify which\n     untracked paths to be included in the result from the working\n     tree contents.\n\nSo\n\n\tgit archive \\\n\t\t--add-untracked=./configure \\\n\t\t--add-untracked='Documentation/**/*.html' \\\n\t\t--add-untracked='Documentation/*.[1-9]' \\\n\t\tHEAD -- . ':!contrib/' ':t/'\n\nmight be a way to package up sources we use without tests but\ninclude the built documentation files.\n"}]}