# [PATCH 0/3] hash-object doc: small fixes 12 messages from 2019-05-20 to 2019-05-28. Participants: Ævar Arnfjörð Bjarmason, Jeff King, Jakub Narebski, Junio C Hamano. Thread: https://gitlist.dev/t/51135 ## Ævar Arnfjörð Bjarmason, 2019-05-20 21:53 Subject: [PATCH 0/3] hash-object doc: small fixes Message-ID: <20190520215312.10363-1-avarab@gmail.com> URL: https://gitlist.dev/e/20190520215312.10363-1-avarab%40gmail.com ``` Small doc fixes. Maybe trivial enough to land in 2.22, but there's no rush. A pair of #leftoverbits I noticed is that we've implemented the "--stdin-paths" option via unquote_c_style() from day one, so our current docs lie (and still do with this series) about wanting \n-delimited files, you can't hash a file called '"foo"' as you'd expect, you need to pass '"\"foo\""'. I wonder if we should document this at this point, or just change it and add a "-z" option. None of our tests fail if I remove this unquote_c_style() codepath, and it's never been documented, but someone in the wild may have organically depended on it. Ævar Arnfjörð Bjarmason (3): hash-object doc: stop mentioning git-cvsimport hash-object doc: elaborate on -w and --literally promises hash-object doc: point to ls-files and rev-parse Documentation/git-hash-object.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) -- 2.21.0.1020.gf2820cf01a ``` ## Ævar Arnfjörð Bjarmason, 2019-05-20 21:53 Subject: [PATCH 1/3] hash-object doc: stop mentioning git-cvsimport Message-ID: <20190520215312.10363-2-avarab@gmail.com> URL: https://gitlist.dev/e/20190520215312.10363-2-avarab%40gmail.com In-Reply-To: <20190520215312.10363-1-avarab@gmail.com> ``` Remove a reference to git-cvsimport in the intro. As can be seen from the history of this command[1] it was originally intended for use with git-cvsimport, but how it uses it (and that it uses it at all) is irrelevant trivia at this point. 1. See 7672db20c2 ("[PATCH] Expose object ID computation functions.", 2005-07-08) and 8b8840e046 ("[PATCH] cvsgit fixes: spaces in filenames and CVS server dialog woes", 2005-08-15). Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/git-hash-object.txt | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Documentation/git-hash-object.txt b/Documentation/git-hash-object.txt index 814e74406a..df9e2c58bd 100644 --- a/Documentation/git-hash-object.txt +++ b/Documentation/git-hash-object.txt @@ -18,9 +18,7 @@ Computes the object ID value for an object with specified type with the contents of the named file (which can be outside of the work tree), and optionally writes the resulting object into the object database. Reports its object ID to its standard output. -This is used by 'git cvsimport' to update the index -without modifying files in the work tree. When is not -specified, it defaults to "blob". +When is not specified, it defaults to "blob". OPTIONS ------- -- 2.21.0.1020.gf2820cf01a ``` ## Ævar Arnfjörð Bjarmason, 2019-05-20 21:53 Subject: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: <20190520215312.10363-3-avarab@gmail.com> URL: https://gitlist.dev/e/20190520215312.10363-3-avarab%40gmail.com In-Reply-To: <20190520215312.10363-1-avarab@gmail.com> ``` Clarify the hash-object docs to explicitly note that the --literally option guarantees that a loose object will be written, but that a normal -w ("write") invocation doesn't. At first I thought talking about "loose object" in the docs was a mistake in 83115ac4a8 ("git-hash-object.txt: document --literally option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object: add --literally option", 2014-09-11) this was intended all along. As seen in 4dd1fbc7b1 ("Bigfile: teach "git add" to send a large file straight to a pack", 2011-05-08) we'll otherwise not guarantee that we write loose objects, so let's explicitly note that, using vague enough language to leave the door open to any arbitrary future storage format, not just loose objects and packs. While I'm at it remove the mention of "--stdin" from the "--literally" documentation. This wasn't correct, the "--literally" option combines with any other option (e.g. "--stdin-paths"), not just "--stdin". Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/git-hash-object.txt | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/Documentation/git-hash-object.txt b/Documentation/git-hash-object.txt index df9e2c58bd..100630d021 100644 --- a/Documentation/git-hash-object.txt +++ b/Documentation/git-hash-object.txt @@ -28,6 +28,12 @@ OPTIONS -w:: Actually write the object into the object database. ++ +No guarantees are made as to how the object is added to the database +(e.g. whether as a loose object, or streamed to a pack), except in the +case of the `--literally` option. How it's written may depend on +heuristics such as the `core.bigFileThreshold` configuration variable +(see linkgit:git-config[1]). --stdin:: Read the object from standard input instead of from a file. @@ -53,10 +59,11 @@ OPTIONS is always implied, unless the `--path` option is given. --literally:: - Allow `--stdin` to hash any garbage into a loose object which might not + Allow for hashing arbitrary data which might not otherwise pass standard object parsing or git-fsck checks. Useful for stress-testing Git itself or reproducing characteristics of corrupt or - bogus objects encountered in the wild. + bogus objects encountered in the wild. When writing objects guarantees + that the written object will be a loose object, for ease of debugging. GIT --- -- 2.21.0.1020.gf2820cf01a ``` ## Ævar Arnfjörð Bjarmason, 2019-05-20 21:53 Subject: [PATCH 3/3] hash-object doc: point to ls-files and rev-parse Message-ID: <20190520215312.10363-4-avarab@gmail.com> URL: https://gitlist.dev/e/20190520215312.10363-4-avarab%40gmail.com In-Reply-To: <20190520215312.10363-1-avarab@gmail.com> ``` Amend the intro to note that it's better to ask the index about files already tracked by it. I've seen uses of this in the wild where the use-case was finding object IDs for files found in a freshly cloned repo, i.e. something like "git ls-files | git hash-object --stdin-paths". Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/git-hash-object.txt | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Documentation/git-hash-object.txt b/Documentation/git-hash-object.txt index 100630d021..077d83ec65 100644 --- a/Documentation/git-hash-object.txt +++ b/Documentation/git-hash-object.txt @@ -20,6 +20,13 @@ work tree), and optionally writes the resulting object into the object database. Reports its object ID to its standard output. When is not specified, it defaults to "blob". +This command won't take the shortcut of seeing if the path(s) to be +hashed are present in the index, in which case their already computed +object ID could be retrieved by linkgit:git-ls-files[1] via the +`--stage` option, or by linkgit:git-rev-parse[1]. Use those instead of +e.g. `--stdin-paths` if the intent is to say find the blob object IDs +of checked-out files. + OPTIONS ------- -- 2.21.0.1020.gf2820cf01a ``` ## Jeff King, 2019-05-22 04:57 Subject: Re: [PATCH 1/3] hash-object doc: stop mentioning git-cvsimport Message-ID: <20190522045704.GA29933@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190522045704.GA29933%40sigill.intra.peff.net In-Reply-To: <20190520215312.10363-2-avarab@gmail.com> ``` On Mon, May 20, 2019 at 11:53:10PM +0200, Ævar Arnfjörð Bjarmason wrote: > Remove a reference to git-cvsimport in the intro. As can be seen from > the history of this command[1] it was originally intended for use with > git-cvsimport, but how it uses it (and that it uses it at all) is > irrelevant trivia at this point. Yeah, this is definitely obscuring the main point for most readers. The patch looks good. -Peff ``` ## Jeff King, 2019-05-22 05:08 Subject: Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: <20190522050839.GB29933@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190522050839.GB29933%40sigill.intra.peff.net In-Reply-To: <20190520215312.10363-3-avarab@gmail.com> ``` On Mon, May 20, 2019 at 11:53:11PM +0200, Ævar Arnfjörð Bjarmason wrote: > Clarify the hash-object docs to explicitly note that the --literally > option guarantees that a loose object will be written, but that a > normal -w ("write") invocation doesn't. I had to double-check here: you mean that _when_ we are writing an object, "--literally" would always write loose, right? > At first I thought talking about "loose object" in the docs was a > mistake in 83115ac4a8 ("git-hash-object.txt: document --literally > option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object: > add --literally option", 2014-09-11) this was intended all along. Hmm. After reading both of those, I do think it's mostly an implementation detail. I would not be at all surprised to find that the test suite relies on this (e.g., cleaning up with rm .git/objects/ab/cd1234). But I suspect we also rely on that for the non-literal case, too. ;) So I am on the fence. In some sense it doesn't hurt to document the behavior, but I'm not sure I would want to lock us in to any particular behavior, even for --literally. The intent of the option (as I recall) really is just "let us write whatever trash we want as an object, ignoring all quality checks". > --literally:: > - Allow `--stdin` to hash any garbage into a loose object which might not > + Allow for hashing arbitrary data which might not > otherwise pass standard object parsing or git-fsck checks. Useful for > stress-testing Git itself or reproducing characteristics of corrupt or > - bogus objects encountered in the wild. > + bogus objects encountered in the wild. When writing objects guarantees > + that the written object will be a loose object, for ease of debugging. I had to read this last sentence a few times to parse it. Maybe a comma before guarantees would help? Or even: When writing objects, this option guarantees that the written object will be a loose object, for ease of debugging. -Peff ``` ## Jeff King, 2019-05-22 05:15 Subject: Re: [PATCH 3/3] hash-object doc: point to ls-files and rev-parse Message-ID: <20190522051523.GC29933@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190522051523.GC29933%40sigill.intra.peff.net In-Reply-To: <20190520215312.10363-4-avarab@gmail.com> ``` On Mon, May 20, 2019 at 11:53:12PM +0200, Ævar Arnfjörð Bjarmason wrote: > Amend the intro to note that it's better to ask the index about files > already tracked by it. > > I've seen uses of this in the wild where the use-case was finding > object IDs for files found in a freshly cloned repo, i.e. something > like "git ls-files | git hash-object --stdin-paths". This example left me scratching my head, because it seems like such a bad way to accomplish the goal that I had trouble figuring out what the goal was in the first place. Of course I am pretty familiar with Git, so my mindset is not necessarily the same as the average reader of the manpage. But the suggested documentation really seems to me to be warning about a convoluted use case that most people would not come up with in the first place. And adding it in the introductory text means that people are more likely to read and be confused by it, when they are really just looking to find details on more mundane usage. Or put another way: I could come up with a hundred bad ways to script Git, but we are probably better off enumerating the _good_ ways to do so, rather than listing the bad ways and leaving the user to figure out what's left. So I'm mildly negative on this patch. -Peff ``` ## Jakub Narebski, 2019-05-24 10:04 Subject: Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: <86woigp3ro.fsf@gmail.com> URL: https://gitlist.dev/e/86woigp3ro.fsf%40gmail.com In-Reply-To: <20190522050839.GB29933@sigill.intra.peff.net> ``` Jeff King writes: > On Mon, May 20, 2019 at 11:53:11PM +0200, Ævar Arnfjörð Bjarmason wrote: > >> Clarify the hash-object docs to explicitly note that the --literally >> option guarantees that a loose object will be written, but that a >> normal -w ("write") invocation doesn't. > > I had to double-check here: you mean that _when_ we are writing an > object, "--literally" would always write loose, right? > >> At first I thought talking about "loose object" in the docs was a >> mistake in 83115ac4a8 ("git-hash-object.txt: document --literally >> option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object: >> add --literally option", 2014-09-11) this was intended all along. > > Hmm. After reading both of those, I do think it's mostly an > implementation detail. I would not be at all surprised to find that the > test suite relies on this (e.g., cleaning up with rm > .git/objects/ab/cd1234). But I suspect we also rely on that for the > non-literal case, too. ;) > > So I am on the fence. In some sense it doesn't hurt to document the > behavior, but I'm not sure I would want to lock us in to any particular > behavior, even for --literally. The intent of the option (as I recall) > really is just "let us write whatever trash we want as an object, > ignoring all quality checks". I thik that this implemetation detail of `--literally` is here to stay; how would you otherwise fix the issue if garbage object makes Git crash? However, I would prefer to have options state _intent_; if there is legitimate need for a tool that creates loose objects, it would be better to have separate `--loose` option to `git hash-object` (which would imply `-w`, otherwise it doesn't have sense). >> --literally:: >> - Allow `--stdin` to hash any garbage into a loose object which might not >> + Allow for hashing arbitrary data which might not >> otherwise pass standard object parsing or git-fsck checks. Useful for >> stress-testing Git itself or reproducing characteristics of corrupt or >> - bogus objects encountered in the wild. >> + bogus objects encountered in the wild. When writing objects guarantees >> + that the written object will be a loose object, for ease of debugging. > > I had to read this last sentence a few times to parse it. Maybe a comma > before guarantees would help? Or even: > > When writing objects, this option guarantees that the written object > will be a loose object, for ease of debugging. I agree that this reads better. Regards, -- Jakub Narębski ``` ## Ævar Arnfjörð Bjarmason, 2019-05-24 10:12 Subject: Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: <87lfywf9fp.fsf@evledraar.gmail.com> URL: https://gitlist.dev/e/87lfywf9fp.fsf%40evledraar.gmail.com In-Reply-To: <86woigp3ro.fsf@gmail.com> ``` On Fri, May 24 2019, Jakub Narebski wrote: > Jeff King writes: >> On Mon, May 20, 2019 at 11:53:11PM +0200, Ævar Arnfjörð Bjarmason wrote: >> >>> Clarify the hash-object docs to explicitly note that the --literally >>> option guarantees that a loose object will be written, but that a >>> normal -w ("write") invocation doesn't. >> >> I had to double-check here: you mean that _when_ we are writing an >> object, "--literally" would always write loose, right? >> >>> At first I thought talking about "loose object" in the docs was a >>> mistake in 83115ac4a8 ("git-hash-object.txt: document --literally >>> option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object: >>> add --literally option", 2014-09-11) this was intended all along. >> >> Hmm. After reading both of those, I do think it's mostly an >> implementation detail. I would not be at all surprised to find that the >> test suite relies on this (e.g., cleaning up with rm >> .git/objects/ab/cd1234). But I suspect we also rely on that for the >> non-literal case, too. ;) >> >> So I am on the fence. In some sense it doesn't hurt to document the >> behavior, but I'm not sure I would want to lock us in to any particular >> behavior, even for --literally. The intent of the option (as I recall) >> really is just "let us write whatever trash we want as an object, >> ignoring all quality checks". > > I thik that this implemetation detail of `--literally` is here to stay; > how would you otherwise fix the issue if garbage object makes Git crash? > > However, I would prefer to have options state _intent_; if there is > legitimate need for a tool that creates loose objects, it would be > better to have separate `--loose` option to `git hash-object` (which > would imply `-w`, otherwise it doesn't have sense). I wonder if we can just remove this option and replace it with a GIT_TEST_* env variable, or even a test-tool helper. I can't see why anyone other than our own test suite wants this, and that's why it was added. So why document it & expose it in a plumbing tool? >>> --literally:: >>> - Allow `--stdin` to hash any garbage into a loose object which might not >>> + Allow for hashing arbitrary data which might not >>> otherwise pass standard object parsing or git-fsck checks. Useful for >>> stress-testing Git itself or reproducing characteristics of corrupt or >>> - bogus objects encountered in the wild. >>> + bogus objects encountered in the wild. When writing objects guarantees >>> + that the written object will be a loose object, for ease of debugging. >> >> I had to read this last sentence a few times to parse it. Maybe a comma >> before guarantees would help? Or even: >> >> When writing objects, this option guarantees that the written object >> will be a loose object, for ease of debugging. > > I agree that this reads better. > > Regards, ``` ## Jeff King, 2019-05-28 06:06 Subject: Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: <20190528060641.GC7946@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190528060641.GC7946%40sigill.intra.peff.net In-Reply-To: <87lfywf9fp.fsf@evledraar.gmail.com> ``` On Fri, May 24, 2019 at 12:12:10PM +0200, Ævar Arnfjörð Bjarmason wrote: > > I thik that this implemetation detail of `--literally` is here to stay; > > how would you otherwise fix the issue if garbage object makes Git crash? > > > > However, I would prefer to have options state _intent_; if there is > > legitimate need for a tool that creates loose objects, it would be > > better to have separate `--loose` option to `git hash-object` (which > > would imply `-w`, otherwise it doesn't have sense). > > I wonder if we can just remove this option and replace it with a > GIT_TEST_* env variable, or even a test-tool helper. I can't see why > anyone other than our own test suite wants this, and that's why it was > added. So why document it & expose it in a plumbing tool? I can think of a few reasons you might want it in the general toolbox: - you could be recreating a known-buggy history and want to overcome consistency checks (e.g., imagine a tool that imports or modifies git history, and needs to recreate objects literally, warts and all). I think we don't do a lot of quality checks in hash-object now, but we have discussed it (and --literally would be the obvious escape hatch). - folks like security researchers who are poking at Git and want to see how it reacts to various almost-correct inputs. They _could_ use the test-tool helper, but being able to demonstrate bugs with the standard toolbox is helpful for communication. - likewise for general non-security debugging ("it behaves in this funny way if I violate this constraint...") So I actually think it's nice to have it as part of the plumbing, as long as it's buried and documented sufficiently that unsuspecting users do not stumble onto it. And at any rate, now that it has been in the wild for a while, we risk breaking somebody who has come to depend on it. -Peff ``` ## Junio C Hamano, 2019-05-28 16:49 Subject: Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: URL: https://gitlist.dev/e/xmqqd0k24j7s.fsf%40gitster-ct.c.googlers.com In-Reply-To: <20190520215312.10363-3-avarab@gmail.com> ``` Ævar Arnfjörð Bjarmason writes: > Clarify the hash-object docs to explicitly note that the --literally > option guarantees that a loose object will be written, but that a > normal -w ("write") invocation doesn't. > > At first I thought talking about "loose object" in the docs was a > mistake in 83115ac4a8 ("git-hash-object.txt: document --literally > option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object: > add --literally option", 2014-09-11) this was intended all along. I have to admit that this "loose only" was the doing of my defeatism. IOW, I was utterly pessimistic that I would be able to add more types (and more importantly, unbounded number of random types) of objects in the packstream. So, "loose object" limitation is a practical one for those of us who cannot think of a reasonable way to cram arbitrary number of random new types into just 3 bits of the "type" bitfield, and not inherent to the "hash-object --literally" command. So I am very happy to see the first hunk of this patch, but I doubt there is much value in the last sentence the second hunk adds. Thanks. ``` ## Junio C Hamano, 2019-05-28 16:56 Subject: Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises Message-ID: URL: https://gitlist.dev/e/xmqq8suq4ix8.fsf%40gitster-ct.c.googlers.com In-Reply-To: <86woigp3ro.fsf@gmail.com> ``` Jakub Narebski writes: > I thik that this implemetation detail of `--literally` is here to stay; > how would you otherwise fix the issue if garbage object makes Git crash? By repacking, presumably ;-). More importantly, there needs a way to extend "enum object_type" to allow unbounded number of arbitrary (garbage) types before we can allow --literally to record such a garbage type in a pack stream. So I'd expect the implementation detail would stay for a long time. But there is nothing that says `--literally` inherently must write loose. It is plausible that a new implementation writes objects of known/valid types to a pack stream, while unknown/garbage types to loose objects. > However, I would prefer to have options state _intent_; if there is > legitimate need for a tool that creates loose objects, it would be > better to have separate `--loose` option to `git hash-object` (which > would imply `-w`, otherwise it doesn't have sense). Yes, I very much agree with that. ```