threads / discuss / 50257

Calculate packfile and idxfile checksums

Subject: Calculate packfile and idxfile checksums

## tl;dr

3 messages between Jan 18, 2019 and Jan 18, 2019.

replies: 2people: 2as markdown or json

Farhan Khan· Jan 18, 2019, 07:24 UTC · lore

Hi all,

I am trying to understand how git's index-pack works, particularly how it calculates the packfile checksum and idxfile checksum. I traced back the packfile checksum in the source to the value char *sha1 that is utilized in write_idx_file() in pack_file.c:45. However, I cannot determine where this value is set.

My printf() debugging has it set at pack-write.c:171 (right before the hashwrite call) but it does not seem to be utilized prior to that point. Please assist.

Thanks! -- Farhan Khan PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE

Ævar Arnfjörð Bjarmason· Jan 18, 2019, 08:36 UTC · re: Farhan Khan · lore

Re: Calculate packfile and idxfile checksums

On Fri, Jan 18 2019, Farhan Khan wrote:

> Hi all,
>
> I am trying to understand how git's index-pack works, particularly how
> it calculates the packfile checksum and idxfile checksum.

It's unclear if this is what you're asking, but the pack checksum is just a 20 bytes of SHA1 of the preceding contents at the end for *.idx and *.pack. As seen with this program:

    $ perl -MFile::Slurp=slurp -MDigest::SHA=sha1 -wE 'my $f = shift; my $c = slurp($f); my $cp = $c; $cp =~ s/.{20}$//s; my $n = $cp . sha1($cp); if ($n eq $c) { say "Computed checksum trailer for $f" } else { say "Failed trailer for new content for $f is different" }' pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx
    Computed checksum trailer for pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx

You can also feed it *.pack files.

> I traced back the packfile checksum in the source to the value char
> *sha1 that is utilized in write_idx_file() in pack_file.c:45. However,
> I cannot

It seems you mean pack-write.c not pack_file.c

> determine where this value is set.

It's initialized in cmd_index_pack() and then passed down to that function.

> My printf() debugging has it set at pack-write.c:171 (right before the
> hashwrite call) but it does not seem to be utilized prior to that
> point. Please assist.

I'm happy to help, but still not quite sure what the source of the confusion is, maybe that the variable in index-pack.c has a different name and is passed down to pack-write.c's function as a pointer?

Farhan Khan· Jan 18, 2019, 23:51 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: Calculate packfile and idxfile checksums

On Fri, Jan 18, 2019 at 3:36 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 36 quoted lines

>
>
> On Fri, Jan 18 2019, Farhan Khan wrote:
>
> > Hi all,
> >
> > I am trying to understand how git's index-pack works, particularly how
> > it calculates the packfile checksum and idxfile checksum.
>
> It's unclear if this is what you're asking, but the pack checksum is
> just a 20 bytes of SHA1 of the preceding contents at the end for *.idx
> and *.pack. As seen with this program:
>
>     $ perl -MFile::Slurp=slurp -MDigest::SHA=sha1 -wE 'my $f = shift; my $c = slurp($f); my $cp = $c; $cp =~ s/.{20}$//s; my $n = $cp . sha1($cp); if ($n eq $c) { say "Computed checksum trailer for $f" } else { say "Failed trailer for new content for $f is different" }' pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx
>     Computed checksum trailer for pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx
>
> You can also feed it *.pack files.
>
> > I traced back the packfile checksum in the source to the value char
> > *sha1 that is utilized in write_idx_file() in pack_file.c:45. However,
> > I cannot
>
> It seems you mean pack-write.c not pack_file.c
>
> > determine where this value is set.
>
> It's initialized in cmd_index_pack() and then passed down to that
> function.
>
> > My printf() debugging has it set at pack-write.c:171 (right before the
> > hashwrite call) but it does not seem to be utilized prior to that
> > point. Please assist.
>
> I'm happy to help, but still not quite sure what the source of the
> confusion is, maybe that the variable in index-pack.c has a different
> name and is passed down to pack-write.c's function as a pointer?

I apologize for my poor email and appreciate your willingness to help. Let me try that again.

My objective is to identify how to calculate the packfile and idxfile checksums and/or where this is performed in the git source code.

You said that this value is initialized in builtin/index-pack.c:cmd_index_pack(). Is it correct that this value is the 'pack_hash' variable? At a high level, how is this value calculated?

That seems to be the packfile index, how and where is that idxfile hash calculated?

Thanks!