threads / patch / 12022

patchTechnical details about the index file format.

Subject: [PATCH] Technical details about the index file format.

## tl;dr

9 messages between Feb 11, 2008 and Sep 1, 2010. Diffs are folded; open one to read it.

replies: 8people: 7as markdown or json

Robin Rosenberg· Feb 11, 2008, 06:28 UTC · lore
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
---
 Documentation/technical/index-format.txt |   91 ++++++++++++++++++++++++++++++
 1 files changed, 91 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/technical/index-format.txt

I believe the main index details are accurate. Anything else to explain. The TREE section probably needs more details.

-- robin
Show changes to Documentation/technical/index-format.txt +91 −0
diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt
new file mode 100644
index 0000000..c57b382
--- /dev/null
+++ b/Documentation/technical/index-format.txt
@@ -0,0 +1,91 @@
+GIT index format
+================
+
+= The git index file has the following format
+
+  All binary numbers are in network byte order.
+
+   - A twelve byte header consisting of
+
+     4 byte signature:
+	The signature is { 'D', 'I', 'R', 'C' }
+
+     4 byte version number:
+	The current version is 2
+
+     32-bit number of index entries.
+
+   - An entry consists of
+
+     32-bit ctime seconds, the last time a file's metadata changed
+	this is stat(2) data
+
+     32-bit ctime nanoseconds (modulo 1G)
+	this is stat(2) data
+
+     32-bit mtime seconds, the last time a file's data changed
+	this is stat(2) data
+
+     32-bit mtime nanoseconds (modulo 1G)
+	this is stat(2) data
+
+     32-bit dev
+	this is stat(2) data
+
+     32-bit uid
+	this is stat(2) data
+
+     32-bit gid
+	this is stat(2) data
+
+     32-bit file size
+	This is the on-disk size from stat(2)
+
+     160-bit SHA-1 for the represented blob
+
+     A 16-bit field split into (high to low bits)
+
+	1-bit assume-valid flag
+
+	1-bit update-needed flag
+
+	2-bit stage (during merge)
+
+	12-bit name length
+
+     Name (variable length) - encoding is undefined
+
+     1-8 nul bytes as necessary to pad the entry to a multiple ot eight bytes
+     while keeping the name NUL-terminated.
+
+  - Extensions
+
+    The only know index extension today is a tree cache. It contains
+    pre-computes hashes for all trees that can be derived from the index
+
+    4 byte extension signature. If the first byte is 'A'..'Z' the
+    extension is optional and can be ignored.
+
+    32-bit size of the extension
+
+    Extension data
+
+  - 160-bit SHA-1 over the content of the index file before this checksum.
+
+
+== Tree cache
+
+  - Extension tag { 'T', 'R', 'E', 'E' }
+
+  - 32-bit size
+
+  - A number of entries
+
+     NUL-terminated tree name
+
+     Blank-terminated ASCII decimal number of entries in this tree
+
+     Newline-terminated position of this tree in the parent tree. 0 for
+     the root tree
+
+     160-bit SHA-1 for this tree and it's children
-- 
1.5.4.rc4.25.g81cc
Tim Stoakes· Feb 11, 2008, 12:00 UTC · re: Robin Rosenberg · lore

Re: [PATCH] Technical details about the index file format.

Robin Rosenberg(robin.rosenberg@dewire.com)@110208-07:28:
> diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt
A couple of typos:
> +     Name (variable length) - encoding is undefined
> +
> +     1-8 nul bytes as necessary to pad the entry to a multiple ot eight bytes
                                                                   ^
of
>  +    The only know index extension today is a tree cache. It contains
>  +    pre-computes hashes for all trees that can be derived from the index
                   ^
computed
full stop
Tim
-- 
Tim Stoakes
Junio C Hamano· Feb 11, 2008, 19:27 UTC · re: Robin Rosenberg · lore

Re: [PATCH] Technical details about the index file format.

Robin Rosenberg <robin.rosenberg@dewire.com> writes:
Show 7 quoted lines
> Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
> ---
>  Documentation/technical/index-format.txt |   91 ++++++++++++++++++++++++++++++
>  1 files changed, 91 insertions(+), 0 deletions(-)
>  create mode 100644 Documentation/technical/index-format.txt
>
> I believe the main index details are accurate. Anything else to explain.

You missed the most important feature. The entries are sorted in a particular order.

Also I do not think we store CE_UPDATE. The tip of 'master' will soon clarify about this.

Also when a name is longer than 12-bit can express, we will have a constant there and the name length could be longer (you need to strlen() it). The tip of 'master' will soon have this fix (we used to just overrun).

Other than these, I think the description is fairly accurate.
Sverre Rabbelier· Aug 31, 2010, 00:59 UTC · re: Robin Rosenberg · lore

Re: [PATCH] Technical details about the index file format.

Heya,
Robin Rosenberg <robin.rosenberg <at> dewire.com> writes:
Show 6 quoted lines
> 
> Signed-off-by: Robin Rosenberg <robin.rosenberg <at> dewire.com>
> ---
>  Documentation/technical/index-format.txt |   91 ++++++++++++++++++++++++++++++
>  1 files changed, 91 insertions(+), 0 deletions(-)
>  create mode 100644 Documentation/technical/index-format.txt

This pretty much got a LGTM from Junio back when it was sent [0], Robin, can you be persuaded to resend this? I think it'd be very good if we had some documentation on the index format.

[0] http://thread.gmane.org/gmane.comp.version-control.git/73471

-- Cheers,

Sverre Rabbelier
Ramkumar Ramachandra· Aug 31, 2010, 07:08 UTC · re: Sverre Rabbelier · lore

Re: [PATCH] Technical details about the index file format.

Hi Sverre,
Sverre Rabbelier writes:
Show 11 quoted lines
> Robin Rosenberg <robin.rosenberg <at> dewire.com> writes:
> > 
> > Signed-off-by: Robin Rosenberg <robin.rosenberg <at> dewire.com>
> > ---
> >  Documentation/technical/index-format.txt |   91 ++++++++++++++++++++++++++++++
> >  1 files changed, 91 insertions(+), 0 deletions(-)
> >  create mode 100644 Documentation/technical/index-format.txt
> 
> This pretty much got a LGTM from Junio back when it was sent [0], Robin, can you 
> be persuaded to resend this? I think it'd be very good if we had some 
> documentation on the index format.
Don't we already have this in Documentation/technical/pack-format.txt?
-- Ram
Nguyen Thai Ngoc Duy· Aug 31, 2010, 22:12 UTC · re: Robin Rosenberg · lore

Re: [PATCH] Technical details about the index file format.

On Mon, Feb 11, 2008 at 5:28 PM, Robin Rosenberg <robin.rosenberg@dewire.com> wrote:

> +     4 byte version number:
> +       The current version is 2
The version could be 3 if extended flags are used.
Show 5 quoted lines
> +     A 16-bit field split into (high to low bits)
> +
> +       1-bit assume-valid flag
> +
> +       1-bit update-needed flag

I think this bit is CE_EXTENDED, an indication that this entry has extended flags

> +
> +       2-bit stage (during merge)
> +
> +       12-bit name length
     A 16-bit field additional flags (high to low bits), only
applicable to version 3
       1-bit reserved for future
       1-bit skip-worktree flag
       1-bit intent-to-add flag (aka "git add -N")
> +  - Extensions
> +
> +    The only know index extension today is a tree cache.

There's also "REUC" extension from read-cache.c. I personally have never touched it, so no comments.

-- 
Duy
Sverre Rabbelier· Sep 1, 2010, 02:28 UTC · re: Nguyen Thai Ngoc Duy · lore

Re: [PATCH] Technical details about the index file format.

Heya,
On Tue, Aug 31, 2010 at 17:12, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> The version could be 3 if extended flags are used.

I suspect you are the person to have most recently messed around with the index, and given your corrections above, could you perhaps pick up the patch?

-- 
Cheers,

Sverre Rabbelier
Nguyen Thai Ngoc Duy· Sep 1, 2010, 02:52 UTC · re: Sverre Rabbelier · lore

Re: [PATCH] Technical details about the index file format.

On Wed, Sep 1, 2010 at 12:28 PM, Sverre Rabbelier <srabbelier@gmail.com> wrote:
Show 8 quoted lines
> Heya,
>
> On Tue, Aug 31, 2010 at 17:12, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> The version could be 3 if extended flags are used.
>
> I suspect you are the person to have most recently messed around with
> the index, and given your corrections above, could you perhaps pick up
> the patch?
Whoa, I did not notice the patch was from 2008. Yes, I'll fix it up and resend.
-- 
Duy

← back to recent threads