git/list[1] front-page[2] threads[3] people[4] search[5] about
 

getting list of objects for packing

From
Brandon Casey <casey@nrlssc.navy.mil>
Date
Oct 31, 2008, 19:32 UTC
Message-ID
<TtAUShKh7lOR5rkf1iyWwpMOWoYpT8Mnw-t3Wemdy3tTCd0XiQHdag@cipher.nrlssc.navy.mil>

I'm trying to write a script that will repack large binary or compressed objects into their own non-compressed, non-delta'ed pack file.

To make the decision about whether an object should go into this special pack file or not, I want the output from 'git cat-file --batch-check'. I get it with something similar to:

   git rev-list --objects --all |
      sed -e 's/^\([0-9a-f]\{40\}\).*/\1/' |
      git cat-file --batch-check
First question: Is the rev-list call correct?
  -If I am understanding things right, then the list of objects produced
   by rev-list will be in the right order for piping to pack-objects. 
  -The sed statement is stripping off anything after the sha1. Any way to
   get rev-list to print out just the sha1 so that sed is not necessary?

Then I want to parse the output from cat-file and use an external program to detect the file format. Here is a simplified version:

  | while read sha1 type size; do
       if [ $type = "blob" ]; then
           if ! ( git cat-file blob "$sha1" | file -b - | grep text ) &&
              [ $size -ge $threshhold ]; then
               # pack into special pack
           else
               # pack normally into normal pack
           fi
       fi
  done

All of this has actually been rewritten into a perl script, so ignore any syntax mistakes.

I have successfully created two of the pack files that I have been trying to make. Where the definition of successful means that after removing the existing packs and objects, and putting in place the two pack files that I generated, 'git fsck --full' prints no errors and exits successfully.

These two packs will be placed into a central repository.
ISSUE TWO:

I have placed these two packs into my own personal repo, and I have unpacked all of the other objects so that they are loose.

I thought I could use a similar sequence of commands to pack those loose objects into a normal and special pack. I added the --unpacked option to my rev-list command, but it still lists many more objects than exist loosely in the repository.

   git rev-list --objects --unpacked --all
The man page says:
   --objects
          Print  the  object  IDs  of any object referenced by the listed
          commits. --objects foo ^bar thus means "send me all object  IDs
          which  I  need to download if I have the commit object bar, but
          not foo".
   --unpacked
          Only useful with --objects; print the object IDs that  are  not
          in packs.

Is this the correct behavior for rev-list --unpacked? Am I mis-reading the --unpacked text, or should it be changed?

-brandon
Next: Nicolas Pitre
Message 1 of 61 in “getting list of objects for packing”
  1. Brandon CaseyOct 31, 2008
  2. Nicolas PitreOct 31, 2008
  3. Brandon CaseyOct 31, 2008
  4. Junio C HamanoOct 31, 2008
  5. Brandon CaseyOct 31, 2008
  6. Jakub NarebskiOct 31, 2008
  7. Brandon CaseyNov 1, 2008
  8. t7700: demonstrate mishandling of objects in packs with a .keep filedrafnel@gmail.com, Nov 2, 2008
  9. 1/3 packed_git: convert pack_local flag into generic bit maskdrafnel@gmail.com, Nov 2, 2008
  10. Shawn O. PearceNov 3, 2008
  11. Brandon CaseyNov 3, 2008
  12. 1/3 t7700: demonstrate mishandling of objects in packs with a .keep fileBrandon Casey, Nov 3, 2008
  13. 2/3 packed_git: convert pack_local flag into a bitfield and add pack_keepBrandon Casey, Nov 3, 2008
  14. 3/3 pack-objects: honor '.keep' filesBrandon Casey, Nov 3, 2008
  15. Shawn O. PearceNov 3, 2008
  16. Brandon CaseyNov 5, 2008
  17. Brandon CaseyNov 6, 2008
  18. Junio C HamanoNov 7, 2008
  19. Brandon CaseyNov 7, 2008
  20. Andreas EricssonNov 7, 2008
  21. Shawn O. PearceNov 7, 2008
  22. recognize loose local objects during repackdrafnel@gmail.com, Nov 10, 2008
  23. Junio C HamanoNov 10, 2008
  24. 1/4 pack-objects: new option --honor-pack-keepBrandon Casey, Nov 7, 2008
  25. 2/4 repack: don't repack local objects in packs with .keep fileBrandon Casey, Nov 7, 2008
  26. 3/4 repack: do not fall back to incremental repacking with [-a|-A]Brandon Casey, Nov 7, 2008
  27. 4/4 builtin-gc.c: use new pack_keep bitfield to detect .keep file existenceBrandon Casey, Nov 7, 2008
  28. Andreas EricssonNov 7, 2008
  29. Andreas EricssonNov 7, 2008
  30. t7700: demonstrate mishandling of objects in packs with a .keep fileBrandon Casey, Nov 3, 2008
  31. Andreas EricssonNov 4, 2008
  32. Brandon CaseyNov 4, 2008
  33. Junio C HamanoNov 4, 2008
  34. Brandon CaseyNov 4, 2008
  35. Andreas EricssonNov 4, 2008
  36. Junio C HamanoNov 4, 2008
  37. Jeff KingNov 12, 2008
  38. Junio C HamanoNov 12, 2008
  39. Jeff KingNov 12, 2008
  40. Brandon CaseyNov 12, 2008
  41. repack and .keep seriesBrandon Casey, Nov 12, 2008
  42. 1/6 t7700: demonstrate mishandling of objects in packs with a .keep fileBrandon Casey, Nov 12, 2008
  43. 2/6 packed_git: convert pack_local flag into a bitfield and add pack_keepBrandon Casey, Nov 12, 2008
  44. 3/6 pack-objects: new option --honor-pack-keepBrandon Casey, Nov 12, 2008
  45. 4/6 repack: don't repack local objects in packs with .keep fileBrandon Casey, Nov 12, 2008
  46. 5/6 repack: do not fall back to incremental repacking with [-a|-A]Brandon Casey, Nov 12, 2008
  47. 6/6 builtin-gc.c: use new pack_keep bitfield to detect .keep file existenceBrandon Casey, Nov 12, 2008
  48. t7700: test that 'repack -a' packs alternate packed objectsBrandon Casey, Nov 13, 2008
  49. Junio C HamanoNov 12, 2008
  50. Junio C HamanoNov 12, 2008
  51. Andreas EricssonNov 3, 2008
  52. Brandon CaseyNov 3, 2008
  53. Andreas EricssonNov 3, 2008
  54. Brandon CaseyNov 3, 2008
  55. Andreas EricssonNov 4, 2008
  56. 2/3 packed_git: add new PACK_KEEP flag and haspackkeep() access macrodrafnel@gmail.com, Nov 2, 2008
  57. 3/3 pack-objects: honor '.keep' filesdrafnel@gmail.com, Nov 2, 2008
  58. Shawn O. PearceNov 3, 2008
  59. 1/3 t7700: demonstrate mishandling of loose objects in an alternate ODBdrafnel@gmail.com, Nov 10, 2008
  60. 2/3 sha1_file.c: split has_loose_object() into local and non-local counterpartsdrafnel@gmail.com, Nov 10, 2008
  61. 3/3 pack-objects: extend --local to mean ignore non-local loose objects toodrafnel@gmail.com, Nov 10, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.