{"thread":{"id":"23805","subject":"many files, simple history","startedAt":"2010-05-14T02:57:22Z","lastAt":"2010-05-14T16:25:52Z","messageCount":5,"participants":["Ali Tofigh","Michael Witten","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"141623","messageId":"AANLkTinHZbJ4obpa1FpT8boFWjNYpgU184HUTvki_A0G@mail.gmail.com","threadId":"23805","inReplyTo":null,"subject":"many files, simple history","fromName":"Ali Tofigh","fromEmail":"alix.tofigh@gmail.com","sentAt":"2010-05-14T02:57:22Z","receivedAt":"2010-05-14T02:57:22Z","isPatch":false,"sender":{"key":"alix.tofigh@gmail.com","avatar":null},"body":"short version: will git handle large number of files efficiently if\nthe history is simple and linear, i.e., without merges?\n\nlonger version: i'm considering using git to keep track of my\ninstalled user programs/files on my linux machine, mainly because i\nwant to be able to uninstall software cleanly and completely (i almost\nalways build from source code and install in non-standard locations).\nso i would want to use git to keep track of every program i install or\nuninstall. this way, i could go back and uninstall a program simply by\nfinding the commit when it was installed, get the list of files that\nwere added as a result, and remove them (and of course, commit the\nremovals into git so i can always undo the uninstall too!)\n\nso the history stored in git will be linear and consist of file\nadditions and removals. but this will potentially involve many files.\nwill git be able to handle this (as yet hypothetical) situation\nefficiently?\n\n/ali\n"},{"id":"141625","messageId":"AANLkTimHB9jZq-SUxD2YUfCXKfTOHSxEnANETDL7i6DB@mail.gmail.com","threadId":"23805","inReplyTo":"AANLkTinHZbJ4obpa1FpT8boFWjNYpgU184HUTvki_A0G@mail.gmail.com","subject":"Re: many files, simple history","fromName":"Michael Witten","fromEmail":"mfwitten@gmail.com","sentAt":"2010-05-14T03:27:42Z","receivedAt":"2010-05-14T03:27:42Z","isPatch":false,"sender":{"key":"mfwitten@gmail.com","avatar":"https://avatars.githubusercontent.com/u/597101?v=4"},"body":"On Thu, May 13, 2010 at 21:57, Ali Tofigh wrote:\n> (i almost\n> always build from source code and install in non-standard locations).\n\nIt would make a lot more sense to use a package manager.\n\nYou might find that Arch Linux's `pacman' provides everything that you\nneed; it's really easy to define your own packages and build from\nsource, allowing pacman to manage dependencies, locate which package\nowns which file, and upgrade/remove packages.\n\nI bet gentoo's `emerge' would also be viable.\n"},{"id":"141626","messageId":"20100514040559.GB6075@coredump.intra.peff.net","threadId":"23805","inReplyTo":"AANLkTinHZbJ4obpa1FpT8boFWjNYpgU184HUTvki_A0G@mail.gmail.com","subject":"Re: many files, simple history","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-14T04:05:59Z","receivedAt":"2010-05-14T04:05:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 13, 2010 at 10:57:22PM -0400, Ali Tofigh wrote:\n\n> short version: will git handle large number of files efficiently if\n> the history is simple and linear, i.e., without merges?\n\nShort answer: large number of files, yes, large files, not really. The\nshape of history is largely irrelevant.\n\nLonger answer:\n\nGit separates the conceptual structure of history (the digraph of\ncommits, and the pointers of commits to trees to blobs) from the actual\nstorage of objects representing that history. Problems with large files\nare usually storage issues. Copying them around in packfiles is\nexpensive, storing an extra copy in the repo is expensive, trying deltas\nand diffs is expensive. None of those things has to do with the shape of\nyour history. So I would expect git to handle such a load with a linear\nhistory about as well as a complex history with merges.\n\nFor large numbers of files, git generally does a good job, especially if\nthose files are distributed throughout a directory hierarchy. But keep\nin mind that the git repo will store another copy of every file. They\nwill be delta-compressed between versions, and zlib compressed overall,\nbut you may potentially be doubling the amount of disk space required if\nyou have a lot of uncompressible binary files.\n\nFor large files, git expects to be able to pull each file into memory.\nSometimes two versions if you are doing a diff. And it will copy those\nfiles around when repacking (which you will want to do for the sake of\nthe smaller files). So files on the order of a few megabytes are not a\nproblem. If you have files in the hundreds of megabytes or gigabytes,\nexpect some operations to be slow (like repacking).\n\nReally, I would start by just \"git add\"-ing your whole filesystem, doing\na \"git repack -ad\", and seeing how long it takes, and what the resulting\nsize is.\n\n-Peff\n"},{"id":"141682","messageId":"AANLkTimwd0DMc2gMbQfv9t_4YeEZrJETz2trszVqq3jd@mail.gmail.com","threadId":"23805","inReplyTo":"20100514040559.GB6075@coredump.intra.peff.net","subject":"Re: many files, simple history","fromName":"Ali Tofigh","fromEmail":"alix.tofigh@gmail.com","sentAt":"2010-05-14T16:18:03Z","receivedAt":"2010-05-14T16:18:03Z","isPatch":false,"sender":{"key":"alix.tofigh@gmail.com","avatar":null},"body":"On Fri, May 14, 2010 at 00:05, Jeff King wrote:\n> On Thu, May 13, 2010 at 10:57:22PM -0400, Ali Tofigh wrote:\n>\n>> short version: will git handle large number of files efficiently if\n>> the history is simple and linear, i.e., without merges?\n>\n> Short answer: large number of files, yes, large files, not really. The\n> shape of history is largely irrelevant.\n\nthank you for the explanation. I will start using git for managing my\ninstalled programs and will try to report back to this list about my\nexperience.\n\n/ali\n\n>\n> Longer answer:\n>\n> Git separates the conceptual structure of history (the digraph of\n> commits, and the pointers of commits to trees to blobs) from the actual\n> storage of objects representing that history. Problems with large files\n> are usually storage issues. Copying them around in packfiles is\n> expensive, storing an extra copy in the repo is expensive, trying deltas\n> and diffs is expensive. None of those things has to do with the shape of\n> your history. So I would expect git to handle such a load with a linear\n> history about as well as a complex history with merges.\n>\n> For large numbers of files, git generally does a good job, especially if\n> those files are distributed throughout a directory hierarchy. But keep\n> in mind that the git repo will store another copy of every file. They\n> will be delta-compressed between versions, and zlib compressed overall,\n> but you may potentially be doubling the amount of disk space required if\n> you have a lot of uncompressible binary files.\n>\n> For large files, git expects to be able to pull each file into memory.\n> Sometimes two versions if you are doing a diff. And it will copy those\n> files around when repacking (which you will want to do for the sake of\n> the smaller files). So files on the order of a few megabytes are not a\n> problem. If you have files in the hundreds of megabytes or gigabytes,\n> expect some operations to be slow (like repacking).\n>\n> Really, I would start by just \"git add\"-ing your whole filesystem, doing\n> a \"git repack -ad\", and seeing how long it takes, and what the resulting\n> size is.\n>\n> -Peff\n>\n"},{"id":"141683","messageId":"AANLkTinZ4MuAofDwhx6FZjlY6fU7_sdgIZlg7RRMH1C8@mail.gmail.com","threadId":"23805","inReplyTo":"AANLkTimHB9jZq-SUxD2YUfCXKfTOHSxEnANETDL7i6DB@mail.gmail.com","subject":"Re: many files, simple history","fromName":"Ali Tofigh","fromEmail":"alix.tofigh@gmail.com","sentAt":"2010-05-14T16:25:52Z","receivedAt":"2010-05-14T16:25:52Z","isPatch":false,"sender":{"key":"alix.tofigh@gmail.com","avatar":null},"body":"On Thu, May 13, 2010 at 23:27, Michael Witten wrote:\n> On Thu, May 13, 2010 at 21:57, Ali Tofigh wrote:\n>> (i almost\n>> always build from source code and install in non-standard locations).\n>\n> It would make a lot more sense to use a package manager.\n\nyou're probably right, but if i want to keep up with all my everyday\nobligations, then i have to chose my battles carefully. for many\ntasks, there are specialized tools that can be used, but these always\ncome at the expense of having to learn them. and if those tasks aren't\nperformed on a daily basis, then i don't stand a chance at remembering\nwhat i've learned and i have to relearn them over and over again.\n\n/ali\n\n>\n> You might find that Arch Linux's `pacman' provides everything that you\n> need; it's really easy to define your own packages and build from\n> source, allowing pacman to manage dependencies, locate which package\n> owns which file, and upgrade/remove packages.\n>\n> I bet gentoo's `emerge' would also be viable.\n>\n"}]}