# Incremental object transfer 3 messages from 2010-11-12 to 2010-11-13. Participants: Enrico Weigelt, Jonathan Nieder, Jakub Narebski. Thread: https://gitlist.dev/t/25724 ## Enrico Weigelt, 2010-11-12 10:34 Subject: Incremental object transfer Message-ID: <20101112103427.GA29057@nibiru.local> URL: https://gitlist.dev/e/20101112103427.GA29057%40nibiru.local ``` Hi folks, as already said some time ago, I'm using git to back up maildirs on a machine w/ relatively low ram. The biggest problem for now is the initial push (maybe later larger subsequent pushes could be also affected too): it takes quite a long time to get everything packed, and if the connection breaks (the box is sitting behind a dynamic-IP DSL link), everything has to be restarted :( So my idea is to incrementally transfer objects in smaller packs, disable gc on remote side and update refs when some commit is complete. Is there any way to do this ? thx -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Jonathan Nieder, 2010-11-12 16:09 Subject: Re: Incremental object transfer Message-ID: <20101112160933.GC15141@burratino> URL: https://gitlist.dev/e/20101112160933.GC15141%40burratino In-Reply-To: <20101112103427.GA29057@nibiru.local> ``` Enrico Weigelt wrote: > as already said some time ago, I'm using git to back up maildirs > on a machine w/ relatively low ram. The biggest problem for now > is the initial push (maybe later larger subsequent pushes could > be also affected too): it takes quite a long time to get everything > packed, and if the connection breaks (the box is sitting behind > a dynamic-IP DSL link), everything has to be restarted :( Maybe use "git bundle" and then transfer by rsync? ``` ## Jakub Narebski, 2010-11-13 18:31 Subject: Re: Incremental object transfer Message-ID: URL: https://gitlist.dev/e/m31v6pf54h.fsf%40localhost.localdomain In-Reply-To: <20101112103427.GA29057@nibiru.local> ``` Enrico Weigelt writes: > As already said some time ago, I'm using git to back up maildirs > on a machine w/ relatively low ram. The biggest problem for now > is the initial push (maybe later larger subsequent pushes could > be also affected too): it takes quite a long time to get everything > packed, and if the connection breaks (the box is sitting behind > a dynamic-IP DSL link), everything has to be restarted :( > > So my idea is to incrementally transfer objects in smaller packs, > disable gc on remote side and update refs when some commit is > complete. > > Is there any way to do this ? This would work only for "dumb" transports: "dumb" HTTP transport and deprecated rsync transport. "Smart" transport (e.g. git://, SSH, new "smart" HTTP transport) all create packs to send on the fly. Those packs would rarely be byte-for-byte the same, even if both server and client have the same objects, unless perhaps on single code (unthreaded). There were discussion about resumable download, by examining what got downloaded in partial pack; but this is a hard problem. So I would recommend either creating a bundle (see git-bundle manpage), which can be resumably downloaded via HTTP, FTP, P2P - it is an ordinary file. Or you can try cloning via rsync (but the repository should be quiescent). -- Jakub Narebski Poland ShadeHawk on #git ```