# Existing utility to track compiled files in another sister repository, for rollouts 3 messages from 2012-08-23 to 2012-08-23. Participants: Ævar Arnfjörð Bjarmason, Junio C Hamano. Thread: https://gitlist.dev/t/31333 ## Ævar Arnfjörð Bjarmason, 2012-08-23 16:28 Subject: Existing utility to track compiled files in another sister repository, for rollouts Message-ID: URL: https://gitlist.dev/e/CACBZZX6SVSn17nfSarQC1qgz-TxcDDn5tvHb0BkyW9S8DcLY4w%40mail.gmail.com ``` I'm planning on using Git for a deployment process where the steps are basically: 1. You log into a deployment host, cd into software.git, do git pull 2. A tool runs "make" for you, creates a deployment-YYYYMMDD-HHMMSS tag 3. That make step will create a bunch of generated (text) files 4. Get a list of these with : git clean -dxfn 5. Copy those to to software-generated.git, removing any that we didn't just create, adding any that are new 6. Commit that, tag it with generated-deployment-YYYYMMDD-HHMMSS 7. Push out both our generated software.git and software-generated.git tag to our servers 8. git reset --hard both of those to our newly pushed out tags 9. Do git clean -dxf on software.git remove old generated files 10. Copy new generated files from generated-software.git to software.git 11. Restart our application to pick up the new software For this I'll need to write some "git snapshot-commit" tool for #5 and #6 to commit whatever the current state of the directory is (with removed/added files), and hack up something to do #9-#10. This should all be relatively easy, I was just wondering if there was any prior art on this that I could use instead of hacking it up myself. For what it's worth the reason I'm using Git like this for deployment is that I'm converting things away from an rsync-based sync process that's becoming increasingly slow since rsync with -c needs to recursively checksum everything we're syncing out (which is quite a lot), since it's all text files Git can be really efficient in just transferring deltas and quickly doing a hard reset to a new commit. ``` ## Junio C Hamano, 2012-08-23 17:57 Subject: Re: Existing utility to track compiled files in another sister repository, for rollouts Message-ID: <7vk3wp4hkh.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vk3wp4hkh.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Ævar Arnfjörð Bjarmason writes: > 5. Copy those to to software-generated.git, removing any that we > didn't just create, adding any that are new > 6. Commit that, tag it with generated-deployment-YYYYMMDD-HHMMSS > 9. Do git clean -dxf on software.git remove old generated files > 10. Copy new generated files from generated-software.git to software.git > > For this I'll need to write some "git snapshot-commit" tool for #5 and > #6 to commit whatever the current state of the directory is (with > removed/added files), and hack up something to do #9-#10. > > This should all be relatively easy, I was just wondering if there was > any prior art on this that I could use instead of hacking it up > myself. FWIW, dodoc script in my todo branch was hacked up to respond to my push into k.org repository, pull 'master' into "documentation build repo" at the k.org machine and check it out, build docs and then "removing any that we didn't create, adding any that are new" to update the html and man "branches", and push it back to the main repository, so in that sense, it has all the logic that was needed (and more, as the rebuilt documentation from two revisions may only differ from version stamp asciidoc-to-html/man toolchain adds, which need to be filtered out when updating html/man branches). There no longer is post-update hook at k.org, so I have to run it manually at the end of day's integration cycle, just before pushing things out. I wrote it myself, not because I looked for suitable reusable code and didn't find any, but because I was too lazy to look for existing tools and evaluating them. ``` ## Ævar Arnfjörð Bjarmason, 2012-08-23 20:45 Subject: Re: Existing utility to track compiled files in another sister repository, for rollouts Message-ID: URL: https://gitlist.dev/e/CACBZZX4RoPDODMgbz1T%2BwEMT5K4X157G5VYtXFp%3De%2Bj0C0hqiQ%40mail.gmail.com In-Reply-To: ``` On Thu, Aug 23, 2012 at 6:28 PM, Ævar Arnfjörð Bjarmason wrote: > I'm planning on using Git for a deployment process where the steps are > basically: > > 1. You log into a deployment host, cd into software.git, do git pull > 2. A tool runs "make" for you, creates a deployment-YYYYMMDD-HHMMSS tag > 3. That make step will create a bunch of generated (text) files > 4. Get a list of these with : git clean -dxfn > 5. Copy those to to software-generated.git, removing any that we > didn't just create, adding any that are new > 6. Commit that, tag it with generated-deployment-YYYYMMDD-HHMMSS > 7. Push out both our generated software.git and > software-generated.git tag to our servers > 8. git reset --hard both of those to our newly pushed out tags > 9. Do git clean -dxf on software.git remove old generated files > 10. Copy new generated files from generated-software.git to software.git > 11. Restart our application to pick up the new software > > For this I'll need to write some "git snapshot-commit" tool for #5 and > #6 to commit whatever the current state of the directory is (with > removed/added files), and hack up something to do #9-#10. > > This should all be relatively easy, I was just wondering if there was > any prior art on this that I could use instead of hacking it up > myself. Here's a quick hack that does #4-6 but not #9-10 yet, although that would be easy: https://gist.github.com/3440792 Suggestions for improvements welcome, particularly whether there's a simpler way to do this to nuke existing files in a repo and replace it with new files all staged for commit: # Go to the target repository, nuke anything already there chdir $to_repository; system "git reset --hard"; system "git clean -dxf"; system "git ls-tree --name-only HEAD -z | xargs -0 rm -rf"; system "git add --update"; # stage any removals Followed by: system "tar xvf incoming.tar"; system "rm incoming.tar"; system "git add * .??* || :"; # Might die if we empty the repo, TODO: make this use status -> add each file system "git commit -m'Bump copy from $from_repository to $to_repository' || :"; # We might have nothing to change! ```