{"thread":{"id":"31333","subject":"Existing utility to track compiled files in another sister repository, for rollouts","startedAt":"2012-08-23T16:28:09Z","lastAt":"2012-08-23T20:45:26Z","messageCount":3,"participants":["Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"197717","messageId":"CACBZZX6SVSn17nfSarQC1qgz-TxcDDn5tvHb0BkyW9S8DcLY4w@mail.gmail.com","threadId":"31333","inReplyTo":null,"subject":"Existing utility to track compiled files in another sister repository, for rollouts","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2012-08-23T16:28:09Z","receivedAt":"2012-08-23T16:28:09Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I'm planning on using Git for a deployment process where the steps are\nbasically:\n\n 1. You log into a deployment host, cd into software.git, do git pull\n 2. A tool runs \"make\" for you, creates a deployment-YYYYMMDD-HHMMSS tag\n 3. That make step will create a bunch of generated (text) files\n 4. Get a list of these with : git clean -dxfn\n 5. Copy those to to software-generated.git, removing any that we\ndidn't just create, adding any that are new\n 6. Commit that, tag it with generated-deployment-YYYYMMDD-HHMMSS\n 7. Push out both our generated software.git and\nsoftware-generated.git tag to our servers\n 8. git reset --hard both of those to our newly pushed out tags\n 9. Do git clean -dxf on software.git remove old generated files\n 10. Copy new generated files from generated-software.git to software.git\n 11. Restart our application to pick up the new software\n\nFor this I'll need to write some \"git snapshot-commit\" tool for #5 and\n#6 to commit whatever the current state of the directory is (with\nremoved/added files), and hack up something to do #9-#10.\n\nThis should all be relatively easy, I was just wondering if there was\nany prior art on this that I could use instead of hacking it up\nmyself.\n\nFor what it's worth the reason I'm using Git like this for deployment\nis that I'm converting things away from an rsync-based sync process\nthat's becoming increasingly slow since rsync with -c needs to\nrecursively checksum everything we're syncing out (which is quite a\nlot), since it's all text files Git can be really efficient in just\ntransferring deltas and quickly doing a hard reset to a new commit.\n"},{"id":"197725","messageId":"7vk3wp4hkh.fsf@alter.siamese.dyndns.org","threadId":"31333","inReplyTo":"CACBZZX6SVSn17nfSarQC1qgz-TxcDDn5tvHb0BkyW9S8DcLY4w@mail.gmail.com","subject":"Re: Existing utility to track compiled files in another sister repository, for rollouts","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-23T17:57:34Z","receivedAt":"2012-08-23T17:57:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n>  5. Copy those to to software-generated.git, removing any that we\n> didn't just create, adding any that are new\n>  6. Commit that, tag it with generated-deployment-YYYYMMDD-HHMMSS\n\n>  9. Do git clean -dxf on software.git remove old generated files\n>  10. Copy new generated files from generated-software.git to software.git\n>\n> For this I'll need to write some \"git snapshot-commit\" tool for #5 and\n> #6 to commit whatever the current state of the directory is (with\n> removed/added files), and hack up something to do #9-#10.\n>\n> This should all be relatively easy, I was just wondering if there was\n> any prior art on this that I could use instead of hacking it up\n> myself.\n\nFWIW, dodoc script in my todo branch was hacked up to respond to my\npush into k.org repository, pull 'master' into \"documentation build\nrepo\" at the k.org machine and check it out, build docs and then\n\"removing any that we didn't create, adding any that are new\" to\nupdate the html and man \"branches\", and push it back to the main\nrepository, so in that sense, it has all the logic that was needed\n(and more, as the rebuilt documentation from two revisions may only\ndiffer from version stamp asciidoc-to-html/man toolchain adds, which\nneed to be filtered out when updating html/man branches).\n\nThere no longer is post-update hook at k.org, so I have to run it\nmanually at the end of day's integration cycle, just before pushing\nthings out.\n\nI wrote it myself, not because I looked for suitable reusable code\nand didn't find any, but because I was too lazy to look for existing\ntools and evaluating them.\n"},{"id":"197734","messageId":"CACBZZX4RoPDODMgbz1T+wEMT5K4X157G5VYtXFp=e+j0C0hqiQ@mail.gmail.com","threadId":"31333","inReplyTo":"CACBZZX6SVSn17nfSarQC1qgz-TxcDDn5tvHb0BkyW9S8DcLY4w@mail.gmail.com","subject":"Re: Existing utility to track compiled files in another sister repository, for rollouts","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2012-08-23T20:45:26Z","receivedAt":"2012-08-23T20:45:26Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Thu, Aug 23, 2012 at 6:28 PM, Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> I'm planning on using Git for a deployment process where the steps are\n> basically:\n>\n>  1. You log into a deployment host, cd into software.git, do git pull\n>  2. A tool runs \"make\" for you, creates a deployment-YYYYMMDD-HHMMSS tag\n>  3. That make step will create a bunch of generated (text) files\n>  4. Get a list of these with : git clean -dxfn\n>  5. Copy those to to software-generated.git, removing any that we\n> didn't just create, adding any that are new\n>  6. Commit that, tag it with generated-deployment-YYYYMMDD-HHMMSS\n>  7. Push out both our generated software.git and\n> software-generated.git tag to our servers\n>  8. git reset --hard both of those to our newly pushed out tags\n>  9. Do git clean -dxf on software.git remove old generated files\n>  10. Copy new generated files from generated-software.git to software.git\n>  11. Restart our application to pick up the new software\n>\n> For this I'll need to write some \"git snapshot-commit\" tool for #5 and\n> #6 to commit whatever the current state of the directory is (with\n> removed/added files), and hack up something to do #9-#10.\n>\n> This should all be relatively easy, I was just wondering if there was\n> any prior art on this that I could use instead of hacking it up\n> myself.\n\nHere's a quick hack that does #4-6 but not #9-10 yet, although that\nwould be easy: https://gist.github.com/3440792\n\nSuggestions for improvements welcome, particularly whether there's a\nsimpler way to do this to nuke existing files in a repo and replace it\nwith new files all staged for commit:\n\n    # Go to the target repository, nuke anything already there\n    chdir $to_repository;\n    system \"git reset --hard\";\n    system \"git clean -dxf\";\n    system \"git ls-tree --name-only HEAD -z | xargs -0 rm -rf\";\n    system \"git add --update\"; # stage any removals\n\nFollowed by:\n\n    system \"tar xvf incoming.tar\";\n    system \"rm incoming.tar\";\n    system \"git add * .??* || :\"; # Might die if we empty the repo,\nTODO: make this use status -> add each file\n    system \"git commit -m'Bump copy from $from_repository to\n$to_repository' || :\"; # We might have nothing to change!\n"}]}