{"thread":{"id":"31805","subject":"looking for suggestions for managing a tree of server configs","startedAt":"2012-10-13T21:20:59Z","lastAt":"2012-10-21T23:33:44Z","messageCount":7,"participants":["david@lang.hm","Junio C Hamano","Drew Northup"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"201089","messageId":"alpine.DEB.2.02.1210131413240.6253@asgard.lang.hm","threadId":"31805","inReplyTo":null,"subject":"looking for suggestions for managing a tree of server configs","fromName":"","fromEmail":"david@lang.hm","sentAt":"2012-10-13T21:20:59Z","receivedAt":"2012-10-13T21:20:59Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"I've got a directory tree that holds config data for all my servers. This \nconsists of one directory per server (which is updated periodically from \nwhat is currently configured on that server), plus higher level summary \nreports and similar information.\n\ntoday I have just a single git tree covering everything, and I make a \ncommit each time one of the per-server directories is updated, and again \nwhen the top-level stuff is created. This works, but it's not really that \ngood at letting me go back and see what happened when on a particular \nserver because of all the other activity going on in git log.\n\nAlso, currently all updates to this repository is serialized, but going \nforward I'm going to be adding some update processes that could result in \ndifferent per-server directories being updated at the same time (and so, \none may be fully populated with new data, while another has just had all \nthe old data deleted prior to the new data being written). I'm concerned \nabout possible issues with having these independant and overlapping update \nprocesses all working in a single git tree\n\nI do want to have everythng share one storage repository, because much of \nthe config on the servers is very standard, so I gain quite a bit by \nletting them delta off each other.\n\nIs what I'm currently doing the best way to do this? or should I do \nsomething like having a submodule per server, or is there some other, \nbetter way of doing this.? Are there any tools that I should be using \ninstead of continuing to use the scripts that I threw together several \nyears ago?\n\nDavid Lang\n"},{"id":"201117","messageId":"7vpq4l1x86.fsf@alter.siamese.dyndns.org","threadId":"31805","inReplyTo":"alpine.DEB.2.02.1210131413240.6253@asgard.lang.hm","subject":"Re: looking for suggestions for managing a tree of server configs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-10-14T04:41:45Z","receivedAt":"2012-10-14T04:41:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"david@lang.hm writes:\n\n> I've got a directory tree that holds config data for all my\n> servers. This consists of one directory per server (which is updated\n> periodically from what is currently configured on that server), plus\n> higher level summary reports and similar information.\n>\n> today I have just a single git tree covering everything, and I make a\n> commit each time one of the per-server directories is updated, and\n> again when the top-level stuff is created.\n\nIt is quite clear to me what you are keeping at the top-level files,\nbut if a large portion of the configuration for these servers are\nshared, it might not be a bad idea to have a canonical \"gold-master\"\nconfiguration branch, to which the shared updates are applied, with\na branch per server that forks from that canonical branch to keep\nthe machine specific tweaks as differences from the canonical stuff,\ninstead of having N subdirectories (one per machine).\n"},{"id":"201120","messageId":"alpine.DEB.2.02.1210132153040.6253@asgard.lang.hm","threadId":"31805","inReplyTo":"7vpq4l1x86.fsf@alter.siamese.dyndns.org","subject":"Re: looking for suggestions for managing a tree of server configs","fromName":"","fromEmail":"david@lang.hm","sentAt":"2012-10-14T04:57:18Z","receivedAt":"2012-10-14T04:57:18Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Sat, 13 Oct 2012, Junio C Hamano wrote:\n\n> david@lang.hm writes:\n>\n>> I've got a directory tree that holds config data for all my\n>> servers. This consists of one directory per server (which is updated\n>> periodically from what is currently configured on that server), plus\n>> higher level summary reports and similar information.\n>>\n>> today I have just a single git tree covering everything, and I make a\n>> commit each time one of the per-server directories is updated, and\n>> again when the top-level stuff is created.\n>\n> It is quite clear to me what you are keeping at the top-level files,\n> but if a large portion of the configuration for these servers are\n> shared, it might not be a bad idea to have a canonical \"gold-master\"\n> configuration branch, to which the shared updates are applied, with\n> a branch per server that forks from that canonical branch to keep\n> the machine specific tweaks as differences from the canonical stuff,\n> instead of having N subdirectories (one per machine).\n\nIn an ideal world yes, but right now these machines are updated by many \ndifferent tools (unforuntantly including 'vi'), so these directories \naren't the config to be pushed out to the boxes (i.e. what they should \nbe), it's instead an archived 'what is', the result of changes from all \nthe tools.\n\nThe systems are all built with a standard image, but the automation tools \nI do have tend to push identical files out to many of the systems (or \nfiles identical except for a couple of lines)\n\nDavid Lang\n"},{"id":"201603","messageId":"CAM9Z-nmHxyqnyq1fChhv7hP_awgsaO2FT1t29PAwrvZkaA-hgg@mail.gmail.com","threadId":"31805","inReplyTo":"alpine.DEB.2.02.1210132153040.6253@asgard.lang.hm","subject":"Re: looking for suggestions for managing a tree of server configs","fromName":"Drew Northup","fromEmail":"n1xim.email@gmail.com","sentAt":"2012-10-21T01:50:19Z","receivedAt":"2012-10-21T01:50:19Z","isPatch":false,"sender":{"key":"n1xim.email@gmail.com","avatar":null},"body":"On Sun, Oct 14, 2012 at 12:57 AM,  <david@lang.hm> wrote:\n> On Sat, 13 Oct 2012, Junio C Hamano wrote:\n>> david@lang.hm writes:\n>>> I've got a directory tree that holds config data for all my\n>>> servers. This consists of one directory per server (which is updated\n>>> periodically from what is currently configured on that server), plus\n>>> higher level summary reports and similar information.\n>>>\n>>> today I have just a single git tree covering everything, and I make a\n>>> commit each time one of the per-server directories is updated, and\n>>> again when the top-level stuff is created.\n>>\n>> It is quite clear to me what you are keeping at the top-level files,\n>> but if a large portion of the configuration for these servers are\n>> shared, it might not be a bad idea to have a canonical \"gold-master\"\n>> configuration branch, to which the shared updates are applied, with\n>> a branch per server that forks from that canonical branch to keep\n>> the machine specific tweaks as differences from the canonical stuff,\n>> instead of having N subdirectories (one per machine).\n>\n> In an ideal world yes, but right now these machines are updated by many\n> different tools (unforuntantly including 'vi'), so these directories aren't\n> the config to be pushed out to the boxes (i.e. what they should be), it's\n> instead an archived 'what is', the result of changes from all the tools.\n>\n> The systems are all built with a standard image, but the automation tools I\n> do have tend to push identical files out to many of the systems (or files\n> identical except for a couple of lines)\n\nDavid,\nIs there any particular reason you aren't using etckeeper?\n\n-- \n-Drew Northup\n--------------------------------------------------------------\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"201605","messageId":"alpine.DEB.2.02.1210201931130.31862@asgard.lang.hm","threadId":"31805","inReplyTo":"CAM9Z-nmHxyqnyq1fChhv7hP_awgsaO2FT1t29PAwrvZkaA-hgg@mail.gmail.com","subject":"Re: looking for suggestions for managing a tree of server configs","fromName":"","fromEmail":"david@lang.hm","sentAt":"2012-10-21T02:34:57Z","receivedAt":"2012-10-21T02:34:57Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Sat, 20 Oct 2012, Drew Northup wrote:\n\n> On Sun, Oct 14, 2012 at 12:57 AM,  <david@lang.hm> wrote:\n>> On Sat, 13 Oct 2012, Junio C Hamano wrote:\n>>> david@lang.hm writes:\n>>>> I've got a directory tree that holds config data for all my\n>>>> servers. This consists of one directory per server (which is updated\n>>>> periodically from what is currently configured on that server), plus\n>>>> higher level summary reports and similar information.\n>>>>\n>>>> today I have just a single git tree covering everything, and I make a\n>>>> commit each time one of the per-server directories is updated, and\n>>>> again when the top-level stuff is created.\n>>>\n>>> It is quite clear to me what you are keeping at the top-level files,\n>>> but if a large portion of the configuration for these servers are\n>>> shared, it might not be a bad idea to have a canonical \"gold-master\"\n>>> configuration branch, to which the shared updates are applied, with\n>>> a branch per server that forks from that canonical branch to keep\n>>> the machine specific tweaks as differences from the canonical stuff,\n>>> instead of having N subdirectories (one per machine).\n>>\n>> In an ideal world yes, but right now these machines are updated by many\n>> different tools (unforuntantly including 'vi'), so these directories aren't\n>> the config to be pushed out to the boxes (i.e. what they should be), it's\n>> instead an archived 'what is', the result of changes from all the tools.\n>>\n>> The systems are all built with a standard image, but the automation tools I\n>> do have tend to push identical files out to many of the systems (or files\n>> identical except for a couple of lines)\n>\n> David,\n> Is there any particular reason you aren't using etckeeper?\n\nnot really, I've thought of that as a tool for managing a single system. \nSome of the data in configs is sensitive (and much of it is not in /etc), \nbut I guess I should be able to work around those issues.\n\nI can e-mail 'patches' to the central server, but I'm then back to the \nsame question that I started out with.\n\nHow can I sanely organize all these different, but similar sets of files \non the central server?\n\nDavid Lang\n"},{"id":"201627","messageId":"CAM9Z-n=VGM4cjrZxMz=n5LwDsvuG6uwXymh4J3Gjq4mYJoGPcg@mail.gmail.com","threadId":"31805","inReplyTo":"alpine.DEB.2.02.1210201931130.31862@asgard.lang.hm","subject":"Re: looking for suggestions for managing a tree of server configs","fromName":"Drew Northup","fromEmail":"n1xim.email@gmail.com","sentAt":"2012-10-21T11:58:52Z","receivedAt":"2012-10-21T11:58:52Z","isPatch":false,"sender":{"key":"n1xim.email@gmail.com","avatar":null},"body":"On Sat, Oct 20, 2012 at 10:34 PM,  <david@lang.hm> wrote:\n> On Sat, 20 Oct 2012, Drew Northup wrote:\n>> On Sun, Oct 14, 2012 at 12:57 AM,  <david@lang.hm> wrote:\n>>> On Sat, 13 Oct 2012, Junio C Hamano wrote:\n>>>> david@lang.hm writes:\n>>>>>\n>>>>> today I have just a single git tree covering everything, and I make a\n>>>>> commit each time one of the per-server directories is updated, and\n>>>>> again when the top-level stuff is created.\n>>>>\n>>>> if a large portion of the configuration for these servers are\n>>>> shared, it might not be a bad idea to have a canonical \"gold-master\"\n>>>> configuration branch, to which the shared updates are applied, with\n>>>> a branch per server that forks from that canonical branch to keep\n>>>> the machine specific tweaks\n>>>\n>>> In an ideal world yes, but right now these machines are updated by many\n>>> different tools (unforuntantly including 'vi'), so\n>>> these directories aren't the config to be pushed out to the boxes, it's\n>>> instead an archived 'what is', the result of changes from all the tools.\n\nSo you need to save what is there before pulling changes from the\nmaster. That's no different from doing development work on an active\ncode base.\n\n>> David,\n>> Is there any particular reason you aren't using etckeeper?\n>>\n> not really, I've thought of that as a tool for managing a single system.\n> Some of the data in configs is sensitive (and much of it is not in /etc),\n> but I guess I should be able to work around those issues.\n\n.gitignore and symlinks have been employed at times to deal with that.\n\n> How can I sanely organize all these different, but similar sets of files on\n> the central server?\n\nThe reason I asked about etckeeper is that you could, with proper\nsecurity in place, push those up to branches in a shared repository\n(set up using gitolite, for instance) and not loose information about\nthe files in the process. This would allow you to make your changes on\none system using vi or whatever else is convenient, push the change up\nto the shared repo, cherry-pick it into the other branches (using a\nfull check-out of all of the branches someplace safe as a workspace),\nand pull that change out to the other systems.\n\nIf you are just looking to gather configuration information in the\nlarge and don't want to engage in any shared management schemes (which\nmay involve symlinks in seemingly odd places to /etc and such) you may\nwish to look at the System Configuration Collector [1] [2] which is a\nnicely organized tool designed specifically to gather just the\nimportant (and not highly confidential) information about common\nsoftware on a server and present it (and changes to it) to the admin\nin a sensible manner. It is outside of the \"Git Universe\" but it does\nwhat it sounds like you are doing now (if not what you wish to be\ndoing).\n\n(1) http://www.qnh.eu/scc/\n(2) http://sourceforge.net/projects/sysconfcollect/\n\n-- \n-Drew Northup\n--------------------------------------------------------------\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"201671","messageId":"alpine.DEB.2.02.1210211553420.31862@asgard.lang.hm","threadId":"31805","inReplyTo":"CAM9Z-n=VGM4cjrZxMz=n5LwDsvuG6uwXymh4J3Gjq4mYJoGPcg@mail.gmail.com","subject":"Re: looking for suggestions for managing a tree of server configs","fromName":"","fromEmail":"david@lang.hm","sentAt":"2012-10-21T23:33:44Z","receivedAt":"2012-10-21T23:33:44Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Sun, 21 Oct 2012, Drew Northup wrote:\n\n> On Sat, Oct 20, 2012 at 10:34 PM,  <david@lang.hm> wrote:\n>> On Sat, 20 Oct 2012, Drew Northup wrote:\n>>> On Sun, Oct 14, 2012 at 12:57 AM,  <david@lang.hm> wrote:\n>>>> On Sat, 13 Oct 2012, Junio C Hamano wrote:\n>>>>> david@lang.hm writes:\n>>>>>>\n>>>>>> today I have just a single git tree covering everything, and I make a\n>>>>>> commit each time one of the per-server directories is updated, and\n>>>>>> again when the top-level stuff is created.\n>>>>>\n>>>>> if a large portion of the configuration for these servers are\n>>>>> shared, it might not be a bad idea to have a canonical \"gold-master\"\n>>>>> configuration branch, to which the shared updates are applied, with\n>>>>> a branch per server that forks from that canonical branch to keep\n>>>>> the machine specific tweaks\n>>>>\n>>>> In an ideal world yes, but right now these machines are updated by many\n>>>> different tools (unforuntantly including 'vi'), so\n>>>> these directories aren't the config to be pushed out to the boxes, it's\n>>>> instead an archived 'what is', the result of changes from all the tools.\n>\n> So you need to save what is there before pulling changes from the\n> master. That's no different from doing development work on an active\n> code base.\n\nI think I've done a poor job of explaining my problem.\n\nI'm not looking for tips on how to manage the systems themselves, I'm \nlooking for suggestions on how to manage this data that I'm already \ngathering on this reporting server.\n\nI have the problem that different departments have their own (different) \npreferred tools for implementing changes. There are 6 different \ndepartments that need to be involved with a single system to build and \nmaintain it. Each department has their 'standard' way of doing things. At \nleast two of these departments are using different, central configuration \n(i.e. puppet like) tools.\n\nAs a result, I am not looking to pull changes from the central location. \nI'm just trying to gather information and be able to produce reports about \nthe systems (Including \"This is what all the different configs files on \nthis server were like at time X\"). I'm not using the distributed features \nof git at this time.\n\nI've got existing tools that do a very similar job to what it sounds like \nsysconfcollect does that gather the non-sensitive info from all my remote \nmachines and sends the data to my central server. These tools send an \nupdate whenever 'significant' changes are made, and in addition do a \nscheduled update to catch less significant changes.\n\nOn my central server I have the directory configs-current that then has a \nsubdirectory details/systemname for each system that contains all the \ninformation about htat system (populated by scripts that parse apart the \ndata mentioned above)\n\nIn other files and directories in configs-current I have lots of more \nglobal data and reports. This includes things like a report of every \ninterface on every machine, the IP address, does it have link, what speed \nis it at, etc.\n\nRight now I have one git tree for configs-current and each time I update a \ndetails/systemname tree I do\n\ngit add -a configs-current/details/$systemname\ngit commit -m'system update from $servername'\n\nthen when I run the summary scripts I do\n\ngit add -a configs-current\ngit commit -m'summary update'\n\nThis has been working for a few years\n\nHowever, trying to go back in history to find a change on one system is a \npain.\n\nRight now the updates accumulate until I manually trigger a processing \ncycle to update the files. I would like to make it so that the updates to \neach system's details/systemname directory is done automatically as the \ne-mail from that system arrives, and this could result in parallel \nupdates. I don't think that git will handle this well in one tree with the \nexisting process (different processes doing git add and git commits in \nparallel will end up mixing their data)\n\nAs one big tree, this has lots of commits (a couple hundred each update), \nand this is making it slow to try and track changes to a particular file \nin a particular system.\n\nI'm thinking that splitting the history tracking per-server should make \neverything faster.\n\nI'm wondering if I should do a subproject for each details/systemname \ndirectory, or if there is something else I can do to make this tracking of \nthe data better.\n\nDoing a single repository with lots of branches doesn't seem like it would \nwork as I need to get at the data from all the branches at the same time. \nI guess I could do something with branches on one repository, with a \ndifferent worktree for each system, but that seems a bit fragile (one \ncommand with the wrong environment variables and it coudl really tangle \nthings up)\n\nDavid Lang\n"}]}