{"thread":{"id":"18525","subject":"GSoC proposal: git mirror-sync","startedAt":"2009-03-25T04:00:32Z","lastAt":"2009-03-25T04:00:32Z","messageCount":1,"participants":["Andrew Wang"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"109307","messageId":"1c7233eb0903242100v406bda32h4803f54cd4c09ebb@mail.gmail.com","threadId":"18525","inReplyTo":null,"subject":"GSoC proposal: git mirror-sync","fromName":"Andrew Wang","fromEmail":"andrew@umbrant.com","sentAt":"2009-03-25T04:00:32Z","receivedAt":"2009-03-25T04:00:32Z","isPatch":false,"sender":{"key":"andrew@umbrant.com","avatar":null},"body":"Greetings,\n\nI'm applying for a Google Summer of Code project for git mirror-sync,\nmentioned on the Ideas wiki page. I've submitted my first draft to the\nGSoC website already, but if any kind mailing list members have any\ncomments or questions I'd be happy to take them. Thanks go out to Sam\nVilian, who already helped me with my app :)\n\nCheers,\nAndrew\n\n==================================\n\nProject Goals\n\nWhat is the goal of your project?\n\nThe goal is to implement the MirrorSync protocol\n(http://code.google.com/p/gittorrent/wiki/MirrorSync and\nhttp://tinyurl.com/c7j3m7) as a continuation of previous Google Summer\nof Code projects involving peer to peer distribution of Git\nrepositories to increase download speeds and decentralize\ndistribution. It's a refinement of the GitTorrent protocol\n(http://gittorrent.utsl.gen.nz/rfc.html), which in turn was based on\nthe popular BitTorrent protocol\n(http://www.bittorrent.org/beps/bep_0003.html). MirrorSync offers a\nsimplified version of GitTorrent more tailored to the design of Git\nand without all the unnecessary BitTorrent cruft.\n\nHow would you measure its success or failure?\n\nThe MirrorSync overview goes over the three parts of functionality\nthat need to be implemented. Completing all three would be highly\ndesirable, but each part is useful in its own right since they add\nmirror downloading functionality. However, the scope of the project\nseems such that all three could be completed in a single Summer of\nCode project, and that's what I'll aim for. There's going to be a\nsignificant amount of work on formalizing the protocol and talking\nwith Git developers, and that'll also be part of my work for the\nsummer.\n\nPart one is Mirror List. This lets a client hit a repository and get a\nlist of mirrors that the client could try downloading from, along with\nthe most recent update to the repository and the signing key for\nverifying updates.\n\nPart two is Mirror Notify. This is how a client can tell a repository\nthat the client has a copy of the repository, and is willing to act as\na mirror.\n\nPart three is Mirror Sync. This is where the actual exchange of\nrepository contents occurs. The peers start by comparing each other's\nlatest packed-refs files, and then start requesting desired packs from\neach other. This means new changes will quickly propagate through the\nnetwork, and since the objects are fragmented reproducibly, downloads\ncan be spread across many peers.\n\nDescribe your project in more detail.\n\nThe necessary steps to implementing all of MirrorSync have already\nbeen laid out in skeleton form by Sam Vilain\n(http://tinyurl.com/crdq9f). The three messages will be implemented\nMirror List first, followed by Mirror Notify and then Mirror Sync.\nThese are major milestones for progress, and more formal documentation\nand specifications on each part should emerge along the way.\n\nMy plan for all three parts is to write the bulk of the code in\nPython, since git-daemon can wrap the Python executables with\nexecl_git_cmd(). This means there are going to be additions to\ngit-daemon to accommodate this, but no big changes.\n\nMirror List requires modifications to git-tag to create the necessary\n\"push objects\".These are tags containing a packed-refs file in the\ncomments section, signed so a mirror can verify the state of the\nrepository. The server has to maintain a list of mirrors and which\nsigning key IDs are allowed to push back to which branches. The client\nhas to make use of this key info to verify push objects, ensuring that\nthe push objects are only changing the allowed refs. Adding keys to\nthe keyring can be done through prompting similarly to how it's\nhandled in SSH. Finally, git-fetch will be modified to have a\n\"--use-mirror\" flag to select from the list to download from a local\nmirror.\n\nMirror Notify has to handle maintaining the list of mirrors and\nresponding appropriately. The main repository should attempt to verify\nnotify requests by checking that the repo exists on the mirror.\nWriting the client commands for notifying the server should be pretty\neasy.\n\nMirror Sync is probably the hardest to implement, since it's the most\ncomplicated and requires the most investigation into Git internals. It\nmakes use of Mirror List to get a list of peers, and then it needs to\nstart advertising bundles it has and bundles it wants. Then the actual\ndownloading and uploading process happens, and this keeps happening\nuntil everyone is in sync.\n\nThe timeline for the three months would look something like this: one\nweek of research  and planning, three weeks for Mirror List, two weeks\nfor Mirror Notify, five weeks for Mirror Sync, and one week for\ncleanups and documentation. This is a total of 12 weeks, the time\nallotted for Summer of Code.\n\nInterfaces\n\nWhat parts of Git will you need to call?\n\nThere won't be much code reuse, since the MirrorSync additions are\nfairly separate from the rest of the Git codebase. The tag creation\nfacilities will be used in making \"push objects\", and the existing\nframework in git-daemon for handling fetch requests will be extended\nto handle the new MirrorSync requests.\n\nWhat parts of Git might you need to change?\n\nThere are two things that will be changed: git-tag and git-daemon.\ngit-tag needs to be changed to accommodate creating the \"push objects\"\nnecessary for peer communication, and the required additions will\nhopefully be minor. git-daemon needs to be modified to call out to the\nnew mirror-* functionality when it's hit with a request. Besides that,\nthree new commands are being added (mirror-list, mirror-notify,\nmirror-sync) so the Makefile will need to be modified.\n\nAbout You\n\nCan you list some prior projects that you have worked on?\n\nI haven't been involved in any open source projects much beyond bug\nreporting and triaging, for KDE and a few other projects. I have a\ncouple very minor patches to Basket Note Pads in their Git tree, a KDE\nproject written in Qt and C++ (http://basket.kde.org/). As with most\nopen source projects, communication within Basket is primarily through\nemail, and I've also been subscribed on and off to development mailing\nlists for KDE, Enlightenment, Gentoo, and now Git.\n\nI've done plenty of coding on my own projects though, the most\nrelevant being Python implementations of a BitTorrent tracker and a\nthreaded XML feed grabber and parser. Both of these were done on my\nown as self learning projects, using mainly the extensive standard\nPython libraries. Python is my goto language for projects big and\nsmall, and I'm pretty comfortable in it.\n\nBesides that, my professional background has been in websites, so I\nhave exposure to all types of web development technologies and related\nprogramming languages. My resume (http://www.linkedin.com/in/aawang)\nis the best place to go for a run down on that part of my skill set.\nI've also been producing copious amounts of Java code for school\nrelated projects; right now we're doing development with Lego NXT\nrobots with communication over Bluetooth with a WiiRemote.\n\nDo you have any prior Git experience? Have you started to get involved?\n\nI don't have any prior Git experience except as a user. I asked Sam\nVilain a bit about MirrorSync when I saw the Git Ideas page, and now\nI'm on the developer mailing list. I've also started browsing through\nthe source code, but I don't have any patches to my name right now.\n"}]}