{"thread":{"id":"12067","subject":"[offtopic?] xdelta patch format wrapper","startedAt":"2008-02-13T01:53:14Z","lastAt":"2008-02-13T17:53:11Z","messageCount":7,"participants":["Martin Langhoff","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"68595","messageId":"47B24D8A.5090703@catalyst.net.nz","threadId":"12067","inReplyTo":null,"subject":"[offtopic?] xdelta patch format wrapper","fromName":"Martin Langhoff","fromEmail":"martin@catalyst.net.nz","sentAt":"2008-02-13T01:53:14Z","receivedAt":"2008-02-13T01:53:14Z","isPatch":false,"sender":{"key":"martin@laptop.org","avatar":null},"body":"This is somewhat offtopic but this list is the best-informed crowd on\ndiff/xdelta matters so here I am, abusing your attention...\n\nI am working on an \"incremental content\" feature for Moodle - and my\nplan is to serve pre-computed \"patchfiles\" based on the xdelta utility\nto the client systems.\n\nNow, I need to provide a wrapper that concats the deltas from the xdelta\nutility with a more verbose header - akin to the header in git's unified\ndiff output. This is because the xdelta utility only handles one file\ndelta at a time - the file is a pure delta (prefaced with a SHA1 of the\nfile it applies to, IIRC).\n\n(From xdelta I like that it's one-way-only, and compressed internally --\nthus saving a lot of space on large changes. There are small statically\nlinked binaries for windows, osx and linux. I did consider using git's\nown diffs, but it involves significantly more work, and portability to\nWin32 is still green for the wide distribution this project is expecting.)\n\nSo my question is what is a good format for the header? My thinking sofar:\n\n - have a prefix to scan for, such as \"xdelta\" at the beginning of\n   the file, or after a newline/whitespace\n\n - keep the <fromsha1> <tosha1> line\n\n - \\0 delimited filenames\n\n - filenames as ambiguous bag'o'bytes or utf-8?\n   (should we have another flamewar on this? ;-) )\n\n - keep file modes and perhaps support copy/move headers\n\n - keep a/ b/ prefixes?\n\n - last line in the header is length: <length-in-bytes>, followed by\n   a newline and the xdelta itself\n\n - one or more newlines follow the end of the xdelta if there is another\n   header coming\n\nSomething along the lines of\n\nxdelta d065883..74cd8e5 100644\na/foo.zip\\0\nb/foo.zip\\0\nlength 1024\n<1024 bytes of xdelta data>\n\nxdelta d065883..74cd8e5 100644\na/bar.zip\\0\nb/bar.zip\\0\nlength 92312\n\n<92312 bytes xdelta data>EOF\n\n  ---\n\nWould the above work as a reasonably solid patch header format? Is there\nsomething else I should be using instead of rolling my own? If there\nisn't a more suitable tool,  and I'm hoping to come up with something\nunambiguous and reliable that is generally useful.\n\n[ I have to confess, it came as a bit of a surprise that xdelta didn't\nsupport this out of the box. Haven't seen any existing wrapper for it\nthat covers this either. ]\n\nAs a kludge I could tar both sides and xdelta across the tars, but it is\nwasteful, and an xdelta-based diff/patch that can handle multiple files\ndoes seem useful. And I don't know how stable the output of tar is (wrt\nfile ordering for example).\n\nTIA for any feedback :-)\n\n\nm\n-- \n"},{"id":"68597","messageId":"7vy79py1it.fsf@gitster.siamese.dyndns.org","threadId":"12067","inReplyTo":"47B24D8A.5090703@catalyst.net.nz","subject":"Re: [offtopic?] xdelta patch format wrapper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-02-13T03:32:26Z","receivedAt":"2008-02-13T03:32:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff <martin@catalyst.net.nz> writes:\n\n> So my question is what is a good format for the header? My thinking sofar:\n>\n>  - have a prefix to scan for, such as \"xdelta\" at the beginning of\n>    the file, or after a newline/whitespace\n>\n>  - keep the <fromsha1> <tosha1> line\n>\n>  - \\0 delimited filenames\n>\n>  - filenames as ambiguous bag'o'bytes or utf-8?\n>    (should we have another flamewar on this? ;-) )\n>\n>  - keep file modes and perhaps support copy/move headers\n>\n>  - keep a/ b/ prefixes?\n>\n>  - last line in the header is length: <length-in-bytes>, followed by\n>    a newline and the xdelta itself\n>\n>  - one or more newlines follow the end of the xdelta if there is another\n>    header coming\n\nI am lost as to your objective because you seem to be keeping a\nwhole LOT more than I would have imagined for a specialized\npurpose file format.\n\nIf you want to reuse that much of git, maybe our binary patch\nformat is good enough for you?  We always produce two xdelta so\nthat we can apply in reverse, but it is Ok to add a one-way\noption.\n"},{"id":"68598","messageId":"47B26830.6090501@catalyst.net.nz","threadId":"12067","inReplyTo":"7vy79py1it.fsf@gitster.siamese.dyndns.org","subject":"Re: [offtopic?] xdelta patch format wrapper","fromName":"Martin Langhoff","fromEmail":"martin@catalyst.net.nz","sentAt":"2008-02-13T03:46:56Z","receivedAt":"2008-02-13T03:46:56Z","isPatch":false,"sender":{"key":"martin@laptop.org","avatar":null},"body":"Junio C Hamano wrote:\n> I am lost as to your objective because you seem to be keeping a\n> whole LOT more than I would have imagined for a specialized\n> purpose file format.\n\nMy source files are 2 zipfiles that I know contain 1 xml file, and then\nmay contain any arbitrary files. As a specialised file format is a\npretty general case ;-) Because of compression, xdeltas of the zipfiles\naren't good. So what I want to do is to diff the 2 unzipped directories\n- nothing git-specific, I could use diff -urN.\n\nGit diff *is* better in that it handles binary files, but we pay a\nsizable cost in being reversible.\n\nSo I am thinking of doing is writing a wrapper that does the equivalent\nof the \"urN\" flags to diff, but uses xdelta as the diffing algorithm.\n\nAs my case is rather general I suspect I'm better off biting the bullet\nand writing something generally useful - it doesn't take that much more\neffort and if it ends up being popular, I'll have some help with its\nmaintenance ;-)\n\nIn other words, I'm trolling for peer review to make sure the tool is\nsane, and will be useful to others ;-)\n\n> If you want to reuse that much of git\n\nI don't think I'll use *any* git code at all for the time being. If it\nwas trivial to produce a statically compiled git-diff.exe and\ngit-apply-patch.exe that work without funny dependencies on any windows\nbox then I would. Don't think any of the windows ports of git are there\n(even though they are excellent!).\n\ncheers,\n\n\n\nm\n-- \n-----------------------------------------------------------------------\nMartin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington\nWEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St\nNZ: +64(4)916-7224    MOB: +64(21)364-017    UK: 0845 868 5733 ext 7224\n      Make things as simple as possible, but no simpler - Einstein\n-----------------------------------------------------------------------\n"},{"id":"68599","messageId":"7vprv1y0e2.fsf@gitster.siamese.dyndns.org","threadId":"12067","inReplyTo":"47B26830.6090501@catalyst.net.nz","subject":"Re: [offtopic?] xdelta patch format wrapper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-02-13T03:56:53Z","receivedAt":"2008-02-13T03:56:53Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff <martin@catalyst.net.nz> writes:\n\n> Junio C Hamano wrote:\n>> I am lost as to your objective because you seem to be keeping a\n>> whole LOT more than I would have imagined for a specialized\n>> purpose file format.\n>\n> My source files are 2 zipfiles that I know contain 1 xml file, and then\n> may contain any arbitrary files. As a specialised file format is a\n> pretty general case ;-) Because of compression, xdeltas of the zipfiles\n> aren't good. So what I want to do is to diff the 2 unzipped directories\n> - nothing git-specific, I could use diff -urN.\n>\n> Git diff *is* better in that it handles binary files, but we pay a\n> sizable cost in being reversible.\n\nDid I forget to say that I am Ok with --oneway option?\n\nIn fact, we started as oneway but we _fixed_ it to make it\nreversible soon after the initial version ;-)  So \"git apply\"\nstill can grok oneway format.\n"},{"id":"68600","messageId":"47B26E60.70005@catalyst.net.nz","threadId":"12067","inReplyTo":"7vy79py1it.fsf@gitster.siamese.dyndns.org","subject":"Re: [offtopic?] xdelta patch format wrapper","fromName":"Martin Langhoff","fromEmail":"martin@catalyst.net.nz","sentAt":"2008-02-13T04:13:20Z","receivedAt":"2008-02-13T04:13:20Z","isPatch":false,"sender":{"key":"martin@laptop.org","avatar":null},"body":"Junio C Hamano wrote:\n> If you want to reuse that much of git\n\nWondering about the confusion over this. When I talk about using xdelta,\nit's not the implementation in git. I intend to ship this xdelta.exe\nhttp://evanjones.ca/software/xdelta-win32.html (for Windows users at\nleast!).\n\nWhat I am sounding out is writing a wrapper written in PHP (I'd write it\nin Perl, but we're already shipping the PHP interpreter) that does all\nthe parsing of the file, splits out the actual \"xdelta\" blobs and calls\nxdelta.exe to apply them to the relevant files.\n\nSomeone more talented than me would write it in perfectly portable C so\nthat on day one works on Win32, OSX, unices and linuces. I can't so I'll\nlook like a wimp but I'll deliver something workable ;-) But there's no\nreason the PHP or Perl implementation can't be considered a working\nprototype for a subsequent C version.\n\nSpecially if the file format makes sense. And we've been complaining\nabout problems and ambiguities in the unified diff header. So... I'll\nrephrase my question\n\n   \"What would a unified diff header that didn't suck look like?\"\n\n(Ah, can't find the threads where the ambiguities of diff headers were\ndiscussed. Alas, the Google Gods aren't with me today.)\n\ncheers,\n\n\n\nm\n-- \n"},{"id":"68634","messageId":"alpine.LSU.1.00.0802131128040.30505@racer.site","threadId":"12067","inReplyTo":"47B26830.6090501@catalyst.net.nz","subject":"Re: [offtopic?] xdelta patch format wrapper","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-02-13T11:33:51Z","receivedAt":"2008-02-13T11:33:51Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 13 Feb 2008, Martin Langhoff wrote:\n\n> I don't think I'll use *any* git code at all for the time being. If it\n> was trivial to produce a statically compiled git-diff.exe and\n> git-apply-patch.exe that work without funny dependencies on any windows\n> box then I would.\n\nIt is trivial.\n\nExcept that we do not have any git-apply-patch.exe.  Maybe you meant \ngit-apply.exe?\n\n> Don't think any of the windows ports of git are there (even though they \n> are excellent!).\n\nHow can you say that they are excellent, and then say they are not there \nyet?\n\nFWIW I just checked.  In msysGit, git-apply.exe and git-diff.exe are \nidentical (no mystery there: they are both builtins), and weigh in with \n2893142 bytes.\n\nIf you're serious about wanting something reliable, quick, but smaller \nthan that, it should be _trivial_ to cut down.  For example, a simple \n\"strip git-diff.exe\" brings it down to 821248 bytes.\n\nAnd that's without removing all the other builtins, which would be \ntrivial, too (just cull \"struct cmd_struct commands\" in git.c, and \n\"BUILT_INS\" and \"BUILTIN_OBJS\" in the Makefile).\n\nHth,\nDscho\n"},{"id":"68662","messageId":"47B32E87.5080403@catalyst.net.nz","threadId":"12067","inReplyTo":"alpine.LSU.1.00.0802131128040.30505@racer.site","subject":"Re: [offtopic?] xdelta patch format wrapper","fromName":"Martin Langhoff","fromEmail":"martin@catalyst.net.nz","sentAt":"2008-02-13T17:53:11Z","receivedAt":"2008-02-13T17:53:11Z","isPatch":false,"sender":{"key":"martin@laptop.org","avatar":null},"body":"Johannes Schindelin wrote:\n> > FWIW I just checked.  In msysGit, git-apply.exe and git-diff.exe are\n> > identical (no mystery there: they are both builtins), and weigh in with\n> > 2893142 bytes.\n\nHmmm. I thought they depended on msys infrastructure. Can I trivially\ncompile a statically linked git-diff.exe and git-apply.exe and expect\nthem to just work? How large would they be then?\n\nmsysGIT is *excellent* to get a full GIT install for development\npurposes. The requirements in this case are different...\n\ncheers,\n\n\n\nm\n-- \n-----------------------------------------------------------------------\nMartin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington\nWEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St\nNZ: +64(4)916-7224    MOB: +64(21)364-017    UK: 0845 868 5733 ext 7224\n      Make things as simple as possible, but no simpler - Einstein\n-----------------------------------------------------------------------\n"}]}