git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 05/16] vcs-svn: learn to maintain a sliding view of a file

From
David Barr <david.barr@cordelta.com>
Date
Mar 19, 2011, 07:20 UTC
Message-ID
<1300519254-20201-6-git-send-email-david.barr@cordelta.com>
In-Reply-To
<1300519254-20201-1-git-send-email-david.barr@cordelta.com>
From: Jonathan Nieder <jrnieder@gmail.com>

Each section of a Subversion-format delta only requires examining (and keeping in random-access memory) a small portion of the preimage. At any moment, this portion is starts at a certain file offset and has a well-defined length, and as the delta is applied, the portion moves from the beginning to the end of the preimage. Add a move_window function to keep track of this view into the preimage.

You can use it like this:
	buffer_init(f, NULL);
	struct sliding_view window = SLIDING_VIEW_INIT(f);
	move_window(&window, 3, 7);	/* (1) */
	move_window(&window, 5, 5);	/* (2) */
	move_window(&window, 12, 2);	/* (3) */
	strbuf_release(&window.buf);
	buffer_deinit(f);

The data structure is called sliding_view instead of _window to prevent confusion with svndiff0 Windows.

In this example, (1) reads 10 bytes and discards the first 3; (2) discards the first 2, which are not needed any more; and (3) skips 2 bytes and reads 2 new bytes to work with.

When move_window returns, the file position indicator is at position window->off + window->width and the data from positions window->off to the current file position are stored in window->buf.

This function performs only sequential access from the input file and never seeks, so it can be safely used on pipes and sockets.

On end-of-file, move_window silently reads less than the caller requested. On other errors, it prints a message and returns -1.

Helped-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: David Barr <david.barr@cordelta.com>
---
 Makefile                 |    5 ++-
 vcs-svn/LICENSE          |    2 +
 vcs-svn/sliding_window.c |   74 ++++++++++++++++++++++++++++++++++++++++++++++
 vcs-svn/sliding_window.h |   17 ++++++++++
 4 files changed, 96 insertions(+), 2 deletions(-)
 create mode 100644 vcs-svn/sliding_window.c
 create mode 100644 vcs-svn/sliding_window.h
diff --git a/Makefile b/Makefile
index 6165609..bac415d 100644
--- a/Makefile
+++ b/Makefile
@@ -1836,7 +1836,8 @@ endif
 XDIFF_OBJS = xdiff/xdiffi.o xdiff/xprepare.o xdiff/xutils.o xdiff/xemit.o \
 	xdiff/xmerge.o xdiff/xpatience.o
 VCSSVN_OBJS = vcs-svn/line_buffer.o vcs-svn/repo_tree.o \
-	vcs-svn/fast_export.o vcs-svn/svndump.o
+	vcs-svn/fast_export.o vcs-svn/svndump.o \
+	vcs-svn/sliding_window.o
 VCSSVN_TEST_OBJS = test-obj-pool.o test-line-buffer.o test-treap.o
 OBJECTS := $(GIT_OBJS) $(XDIFF_OBJS) $(VCSSVN_OBJS)
 
@@ -1962,7 +1963,7 @@ xdiff-interface.o $(XDIFF_OBJS): \
 
 $(VCSSVN_OBJS) $(VCSSVN_TEST_OBJS): $(LIB_H) \
 	vcs-svn/line_buffer.h vcs-svn/repo_tree.h vcs-svn/fast_export.h \
-	vcs-svn/svndump.h
+	vcs-svn/svndump.h vcs-svn/sliding_window.h
 
 test-svn-fe.o: vcs-svn/svndump.h
 endif
diff --git a/vcs-svn/LICENSE b/vcs-svn/LICENSE
index 533f585..eb91858 100644
--- a/vcs-svn/LICENSE
+++ b/vcs-svn/LICENSE
@@ -1,6 +1,8 @@
 Copyright (C) 2010 David Barr <david.barr@cordelta.com>.
 All rights reserved.
 
+Copyright (C) 2010 Jonathan Nieder <jrnieder@gmail.com>.
+
 Copyright (C) 2005 Stefan Hegny, hydrografix Consulting GmbH,
 Frankfurt/Main, Germany
 and others, see http://svn2cc.sarovar.org
diff --git a/vcs-svn/sliding_window.c b/vcs-svn/sliding_window.c
new file mode 100644
index 0000000..9ce399c
--- /dev/null
+++ b/vcs-svn/sliding_window.c
@@ -0,0 +1,74 @@
+/*
+ * Licensed under a two-clause BSD-style license.
+ * See LICENSE for details.
+ */
+
+#include "git-compat-util.h"
+#include "sliding_window.h"
+#include "line_buffer.h"
+#include "strbuf.h"
+
+static int input_error(struct line_buffer *file)
+{
+	if (!buffer_ferror(file))
+		return error("delta preimage ends early");
+	return error("cannot read delta preimage: %s", strerror(errno));
+}
+
+static int skip_or_whine(struct line_buffer *file, off_t gap)
+{
+	const off_t nread = buffer_skip_bytes(file, gap);
+	return nread == gap ? 0 : input_error(file);
+}
+
+static int read_to_fill_or_whine(struct line_buffer *file,
+				struct strbuf *buf, size_t width)
+{
+	buffer_read_binary(file, buf, width - buf->len);
+	return buf->len == width ? 0 : input_error(file);
+}
+
+static int check_overflow(off_t a, size_t b)
+{
+	if (b > maximum_signed_value_of_type(off_t))
+		return error("unrepresentable length in delta: "
+				"%"PRIuMAX" > OFF_MAX", (uintmax_t) b);
+	if (signed_add_overflows(a, (off_t) b))
+		return error("unrepresentable offset in delta: "
+				"%"PRIuMAX" + %"PRIuMAX" > OFF_MAX",
+				(uintmax_t) a, (uintmax_t) b);
+	return 0;
+}
+
+int move_window(struct sliding_view *view, off_t off, size_t width)
+{
+	off_t file_offset;
+	assert(view);
+	assert(view->width <= view->buf.len);
+	assert(!check_overflow(view->off, view->buf.len));
+
+	if (check_overflow(off, width))
+		return -1;
+	if (off < view->off || off + width < view->off + view->width)
+		return error("invalid delta: window slides left");
+
+	file_offset = view->off + view->buf.len;
+	if (off < file_offset) {
+		/* Move the overlapping region into place. */
+		strbuf_remove(&view->buf, 0, off - view->off);
+	} else {
+		/* Seek ahead to skip the gap. */
+		if (skip_or_whine(view->file, off - file_offset))
+			return -1;
+		strbuf_setlen(&view->buf, 0);
+	}
+
+	if (view->buf.len > width)
+		; /* Already read. */
+	else if (read_to_fill_or_whine(view->file, &view->buf, width))
+		return -1;
+
+	view->off = off;
+	view->width = width;
+	return 0;
+}
diff --git a/vcs-svn/sliding_window.h b/vcs-svn/sliding_window.h
new file mode 100644
index 0000000..ed0bfdd
--- /dev/null
+++ b/vcs-svn/sliding_window.h
@@ -0,0 +1,17 @@
+#ifndef SLIDING_WINDOW_H_
+#define SLIDING_WINDOW_H_
+
+#include "strbuf.h"
+
+struct sliding_view {
+	struct line_buffer *file;
+	off_t off;
+	size_t width;
+	struct strbuf buf;
+};
+
+#define SLIDING_VIEW_INIT(input)	{ (input), 0, 0, STRBUF_INIT }
+
+extern int move_window(struct sliding_view *view, off_t off, size_t width);
+
+#endif

-- 
1.7.3.2.846.gf4b062
Previous: David BarrNext: David Barr
Message 26 of 72 in “vcs-svn: purge obsolete data structures and code”
  1. David BarrMar 19, 2011
  2. 1/9 vcs-svn: pass paths through to fast-importDavid Barr, Mar 19, 2011
  3. Jonathan NiederMar 19, 2011
  4. 2/9 vcs-svn: avoid using ls command twiceDavid Barr, Mar 19, 2011
  5. Jonathan NiederMar 19, 2011
  6. 3/9 vcs-svn: implement perfect hash for node-prop keysDavid Barr, Mar 19, 2011
  7. Jonathan NiederMar 19, 2011
  8. 1/3 vcs-svn: implement perfect hash for node-prop keysDavid Barr, Mar 21, 2011
  9. 2/3 vcs-svn: implement perfect hash for top-level keysDavid Barr, Mar 21, 2011
  10. 3/3 vcs-svn: use switch rather than cascading ifsDavid Barr, Mar 21, 2011
  11. [PATCHv2] vcs-svn: use switch rather than cascading ifsDavid Barr, Mar 21, 2011
  12. 4/9 vcs-svn: implement perfect hash for top-level keysDavid Barr, Mar 19, 2011
  13. Jonathan NiederMar 19, 2011
  14. 5/9 vcs-svn: factor out usage of string_poolDavid Barr, Mar 19, 2011
  15. Jonathan NiederMar 19, 2011
  16. 6/9 vcs-svn: drop string_poolDavid Barr, Mar 19, 2011
  17. 7/9 vcs-svn: drop trp.hDavid Barr, Mar 19, 2011
  18. 8/9 vcs-svn: drop obj_pool.hDavid Barr, Mar 19, 2011
  19. 9/9 vcs-svn: use strchr to find RFC822 delimiterDavid Barr, Mar 19, 2011
  20. Jonathan NiederMar 19, 2011
  21. vcs-svn: integrate support for text deltasDavid Barr, Mar 19, 2011
  22. 01/16 vcs-svn: improve support for reading large filesDavid Barr, Mar 19, 2011
  23. 02/16 vcs-svn: make buffer_skip_bytes return length readDavid Barr, Mar 19, 2011
  24. 03/16 vcs-svn: make buffer_copy_bytes return length readDavid Barr, Mar 19, 2011
  25. 04/16 vcs-svn: improve reporting of input errorsDavid Barr, Mar 19, 2011
  26. 05/16 vcs-svn: learn to maintain a sliding view of a fileDavid Barr, Mar 19, 2011
  27. 06/16 vcs-svn: skeleton of an svn delta parserDavid Barr, Mar 19, 2011
  28. Jonathan NiederMar 28, 2011
  29. 07/16 vcs-svn: parse svndiff0 window headerDavid Barr, Mar 19, 2011
  30. 08/16 vcs-svn: read the preimage when applying deltasDavid Barr, Mar 19, 2011
  31. 09/16 vcs-svn: read inline data from deltasDavid Barr, Mar 19, 2011
  32. 10/16 vcs-svn: read instructions from deltasDavid Barr, Mar 19, 2011
  33. 11/16 vcs-svn: implement copyfrom_data delta instructionDavid Barr, Mar 19, 2011
  34. 12/16 vcs-svn: verify that deltas consume all inline dataDavid Barr, Mar 19, 2011
  35. 13/16 vcs-svn: let deltas use data from postimageDavid Barr, Mar 19, 2011
  36. 14/16 vcs-svn: let deltas use data from preimageDavid Barr, Mar 19, 2011
  37. 15/16 vcs-svn: microcleanup in svndiff0 window-reading codeDavid Barr, Mar 19, 2011
  38. 16/16 vcs-svn: implement text-delta handlingDavid Barr, Mar 19, 2011
  39. Jonathan NiederMar 28, 2011
  40. David BarrMar 28, 2011
  41. 00/11 vcs-svn: purge obsolete data structures and codeDavid Barr, Mar 21, 2011
  42. 01/11 vcs-svn: use strbuf for revision logDavid Barr, Mar 21, 2011
  43. 02/11 vcs-svn: pass paths through to fast-importDavid Barr, Mar 21, 2011
  44. 03/11 vcs-svn: avoid using ls command twiceDavid Barr, Mar 21, 2011
  45. 04/11 vcs-svn: implement perfect hash for node-prop keysDavid Barr, Mar 21, 2011
  46. 05/11 vcs-svn: implement perfect hash for top-level keysDavid Barr, Mar 21, 2011
  47. 06/11 vcs-svn: use switch rather than cascading ifsDavid Barr, Mar 21, 2011
  48. 07/11 vcs-svn: factor out usage of string_poolDavid Barr, Mar 21, 2011
  49. 08/11 vcs-svn: drop string_poolDavid Barr, Mar 21, 2011
  50. =??q?=5BPATCH=2009/11=5D=20vcs-svn=3A=20drop=20trp=2Eh?=David Barr, Mar 21, 2011
  51. 10/11 vcs-svn: drop obj_pool.hDavid Barr, Mar 21, 2011
  52. 11/11 vcs-svn: use strchr to find RFC822 delimiterDavid Barr, Mar 21, 2011
  53. [PULL svn-fe] vcs-svn: simplifications, error handling improvementsJonathan Nieder, Mar 23, 2011
  54. Junio C HamanoMar 23, 2011
  55. Junio C HamanoMar 23, 2011
  56. Jonathan NiederMar 26, 2011
  57. t0081-line-buffer.sh hangs (Re: [PULL svn-fe] vcs-svn: simplifications, error handling improvements)Jonathan Nieder, Mar 26, 2011
  58. David BarrMar 23, 2011
  59. fixup! vcs-svn: improve reporting of input errorsDavid Barr, Mar 24, 2011
  60. Jonathan NiederMar 25, 2011
  61. 0/4 vcs-svn: null bytes in propertiesJonathan Nieder, Mar 25, 2011
  62. 1/4 vcs-svn: make reading of properties binary-safeJonathan Nieder, Mar 25, 2011
  63. tbMar 28, 2011
  64. Jonathan NiederMar 28, 2011
  65. Torsten BögershausenMar 28, 2011
  66. Jonathan NiederMar 28, 2011
  67. 2/4 vcs-svn: remove buffer_read_stringJonathan Nieder, Mar 25, 2011
  68. 3/4 vcs-svn: avoid unnecessary copying of log message and authorJonathan Nieder, Mar 25, 2011
  69. 4/4 vcs-svn: handle log message with embedded null bytesJonathan Nieder, Mar 25, 2011
  70. Jonathan NiederMar 26, 2011
  71. Junio C HamanoMar 26, 2011
  72. vcs-svn: add missing cast to printf argumentJonathan Nieder, Mar 28, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.