git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 3/7] bulk-checkin: refactor deflate routine to accept a `bulk_checkin_source`

From
Taylor Blau <me@ttaylorr.com>
Date
Oct 19, 2023, 17:28 UTC
Message-ID
<d5bbd7810ee8f90be4cf6028f6420f52d72d24d0.1697736516.git.me@ttaylorr.com>
In-Reply-To
<cover.1697736516.git.me@ttaylorr.com>

Prepare for a future change where we will want to use a routine very similar to the existing `deflate_blob_to_pack()` but over arbitrary sources (i.e. either open file-descriptors, or a location in memory).

Extract out a common "deflate_obj_to_pack()" routine that acts on a bulk_checkin_source, instead of a (int, size_t) pair. Then rewrite `deflate_blob_to_pack()` in terms of it.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 bulk-checkin.c | 52 ++++++++++++++++++++++++++++++--------------------
 1 file changed, 31 insertions(+), 21 deletions(-)
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 7e6b52112e..28bc8d5ab4 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -285,30 +285,23 @@ static void prepare_to_stream(struct bulk_checkin_packfile *state,
 		die_errno("unable to write pack header");
 }
 
-static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,
-				struct object_id *result_oid,
-				int fd, size_t size,
-				const char *path, unsigned flags)
+
+static int deflate_obj_to_pack(struct bulk_checkin_packfile *state,
+			       struct object_id *result_oid,
+			       struct bulk_checkin_source *source,
+			       enum object_type type,
+			       off_t seekback,
+			       unsigned flags)
 {
-	off_t seekback, already_hashed_to;
+	off_t already_hashed_to = 0;
 	git_hash_ctx ctx;
 	unsigned char obuf[16384];
 	unsigned header_len;
 	struct hashfile_checkpoint checkpoint = {0};
 	struct pack_idx_entry *idx = NULL;
-	struct bulk_checkin_source source = {
-		.type = SOURCE_FILE,
-		.fd = fd,
-		.size = size,
-		.path = path,
-	};
 
-	seekback = lseek(fd, 0, SEEK_CUR);
-	if (seekback == (off_t) -1)
-		return error("cannot find the current offset");
-
-	header_len = format_object_header((char *)obuf, sizeof(obuf),
-					  OBJ_BLOB, size);
+	header_len = format_object_header((char *)obuf, sizeof(obuf), type,
+					  source->size);
 	the_hash_algo->init_fn(&ctx);
 	the_hash_algo->update_fn(&ctx, obuf, header_len);
 	the_hash_algo->init_fn(&checkpoint.ctx);
@@ -317,8 +310,6 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,
 	if ((flags & HASH_WRITE_OBJECT) != 0)
 		CALLOC_ARRAY(idx, 1);
 
-	already_hashed_to = 0;
-
 	while (1) {
 		prepare_to_stream(state, flags);
 		if (idx) {
@@ -327,7 +318,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,
 			crc32_begin(state->f);
 		}
 		if (!stream_obj_to_pack(state, &ctx, &already_hashed_to,
-					&source, OBJ_BLOB, flags))
+					source, type, flags))
 			break;
 		/*
 		 * Writing this object to the current pack will make
@@ -339,7 +330,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,
 		hashfile_truncate(state->f, &checkpoint);
 		state->offset = checkpoint.offset;
 		flush_bulk_checkin_packfile(state);
-		if (bulk_checkin_source_seek_to(&source, seekback) == (off_t)-1)
+		if (bulk_checkin_source_seek_to(source, seekback) == (off_t)-1)
 			return error("cannot seek back");
 	}
 	the_hash_algo->final_oid_fn(result_oid, &ctx);
@@ -361,6 +352,25 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,
 	return 0;
 }
 
+static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,
+				struct object_id *result_oid,
+				int fd, size_t size,
+				const char *path, unsigned flags)
+{
+	struct bulk_checkin_source source = {
+		.type = SOURCE_FILE,
+		.fd = fd,
+		.size = size,
+		.path = path,
+	};
+	off_t seekback = lseek(fd, 0, SEEK_CUR);
+	if (seekback == (off_t) -1)
+		return error("cannot find the current offset");
+
+	return deflate_obj_to_pack(state, result_oid, &source, OBJ_BLOB,
+				   seekback, flags);
+}
+
 void prepare_loose_object_bulk_checkin(void)
 {
 	/*
-- 
2.42.0.405.g86fe3250c2
Previous: Taylor BlauNext: Taylor Blau
Message 6 of 63 in “merge-ort: implement support for packing objects together”
  1. 0/7 merge-ort: implement support for packing objects togetherTaylor Blau, Oct 19, 2023
  2. 1/7 bulk-checkin: extract abstract `bulk_checkin_source`Taylor Blau, Oct 19, 2023
  3. Jeff KingOct 20, 2023
  4. Junio C HamanoOct 20, 2023
  5. 2/7 bulk-checkin: generify `stream_blob_to_pack()` for arbitrary typesTaylor Blau, Oct 19, 2023
  6. 3/7 bulk-checkin: refactor deflate routine to accept a `bulk_checkin_source`Taylor Blau, Oct 19, 2023
  7. 4/7 bulk-checkin: implement `SOURCE_INCORE` mode for `bulk_checkin_source`Taylor Blau, Oct 19, 2023
  8. Patrick SteinhardtOct 23, 2023
  9. Jeff KingOct 23, 2023
  10. Patrick SteinhardtOct 24, 2023
  11. Junio C HamanoOct 24, 2023
  12. 5/7 bulk-checkin: introduce `index_blob_bulk_checkin_incore()`Taylor Blau, Oct 19, 2023
  13. 6/7 bulk-checkin: introduce `index_tree_bulk_checkin_incore()`Taylor Blau, Oct 19, 2023
  14. 7/7 builtin/merge-tree.c: implement support for `--write-pack`Taylor Blau, Oct 19, 2023
  15. Junio C HamanoOct 19, 2023
  16. Jeff KingOct 20, 2023
  17. Junio C HamanoOct 20, 2023
  18. Patrick SteinhardtOct 23, 2023
  19. 0/5 merge-ort: implement support for packing objects togetherTaylor Blau, Oct 23, 2023
  20. 1/5 bulk-checkin: extract abstract `bulk_checkin_source`Taylor Blau, Oct 23, 2023
  21. Jeff KingOct 25, 2023
  22. Taylor BlauOct 25, 2023
  23. Junio C HamanoOct 27, 2023
  24. 2/5 bulk-checkin: generify `stream_blob_to_pack()` for arbitrary typesTaylor Blau, Oct 23, 2023
  25. 3/5 bulk-checkin: introduce `index_blob_bulk_checkin_incore()`Taylor Blau, Oct 23, 2023
  26. Patrick SteinhardtOct 25, 2023
  27. Taylor BlauOct 25, 2023
  28. Eric SunshineOct 25, 2023
  29. Patrick SteinhardtOct 26, 2023
  30. Elijah NewrenNov 11, 2023
  31. 4/5 bulk-checkin: introduce `index_tree_bulk_checkin_incore()`Taylor Blau, Oct 23, 2023
  32. 5/5 builtin/merge-tree.c: implement support for `--write-pack`Taylor Blau, Oct 23, 2023
  33. Patrick SteinhardtOct 25, 2023
  34. Taylor BlauOct 25, 2023
  35. Elijah NewrenNov 10, 2023
  36. Junio C HamanoNov 11, 2023
  37. Taylor BlauNov 11, 2023
  38. Taylor BlauNov 11, 2023
  39. Jeff KingNov 13, 2023
  40. Junio C HamanoNov 14, 2023
  41. Elijah NewrenNov 14, 2023
  42. Jeff KingNov 14, 2023
  43. Elijah NewrenNov 14, 2023
  44. Jeff KingNov 13, 2023
  45. Taylor BlauNov 13, 2023
  46. Elijah NewrenNov 14, 2023
  47. Jeff KingNov 14, 2023
  48. Jeff KingNov 14, 2023
  49. Junio C HamanoOct 23, 2023
  50. Johannes SchindelinNov 6, 2023
  51. Junio C HamanoNov 6, 2023
  52. Jeff KingNov 7, 2023
  53. Taylor BlauNov 7, 2023
  54. 0/3 replay: implement support for writing new objects to a packTaylor Blau, Nov 7, 2023
  55. 1/3 merge-ort.c: finalize ODB transactions after each stepTaylor Blau, Nov 7, 2023
  56. Elijah NewrenNov 11, 2023
  57. 2/3 tmp-objdir: introduce `tmp_objdir_repack()`Taylor Blau, Nov 7, 2023
  58. Patrick SteinhardtNov 8, 2023
  59. Taylor BlauNov 9, 2023
  60. 3/3 builtin/replay.c: introduce `--write-pack`Taylor Blau, Nov 7, 2023
  61. Elijah NewrenNov 11, 2023
  62. Elijah NewrenNov 11, 2023
  63. Patrick SteinhardtOct 25, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.