git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v7 6/7] git-p4: add support for large file systems

From
Luke Diamand <luke@diamand.org>
Date
Sep 25, 2015, 09:09 UTC
Message-ID
<CAE5ih7890uN-5uP9YkwqGi8AV2P+s=Heban6fvbdJ4DG-WrurA@mail.gmail.com>
In-Reply-To
<1442875273-48610-7-git-send-email-larsxschneider@gmail.com>
One tiny comment, otherwise this looks good to me.
Can we use test_path_is_missing in place of !test_path_is_file ?
On 21 September 2015 at 23:41,  <larsxschneider@gmail.com> wrote:
Show 59 quoted lines
> From: Lars Schneider <larsxschneider@gmail.com>
>
> Perforce repositories can contain large (binary) files. Migrating these
> repositories to Git generates very large local clones. External storage
> systems such as Git LFS [1], Git Fat [2], Git Media [3], git-annex [4]
> try to address this problem.
>
> Add a generic mechanism to detect large files based on extension,
> uncompressed size, and/or compressed size.
>
> [1] https://git-lfs.github.com/
> [2] https://github.com/jedbrown/git-fat
> [3] https://github.com/alebedev/git-media
> [4] https://git-annex.branchable.com/
>
>
> diff --git a/t/t9823-git-p4-mock-lfs.sh b/t/t9823-git-p4-mock-lfs.sh
> new file mode 100755
> index 0000000..5684ee3
> --- /dev/null
> +++ b/t/t9823-git-p4-mock-lfs.sh
> @@ -0,0 +1,170 @@
> +#!/bin/sh
> +
> +test_description='Clone repositories and store files in Mock LFS'
> +
> +. ./lib-git-p4.sh
> +
> +test_file_is_not_in_mock_lfs () {
> +       FILE="$1" &&
> +       CONTENT="$2" &&
> +       echo "$CONTENT" >expect_content &&
> +       test_path_is_file "$FILE" &&
> +       test_cmp expect_content "$FILE"
> +}
> +
> +test_file_is_in_mock_lfs () {
> +       FILE="$1" &&
> +       CONTENT="$2" &&
> +       LOCAL_STORAGE=".git/mock-storage/local/$CONTENT" &&
> +       SERVER_STORAGE=".git/mock-storage/remote/$CONTENT" &&
> +       echo "pointer-$CONTENT" >expect_pointer &&
> +       echo "$CONTENT" >expect_content &&
> +       test_path_is_file "$FILE" &&
> +       test_path_is_file "$LOCAL_STORAGE" &&
> +       test_path_is_file "$SERVER_STORAGE" &&
> +       test_cmp expect_pointer "$FILE" &&
> +       test_cmp expect_content "$LOCAL_STORAGE" &&
> +       test_cmp expect_content "$SERVER_STORAGE"
> +}
> +
> +test_file_is_deleted_in_mock_lfs () {
> +       FILE="$1" &&
> +       CONTENT="$2" &&
> +       LOCAL_STORAGE=".git/mock-storage/local/$CONTENT" &&
> +       SERVER_STORAGE=".git/mock-storage/remote/$CONTENT" &&
> +       echo "pointer-$CONTENT" >expect_pointer &&
> +       echo "$CONTENT" >expect_content &&
> +       ! test_path_is_file "$FILE" &&
Perhaps use test_path_is_missing instead here?
Show 136 quoted lines
> +       test_path_is_file "$LOCAL_STORAGE" &&
> +       test_path_is_file "$SERVER_STORAGE" &&
> +       test_cmp expect_content "$LOCAL_STORAGE" &&
> +       test_cmp expect_content "$SERVER_STORAGE"
> +}
> +
> +test_file_count_in_dir () {
> +       DIR="$1" &&
> +       EXPECTED_COUNT="$2" &&
> +       find "$DIR" -type f >actual &&
> +       test_line_count = $EXPECTED_COUNT actual
> +}
> +
> +test_expect_success 'start p4d' '
> +       start_p4d
> +'
> +
> +test_expect_success 'Create repo with binary files' '
> +       client_view "//depot/... //client/..." &&
> +       (
> +               cd "$cli" &&
> +
> +               echo "content 1 txt 23 bytes" >file1.txt &&
> +               p4 add file1.txt &&
> +               echo "content 2-3 bin 25 bytes" >file2.dat &&
> +               p4 add file2.dat &&
> +               p4 submit -d "Add text and binary file" &&
> +
> +               mkdir "path with spaces" &&
> +               echo "content 2-3 bin 25 bytes" >"path with spaces/file3.bin" &&
> +               p4 add "path with spaces/file3.bin" &&
> +               p4 submit -d "Add another binary file with same content and spaces in path" &&
> +
> +               echo "content 4 bin 26 bytes XX" >file4.bin &&
> +               p4 add file4.bin &&
> +               p4 submit -d "Add another binary file with different content"
> +       )
> +'
> +
> +test_expect_success 'Store files in Mock LFS based on size (>24 bytes)' '
> +       client_view "//depot/... //client/..." &&
> +       test_when_finished cleanup_git &&
> +       (
> +               cd "$git" &&
> +               git init . &&
> +               git config git-p4.useClientSpec true &&
> +               git config git-p4.largeFileSystem MockLFS &&
> +               git config git-p4.largeFileThreshold 24 &&
> +               git config git-p4.pushLargeFiles True &&
> +               git p4 clone --destination="$git" //depot@all &&
> +
> +               test_file_is_not_in_mock_lfs file1.txt "content 1 txt 23 bytes" &&
> +               test_file_is_in_mock_lfs file2.dat "content 2-3 bin 25 bytes" &&
> +               test_file_is_in_mock_lfs "path with spaces/file3.bin" "content 2-3 bin 25 bytes" &&
> +               test_file_is_in_mock_lfs file4.bin "content 4 bin 26 bytes XX" &&
> +
> +               test_file_count_in_dir ".git/mock-storage/local" 2 &&
> +               test_file_count_in_dir ".git/mock-storage/remote" 2
> +       )
> +'
> +
> +test_expect_success 'Store files in Mock LFS based on extension (dat)' '
> +       client_view "//depot/... //client/..." &&
> +       test_when_finished cleanup_git &&
> +       (
> +               cd "$git" &&
> +               git init . &&
> +               git config git-p4.useClientSpec true &&
> +               git config git-p4.largeFileSystem MockLFS &&
> +               git config git-p4.largeFileExtensions dat &&
> +               git config git-p4.pushLargeFiles True &&
> +               git p4 clone --destination="$git" //depot@all &&
> +
> +               test_file_is_not_in_mock_lfs file1.txt "content 1 txt 23 bytes" &&
> +               test_file_is_in_mock_lfs file2.dat "content 2-3 bin 25 bytes" &&
> +               test_file_is_not_in_mock_lfs "path with spaces/file3.bin" "content 2-3 bin 25 bytes" &&
> +               test_file_is_not_in_mock_lfs file4.bin "content 4 bin 26 bytes XX" &&
> +
> +               test_file_count_in_dir ".git/mock-storage/local" 1 &&
> +               test_file_count_in_dir ".git/mock-storage/remote" 1
> +       )
> +'
> +
> +test_expect_success 'Remove file from repo and store files in Mock LFS based on size (>24 bytes)' '
> +       client_view "//depot/... //client/..." &&
> +       (
> +               cd "$cli" &&
> +               p4 delete file4.bin &&
> +               p4 submit -d "Remove file"
> +       ) &&
> +
> +       client_view "//depot/... //client/..." &&
> +       test_when_finished cleanup_git &&
> +       (
> +               cd "$git" &&
> +               git init . &&
> +               git config git-p4.useClientSpec true &&
> +               git config git-p4.largeFileSystem MockLFS &&
> +               git config git-p4.largeFileThreshold 24 &&
> +               git config git-p4.pushLargeFiles True &&
> +               git p4 clone --destination="$git" //depot@all &&
> +
> +               test_file_is_not_in_mock_lfs file1.txt "content 1 txt 23 bytes" &&
> +               test_file_is_in_mock_lfs file2.dat "content 2-3 bin 25 bytes" &&
> +               test_file_is_in_mock_lfs "path with spaces/file3.bin" "content 2-3 bin 25 bytes" &&
> +               test_file_is_deleted_in_mock_lfs file4.bin "content 4 bin 26 bytes XX" &&
> +
> +               test_file_count_in_dir ".git/mock-storage/local" 2 &&
> +               test_file_count_in_dir ".git/mock-storage/remote" 2
> +       )
> +'
> +
> +test_expect_success 'Run git p4 submit in repo configured with large file system' '
> +       client_view "//depot/... //client/..." &&
> +       test_when_finished cleanup_git &&
> +       (
> +               cd "$git" &&
> +               git init . &&
> +               git config git-p4.useClientSpec true &&
> +               git config git-p4.largeFileSystem MockLFS &&
> +               git config git-p4.largeFileThreshold 24 &&
> +               git config git-p4.pushLargeFiles True &&
> +               git p4 clone --destination="$git" //depot@all &&
> +
> +               test_must_fail git p4 submit
> +       )
> +'
> +
> +test_expect_success 'kill p4d' '
> +       kill_p4d
> +'
> +
> +test_done
> --
> 2.5.1
>
Previous: larsxschneider@gmail.comNext: larsxschneider@gmail.com
Message 10 of 21 in “git-p4: add support for large file systems”
  1. 0/7 git-p4: add support for large file systemslarsxschneider@gmail.com, Sep 21, 2015
  2. 1/7 git-p4: add optional type specifier to gitConfig readerlarsxschneider@gmail.com, Sep 21, 2015
  3. Junio C HamanoSep 22, 2015
  4. Lars SchneiderSep 23, 2015
  5. 2/7 git-p4: add gitConfigInt readerlarsxschneider@gmail.com, Sep 21, 2015
  6. 3/7 git-p4: return an empty list if a list config has no valueslarsxschneider@gmail.com, Sep 21, 2015
  7. 4/7 git-p4: add file streaming progress in verbose modelarsxschneider@gmail.com, Sep 21, 2015
  8. 5/7 git-p4: check free space during streaminglarsxschneider@gmail.com, Sep 21, 2015
  9. 6/7 git-p4: add support for large file systemslarsxschneider@gmail.com, Sep 21, 2015
  10. Luke DiamandSep 25, 2015
  11. 7/7 git-p4: add Git LFS backend for large file systemlarsxschneider@gmail.com, Sep 21, 2015
  12. Junio C HamanoSep 22, 2015
  13. Lars SchneiderSep 23, 2015
  14. Luke DiamandSep 23, 2015
  15. Lars SchneiderSep 23, 2015
  16. Lars SchneiderSep 23, 2015
  17. Luke DiamandSep 23, 2015
  18. Lars SchneiderSep 23, 2015
  19. Luke DiamandSep 24, 2015
  20. Lars SchneiderSep 25, 2015
  21. Luke DiamandSep 25, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.