{"thread":{"id":"63326","subject":"Verifying data integrity of two git repositories","startedAt":"2025-04-22T07:20:03Z","lastAt":"2025-04-22T20:33:57Z","messageCount":2,"participants":["Akash S","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"516467","messageId":"SA1PR19MB7013FAD97E5E89B56AD7BF04C0BB2@SA1PR19MB7013.namprd19.prod.outlook.com","threadId":"63326","inReplyTo":null,"subject":"Verifying data integrity of two git repositories","fromName":"Akash S","fromEmail":"akashs@commvault.com","sentAt":"2025-04-22T07:19:57Z","receivedAt":"2025-04-22T07:20:03Z","isPatch":false,"sender":{"key":"akashs@commvault.com","avatar":null},"body":"I have a bare repository of size 5.7 GB in my local disk.\n\nI need to push this to Azure DevOps. I usually do it with the command \"git push --mirror\" but unfortunately, Azure DevOps has a single push size limit of 5GB.\n\nSo I have to push repos larger than 5GB in chunks.\n\nI used this stackoverflow answer (https://stackoverflow.com/questions/79167276/splitt-git-push-to-azure-devops) asmy basis and created a script to push each branch in batches of commits.\n\nI pushed my repository in batches to lets say remote repo \"A\".\n\nI did a \"git clone --bare\" from remote repo A to my local disk. I verified the size of this bare and it seems to be of size 5 GB only.\n\n\ti) I counted the number of objects using this command \"git rev-list --objects --all | wc -l\" in both repos, both are same.\n\n\tii) There is only 1 branch master in both repos and the last commit id of both master branches are matching (read an article that data integrity can be checked like this also since git also works like Blockchain)\n\n\tiii) git fsck --full in both repos,  both gave the same output: \n\n\t\tChecking object directories: 100% (256/256), done.\n\t\tChecking objects: 100% (10793794/10793794), done.\n\t\tChecking connectivity: 10793794, done.\n\n\t\tBut original repo on disk had this extra line in the end (which the remote bare on disk did not display)\n\n\t\tVerifying commits in commit graph: 100% (1351940/1351940), done.\n\t\n\tiv) I create a bundle of the original repo on disk using command \"git bundle create repo.bundle --all\" and then in the remote cloned repo on disk I ran, \"git bundle verify ../repo.bundle\". Output:\n\n\t\tThe bundle contains these 883 refs:\n\t\t<All Refs>\n\t\tThe bundle records a complete history.\n\t\tThe bundle uses this hash algorithm: sha1\n\t\t/home/repo.bundle is okay\n\n\tii) I checked the repo size using this command \"git count-objects -vH\", the size-pack differs (original repo says 5.62 GB and the remote cloned repo on disk says 4.93 GB)\n\nNote: My repository does not have lfs/objects also. So I do not have any lfs objects to begin with. So that is out of the question.\n\nWhy is there a change in size? Also how do I validate if two repos are the same or not?\n\nScript being used to push in batches of commits:\n\n#!/bin/bash\nset -e\n\n# === CONFIGURATION ===\nRepositoryFolderPathForBareCloneBAK=\"/root/linux\"\nBackupRepositoryHttpsURL=\"<REMOTE_URL> \"\nremoteName=\"origin\"\nmaxPushSizeInMB=$((4 * 1024)) # 4GB\nsplitPushCommitsCount=35000\nsplitPush=false\n\nALocation=$(pwd)\n\nif [ ! -d \"$RepositoryFolderPathForBareCloneBAK\" ]; then\n    echo \"Error: Bare clone folder not found at $RepositoryFolderPathForBareCloneBAK\"\n    exit 1\nfi\n\ncd \"$RepositoryFolderPathForBareCloneBAK\"\ngit config http.postBuffer 524288000\n\ndoSplitPush=$splitPush\n\n# Check repo size and decide whether to split push\nif [ \"$doSplitPush\" = false ]; then\n    echo \"Checking repository size...\"\n    repositorySize=0\n    while read -r line; do\n        echo \"$line\"\n        if [[ \"$line\" =~ ^size-pack:\\ ([0-9]+(\\.[0-9]+)?)\\ ([A-Za-z]+) ]]; then\n            value=${BASH_REMATCH[1]}\n            unit=${BASH_REMATCH[3]}\n            case \"$unit\" in\n                bytes) repositorySize=$(echo \"$value / 1024 / 1024\" | bc) ;;\n                KiB)   repositorySize=$(echo \"$value / 1024\" | bc) ;;\n                MiB)   repositorySize=$(echo \"$value\" | bc) ;;\n                GiB)   repositorySize=$(echo \"$value * 1024\" | bc) ;;\n                *)     repositorySize=$(echo \"$value\" | bc) ;;\n            esac\n        fi\n    done < <(git count-objects -vH)\n\n    # Round down to integer\n    repositorySize=${repositorySize%.*}\n\n    echo \"Repo size: $repositorySize MiB\"\n\n    if [ \"$repositorySize\" -ge \"$maxPushSizeInMB\" ]; then\n        doSplitPush=true\n    fi\nfi\n\n# Unset mirror config to allow partial pushes if needed\nif git config --get remote.origin.mirror >/dev/null; then\n    git config --unset remote.origin.mirror\nfi\n\n# Setup remote\nNewREMOTE=\"push_remote\"\nif git remote | grep -q \"$NewREMOTE\"; then\n    git remote remove \"$NewREMOTE\"\nfi\ngit remote add \"$NewREMOTE\" \"$BackupRepositoryHttpsURL\"\n\nif [ \"$doSplitPush\" = false ]; then\n    echo \"Performing full push to $BackupRepositoryHttpsURL\"\n    git push \"$NewREMOTE\" --mirror\nelse\n    echo \"Performing split push to $BackupRepositoryHttpsURL\"\n\n    git for-each-ref --format=\"%(refname)\" --sort='authordate' | while read -r ref; do\n        if [[ \"$ref\" == refs/heads/* ]]; then\n            BRANCH=\"${ref#refs/heads/}\"\n            echo \"Processing branch: $BRANCH\"\n\n            git symbolic-ref HEAD \"$ref\"\n\n            if git show-ref --quiet --verify \"refs/remotes/$NewREMOTE/$BRANCH\"; then\n                range=\"$NewREMOTE/$BRANCH..HEAD\"\n            else\n                range=\"HEAD\"\n            fi\n\n            n=$(git log --first-parent --format=\"format:x\" $range | wc -l)\n            echo \"$n commits to push\"\n\n            splitPushCommitsCount=$(( (maxPushSizeInMB * n) / repositorySize ))\n            [ \"$splitPushCommitsCount\" -gt 20000 ] && splitPushCommitsCount=20000\n\n            echo \"Calculated splitPushCommitsCount: $splitPushCommitsCount\"\n\n            if [ \"$n\" -gt 0 ]; then\n                loopCount=$((n / splitPushCommitsCount))\n                for ((i=1; i<=loopCount; i++)); do\n                    h=$(git log --first-parent --reverse --format=format:%H --skip $((n - (i * splitPushCommitsCount))) -n1)\n                    echo \"Batch commit: $h\"\n                    git push \"$NewREMOTE\" --force \"$h:refs/heads/$BRANCH\"\n                    echo \"sleeping for 5 minutes\"\n                    sleep 300\n                done\n                echo \"Final push: HEAD:refs/heads/$BRANCH\"\n                git push \"$NewREMOTE\" --force \"HEAD:refs/heads/$BRANCH\"\n            else\n                echo \"No commits to push for $BRANCH\"\n            fi\n        fi\n    done\n\n    echo \"Pushing tags\"\n    git push \"$NewREMOTE\" --force 'refs/tags/*'\n\n    echo \"Pushing replace refs (if any)\"\n    git push \"$NewREMOTE\" --force 'refs/replace/*'\nfi\n\n# === LFS Push ===\necho \"Pushing Git LFS objects...\"\nGet_LFS_Objects() {\n    lfs_objects_dir=\"$1/lfs/objects\"\n    if [ -d \"$lfs_objects_dir\" ]; then\n        lfs_objects=$(find \"$lfs_objects_dir\" -type f -printf \"%f \")\n        if [ -z \"$lfs_objects\" ]; then\n            lfs_objects=\"NO_OBJECTS\"\n        fi\n    else\n        lfs_objects=\"NO_OBJECTS\"\n    fi\n}\nGet_LFS_Objects \"$RepositoryFolderPathForBareCloneBAK\"\nif [[ \"$lfs_objects\" != \"NO_OBJECTS\" ]]; then\n    LFS_SPECIFIER=\"--object-id $lfs_objects\"\n    echo \"Running lfs\"\n\n    git lfs push \"$NewREMOTE\" $LFS_SPECIFIER\n    retCode=$?\n    echo \"LFS push exited with code: $retCode\"\nelse\n    echo \"No LFS objects to push.\"\nfi\n\ncd \"$ALocation\"\necho \"All done! Git and LFS data pushed successfully.\"\n\n\n\n"},{"id":"516531","messageId":"a736beec-2e9b-4ae2-a611-2a1187042bcf@kdbg.org","threadId":"63326","inReplyTo":"SA1PR19MB7013FAD97E5E89B56AD7BF04C0BB2@SA1PR19MB7013.namprd19.prod.outlook.com","subject":"Re: Verifying data integrity of two git repositories","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2025-04-22T19:57:20Z","receivedAt":"2025-04-22T20:33:57Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 22.04.25 um 09:19 schrieb Akash S:\n> ii) There is only 1 branch master in both repos and the last commit\n> id of both master branches are matching (read an article that data\n> integrity can be checked like this also since git also works like\n> Blockchain)\n> \n> \tiii) git fsck --full in both repos,  both gave the same output: \n> \n> \t\tChecking object directories: 100% (256/256), done.\n> \t\tChecking objects: 100% (10793794/10793794), done.\n> \t\tChecking connectivity: 10793794, done.\n\nThe facts that no errors were reported and that the commit ids are\nidentical are sufficient evidence that both repositories are identical.\n\n> But original repo on disk had this extra line in the end (which the\n> remote bare on disk did not display)\n> \n> \t\tVerifying commits in commit graph: 100% (1351940/1351940), done.\n\nA commit graph is an optional data structure. Its absence doesn't\ninvalidate the repository.\n\n> Why is there a change in size? Also how do I validate if two repos\n> are the same or not?\nMost likely, the two repositories have been packed in different manners.\nThis has no bearing on the validity of the repository at all as long as\n`git fsck --full` reports no error. The different sizes should not worry\nyou.\n\n-- Hannes\n\n"}]}