Can't push to GitHub because of large file which I already deleted

Question

Currently I have

1. Empty GitHub repo
2. SSH server repo (main)
3. Local Repo

SSH server repo was the most up-to-date repo (production site) so I did a Git clone from there to local. I then tried to do a git push to GitHub.

Everything went OK but then it said something about filename.gz being too large for GitHub. I didn't need this file so I ran several Git commands to get rid of it from Git cache then pushed back to SSH server.

I don't see the large file locally but it's still on SSH server even though git diff returns nothing and git push returns "Everything is up-to-date" - And even though the file is not visible in local repo when I try to push to GitHub I still get error about it

remote: error: File fpss.tar.gz is 135.17 MB; this exceeds GitHub's file size limit of 100 MB

I followed steps under "fixing the problem" listed on GitHub help so shouldn't that have been enough?

How is the file still in the ether when it's not local or listed in git status/diff/push?

Automata

The file is still there in history. You need to destroy the history, possibly by squashing the commits that added and removed the file. – Vaudois

@Vaudois I followed steps under "fixing the problem" listed on this site...shouldn't that have been enough? help.github.com/articles/working-with-large-files – Automata

The command there is more advanced than my knowledge of git, so I can't really tell. Anyway, if git log -- the_big_file is returning you anything, then the file is still in the history. – Vaudois

@Vaudois that returns nothings >< – Automata

Could it be that you are also pushing other branches where the file exists? Also, if the file is still on the server, why would git push say everything is up-to-date? Since you changed history, it should have complained that the push is not possible and that you would have to force it. – Vaudois

Exactly, it's not making sense. There is only 1 branch master. Remote ssh has master + a dummy branch (i created so i could push to it's master). @Vaudois – Automata

If you have the terminal where you did all these open, it would probably help if you paste the exact commands and messages (removing personal info). There could be something that to you seems unimportant but in reality is. – Vaudois

May do that, but here's a Q. If everything in it's current state is ok and I do not need the previous histories. Can I just delete git folder and redo git init on local then try pushing to GitHub without any issues? @Vaudois – Automata

Yes think I may do this @Vaudois #9683779 – Automata

If your history doesn't matter to you, sure you can do that. It wouldn't be nice though. One thing you could do is create a branch specifically for github, squash all your history in a single commit (effectively the same as what you said, but not deleting other branches) and only push that particular branch to github. Later, when there are commits in the master branch for example, you can cherry-pick all of them and apply to the github branch. (not sure if merge would work, but if it could, then that would be even better) – Vaudois

Wholeheartedly f*** Github for this. – Ebullition

Why? Do you think they should allow storage of extremely large files like videos over 2gb on a completely free repository service? @TobiasWilfert – Automata

@Automata First and foremost I am not asking for 2GB I am asking for more than 100MB. Secondly, Github Team currently costs 44$/y it costs less than 3 cents to store 100MB on AWS (S3 Standard). So maybe they could make the price 44.03$/y and give us 200 MB? – Ebullition

Gothca, I had this issue on public repot... in that case I agree...they should include storage for GitHub teams including large videos as a sort of all-in-one service.. @TobiasWilfert – Automata

I just want to say, I have found myself adding specifically model weights by accident so many times... feels like a huge bug in git that you can't undo this easily, ack! 🙀 – Boise

Answer 1

You can use

git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch <file/dir>' HEAD

This will delete everything in the history of that file. The problem is that the file is present in the history.

This command changes the hashes of your commits which can be a real problem, especially on shared repositories. It should not be performed without understanding the consequences.

Edit: The git project now recommends that users use git filter-repo instead of git filter-branch.

---

Using git filter-repo

WARNING: git-filter-branch has a glut of gotchas generating mangled history
         rewrites.  Hit Ctrl-C before proceeding to abort, then use an
         alternative filtering tool such as 'git filter-repo'
         (https://github.com/newren/git-filter-repo/) instead.  See the
         filter-branch manual page for more details; to squelch this warning,
         set FILTER_BRANCH_SQUELCH_WARNING=1.

Installation

[brew|pip3|...] install git-filter-repo

Usage

To remove any file with the path prefix example/path/to/something, you can run

git filter-repo --path example/path/to/something--invert-paths

To remove any file without the path prefix example/path/to/something, you can run

git filter-repo --path example/path/to/something

Erdah

Worked for me but I had to 'force' it: git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch <file/dir>' -f HEAD – Hurt

This command changes the hashes of your commits which can be a real problem, especially on shared repositories. It should not be performed without understanding the consequences. – Tillis

Are you supposed to replace <file/dir> with the name of the file or dir that is causing the problem? – Peters

The answer is, "yes, you are", and thanks so much for your answer. – Peters

Note that if you want apply these changes to ALL branches, you need to use a --all flag instead of HEAD – Stupefy

This was the only solution that worked for me. Then I was able to git push origin master, and finally removed any reference of the too large files in the repo. – Jobyna

I am getting: Rewrite 657560fa18c030bcfac9132ce1c3541e84a5bc2c (1/10) (0 seconds passed, remaining 0 predicted) /usr/lib/git-core/git-filter-branch: 1: eval: Syntax error: end of file unexpected – Ferino

this is absolutely magical. I was trying to push code from last three days. saved my life. cheers. – Odeliaodelinda

CAUTION ! Only run the command in the answer IF YOU DO NOT NEED THE FOLDER/FILE anymore ! I just ran it and lost an entire folder of code. Luckily it was backed up on the remote server. – Decorous

I am getting the error:: "cannot rewrite branches: you have unstaged changes. I tried staging everything, but it still does not work. I have tried many things, I have asked a related question about using git-amend but I don't understand either how that works #48699817 – Breeches

Note to Windows users, use double quotes " instead of single quotes. – Nettle

This one messed up my repo, cant pull or push. Completely screwed it. – Tarpley

i'm getting this : fatal: ambiguous argument 'rm': unknown revision or path not in the working tree. is that because the file is deleted? – Achernar

@T.Nel, my file was deleted, but no the folder. Maybe you removed a folder too? You may have to try with the -f to force the removal. – Mawkin

DO NOT USE THIS ON A SHARED REPO. IN FACT, DO NOT USE THIS AT ALL. – Ecdysiast

If you're on Windows you'll need to use "...." instead of '...' , like so: git filter-branch --index-filter "git rm -r --cached --ignore-unmatch folder/file" HEAD -- as per this related question: #21395063 – Barely

Now issues the warning: WARNING: git-filter-branch has a glut of gotchas generating mangled history rewrites. Hit Ctrl-C before proceeding to abort, then use an alternative filtering tool such as 'git filter-repo' (https://github.com/newren/git-filter-repo/) instead. See the filter-branch manual page for more details; to squelch this warning, set FILTER_BRANCH_SQUELCH_WARNING=1. – Japha

IF YOU MESS THIS UP IT WILL NUKE YOUR ENTIRE REPO. PLEASE TRY ALTERNATIVES – Protuberancy

This will not work on a repo that has not been pushed to an origin – Orvil

WARNING --- THIS DELETED MY ENTIRE REPO -- THE COMMAND ``` git filter-repo --path example/path/to/something--invert-paths ``` DELETED MY ENTIRE REPO DO NOT DO THIS – Suboceanic

THIS IS A INCORRECT ANSWER - --invert-paths REMOVES EVERYTHING ****EXCEPT**** THAT PATH – Suboceanic

Repo no longer pushes or pulls at all after doing this. There needs to be a real solution to this problem from git/github's end, this problem is over 10 years old. Starting new repo from scratch. – Boise

Answer 2

I found squashing more useful than filter-branch. I did the following:

1. Locally delete large files.
2. Commit the local deletes.
3. Soft reset back X number of commits (for me it was 3): git reset --soft HEAD~3.
4. Then recommit all the changes together (AKA squash) git commit -m "New message for the combined commit"
5. Push squashed commit.

Special case (from user @lituo): If above doesn't work, then you may have this case. Commit 1 included the large file and Commit 1's push failed due to large file error. Commit 2 removed the large file by git rm --cached [file_name] but Commit 2's push still failed. You can follow the same steps above but instead of using HEAD~3, use HEAD~2.

Prae

This is MUCH better than the top answer. The top answer screws up your whole commit history. – Evita

Better is a strong word. This is simple a different solution. Which one you should choose would depend on what your desired outcome is and/or you current state. – Corelli

If you get an error when you push the combined commit, you can append a + sign to the branch. git push origin +name-of-branch The plus sign forces the remote branch to accept your rewritten history, otherwise you will end up with divergent branches – Samuelsamuela

Worked perfectly for me. I was just doing this initially -> git reset --soft HEAD~1. I then realized that I had 3 commits and changed it to git reset --soft HEAD~3 which worked. Thank you. I upvoted your answer. – Guildsman

For the record: The 5th step should be: git push --force . It is working for me. to remove the large file history from the log. – Pubilis

Please consider this one before you try the accepted answer, it's much safer! – Thierry

Def use --force as Ben says, or else you'll deal with your branches separating – Asternal

Thank you! Saved my day! I actually had to to do git reset --soft HEAD~3 three times, since I had 4 commits waiting inline (Shame on me yes.....) and the large file blocked me from pushing. – Manama

Answer 3

Here's something I found super helpful if you've already been messing around with your repo before you asked for help. First type:

git status

After this, you should see something along the lines of

On branch master
Your branch is ahead of 'origin/master' by 2 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

The important part is the "2 commits"! From here, go ahead and type in:

git reset HEAD~<HOWEVER MANY COMMITS YOU WERE BEHIND>

So, for the example above, one would type:

git reset HEAD~2

After you typed that, your "git status" should say:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

From there, you can delete the large file (assuming you haven't already done so), and you should be able to re-commit everything without losing your work.

Excitor

I appreciate the advice for figuring out how many commits to go back, but what does not including the '--soft' option do (or not do)? – Brumal

If your branch has not been pushed, there would be no "Your branch is ahead of 'origin/master' by 2 commits.", here you can use git log and then count the number of commits since you branched off. – Fruiter

This worked for me! The previous two solutions did not! – Moffitt

Thank you kind stranger! I still hate git with the fury of a thousand suns, but thanks to you I can at least go home this evening. – Mckim

Answer 4

If the file was added with your most recent commit, and you have not pushed to the remote repository, you can delete the file and amend the commit, Taken from here:

git rm --cached giant_file
    # Stage "giant_file" for removal with "git rm"
    # Leave it on disk with "--cached". if you want to remove it from disk
    # then ignore the "--cached" parameter
git commit --amend -CHEAD
    # Commit the current tree without the giant file using "git commit"
    # Amend the previous commit with your change "--amend" 
    # (simply making a new commit won't work, as you need
    # to remove the file from the unpushed history as well)
    # Use the log/authorship/timestamp of the last commit (the one we are
    # amending) with "-CHEAD", equivalent to --reuse-message=HEAD
git push
    # Push our rewritten, smaller commit with "git push"

Concubine

This solution will not work since the file is not anymore in the git index (it results as untracked file list at git status. – Jobyna

Nothing happening. After applying this it reduced total number of file count but after showing process 99% it stuck again. Any suggestion what I'm missing ? – Zenger

what does -CHEAD mean? – Unfinished

What if I want to try this from a specific commit--not the very last commit? I tried git rm --cached giant_file commit_id but it didn't work :( – Diluent

@Diluent I would revert to the previous commit, do these steps, and then merge with the current one. I'm not sure if this is the best approach, I'm not a Git expert – Concubine

Answer 5

Why is GitHub rejecting my repo, even after I deleted the big file?

Git stores the full history of your project, so even if you 'delete' a file from your project, the Git repo still has a copy of the file in it's history, and if you try to push to another repository (like one hosted at GitHub) then Git requires the remote repo has the same history that your local repo does (ie the same big files in it's history).

How can I can I get GitHub to accept my repo?

You need to clean the Git history of your project locally, removing the unwanted big files from all of history, and then use only the 'cleaned' history going forward. The Git commit ids of the affected commits will change.

How do I clean big files out of my Git repo?

The best tool for cleaning unwanted big files out of Git history is the BFG Repo-Cleaner - it's a simpler, faster alternative to git-filter-branch specifically designed for removing unwanted files from Git history.

Carefully follow the usage instructions, the core part is just this:

$ java -jar bfg.jar --strip-blobs-bigger-than 100M my-repo.git

Any files over 100MB in size (that aren't in your latest commit) will be removed from your Git repository's history. You can then use git gc to clean away the dead data:

$ git gc --prune=now --aggressive

The BFG is typically at least 10-50x faster than running git-filter-branch, and generally a lot easier to use.

Full disclosure: I'm the author of the BFG Repo-Cleaner.

Jargonize

My case had additional complications that precluded squashing. The BFG tool worked great. Thanks. – Shiest

This is a phenomenal solution – Backswept

Worth noting that BFG hasn't really been maintained for a number of years now – Corelli

You tool is a magic. Thanks for the help – Nestorius

Your tool is the only one that finally worked for me. And it worked in flash almost like magic. Many many thanks for your help and the awesome tool. – Fauman

Answer 6

I had a similar issue and used the step above to remove the file. It worked perfectly.

I then got an error on a second file that I needed to remove: remote: error: File <path/filename> is 109.99 MB; this exceeds GitHub's file size limit of 100.00 MB

I tried the same step, got an error: "A previous backup already exists in <path/filename>"

From research on this website I used the command: git filter-branch --force --index-filter "git rm --cached --ignore-unmatch <path/filename>" --prune-empty --tag-name-filter cat -- --all

Worked great, and the large files were removed.

Unbelievably, the push still failed with another error: error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104 fatal: The remote end hung up unexpectedly

This I fixed by directly modifying the .git config file - postBuffer = 999999999

After that the push went through!

Monosepalous

an additional gotcha I had to contend with removing a large file (as above) was that one of the folders had a hash # character in it. This caused no problems at all for normal git operation however for the git rm I needed to give the full repository path name for the file and to escape the # with a backslash to get it to work – Glaciology

this worked for me, too. I avoided the reset hard step at the bottom of the page with a simple push. czettner.com/2015/07/16/… – Southerly

This worked after also running 'git push -f origin' – Mezzorilievo

Answer 7

I have tried all above methods but none of them work for me.

Then I came up with my own solution.

1. First of all, you need a clean, up-to-date local repo. Delete all the large files.

2. Now create a new folder OUTSIDE of your repo folder and use "Git create repository here" to make it a new Git repository, let's call it new_local_repo. This is it! All above methods said you have to clean the history..., well, I'm sick of that, let's create a new repo which has no history at all!

3. Copy the files from your old, messed up local repo to the new, beautiful repo. Note that the green logo on the folder icon will disappear, this is promising because this is a new repo!

4. Commit to the local branch and then push to remote new branch. Let's call it new_remote_branch. If you don't know how to push from a new local repo, Google it.

5. Congrats! You have pushed your clean, up-to-date code to GitHub. If you don't need the remote master branch anymore, you can make your new_remote_branch as new master branch. If you don't know how to do it, Google it.

6. Last step, it's time to delete the messed up old local repo. In the future you only use the new_local_repo.

Avitzur

Answer 8

I got the same problem and none of the answers work for me. I solved by the following steps:

1. Find which commit(s) contains the large file

git log --all -- 'large_file`

The bottom commit is the oldest commit in the result list.

2. Find the one just before the oldest.

git log

Suppose you got:

commit 3f7dd04a6e6dbdf1fff92df1f6344a06119d5d32

3. Git rebase

git rebase -i 3f7dd04a6e6dbdf1fff92df1f6344a06119d5d32

Tips:

1. List item
2. I just choose drop for the commits contains the large file.
3. You may meet conflicts during rebase fix them and use git rebase --continue to continue until you finish it.
4. If anything went wrong during rebase use git rebase --abort to cancel it.

Glaciate

First step didn't work for me. I needed to use : git log --all --full-history -- "**/large_file" – Tempe

Answer 9

git lfs migrate import --include="fpss.tar.gz"

this should re-write your local commits with the new lfs refs

https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-migrate.1.ronn?utm_source=gitlfs_site&utm_medium=doc_man_migrate_link&utm_campaign=gitlfs#examples

Paranymph

Answer 10

The solution to keep the large files/folders within the working folder

This is the line that worked to solve the problem asked here (from answer 1):

git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch <file/dir>' HEAD

This command also delete the file/dir if the file/dir is within the working tree.

If you want to keep the file/folder within the working tree I propose taking the following steps.

1. After that error run git reset HEAD^

2. Add the file/folder in question into ``.gitignore``` file.

3. Proceed as usual git add . which might capture other files/folders but must capture .gitignore file. Next is git commit -m"message" and finally git push origin <branch_name>

Lamrouex

Answer 11

I thing it because your deleted file may be already exist in your commit ,to check this first use

git log

this will return list of your commit in current branch , find id for exact commit you look for,

Then use ,

git show <commit_id>

This should show the details of commit with files included in it

Now to fix your problem, use

git reset --soft HEAD~1

Here 1 on HEAD~1 represent for previous commit, You can use different number according to your required commit , If you need second last commit just use, git reset --soft HEAD~2

This will reset your Head to previous commit,if this commit don't have the large file Then you can do,

git add .

git commit -m <message_for_commit>

git push origin <repo_name>

else

if you want to reset to a specific commit that not contain your file, just use

git reset --soft <commit_id>

Then create a new commit from here with out the deleted file and push it

Delwyn

Answer 12

My unorthodox but simple solution from scratch:

• just forget about your recent problematic local git repository and git clone your repository into a fresh new directory.
• git remote add upstream <your github rep here>
• git pull upstream master
• at this point just copy your new files to commit, to your new local rep from the old one may be including your now reduced giant file(s).
• git add .
• git commit -m "your commit text here"
• git push origin master

voila! worked like a charm in my case.

Naos

Answer 13

if you are uploading your own project then just go the file where directory is present. Delete large file. then click on the "view"(window file) view-> check hidden folder then you will be able to see '.git' file delete .git file this will delete all your commit history Then you can push your repo like new one...

Stake

Answer 14

Somehow this worked for me. I tried all solutions but this command saved my time. Hope this helps you too.

git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch <file/folder path>' --prune-empty --tag-name-filter cat -- --all

Please run this at your own risk. I'll not be responsible for your actions.

Run this command from the root directory

June

you should explain the solution – Gloxinia

Answer 15

Re-initializing the repo worked for me:

1. rm -rf .git
2. git init
3. git remote add origin https://github.com/{user}/{repo}
4. git commit -am 'fix: reinit repo'
5. git push -u origin {branch} --force

Dorweiler

Careful! While this works, you will loose the complete commit history for the project, which might not be what you want. – These

Answer 16

For me the fix was easier, just created a new folder, cloned the repo, added the new files from the last commit and just pushed.

Acne

I'll add details: In the the repository directory... 1) Do git log to show, then copy commits done since that last push. 2) Delete (only) the .git folder (rm -rf). Then... 1) Clone the repo from GitHub in a temporary directory. 2) Move the newly cloned .git folder from the temporary directory into the repository directory. 3) Do a commit. (I included the commit messages from the log copy made above, plus an explanation of the commit message discontinuity.) 4) Push. QED – Gooch

Answer 17

this worked for me. documentation from github Squashing Git Commits git reset origin/master

git checkout master && git pull;
git merge feature_branch;
git add . --all;
git commit -m "your commit message"

find documentation here

Rafael

Answer 18

So I encountered a particular situation: I cloned a repository from gitlab, which contained a file larger than 100 mb, but was removed at some point in the git history. Then later when I added a new github private repo and tried to push to the new repo, I got the infamous 'file too large' error. By this point, I no longer had access to the original gitlab repo. However, I was still able to push to the new private github repo using bfg-repo-cleaner on a LOCAL repository on my machine:

$ cd ~
$ curl https://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar > bfg.jar
$ cd my-project
$ git gc
$ cd ../
$ java -jar bfg.jar --strip-blobs-bigger-than 100M my-project
$ cd my-project
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
$ git remote -v # confirm origin is the remote you want to push to
$ git push origin master

Cranage

Answer 19

Sometimes the file is kept in tracking history, try the following steps:

1. git commit, If you are seeing create mode with the big file listed, then do:

2. git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch filename' HEAD. You should see a bunch of Rewrites shown in your console which ends with:

   rm 'filename' and

   the last line Ref was rewritten.

It's done.

Saran

Answer 20

rather than doing complicated stuff, copy your repo (on your computer) to another place. delete the large file. do a couple of push and pull. Then some of your files will be messed up having things like "<<<<<< HEAD". Just copy your backup into the old folder on the disk. Do another add, commit, push!

Emogeneemollient

Answer 21

I had this issue when I didn't have a gitignore file with my iOS project

I think maybe it was trying to push a humungous file to github and github was probably rejecting that giant file or (files)

Brotherson

Answer 22

What worked for me:

1. rename my GitHub project folder to something else
2. re-clone the repo with the correct folder name
3. delete the .git folder in my renamed repo (may have to turn on allow to see hidden files in Windows)
4. move the .git folder from the correct folder name to the renamed one
5. delete the re-cloned repo folder, rename the original repo folder to the correct name
6. commit your changes (without the large files) and push

Solemn

Answer 23

Check your last commit id on GitHub and then use this

git reset --soft <commit_id>

I think it's work. Thanks

Impossibility

Answer 24

The solution that worked for me in cases that you don't mind to lose the commit that contains the problem was, use the following command in order to find the id of the commit:

git log --all -- 'insert/the/path/of/the/large-file.extension'

and then use the commit id to drop it. if you have some important files next to this you can always copy them in another localtion in order to back up, drop the commit and then add it back.

Note: I was using vscode with the extension Git Graph, it was easy for me to :

1. find the commit with the the id by using the search bar
2. right click on the commit and the drop it
3. resolve the conflicts if you have some and then push

Pharmacognosy

Answer 25

I had the same issue.

To fix this, I undid the commit. Then, I recommitted each file individually.

Ewers

Answer 26

Assuming you already pushed your previous commits, just delete your .git folder, initialize a new repo, add the origin, add your files (not including the deleted file), fetch, checkout new branch, and push to new branch.

Scandura

Answer 27

I am adding to the first answer.

git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch ' HEAD

There will be some merge conflict from origin/master.

Your branch and 'origin/master' have diverged, and have 114 and 109 different commits each, respectively. (use "git pull" to merge the remote branch into yours)

Please run this

git reset --hard origin/master

It will throw away all my staged and unstaged changes, forget everything on my current local branch and make it exactly the same as origin/master.

Minify

Answer 28

Deleting Repo And Starting Over Worked For Me

Similar to @Shuaibin Chang none of these things worked for me. One of the big problems I had was that mine happened on the initial git commit so I couldn't go back X commits because there were not any other commits to go back to.

I went to the repo where I wanted to remove my broken repository and install a new one and did the following:

• sudo rm .git - This will remove the old .git repo giving you problems. It may or may not ask you about removing write protected files. If it does answer y to each of them there will probably be a bunch in this case.
• git init - Initialize a new repository where the old one used to live
• From here follow whatever steps you normally take to create and push your git repositories.

Constantine

Answer 29

Make your local repo match the remote repo with (all locals changes will be lost):

git reset --hard origin/master

Then push again.

Algoid

This removed everything in my local directory!!!!!!!!!!!!!!!!!!!!!!!!!!! – Biddick

@Biddick Read again "Make your local repo match the remote repo". – Algoid