Can I optimize a Mercurial clone?

Question

My Mercurial clone has become incredibly slow, presumably due to on-disk fragmentation. Is there a way to optimize it?

The obvious way it to make a new clone, then copy my MQ, saved bundles, hgrc, etc, to the new clone and delete the old one. But it seems like someone might have run into this problem before and made an extension to do it?

Fourlegged

From what I know about Mercurial, there should be nothing that Mercurial does that should make it slow in regards to fragmentation, etc. Have you tried running a normal disk defragmenter on your disk? Also, what kinds of operations are slow, and how slow, etc.? – Duchess

hg qref is the really slow thing. Takes maybe 50 seconds on a repo with 70000ish commits. I agree that it's not a mercurial specific problem (though I think mercurial access patterns exacerbate it), but Mac doesn't have a built-in defragmenter (AFAICT). – Fourlegged

Answer 1

I deleted the repo and recloned, and that improved performance.

Fourlegged

Which means: you lose all history. This is rarely and advantageous solution. – Hoag

Abel: why would you lose all history? – Fourlegged

Oh, sorry, I misunderstood, thought you meant a copy of tip only. Still not sure how you can reclone what you deleted. Or you mean leaving the server state as is and just taking a new clone? I.e., you meant deleting your local clone only? Isn't that the same as purge? – Hoag

The idea is that when you do a clone, they'll be created on-disk with low fragmentation (that is, they'll be right next to each other, which means they'll be fast to access). However, when add new files later, they'll be stored in a different part of the disk, which will be slower to access. Over time this will become slow. By starting fresh, you can get all the files in the same part of the disk, making it much faster (same as a defragmenter on Windows, kinda). – Fourlegged

Ah, maybe that should be reflected in your answer)). Wouldn't help my situation though, using a ram drive. – Hoag

Answer 2

If the manifest gets particularly large then it can result in slow performance. Mercurial has an alternative repository format - generaldelta - that can often result in much smaller manifests.

You can check the size of your manifest using:

ls -lh .hg/store/*manifest*

To get maximum value from generaldelta:

1. Install Mercurial 2.7.2 or later (2.7.2 includes a fix to a bug in generaldelta that could result in larger manifest sizes - but there's a good chance you won't hit the bug with an earlier version).

2. Execute hg --config format.generaldelta=1 clone --pull orig orig.gd.

This may give some improvement in the manifest size, but not the full benefit.

1. Execute hg --config format.generaldelta=1 clone --pull orig.gd orig.gd.gd.

The clone of the clone may give a much greater improvement in the manifest size. This is because when pulling from a generaldelta repo things will be reordered to optimise the manifest size.

As an example of the potential benefits of generaldelta, I recently converted a repo that was ~55000 SVN commits (pulled using hgsubversion) plus ~1000 Mercurial commits/merges/grafts, etc. The manifest in the original repo was ~1.4GB. The manifest in the first clone was ~600MB. The manifest in the clone of the clone was ~30MB.

There isn't a lot of information about generaldelta online - there's still work to be done before it can become the default format, but it works well for many projects. The first few Google search results have some information from when it was first introduced, and there was some recent discussion on the mercurial-dev mailing list.

Luanneluanni

+1 Even for a smaller repo of ~3400 commits and a much smaller ~17MB original manifest size the two clones cut some hg operations down to a third of the original time (final manifest was 11MB). – Goldia

Answer 3

Turn off real-time anti-virus monitoring of the folder the repo is cloned to and defrag. There's not much else you can do.

Hejira