How to speed up the eclipse project 'refresh'

Question

I have a fairly large PHP codebase (10k files) that I work with using Eclipse 3.4/PDT 2 on a windows machine, while the files are hosted on a Debian fileserver. I connect via a mapped drive on windows.

Despite having a 1gbit ethernet connection, doing an eclipse project refresh is quite slow. Up to 5 mins. And I am blocked from working while this happens.

This normally wouldn't be such a problem since Eclipse theoretically shouldn't have to do a full refresh very often. However I use the subclipse plugin also which triggers a full refresh each time it completes a switch/update.

My hunch is that the slowest part of the process is eclipse checking the 10k files one by one for changes over samba.

There is a large number of files in the codebase that I would never need to access from eclipse, so I don't need it to check them at all. However I can't figure out how to prevent it from doing so. I have tried marking them 'derived'. This prevents them from being included in the build process etc. But it doesn't seem to speed up the refresh process at all. It seems that Eclipse still checks their changed status.

I've also removed the unneeded folders from PDT's 'build path'. This does speed up the 'building workspace' process but again it doesn't speed up the actual refresh that precedes building (and which is what takes the most time).

Minneapolis

have the same problem here: wins7 + samba + virtual box + eclipse == very slow. have tried WEBdrive before samba and it was only worse. trying now with "offline" trick @denis suggested.. does wins have problems with samba aswell? – Karlow

@Karlow IF you use virtualbox or other VM, I suggest you to use mount folder feature from VM. To reduce tones of overhead from SMB, network stack. – Spear

Answer 1

Thanks all for your suggestions. Basically, JW was on the right track. Work locally.

To that end, I discovered a plugin called FileSync: http://andrei.gmxhome.de/filesync/

This automatically copies the changed files to the network share. Works fantastically. I can now do a complete update/switch/refresh from within Eclipse in a couple of seconds.

Minneapolis

Thanks for the pointer. That could come in handy when I don't want to run the web server locally. – Canice

Filesynch worked for me for a while but then when I upgraded eclipse it stopped working, I found setting up ftp using Aptana's inbuilt synching seemed to work for me, but not a great solution, just a workaround. eventually I changed to phpstorm and it just works fast too! – Illusionism

Answer 2

Do you have to store the files on a share? Maybe you can set up some sort of automatic mirroring, so you work with the files locally, and they get automatically copied to the share. I'm in a similar situation, and I'd hate to give up the speed of editing files on my own machine.

Canice

I can't work locally because the apache/php setup on the dev server needs to be as close to production as possible. Mirroring is an option. I've looked into this before but never found a way that is instantaneous enough to be usable. – Minneapolis

Mirroring would need to be fast enough that changes are propagated to the fileserver pretty much straight away. I've tried RSE (couldn't make head nor tail of it). Rsync is too intermittent. I even tried using ANT to monitor the change deltas, but couldn't quite get it working. – Minneapolis

I've also tried WinSCP which has a realtime mirroring thing that does actually work for small folders, but it freaks out at the size of this project. And seems to take a lot of resources to monitor. – Minneapolis

Answer 3

Given it's subversioned, why not have the files locally, and use a post commit hook to update to the latest version on the dev server after every commit? (or have a specific string in the commit log (eg '##DEPLOY##') when you want to update dev, and only run the update when the post commit hook sees this string).

Apart from refresh speed-ups, the advantage of this technique is that you can have broken files that you are working on in eclipse, and the dev server is still ok (albeit with an older version of the code).

The disadvantage is that you have to do a commit to push your saved files onto the dev server.

Hungary

That would work but it's not my first preference. Commits can be slow with subclipse as it first gets the status from svn and then asks you to confirm. And I often don't really want to commit a file that I'm still working on. – Minneapolis

Answer 4

I solwed this problem by changing "File Transfer Buffer Size" at:

Window->Preferences->Remote Systems-Files and change "File transfer buffer size"-s Download (KB) and Upload (KB) values to high value, I set it to 1000 kb, by default it is 40 kb

Descent

Note: after change these settings for me restart was required and then used it for projects – Descent

Answer 5

Use offline folder feature in Windows by right-click and select "Make availiable offline".

It could save a lot of time and round trip delay in the file sharing protocol.

Spear

That still only synchronises periodically does it not? – Minneapolis

No. It keep everything up to date unless your network is broken. – Spear

Hmm. Unfortunately this seemed to make things much worse. Eclipse just bugs out when trying to refresh the workspace... it thinks there are always more files to refresh. 'Refresh process' count goes down for a bit and then jumps back up to like 500 without ever finishing. – Minneapolis

Answer 6

The use of svn externals with the revision flag for the non changing stuff might prevent subclipse from refreshing those files on update. Then again it might not. Since you'd have to make some changes to the structure of your subversion repository to get it working, I would suggest you do some simple testing before doing it for real.

Hungary

I've already found out from the subclipse mailing list that a full project refresh is unavoidable when switching. Basically subclipse has to tell eclipse to refresh the entire project so that the hidden .svn folders are rescanned. – Minneapolis