Why is debugging in eclipse/pydev so slow for my python program?

Question

I have a relatively simple (no classes) python 2.7 program. The first thing the program does is read an sqlite dbase into a dictionary. The database is large, but not huge, around 90Meg on disk. It takes about 20 seconds to read in. After reading in the database I initialize some variables, e.g.

localMax = 0
localMin = 0
firstTime = True

When I debug this program in Eclipse-3.7.0/pydev - even these simple lines - each single-step in the debugger eats up 100% of a core, and takes between 5 and 10 seconds. I can see the python process goes to 100% cpu for 10 seconds. Single-step... wait 10 seconds... single-step... wait 10 seconds... If I debug at the command line just using pdb, no problems. If I'm not debugging at all, the program runs at "normal" speed, nothing strange like in Eclipse.

I've reproduced this on a dual core Win7 PC w/ 4G memory, my 8 core Ubuntu box w/ 8G of memory, and even my Mac Air. How's that for multi-platform development! I kept thinking it would work somewhere. I'm never even close to running out of memory at any time.

On each Eclipse single-step, why does the python process jump to 100% CPU, and take 10 seconds?

Lucielucien

no idea, but I have a hunch that it might be related to Eclipse debugger view which tries to output local variables in the watchdog window and somehow this process is failing (e.g. it tries to print a dump of SQlite database as a string). The only way to go forward with this one is that you post a full program source code which expresses the problem in a repetable manner. – Figwort

This may save your day – Blimey

In my experience I just have raged so many times with eclipse taking so much memory and been slow in some parts that I decided to learn pdb. Its the old way but its fast and it will never fail you. I use an enhanced version called ipdb btw. – Mareld

It would make sense if Eclispe was trying to "watch" my large dictionary, Eclipse would "get busy" on each single-step. Is there any way to have Eclipse ignore my large Dictionary? Maybe if I just close the Variable window in the Eclipse GUI... – Lucielucien

I use Eclipse on Windows and Linux to develop, manage and debug huge Java codebase and very simple Python projects. In two years I have never had slowness issues working with Java, while in this regard PyDev development resulted in a PITA, even on multi core systems. Unfortunately I simply came to the conclusion that actually, without accepting (sometimes big) compromises, the Eclipse platform is not the best tool for Python programming. – Denitadenitrate

Answer 1

Here is a good enough workaround, based on Mikko Ohtamaa's hint. I just verified the following on my Mac Air:

• If I simply close the 'Variables' window in the Eclipse GUI, I can single step through the code at normal speed. Which is great, but, uh, I don't have the Variables window.
• For any variable I want to see, I can hover my cursor over the variable and see the value. I didn't attempt to hover over my large dictionary that is the culprit here.
• I can also right-click on any variable and add a 'Watch', which brings up an 'Expressions' window. In this case the variable is just a degenerate case (very simple case) of an 'expression.

So, the workaround for me is to close the Eclipse Variable window, and use the Expressions window to selectively view variables. A pain, but for the debugging I'm doing it is better than pdb.

Lucielucien

Answer 2

I simply commented this line out:

np.set_printoptions(threshold = 'nan')

It seems eclipse is trying to keep up with too much information.

Isidraisidro

where is this line located? – Denitadenitrate

Answer 3

In my case I have the qtThread debug set enter image description here set

Keitt

The link is automatically added by stackoverflow – Keitt