Navigating a big Python codebase faster

Question

As programmers we read more than we write. I've started working at a company that uses a couple of "big" Python packages; packages or package-families that have a high KLOC. Case in point: Zope.

My problem is that I have trouble navigating this codebase fast/easily. My current strategy is

• I start reading a module I need to change/understand
• I hit an import which I need to know more of
• I find out where the source code for that import is by placing a Python debug (pdb) statement after the imports and echoing the module, which tells me it's source file
• I navigate to it, in shell or the Vim file explorer.
• most of the time the module itself imports more modules and before I know it I've got 10KLOC "on my plate"

Alternatively:

• I see a method/class I need to know more of
• I do a search (ack-grep) for the definition of that method/class across the whole codebase (which can be a pain because the codebase is partly in ~/.buildout-eggs)
• I find one or more pieces of code that define that method/class
• I have to deduce which one of them is the one I need to read

This costs a lot of time, which is understandable for a big codebase. But I get the feeling that navigating a large and unknown Python codebase is a common enough problem.

So I'm looking for technical tools or strategic solutions for this problem. ...

I just can't imagine hardcore Python programmers using the strategies outlined above.

Chewning

github.com/mkomitee/vim-gf-python may interest you, or more simply: vim.wikia.com/wiki/Automatically_add_Python_paths_to_Vim_path – Allomerism

Haha, as they say, "first world problems" ... if we could do this with projects in C (Which we usually can't unless raw machine code is your cup of tea) the KLOC would probably be closer to 100K ;p – Spelunker

Answer 1

on Vim, I like NERDTree (a file browser) and taglist.vim (source code browser --> http://www.vim.org/scripts/script.php?script_id=273)

also in Vim, you can use CTRL-] to jump to a definition (:h CTRL-]):

1. download exuberant ctags http://ctags.sourceforge.net/
2. follow the install directions and put it somewhere on your PATH
3. from the 'root' directory of your source code, make a tags file from the shell: "ctags -R"
4. (make sure you have :set noautochdir, and make sure :pwd is the root directory from step 3)
5. go into Vim, cursor over some function or class name, hit CTRL-]

by default, if there's multiple matches for the tag, it shows you everywhere it was imported, and where it was declared

if the tag only has one match, it immediately jumps to it

...then use Ctrl+O and Ctrl+I to move back and forth from where you were

(repeat above steps for the source code of particular libraries you use, i usually keep a separate Vim window open to study stuff)

Gresham

+1 for Exuberant Ctags. The original "IDE" was vi plus ctags, and vim is even better than vi and Exuberant Ctags is even better than the original ctags. – Zinc

Note that ctags -R may not work on OSX, which ships with a version of ctags without the -R option. See #8996887. – Namely

Just to add - I use the ranger file manager which uses vim navigation and integrates well. – Quick

Answer 2

I use ipython's ?? command

You just need to figure out how to import the things you want to look for, then add ?? to the end of the module or class or function or method name to view their source code. And the command completion helps on figuring out long names as well.

Lubow

Answer 3

Try red pill: https://github.com/klen/python-mode

Krissy

This is a bad answer. It is 1) a link only answer 2) to the poster's own project 3) which is dead. – Public

Update: This link is not dead. I've heard a lot of good things about python-mode. I've been using it for a few days now and really like it. – Kentiga