You basically need to have an understanding where to look (API function, hook, system... ) when you want to do X. There is really no need to memorize all hooks/functions in detail with all the arguments and stuff. That's something you can easily look up. Especially if you're using an IDE with I suggest (Using Netbeans myself).
Especially when you're altering stuff, try to develop some techniques to quickly figure out what code is responsible for the stuff you want to change. One example is to look at the hook_menu() definition of the module that does it and then check out the page callback and skim through the code. Things to look up: Are there hooks you can use, is it a form (if yes, what is the form_id, how is the form structured) and so on.
The best and maybe only way to get there (knowing where too look) is exercise. Every time you do something, you'll be faster the next time when you have to do something similar. I think what also helps is working on core/contrib modules together with others. You not only get to learn these modules better, you also learn how to read and understand code written by others better and you improve your own coding style.
Try to utilize proven, generic "building block" modules like Views, Flags, Panels, CCK/Field and so on. Then, the heavy lifting is done by these modules and you only need to provide the glue code to properly integrate them with your site. Might take a bit more time the first time you use these modules but you will likely save a lot time after that.
That having said, I'm not sure if the goal should be to build modules fast. I'd say the goal is to build modules better. Try to make them generic, secure, flexible, theme-able and so on with the goal to re-use these modules on the next site your building, when you need something similar.