How do I extend a python module? Adding new functionality to the `python-twitter` package

Question

What are the best practices for extending an existing Python module – in this case, I want to extend the python-twitter package by adding new methods to the base API class.

I've looked at tweepy, and I like that as well; I just find python-twitter easier to understand and extend with the functionality I want.

I have the methods written already – I'm trying to figure out the most Pythonic and least disruptive way to add them into the python-twitter package module, without changing this modules’ core.

Scriabin

You have to ask yourself what it is you are trying to accomplish that cannot be solved for you by subclassing the functionality from the existing module. – Pompea

Interesting thread on the Python news group on what I originally was searching for that led me to this Q&A, thread is from like, 13 years back: mail.python.org/pipermail/python-dev/2002-June/024839.html – Chickabiddy

Answer 1

A few ways.

The easy way:

Don't extend the module, extend the classes.

exttwitter.py

import twitter

class Api(twitter.Api):
    pass 
    # override/add any functions here.

Downside : Every class in twitter must be in exttwitter.py, even if it's just a stub (as above)

A harder (possibly un-pythonic) way:

Import * from python-twitter into a module that you then extend.

For instance :

basemodule.py

 class Ball():
    def __init__(self,a):
        self.a=a
    def __repr__(self):
        return "Ball(%s)" % self.a

def makeBall(a):
    return Ball(a)

def override():
    print "OVERRIDE ONE"

def dontoverride():
    print "THIS WILL BE PRESERVED"

extmodule.py

from basemodule import *
import basemodule

def makeBalls(a,b):
    foo = makeBall(a)
    bar = makeBall(b)
    print foo,bar

def override():
    print "OVERRIDE TWO"

def dontoverride():
    basemodule.dontoverride()
    print "THIS WAS PRESERVED"

runscript.py

import extmodule

#code is in extended module
print extmodule.makeBalls(1,2)
#returns Ball(1) Ball(2)

#code is in base module
print extmodule.makeBall(1)
#returns Ball(1)

#function from extended module overwrites base module
extmodule.override()
#returns OVERRIDE TWO

#function from extended module calls base module first
extmodule.dontoverride()
#returns THIS WILL BE PRESERVED\nTHIS WAS PRESERVED

I'm not sure if the double import in extmodule.py is pythonic - you could remove it, but then you don't handle the usecase of wanting to extend a function that was in the namespace of basemodule.

As far as extended classes, just create a new API(basemodule.API) class to extend the Twitter API module.

Maternity

thanks, I was looking for something like that to override json.dumps over all my project. (while keeping the other functions) – Pelf

I wouldn't consider this to be unpythonic. It's useful tricks like this that can make using python so powerful. Just be obvious about what you are doing. Try doing that in Java or C++ :-) – Cynthiacynthie

Answer 2

Don't add them to the module. Subclass the classes you want to extend and use your subclasses in your own module, not changing the original stuff at all.

Heir

Ok, I'm not familiar with sub-classing in python yet, (I'm an engineer, still getting used to OO style), do you have an favorite resources? So, basically there is nothing special about a module... I can just write a new module that uses python-twitter API class as a base class and then add in the methods I want..? – Scriabin

@user319045, To subclass you'd just write your own module that creates a new class using the syntax class SomeName(othermodule.TheClassYouAreExtending):... and SomeName will depend back on the original class in a well-defined way. This is called inheritance and is how you would normally add methods to a class. This and other topics are covered in tinyurl.com/thinkcspy2e, which is a book that will introduce you to Python and OOP. It is available free online and in print and is better than some of the other options I've seen. – Heir

The official tutorial docs.python.org/tut is also pretty good, but may prove harder for people who are not experienced programmers with good understanding of the programming paradigms they are using. – Heir

@user319045, Inheritance is probably overused. There's a real chance that what you really want is just to define functions that take an instance of one of python-twitter's classes as an argument or to define a new class that has an instance of one of python-twitter's classes as an attribute. (The latter technique is called composition when you need to distinguish it from inheritance.) I cannot tell you for sure what you want without understanding what problem you are solving exactly. – Heir

Answer 3

Here’s how you can directly manipulate the module list at runtime – spoiler alert: you get the module type from types module:

from __future__ import print_function
import sys
import types
import typing as tx

def modulize(namespace: tx.Dict[str, tx.Any],
             modulename: str,
             moduledocs: tx.Optional[str] = None) -> types.ModuleType:

    """ Convert a dictionary mapping into a legit Python module """

    # Create a new module with a trivially namespaced name:
    namespacedname: str = f'__dynamic_modules__.{modulename}'
    module = types.ModuleType(namespacedname, moduledocs)
    module.__dict__.update(namespace)

    # Inspect the new module:
    name: str = module.__name__
    doc: tx.Optional[str] = module.__doc__
    contents: str = ", ".join(sorted(module.__dict__.keys()))
    print(f"Module name:      {name}")
    print(f"Module contents:  {contents}")
    if doc:
        print(f"Module docstring: {doc}")

    # Add to sys.modules, as per import machinery:
    sys.modules.update({ modulename : module })

    # Return the new module instance:
    return module

… you could then use such a function like so:

ns = {
         'func' : lambda: print("Yo Dogg"), # these can also be normal non-lambda funcs
    'otherfunc' : lambda string=None: print(string or 'no dogg.'),
      '__all__' : ('func', 'otherfunc'),
      '__dir__' : lambda: ['func', 'otherfunc'] # usually this’d reference __all__
}

modulize(ns, 'wat', "WHAT THE HELL PEOPLE")
import wat

# Call module functions:
wat.func()
wat.otherfunc("Oh, Dogg!")

# Inspect module:
contents = ", ".join(sorted(wat.__dict__.keys()))
print(f"Imported module name:      {wat.__name__}")
print(f"Imported module contents:  {contents}")
print(f"Imported module docstring: {wat.__doc__}")

… You could also create your own module subclass, by specifying types.ModuleType as the ancestor of your newly declared class, of course; I have never personally found this necessary to do.

(Also, you don’t have to get the module type from the types module – you can always just do something like ModuleType = type(os) after importing os – I specifically pointed out this one source of the type because it is non-obvious; unlike many of its other builtin types, Python doesn’t offer up access to the module type in the global namespace.)

The real action is in the sys.modules dict, where (if you are appropriately intrepid) you can replace existing modules as well as adding your new ones.

Pietro

Answer 4

Say you have an older module called mod that you use like this:

import mod

obj = mod.Object()
obj.method()
mod.function()
# and so on...

And you want to extend it, without replacing it for your users. Easily done. You can give your new module a different name, newmod.py or place it by same name at a deeper path and keep the same name, e.g. /path/to/mod.py. Then your users can import it in either of these ways:

import newmod as mod       # e.g. import unittest2 as unittest idiom from Python 2.6

or

from path.to import mod    # useful in a large code-base

In your module, you'll want to make all the old names available:

from mod import *

or explicitly name every name you import:

from mod import Object, function, name2, name3, name4, name5, name6, name7, name8, name9, name10, name11, name12, name13, name14, name15, name16, name17, name18, name19, name20, name21, name22, name23, name24, name25, name26, name27, name28, name29, name30, name31, name32, name33, name34, name35, name36, name37, name38, name39

I think the import * will be more maintainable for this use-case - if the base module expands functionality, you'll seamlessly keep up (though you might shade new objects with the same name).

If the mod you are extending has a decent __all__, it will restrict the names imported.

You should also declare an __all__ and extend it with the extended module's __all__.

import mod
__all__ = ['NewObject', 'newfunction']
__all__ += mod.__all__   
# if it doesn't have an __all__, maybe it's not good enough to extend
# but it could be relying on the convention of import * not importing
# names prefixed with underscores, (_like _this)

Then extend the objects and functionality as you normally would.

class NewObject(object):
    def newmethod(self):
        """this method extends Object"""

def newfunction():
    """this function builds on mod's functionality"""

If the new objects provide functionality you intend to replace (or perhaps you are backporting the new functionality into an older code base) you can overwrite the names

Chickabiddy

Answer 5

May I suggest not to reinvent the Wheel here? I'm building a >6k line Twitter Client for 2 month now, at first I checked python-twitter too, but it's lagging a lot behind the recent API changes,, Development doesn't seem to be that active either, also there was(at least when I last checked) no support for OAuth/xAuth).

So after searching around a bit more I discovered tweepy:
http://github.com/joshthecoder/tweepy

Pros: Active development, OAauth/xAuth and up to date with the API.
Chances are high that what you need is already in there.

So I suggest going with that, it's working for me, the only thing I had to add was xAuth(that got merge back to tweepy :)

Oh an a shameless plug, if you need to parse Tweets and/or format them to HTML use my python version of the twitter-text-* libraries:
http://github.com/BonsaiDen/twitter-text-python

This thing is unittestetd an guaranteed to parse Tweets just like Twitter.com does it.

Promise

Thanks, I've looked at tweepy, I just seem to like the style of python-twitter better. I'm not building a twitter client, just a tool suite to handle a few things very well with regard to research. – Scriabin

Answer 6

Define a new class, and instead of inherit it from the class you want to extend from the original module, add an instance of the original class as an attribute to your new class. And here comes the trick: intercept all non-existing method calls on your new class and try to call it on the instance of the old class. In your NewClass just define new or overridden methods as you like:

import originalmodule

class NewClass:
    def __init__(self, *args, **kwargs):
        self.old_class_instance = originalmodule.create_oldclass_instance(*args, **kwargs)

    def __getattr__(self, methodname):
        """This is a wrapper for the original OldClass class.

        If the called method is not part of this NewClass class,
        the call will be intercepted and replaced by the method
        in the original OldClass instance.
        """
        def wrapper(*args, **kwargs):
            return getattr(self.old_class_instance, methodname)(*args, **kwargs)
        return wrapper

    def new_method(self, arg1):
        """Does stuff with the OldClass instance"""
        thing = self.old_class_instance.get_somelist(arg1)
        # returns the first element only
        return thing[0]

    def overridden_method(self):
        """Overrides an existing method, if OldClass has a method with the same name"""
        print("This message is coming from the NewClass and not from the OldClass")

In my case I used this solution when simple inheritance from the old class was not possible, because an instance had to be created not by its constructor, but with an init script from an other class/module. (It is the originalmodule.create_oldclass_instance in the example above.)

Gastrotrich