Using **kwargs with SimpleXMLRPCServer in python

Question

I have a class that I wish to expose as a remote service using pythons SimpleXMLRPCServer. The server startup looks like this:

server = SimpleXMLRPCServer((serverSettings.LISTEN_IP,serverSettings.LISTEN_PORT))

service = Service()

server.register_instance(service)
server.serve_forever()

I then have a ServiceRemote class that looks like this:

def __init__(self,ip,port):
    self.rpcClient = xmlrpclib.Server('http://%s:%d' %(ip,port))

def __getattr__(self, name):
    # forward all calls to the rpc client
    return getattr(self.rpcClient, name)

So all calls on the ServiceRemote object will be forwarded to xmlrpclib.Server, which then forwards it to the remote server. The problem is a method in the service that takes named varargs:

@useDb
def select(self, db, fields, **kwargs):
    pass

The @useDb decorator wraps the function, creating the db before the call and opening it, then closing it after the call is done before returning the result.

When I call this method, I get the error "call() got an unexpected keyword argument 'name'". So, is it possible to call methods taking variable named arguments remotely? Or will I have to create an override for each method variation I need.

---

Thanks for the responses. I changed my code around a bit so the question is no longer an issue. However now I know this for future reference if I indeed do need to implement positional arguments and support remote invocation. I think a combination of Thomas and praptaks approaches would be good. Turning kwargs into positional args on the client through xmlrpclient, and having a wrapper on methods serverside to unpack positional arguments.

Epicene

Answer 1

You can't do this with plain xmlrpc since it has no notion of keyword arguments. However, you can superimpose this as a protocol on top of xmlrpc that would always pass a list as first argument, and a dictionary as a second, and then provide the proper support code so this becomes transparent for your usage, example below:

Server

from SimpleXMLRPCServer import SimpleXMLRPCServer

class Server(object):
    def __init__(self, hostport):
        self.server = SimpleXMLRPCServer(hostport)

    def register_function(self, function, name=None):
        def _function(args, kwargs):
            return function(*args, **kwargs)
        _function.__name__ = function.__name__
        self.server.register_function(_function, name)

    def serve_forever(self):
        self.server.serve_forever()

#example usage
server = Server(('localhost', 8000))
def test(arg1, arg2):
    print 'arg1: %s arg2: %s' % (arg1, arg2)
    return 0
server.register_function(test)
server.serve_forever()

Client

import xmlrpclib

class ServerProxy(object):
    def __init__(self, url):
        self._xmlrpc_server_proxy = xmlrpclib.ServerProxy(url)
    def __getattr__(self, name):
        call_proxy = getattr(self._xmlrpc_server_proxy, name)
        def _call(*args, **kwargs):
            return call_proxy(args, kwargs)
        return _call

#example usage
server = ServerProxy('http://localhost:8000')
server.test(1, 2)
server.test(arg2=2, arg1=1)
server.test(1, arg2=2)
server.test(*[1,2])
server.test(**{'arg1':1, 'arg2':2})

Hiroshige

The idea is good, but things don't work if there is more than one level of api calls (like server.level.test()). In such cases, the getattr function will not know if the "level" is not a function call and returns a proxy function for that as well, which fails. But this is a good idea and in the right direction! I ended up using this solution, along with a modification of Thomas Wouters answer below. Thanks! – Rebroadcast

Answer 2

XML-RPC doesn't really have a concept of 'keyword arguments', so xmlrpclib doesn't try to support them. You would need to pick a convention, then modify xmlrpclib._Method to accept keyword arguments and pass them along using that convention.

For instance, I used to work with an XML-RPC server that passed keyword arguments as two arguments, '-KEYWORD' followed by the actual argument, in a flat list. I no longer have access to the code I wrote to access that XML-RPC server from Python, but it was fairly simple, along the lines of:

import xmlrpclib

_orig_Method = xmlrpclib._Method

class KeywordArgMethod(_orig_Method):     
    def __call__(self, *args, **kwargs):
        if args and kwargs:
            raise TypeError, "Can't pass both positional and keyword args"
        args = list(args) 
        for key in kwargs:
            args.append('-%s' % key.upper())
            args.append(kwargs[key])
       return _orig_Method.__call__(self, *args)     

xmlrpclib._Method = KeywordArgMethod

It uses monkeypatching because that's by far the easiest method to do this, because of some clunky uses of module globals and name-mangled attributes (__request, for instance) in the ServerProxy class.

Kosse

What is monkeypatching? Quick google search didn't give me a defintion, only an example using decorators. – Epicene

monkeypatching is modifying a module or other piece of code without changing the source of that code. See en.wikipedia.org/wiki/Monkey_patch – Kosse

monkeypatching is often considered harmful – Jardine

Indeed it is. However, try to replace the '_Method' class used by xmlrpclib without it. Please :-) You'll see that there are more harmful things than monkeypatching. – Kosse

@Thomas, I think I did, what's so harmful about it? – Jardine

What's harmful is that the ServerProxy class uses the _Method global directly (no way to override that choice) and the only way to change that is to subclass and override getattr. But getattr needs access to self.__request, which because of name-mangling it can't. How did you do it? – Kosse

Nice solution. Just a minor point: there is no need to restrict passing only one of positional and keyword args. I ended up sending the args like this: ("hello", "world", "**kwargs", "text", "Browser"), which is nothing but ("hello", "world", text="Browser"). The server automatically translates this. If someone is willing to monkey patch, they can do as Thomas Wouters pointed or they could simply call it in the expanded way. – Rebroadcast

Answer 3

As far as I know, the underlying protocol doesn't support named varargs (or any named args for that matter). The workaround for this is to create a wrapper that will take the **kwargs and pass it as an ordinary dictionary to the method you want to call. Something like this

Server side:

def select_wrapper(self, db, fields, kwargs):
    """accepts an ordinary dict which can pass through xmlrpc"""
    return select(self,db,fields, **kwargs)

On the client side:

def select(self, db, fields, **kwargs):
    """you can call it with keyword arguments and they will be packed into a dict"""
    return self.rpcClient.select_wrapper(self,db,fields,kwargs)

Disclaimer: the code shows the general idea, you can do it a bit cleaner (for example writing a decorator to do that).

Amato

Answer 4

Using the above advice, I created some working code.

Server method wrapper:

def unwrap_kwargs(func):
    def wrapper(*args, **kwargs):
        print args
        if args and isinstance(args[-1], list) and len(args[-1]) == 2 and "kwargs" == args[-1][0]:
            func(*args[:-1], **args[-1][1])
        else:
            func(*args, **kwargs)
    return wrapper

Client setup (do once):

_orig_Method = xmlrpclib._Method

class KeywordArgMethod(_orig_Method):     
    def __call__(self, *args, **kwargs):
        args = list(args) 
        if kwargs:
            args.append(("kwargs", kwargs))
        return _orig_Method.__call__(self, *args)

xmlrpclib._Method = KeywordArgMethod

I tested this, and it supports method with fixed, positional and keyword arguments.

Epicene

I dislike code that intrudes into other modules. I tend to wrap completely around other code, that is more flexible, less cryptic, and less likely to blow up in your face. – Jardine

I agree, but still a python newb, so just trying the path off least resistance. And since everything is in the same project atm. it's not that big off and issue. – Epicene

Answer 5

As Thomas Wouters said, XML-RPC does not have keyword arguments. Only the order of arguments matters as far as the protocol is concerned and they can be called anything in XML: arg0, arg1, arg2 is perfectly fine, as is cheese, candy and bacon for the same arguments.

Perhaps you should simply rethink your use of the protocol? Using something like document/literal SOAP would be much better than a workaround such as the ones presented in other answers here. Of course, this may not be feasible.

Spoil