Suds is not reusing cached WSDLs and XSDs, although I expect it to

Question

I'm pretty sure suds is not caching my WSDLs and XSDs like I expect it to. Here's how I know that cached objects are not being used:

1. It takes about 30 seconds to create a client: client = Client(url)
2. The logger entries show consistent digestion of the XSD and WSDL files during the entire 30 seconds
3. Wireshark is showing consistent TCP traffic to the server storing the XSD and WSDL files during the entire 30 seconds
4. I see the files in the cache being updated each time I run my program

I have a small program that creates a suds client, sends a single request, gets the response, then ends. My expectation is that each time I run the program, it should fetch the WSDL and XSD files from the file cache, not from the URLs. Here's why I think that:

1. client.options.cache.duration is set to ('days', 1)
2. client.options.cache.location is set to c:\docume~1\mlin\locals~1\temp\suds and I see the cache files being generated and re-generated each time I run the program
3. For a moment I thought that maybe the cache is not reused between runs of a program, but I don't think a file cache would be used if that were the case, because an in-memory cache would do just fine

Am I misunderstanding how suds caching is supposed to work?

Sweetbrier

When I copied the WSDL and XSDs for this to the local file system, it only took about 3 seconds to load from there. Still too slow considering the small size of this web service definition. I have one web service that takes over 2 minutes for suds to load from the local file system. You don't want to know how long it takes to load from the URLs! – Sweetbrier

Answer 1

The problem is in the suds library itself. In cache.py, although ObjectCache.get() is always getting a valid file pointer, it's hitting an exception (EOFError) doing pickle.load(fp). When that happens, the file is just downloaded again.

Here's the sequence of events:

DocumentReader.open():

1. Trying http://172.28.50.249/wsdl/billingServices/v3.0/RequestScrubAddress.wsdl
2. Loading ObjectCache 51012453-document
3. Loading pickled object...
4. Exception raised:
5. Got None from cache
6. Downloading... Done
7. Saving FileCache 51012453-document... Done

So it doesn't really matter that the new cache file was saved, because the same thing happens the next time I run. This happens for ALL of WSDL and XSD files.

I fixed that problem by opening the cache file in binary mode when reading and writing. Specifically, the changes I made were in cache.py:

1) In FileCache.put(), change this line:

f = self.open(fn, 'w')

to

f = self.open(fn, 'wb')

2) In FileCache.getf(), change this line:

return self.open(fn)

to

return self.open(fn, 'rb')

I don't know the codebase well enough to know if these changes are safe, but it is pulling the objects from the file cache, the service is still running successfully, and loading the client went from 16 seconds down to 2.5 seconds. Much better if you ask me.

Hopefully this fix, or something similar can be introduced back into the suds main line. I've already sent this to the suds mailing list (fedora-suds-list at redhat dot com).

Sweetbrier

Worked like a charm. This issue has been very frustrating for me. Thank you! – Carroty

Did you get any feedback on this from the suds developers? I couldn't find your ticket... – Michaeline

I got feedback from the mailing list. We even had the fixes in a branch on BitBucket (which was since deleted), but the changes were never pulled into the project. Feel free to resubmit it. Follow the email thread here: lists.fedoraproject.org/pipermail/suds/2011-July/001481.html – Sweetbrier

Great, I found your answer after debugging for an hour and coming to exactly the same conclusions myself. ;) No seriously, thanks! – Piddle

I also pulled this into the suds fork I maintain with some patches I need: github.com/hheimbuerger/suds-gzip – Piddle

Original ticket & recently updated fork, seemingly lacking this patch. – Urbanism