How do you deserialize an object from bytes in osgi

Question

In my osgi application I have three bundles, travel.api, table.api and utils. travel.api depends on table.api which depends on utils. Note that travel.api doesn't directly depend on utils. I use aQute Bnd to generate the manifests and I believe it is working fine. The manifests are displayed below.

There is a class called PageData that has a field of type TableData, which in turn has a field of type TestObject. PageData is located in travel.api, TableData is located in table.api and TestObject is located in utils. This all works fine when the bundles are loaded. The problem comes when I receive an array of bytes representing a PageData object. I have to deserialize it in the travel.api bundle. This shouldn't be a problem as that is where it is defined. I use org.jboss.netty.handler.codec.serialization.ObjectDecoderInputStream and pass in the classloader from the travel.api bundle. The exception shown below is thrown but basically it says:

Caused by: java.lang.ClassNotFoundException: com.openaf.utils.TestObject not 
    found by travel.api [9].

Now this makes sense because if you look at the Import-Package for travel.api you will see that com.openaf.utils (where TestObject is located) isn't listed. If I add this package then it is correctly deserialized. However, this doesn't seem like a good general solution as I would have to go through every field that PageData uses and ensure that they are all imported in this module, and recursively on every field contained by those fields etc.

Am I doing something completely wrong here?

What is the best way to deserialize an object when using OSGi?

If I'm doing it correctly and I have to specify all the "deep" imports, is there a way to get Bnd to do a "deep" generation?

Any help would be greatly appreciated!

I'm using felix v4 as my osgi library.

Manifest-Version: 1
Bnd-LastModified: 1355404320862
Bundle-ManifestVersion: 2
Bundle-Name: travel.api
Bundle-SymbolicName: travel.api
Bundle-Version: 0
Created-By: 1.7.0_07 (Oracle Corporation)
Export-Package: com.openaf.travel.api;uses:="scala.runtime,scala,scala.c
 ollection,com.openaf.pagemanager.api,scala.reflect,com.openaf.table.api
 ";version="0.0.0"
Import-Package: com.openaf.pagemanager.api,com.openaf.table.api,scala,sc
 ala.collection,scala.reflect,scala.runtime
Tool: Bnd-1.44.0

Manifest-Version: 1
Bnd-LastModified: 1355404158858
Bundle-ManifestVersion: 2
Bundle-Name: table.api
Bundle-SymbolicName: table.api
Bundle-Version: 0
Created-By: 1.7.0_07 (Oracle Corporation)
Export-Package: com.openaf.table.api;uses:="scala.runtime,scala,scala.co
 llection,scala.reflect,scala.collection.immutable,scala.collection.gene
 ric,com.openaf.utils";version="0.0.0"
Import-Package: com.openaf.utils,scala,scala.collection,scala.collection
 .generic,scala.collection.immutable,scala.reflect,scala.runtime
Tool: Bnd-1.44.0

Manifest-Version: 1
Bnd-LastModified: 1355404158801
Bundle-ManifestVersion: 2
Bundle-Name: utils
Bundle-SymbolicName: utils
Bundle-Version: 0
Created-By: 1.7.0_07 (Oracle Corporation)
Export-Package: com.openaf.utils;uses:="scala.runtime,scala,scala.collec
 tion,scala.reflect";version="0.0.0"
Import-Package: scala,scala.collection,scala.reflect,scala.runtime
Tool: Bnd-1.44.0

java.io.InvalidClassException: failed to read class descriptor
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1585)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.jboss.netty.handler.codec.serialization.ObjectDecoderInputStream.readObject(ObjectDecoderInputStream.java:115)
at com.openaf.rmi.common.DefaultObjectEncoder$.decode(RMICommon.scala:33)
at com.openaf.rmi.client.ClientHandler.messageReceived(ClientPipelineFactory.scala:43)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:363)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:345)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:211)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:372)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:246)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: com.openaf.utils.TestObject not found by travel.api [9]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.jboss.netty.handler.codec.serialization.ClassLoaderClassResolver.resolve(ClassLoaderClassResolver.java:30)
at org.jboss.netty.handler.codec.serialization.CachingClassResolver.resolve(CachingClassResolver.java:39)
at org.jboss.netty.handler.codec.serialization.CompactObjectInputStream.readClassDescriptor(CompactObjectInputStream.java:55)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
... 28 more

Thanks, Nick.

Accent

In some scenarios it is possible to use host and fragment bundles. This way you only have a single classloader and the problem is avoided. Like I said, it only covers a small set of scenarios. – Desecrate

We think we are striking the same issue with our application but we don't get the ClassNotFoundException. Instead we get a java.io.StreamCorruptedException: invalid type code: 00 error. How did you fix it in the end? – Coliseum

Answer 1

This actually sounds like a serious shortcoming in deserialization? A decent deserializer should use the class loader of the class that causes the load. The given class loader should only be used for the top level object since the there is no parent object yet.

So in this case the given class loader is used to load PageData. PageData's loader is used to load TableData, and TableData's loader must be used to load TestObject. There is no logical reason why this should fail unless the deserializer you use is really brain damaged since this is the model the VM uses to load classes. I am surprised that the Java deserializer does this, I consider this behavior a serious error since it uses different rules than the VM.

Serialization is a problem in OSGi because modularity is about hiding implementation classes; deserialization has a tendency to want to access these private classes, the antithesis of modularity. However, there are very good solutions to this (which does not include Dynamic-ImportPackage, that is reverting to JAR hell in a more complicated and expensive way than just using plain Java). The basic trick is to have a root object from a public API that has access to the private/transiently needed classes. Hmm, doesn't this sound like a service?

Solution

Looking at how negative people are about this, a small example how you can solve the problem with Java Serialization (that is, ObjectInputStream, and ObjectOutputStream). In your question you mention ObjectDecoderInputStream, a class I am not familiar with.

The setup is:

Bundle A:    class a.A { B b; }   (import b)
Bundle B:    class b.B { C c; }   (import c)
Bundle C:    class c.C { }

So lets first serialize an object:

ByteArrayOutputStream bous = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bous);

oos.writeObject(this);
oos.close();

Now the hard part. We override the resolveObject method, this gives us a chance to to actually do proper class loading ...

ObjectInputStream in = new ObjectInputStream(new ByteArrayInputStream(bous.toByteArray())) {
    Set<ClassLoader>    lhs = new LinkedHashSet<ClassLoader>();
    {
        // Keep a set if discovered class loaders
        lhs.add(getClass().getClassLoader());
    }

    @Override
    protected Class< ? > resolveClass(ObjectStreamClass desc) 
         throws ClassNotFoundException, IOException {

         for (ClassLoader cl : lhs) try {
             Class< ? > c = cl.loadClass(name);

             // we found the class, so we can use its class loader,
             // it is in the proper class space  if the uses constraints 
             // are set properly (and you're using bnd so you should be ok)

             lhs.add(c.getClassLoader());

             // The paranoid among us would check
             // the serial uuid here ...
             // long uuid = desc.getSerialVersionUID();
             // Field field = c.getField("serialVersionUID");
             // assert uuid == field.get(null)

             return c;
         } catch (Exception e) {
           // Ignore
         }

         // Fallback (for void and primitives)
         return super.resolveClass(desc);
     }
 };

 // And now we've successfully read the object ...

 A clone = (A) in.readObject();

Please not that this only works as long as the transient graph is properly exported. I.e. if you can do new TableData then this should also work. An example that does not work is if you for example get an implementation from an interface. The interface class is not connected to the impl. class. I.e. if you had a TableDataImpl that extended TableData you would be screwed. In those cases you need some service to find the "domain" of the implementation.

Good luck.

Nicotiana

Thanks for the answers. Peter, does Bnd have an option that allows it to find the transiently needed classes? – Accent

No, because this would violate the law of Demeter. A module depends on its neighbors but it should depend on its neighbor's neighbors. Again, this is a problem with your serialization provider. Use another serialization mechanism (JSON is much nicer). A very interesting serializer that handles this problem correctly is actuall in bnd: aQute.lib.json – Nicotiana

I should have mentioned that the deserializer I'm using is this one org.jboss.netty.handler.codec.serialization.ObjectDecoderInputStream as I'm getting the objects over the wire. Now I understand I can use your example when using ObjectDecoderInputStream. Thanks. – Accent

Answer 2

There is no other way how to do that AFAIK.

You have to explicitly state all dependencies the deserialized object tree contains in that bundle where you are trying to do that.

You could try to put all domain object into one bundle, let's say model and then let all other bundles depend on it.

Sexagenarian

Not necessary ... in this case all class are reachable from the root so there is NO need to do hard work! – Nicotiana

If you deserialize an object, then it's used the actual class loader and all classes in the deserialized object tree(including the object itself) must be reachable to that classloader, isn't that right? If so, then in this case all classes are not reachable as it throws ClassNotFoundException. – Sexagenarian

No, it must be reachable through the class's class loader. So graphs of class loaders work fine if you can hop from class to class. I.e. A -> B -> C is ok when you deserialize A. Since A's class loader must see B, and B's class loader must see C. There is no need for A to see C. – Nicotiana

Look up about 600 pixels on your screen ... :-) – Nicotiana

Answer 3

Yes, this is a tricky one. In many cases the problem is even worse, it might not even be known which bundles will be required to deserialize a stream. For these, the compile time dependencies are just not the same as runtime dependencies.

To tackle these situations, I've used either DynamicImports-Package or use the BundleWiring API. Both worked out pretty well, Dynamic imports are easier though.

I'd say isolate the part that needs this class loading as much as you can in a separate bundle, and have that bundle use a DynamicImport.

Good luck, Frank

Schnorkle

a) in this case there are no private fields b) as I argued in the other issue, for API -> impl. deserialization there are good solutions. – Nicotiana