Recommended Pattern for Lazy-loading Portions of Object Graph from Cache

Question

I'm using memcache behind a web app to minimize the hits to our SQL database. I'm storing C# objects into this cache by marking them with SerializableAttribute. We make heavy use of dependency injection via Ninject in our app.

Some of these objects are large, and I'd like to break them up. However, they come from a single stored procedure call (i.e. one stored procedure call gets cooked into the full object graph), and I'd like to be able to break these objects up and lazy-load specific subgraphs from the cache separately rather than load the entire object graph into memory all at once.

What are some patterns that would help me accomplish this?

Moorish

I always check with Martin Fowler first: martinfowler.com/eaaCatalog/lazyLoad.html – Chuipek

So I guess one my questions would be, why is the object so large -- or rather, how is it being used that you can't just send back the fully loaded object? Is it being returned over a service or something? If that's the case, then I think you need to segregate those contracts more appropriately, which sounds like it will require some pretty hefty refactoring. – Corncob

If I understand you correctly, I'm pretty sure you can do this with our library Patterns.NET @ www.nubilosoft.com . It sounds as simple as doing Lazy<T>.Create(..), which basically wraps a lazy-loading wrapper around an object. Also not sure if it applies here, but I'd also have a look at the Flyweight. – Triny

@Sinaesthetic: In this case, the large-grain problem of the stored proc call is a reflection of a poorly-designed legacy schema, the modification of which is out-of-scope. – Moorish

How is this/these object(s) being used in the end? Are they being piped out of a service? I ask because I mean, if the object is already pulled out of the database and built, and you cache it, then... it's already there, so I'm not sure I see the benefit of attempting to "lazy load" something that's already in memory. ?? – Corncob

If what you mean is that you're building many objects and caching them but are just trying to find a pattern that allows you to lazy load THOSE objects into another object which you have created, then the pattern below will still work. But instead of the database being your repository, you use your cache instead. Then just use the typical getter/setter method get{ return _myProperty ?? (_myProperty = lazyGetFromCacheRepository()) }; – Corncob

In this scenario, it's a web app. Some requests use some subgraphs of the big object, other requests use other subgraphs, often with quite a bit of overlap. Often, the root of the object graph is all that's needed, though. – Moorish

I might actually approach this the same way. Since your big momma object is being built from the database whether you want it or not, just cache that and transfer the pieces that you need from it into dtos that are sent back to the client, using the same pattern that I put in my answer. So if Obj1 contains member1 and member2, and the client only needs member2, then create a dto class for member2 that only has what the client needs, then build that from your cached object and return it. – Corncob

This wouldn't actually require lazy loading and would give you the opportunity to clean up the overall service pattern. Have the client request the specific endpoint that calls the right operation to build the correct dto. – Corncob

I added another diagram to my answer – Corncob

Answer 1

As far as patterns go, I'd say the one large complex object that's built from a single stored procedure is suspect. I'm not sure if your caching is a requirement or just the current state of its implementation.

The pattern that I'm used to is a type of repository pattern, using operations that fill specific contracts. And those operations house one or many datasources that call stored procedures in the database that will be used to build ONE of those sub-graphs you speak of. With that said, if you're going to lazy load data from a database, then I can only assume that many of the object members are not used much of the time which furthers my point - break that object up.

A couple things about it:

• It can be chatty if the entire object is being used regularly
• It is fully injectable via the Operations
• The datasources contain the reader for the specific object, thus only performing ONE task (SOLID)
• Can be modified to use Entity Framework, without too much fuss
• Can be designed to implement an interface, making it more reusable
• Will require you to break up that proc into smaller, chewable pieces, which will likely only benefit you in the long run.
• The complex object shown in this diagram really shouldn't exist if only parts of it are going to be used. Instead, consider segregating those objects. However, it really depends on how this object is being used.

UPDATE:

Using your cache as the repository, I would probably approach it like this:

So basically, you store the legacy object, but in your operations, you use them to build more relavent DTOs that are returned to the client.

Corncob

"[T]he one large complex object that's built from a single stored procedure is suspect." You're absolutely correct; however, this is a design constraint for me. I can't change this. – Moorish

Answer 2

I know NHibernate does lazy loading buy replacing objects with proxy objects. Then in the proxy object there is some kind of check that causes the loading of the real object the first time you try to access the object.

I'm not sure of any Design Patterns that would cover that, but you could look at the Nhibernate source code.

A down side of using proxy objects is you have to be careful with inheritance and type checks as you could be checking the type of the proxy and not the actual object.

Darrow