We run a relatively high volume content site. Like most content sites, the majority of each page is relatively static. The articles rarely change, making them good candidates for some form of static/edge caching. There are two big problems, though. Secondary page elements (nav, recent content lists, etc) change pretty frequently, quickly invalidating "full" cached pages. It's also quite common that we include more dynamic bits in a page, like user specific information, etc.
It would be really neat to have a reverse-proxy/load balancer that post-processed content and let us handle includes at the proxy/edge. The initial request to the backend would return a rough template, then the proxy software could process that template to complete it. The markup might look something like this:
<html>
<body>
<div id="content">
Lorem ipsum whackem smackem.
<%
dynamic "http://related.content.service/this/story"
%>
</div>
<div id="sidebar">
<%
dynamic do |request|
url = "http://my.user.service/user-widget.html"
if request.cookies.contains?("user_token")
url = "http://my.user.service/" + request.cookies["user_token"] + "/user-widget.html"
end
error_text = "User service not available"
{ :url => url, :timeout => 500, :error => error_text }
end
%>
</div>
</body>
</html>
What you'll see in that example is a small bit of Ruby that determines the included file based on a cookie value, then returns a hash with the URL to pull in, a timeout, and some default text to show in the event of an error. In theory, all the includes could be requested asynchronously as well.
My understanding is that Amazon does something like this. Various page components are generated by backend services, with strict timeout limits to ensure overall page speed. I was hoping their CDN service would include something like this, but it's not to be!
There's a W3 spec for Edge Side Includes (ESI) is almost what I want. There's very little support for it out there, however. It's available through Akamai, there's some Oracle software that does it, and the open source Varnish cache has a very basic implementation. It's also a really ugly XML format.
So the question is: what out there will let me do what I want? Is anyone else doing things in this way?