What is the best way to handle release management for software with multiple versions?

Question

My company builds a web application for Real Estate Institutions -- initially coded in classic ASP with an incremental migration to .NET. Essentially, its a website with backend DB, mixed with custom windows services/dll's. Pretty standard for .NET apps.

In my past companies, we had a traditional software design life cycle. We built releases of our products, and when we released, ALL customers received the same code. Product requirements were filtered through our engineering team, sent to QA for testing on local staging environments, then pushed to production.

This company has multiple versions of our product for multiple customers. Basically, customer A can be on release 1.5, customer B on 1.6, and customer C on 2.0. We do this because the institutions that use our application have strict requirements on anything that changes that affects their users. If a customer is happily fine with release 1.5, they stay there, even though release 2.0 has all the latest bells and whistles. Customers actually push back on the upgrade because the new 'features' actually HURT their user base by causing confusion.

Supporting this type of life cycle is fine when you're small, but as you grow to dozens or hundreds of customers, its a strain on our DEV, DBA's, QA, not to mention our support team. Now we're in a situation where we can only schedule 6-8 sites a week that can be updated as requirements come in. This forces us to make other sites wait 2-4 months to get even minor updates on their sites. Any production issue or bug that needs immediate attention screws things up even more -- because a site already scheduled to receive an update needs to get de-prioritized to make time.

Sorry this is so long, but ANY help is appreciated. The earlier we make some changes to get us on a better release schedule the better. Thanks!

Dexterdexterity

What version control and automated build system are you using right now? If I understand correctly, since you are getting so many version of the same website/app to maintain you are not being able to attend enough of them at the same time, hence the delay. Is that right? – Excursive

We use a combination of things. We use a product called Vault for our version control and we use a plugin called CC.NET (CruiseControl.Net) which is a continuous integration/build server which monitors our repositories and generates builds. – Dexterdexterity

Answer 1

As it turns out, I'm working in exactly the same environment. Here's how we have our system setup:

1. Each client has their own database. Effectively you are discussing a multi-tenant solution. While having one-database-to-rule-them-all has its advantages, it becomes problematic when you want to move clients or when clients want to host their own systems or when you have schema differences between versions (which is common). The only other approach to this problem is to use a single database but segment by schema name (e.g. ClientA.Table1, ClientB.Table1...).

2. We use a single code base. We use virtual directories to point to the version of the code the client is using. All clients on the same build are pointing to the same bits. However, we might have multiple versions out in the wild. Thus, one client might be pointed to 1.0.0.0 and another might be pointed to 1.0.1.1.

3. When we want to update someone, we point their virtual directory to the requested released version. The physical folders are named for their complete version name (i.e. 1.0.3.4). The application is designed to detect a discrepancy between the database version it expects and the database version it has. If there is a difference, it redirects to a page where an administrator can update the database schema. All database changes are scripted and are designed to be executed in the proper sequence. We use the third octet of the version number to indicate the database version. So 1.0.1.1 indicates that the database schema changed from 1.0.0.0.

4. A question that might arise is, "Why use virtual directories?" This gets into a core rule with respect to development: the application code cannot include any client specific data or settings. Specifically, nothing from the root of the application folder down can be client specific. So, what about connection strings? This is why we use virtual directories. Our virtual setup looks something like:

clientsite (or vdir)
    /appname_vdir

The client site root folder contains a web.config file with any non-database driven client-specific setting such as connection strings. When we point /appname_vdir to a new version folder, we do not have to worry about updating connection strings or any other client specific data. What makes this tough is that every change has be evaluated in the light of whether every client will want that change in the same way or not. If not, then there has to be a means to disable (or not install) that change.

Ideally, you want to host as many clients as you can because you can build tools that automate or control the process of pointing clients to an updated version and executing the database update.

At some point, our plan is to build a mechanism to let an administrator be notified that a new version is available. If they are in our hosted environment, updating would involve changing the appropriate virtual directory path. If they are hosting their own, it would provide a means to download the bits and then do the path change. That part we haven't yet had time to build. In addition, it is also possible to build a management tool to let support folks update people and determine what version they are using.

Ciliary

Wow, this sounds almost exactly like our environment. So, I think my question for you has to do with how you actually host the different versions. I believe we generate the build -- do a diff between the current version of their site -- and PUSH (overwrite) their current site with the new files. In conjunction with that, we run the appropriate database scripts to update the DB. Once those steps are done, they are on the 'current' release. You solution uses virtual directories instead.. How do you think I can incorporate that into our setup? – Dexterdexterity

@Dexterdexterity - We host multiple versions by having multiple folders (named with a full version num). Two different client's might have their app vdir pointing to different version folders. If two clients are running the same build, their app vdirs point to the same version folder. – Ciliary

@Dexterdexterity - Btw, it doesn't have to be vdirs; you could just change the root folder of the site when you update someone. However, this brings to light another important aspect to our setup with regards to client specific setups. I'll amend my response to provide more. – Ciliary

@Ciliary - So, after investigating more -- one of the main reasons they've been doing individual releases is because the each site uses a customized web.config file containing mainly connection strings to their DB. Also, since we have 3 different environments (QA, DEV, Production..) even those strings need to be updated manually. Also, each site has its own configuration file which includes variables that turn on/off features on their site. Do you think those things can be addressed with the Virtual Dir solution? – Dexterdexterity

@Ciliary - Also, another curveball maybe that we run a .NET application -- this includes dll's in our /bin directory and .aspx files.. Can the dll's, .aspx files, and the web.config live in this virtual dir world? – Dexterdexterity

@Dexterdexterity - We also are using .NET. In fact, we rely on the fact that .NET will combine web.config files from child and parent folders. That's how we can put client-specific settings at the root web.config file, and keep all client-specific settings out of the app folder. Obviously, this changes the path. Instead of "/default.aspx" for example, you'll have "/appname/default.aspx". However, if you use ~ for all your paths, it will work fine. – Ciliary

@Dexterdexterity - Given what you a few posts ago, setting up a similar folder structure where you can keep the connection strings outside the app folder will make it significantly easier for DEV, QA and multiple releases. In our environment, each developer has their own set of connection strings that at any given time might point to different copies of the database (i.e. for debugging, dev testing etc). – Ciliary

@Dexterdexterity - With respect to client-specific settings, you have two choices: the client's root web.config file or a database table (assuming each client has their own db). Putting them in the db makes them easier to manage and makes it easier to make a maintenance screen for them. If they are in the client's web.config, you have to manually go to the server to change someone's settings. Some settings, like connection strings, are much harder to manage outside the config file and thus those are easier to keep there. – Ciliary

@Thomas- Hey Thomas, how would you solve this for a public facing website where you would want to release the new features to only a subset of users - my question - #8130340 – Horwath

@Horwath - That's a different type of versioning. In the situation I mentioned above, you have multiple installations pointing to different databases. In that scenario, you can segment versions based on client/tenant/customer (customer 1,2, and 3 get version 1.0, customer 4 gets 1.1) because there is a high degree of separation between customer's databases. However, in the link you provided, the desire is to segment by feature within the same codebase and installation which is much more challenging. – Ciliary

@Horwath - I'll think about how one might version the system in the bounty question you provided. I will say that that question is more about staggering the versioning of schema rather than versioning SP per se. – Ciliary

@Thomas- fyi- the bounty ends tomorrow :) – Horwath

Answer 2

The way i see it you have two options:

• host all the versions yourself and intelligently differentiate which version the client should be using
• get the clients to host the application themselves

Hosting it yourself

I'm thinking you could do it this way:

• install all the different versions that are currently in use
• have one domain, each client gets a subdomain of that. Client ABC logs into abc.mydomain.com, etc.
• use a redirection or URL rewrite mechanism to silently redirect them to the appropriate version of your product
• when the client wants to upgrade, migrate their data to the later instance of the product, and change the rewrite rule for their subdomain

Note that i am lacking a little bit of detail here as i don't know too much about your setup, like is each client using their own database, or do you have a multi-tenanted approach? Are they all using an individual installation of the product, or are they using the same installed instance and just getting different data?

Client hosting

This can be relatively straight forward to manage. You can package the ASP.NET part of the application up in an MSI installer, creating virtual directories and app pools etc can all be done from within the MSI. Then you can use a database upgrade tool like DBGhost to package you database - it takes you template (source) database, compares it against the target, and then generates the DDL required to bring the target in line with the source. I understand there is also a tool for this included with one of the versions of VS2010, but i've not used it and cannot comment on it.

---

You dev staff can help mitigate a lot of the upgrade pain by supplying the updates to the support team in a fashion that is easy for them to use. Use MSI packages where possible. Use industrial strength database creation/generation tools where possible. If you can't use a database upgrade tool for some reason, then make sure the database upgrades are scripted in a good manner (i.e if object exists then alter else create ). If necessary enforce the upgrade path that must be followed - if a client wants to jump from 1.6 to 2.0, then they have to go through 1.8 first - this makes upgrades more granular, predictable and easy to manage (you maintain an upgrade script for 1.6->1.8 and 1.8->2.0, you don't have to have one for 1.6->2.0).

I hope this gives you enough to think about - if you have more details then edit your post and add them.

Joijoice

thanks for the feedback. Unfortunately, we MUST host our sites and DB's as the information is sensitive. What do you mean 'install all the versions currently in use'? Does that mean you install each version on its own virtual directory in IIS? – Dexterdexterity

@tres - are you multi-tenanting the databases (several clients use same physical database, their respective info is identified by a client ID)? Or individual databases for each client? If you have multiple clients running v2.0 of the product, are they all running the same build, or are there minor differences? – Joijoice

We use individual db's for each client -- with their connection strings in their individual web.configs. If multiple clients are on v.2.0 i believe the code is the same. – Dexterdexterity

Answer 3

Have you thought about automating the updates and deployments of the application. If you can come up with a standardized way of packaging the application regardless the version, it will be very easy to automation with a tool such as Nolio ASAP (http://www.noliosoft.com). This will enable you to create a deployment process per version or even per account, and enable "almost same" deployments where differences can be entered as input per deployment.

Palaver