Complicated application stack refactor/redesign strategies

Question

I am involved in a project maintenance plan that involves either major refactoring or complete redesign. We have a very complicated existing technology stack that has evolved organically over a period of five years. I myself joined on three years ago, and was 'caught up' in the madness. This stack was put together based on a client's evolving needs, and is fast becoming an unmanageable behemoth consisting of many technologies and uncontrollable collection of data.

Our primary goal is to make the stack more workable and manageable, and come up with a better system to manage data in the future. We would like to convert a tightly integrated Business Model and REST Controller system written primarily in PHP into a coherent REST API.

Our PHP Business Model relies on the eXist XML database, which contains most of the data, and MySQL which supports 2 of our sub-applications' data. We have a library of SQL and XQuery source codes to manage the data in the data stores dynamically. There are also parts of the business model that relies on Java code by using the PHP-Java Bridge and SAXON and FOP to create PDF facsimiles of documents stored in the XML database. Since this is a web application, we use a combination of PHPTAL, XSLT, CSS, XHTML, and JavaScript to facilitate the client UI. Lastly, we have a bunch of housekeeping scripts that manage PHP, Apache, functions written in Perl, that are managed using Ant tasks.

The basic functions of the stack are to facilitate electronic forms for a variety of user models. The simple philosophy behind the application was undermined over the years by a combination of client fiddling, excessive end-user hand holding, lack of systems analyses oversight, bad testing methodologies and lack of future goals and well-defined project scope.

We are currently planning to arrest any further development, better describe the system which may or may not include better unit test plans, and create the foundation for an API using the same technology components. My gut reaction is to come up with a solid scope and redesign the system using a well-supported programming language and framework that is well suited for writing web APIs, possibly loose some functionality, but retain the functionality that matters the most. I would also recommend migrating the existing data to a separate 'read-only' platform.

My question to those who have been in similar situations are:

• how did you get out of this kind of situation?
• what sort of planning steps would be useful to employ in these kinds of situation?
• are there source code management techniques for these kinds of stacks? what do I look for? - search keywords

I realize that there are no perfect solutions, and that to answer these questions, I would need to specify our project issues further. I can not really do that for a variety of reasons. I am having problems finding useful resources to help me approach this issue, and I would appreciate advice on how to proceed.

Gregoor

A very broad question, but I am inclined to +1 to empathise! and you've clearly thought about it. My inclination, based on (a presumed) need to keep operations going, is to suggest you refactor to a better approach gently and incrementally. For example if you swap to a better web framework, rewrite screen-by-screen (or call by call in the case of an API) and use URI rewriting (using mod_rewrite or similar) to patch it in. That way you can avoid the 'big bang' approach that has much more to go wrong. – Anthropomorphic

I'd be inclined to decrement the language count if you can too. The Java stuff may be too large to bring back into your PHP fold, but can you move your housekeeping Perl to PHP or Java? At the moment your language sprawl would, I should think, make hiring more difficult - or at least more expensive. – Anthropomorphic

Don't forget about client expectations. Who's paying for the overhaul? If the users are generally happy, then there's not much you can do but to make things easier on yourself with the existing platform. There may be bugs that users expect to be there, and may get frustrated when they are actually fixed. This is really more about managing the user's expectations at that point than coding. As ass-ugly as the current development situation is, if it works, that's all that matters to the client. For this to be done, expect some hefty negotiations. – Heathenize

Following on from @JayC, any redesign needs to a degree to get the support of the dev team also. Programmers often are precious about their handiwork (I know I am) and chucking it out wholesale can cause friction. That's not to say you shouldn't junk old code - just take the current team's wishes into account, and see how your new technologies would fit in with their skills. – Anthropomorphic

I agree with @halfer. I've been through (and am still going through) a similar process, though it sounds like our legacy code was less entrenched. Prioritize your goals, rebuild/refactor one piece at a time. When a new piece is swapped in, you can begin working on the next, while your users inevitably find bugs in the newly deployed code. Once that feels stable, swap in the next piece, and so on. I would try to reduce the overall number of stacks, languages, and moving parts in general. I would start with the low hanging fruit, then move to the big stuff. – Previse

Folks, these are all good comments. Thank you! @JayC: Let's say there's enough money to finance 1 year's worth of work between the staff that we currently have, and perhaps add one more to beef up unit testing in kept languages. – Gregoor

This is no trolling, spamming, sidestepping your question, or whatever else it may seem, it's my most sincere advice to you after i've been in a quite similar situation myself (well, if i understand your situation well enough to properly draw this parallel): JUST QUIT THIS STUPID JOB. You don't improve on any of the skills that currently (or in the foreseeable future will) matter in the serious software industry, so you're quite literally wasting your time. And we only have one life. Based on the way you talk you should aim higher. – Birnbaum

@halfer: We will be refactoring the PHP model and controller so that they are not as tightly integrated with each other and more tightly follow some sort of MVC framework. Not sure how successful we will be doing that, but it seems to be the place to start right now. Do you think it's worth bringing a experts on short-term basis to comment on the technology of their strengths? – Gregoor

Yes, bringing someone in can certainly help, perhaps to give you a few places to start. However it's not essential, since your team will have to do some testing anyway, to ensure they can work with the new technology, and to try some existing functionality to see how it might be implemented. I went through a similar process when looking to upgrade an ASP site, and tried a variety of PHP frameworks (spending a couple of days on each) to see what performance was like, what could connect to our database, how to use the bundled/recommended ORM, etc. – Anthropomorphic

@Birnbaum - tackling hard technology problems can be an excellent learning experience. Also, I'd wager that if programmers refused to work in legacy environments, 80% of us would be out of a job! – Anthropomorphic

@topmulch: it's worth giving some thought to the database as well, given my comments about swapping out application pieces piecemeal. One approach is to redesign your table structure first, and integrate that into your existing code. This should give you data correctness and/or performance advantages even without transferring to the new technology. If that's too much work, in some cases it is possible to have a new database with a new design and to sync between them in non-real time, but that won't work for all systems. – Anthropomorphic

@Gyll: thank you, I appreciate your advice. I've thought about moving on, but there are some good things about this job that has kept me here. I am always looking to see if there is anything better, but given my personal and professional circumstances at this time, I need to be at this job. We will see if I am still here next year.. – Gregoor

@wst: yeah, it would be nice if we can approach things linearly like that (meaning, one piece at a time). We made a decision at some point to use this XML database and use XQuery, which yielded powerful results beyond what traditional SQL could do. Unfortunately, we could not let go of SQL completely - thus our problem of separating the two. – Gregoor

@halfer: designing the tables. I think we are looking at potentially moving to a completely SQL model again. That would depend on whether or not we could forego some features. Unfortunately, we could not get the okay until we've exhaustively documented how complicated the current data model is. Hopefully by the end of this year, we'd have accomplished enough to say something to that effect. I think having everything stored in SQL would make things A LOT simpler. – Gregoor

Ah, of course, you're on an XML database at the moment. Could you move parts of that to a SQL database and then seamlessly recreate the API/functionality that consumes it, a bit at a time? I'm not familiar with XML databases (I should be fascinated to know how joins work!) but yes, SQL databases are proven, and the skills are greatly easier to obtain. – Anthropomorphic

@halfer: I'm getting the "Please avoid extended discussions in comments." warning, so I will end my comments by saying "perhaps". This is dependent on a lot of unanswered questions right now. Thanks again for the suggestions and words of support. I'll keep searching, I guess... – Gregoor

@topmulch: the rule to 'avoid extended discussions in comments' isn't really enforced, and imo it's just to try to trim down excessive support requests from folks who haven't clearly stated their problem or haven't done any prior research. That doesn't apply here, so (imo again) don't worry about it :). – Anthropomorphic

@halfer: okay, thanks. BTW, here's some info on eXist-DB (exist-db.org/exist/index.xml and exist-db.org/exist/documentation.xml). The original rationale to use an XML database was because we needed to be free of traditional SQL constraints. I think it is possible to define a dynamic form structure in SQL. – Gregoor

@Anthropomorphic Don't underestimate XML databases (done correctly, join-type operations aren't strictly necessary). I can't imagine there being features that you have in SQL that couldn't be reproduced in XQuery with less hassle. This might be worth discussing with an expert. Also, eXist isn't the only game in town. But eXist and others offer consulting services. – Previse

Answer 1

Here are some questions I would be asking or thinking about if I was in your shoes:

Database design:

• What information do you need to capture?
• How does it all need to be tied together?
• What kind of reporting do you need to do?
• What kind of metrics do you want to collect?
• When should old data be purged permanently to long-term archival?

Application design:

• What can you design into the system now that will make it less likely clients will need to be "hand held"?
• What functionality is currently missing or deficient?
• Are there any easy changes you could make that would expand your customer base?
• Do you have a formal coding standard? Is it enforced?
• What kind of testing do you want to require? (Unit testing, formal QA, etc.)
• What kind of documentation do you want to require? (Source code, end-user, etc.)
• What kind of metrics are you using to determine how well the application(s) are working?
• Looking back three years to when you started - knowing what you know now, what design decisions do you wish you had implemented back then?

Software:

• Is it possible to provide the same services at the same level using a smaller set of technologies?
• What technologies are most familiar to your existing staff?
• If you need to hire more people, what technologies are easiest to bring in on a permanent or consultant basis?

Hardware:

• Does your existing hardware meet your needs? If not, what do you need? More servers? Bigger servers? Load balancers? etc.
• Are you currently taking advantage of virtualization?

Personnel:

• Do you have "buy in" from all the stakeholders? If not all of them, do you at least have buy in from the decision makers?
• Do the stakeholders understand how big an undertaking this will be?

Disaster prevention/recovery:

• How will you provide redundancy in the event of the primary server(s)/network(s) being unavailable?
• How will you backup the data (onsite, offsite, short-term, long-term)?
• What kind of monitoring do you need to set up? How should you be notified in the event of a problem? Who should be notified?

Sorry, I couldn't fit that into a comment :-) And I'm out of time to elaborate on any of it, I was just killing time waiting for a meeting.

Cloak

Thanks @Benny Hill: this is a good list to look at - I think I need to inject the following question you suggested: "Looking back three years to when you started - knowing what you know now, what design decisions do you wish you had implemented back then?" into our group discussions. – Gregoor

@Gregoor It's a really good place to start, at least then you won't repeat any "mistakes" you've made in the last few years. halfer - thank you for fixing my formatting. – Cloak