Concrete symptoms of over-engineering [closed]

Question

I have recently found myself in the position of explaining an (In-House) application I have written to two candidates my company likes to hire in order to assist in maintenance and adding minor features.

It is the first "production" application I have written, it has 45k LOCs and I spent almost two years of "solo" development on it. I am fairly young (18) and wrote the application from scratch while being contracted as stand-in for a former developer who left the company. Unexperienced in designing applications of this size, I tried to use common architecture- and design-patterns.

Today I know I have done some serious over-engineering, e.g. using a disconnected change tracking architecture instead of the Unit Of Work pattern, which the chosen ORM has already implemented. I will probably never have to go "real" three tiers.

Both candidates have 10 years+ background in In-House Application Development with the relevant platform. Being half their age and having little experience I do respect their opinion. When I was explaining the application architecture to them, comments were along the lines of:

• Jeez, no one would pay me to do stuff like that, I have to get things done
• Stick with what the framework does, don't use fancy libraries/technologies
• Don't wrap framework code. On a team, everyone will write his own wrapper code anyway.
• You're using .NET 3.5? Well, we are using 2.0.
• What does that LINQ stuff buy me? All this query composition and projection seems too complicated.

Now I am asking myself:
Am I an architecture astronaut? How do I know I am going too far with architecture? What are common symptoms of over-engineering?

Interviewer

This might be best off as a community wiki - there probably isn't a broad consensus about what counts as over-engineering. – Katinakatine

Sometimes the guy with 10+ years of experience is the best on his team, sometimes he's the career amateur. Sometimes the high school kid with no degree performs at a senior level, sometimes they've never written a line of code in their life and think programming looks like this: youtube.com/watch?v=gwlE1aASc4g . For what its worth, we're all prone to over-engineering. I wouldn't sweat it, especially if those guys haven't actually seen the real code. Just do a few clean sweeps to remove anything that looks like DailyWTF material and call it good. – Fai

Agree with Juliet - if you're learning from it and it works, then don't worry about it. Nearly everybody goes towards too complicated before figuring out how to code in a more simple fashion as you get more experience. At least you thought about it... – Jegger

Answer 1

What are common symptoms of over-engineering?

Code that solves problems you don't have.

Katinakatine

Some of the Infrastructure is overkill as I said. Should I consider a (possible dangerous/expensive) refactoring? – Interviewer

I agree with Juliet - you don't need to sweat it unless it's getting in the way of extending or maintaining your code. But it's worth fighting the impulse in the future! – Katinakatine

will it solve a real problem? – Loganiaceous

The corollary to that may be "Code that creates problems you didn't have before." @Just it may solve a real problem, but is that problem worth solving or even relevant? – Audie

@Johannes, could you get someone to pair with you on doing the refactoring? That would be my suggestion as that gives a couple of benefits to the refactoring: Someone else learns about what you did, you learn about how someone else does the work, and the application is improved for 3 benefits to doing it. – Colorist

To a customer, when you create needs that not exist, is called marketing. If you have both (marketing + over-engineering) is called Apple. – Frogmouth

Answer 2

One very strong warning sign of overengineering is when everything goes through so much indirection that it's hard to find the piece of code that actually implements some concrete, domain-level piece of functionality. If you find that most of your functions do very little concrete work and just call other virtual functions, you may have a problem.

Depositary

Answer 3

Boredom

Boredom is good precursor to over-engineered code. I'll admit, when I got my first job, I felt so underutilized. I was just bored. And when I got bored, I wrote code. Not just any code -- CATHEDRALS OF CODE.

No seriously, I had a mental picture of my code and abstractions as large towers with golden jutting spires, flying buttresses of glassy onyx, a wonderful vault supporting by arched domes topped with beautiful geometrical tracery, etc etc etc.

It was really fascinating to see the patterns working together for myself, but in retrospect, I am completely ashamed of the ungodly mess I left behind.

If you're writing your own frameworks and DSLs code to while away the less stimulating hours at work, just stop. Time is better spent reading Wards Wiki, or writing an open source book, or you may just want to ask management for more work.

Fai

Can I use that as a domain name - Cathedralsofcode.com? That's beautiful. – Jago

This phrase "cathedrals of code" has stuck with me for years, so much so that I just googled to find where I'd read it and got back here :) – Sciomancy

Maybe more like "scaffolds of code", as abstractions and patterns takes us away from the system architecture (the beautiful cathedral) and places a very organized, well working framework on all sides of it. A typical software engineering feat. – Isidor

Answer 4

Writing your own Framework

Odds are, someone's already done it. More than that, they've already done it 1000x better than you ever could. More than that, whatever they've done is probably already an industry standard, so that learning the technology will make you more competitive at other jobs.

At the last company where I worked, a programmer had worked solo on his projects for most of his tenure. He wrote one of the company's more popular apps and was widely regarded as the best on the team -- but in my opinion, he had a nasty habit of writing everything he needed from scratch.

He'd written his own dependency injection framework, his own ORM, a unit testing framework (which, inexplicably, looked and acted very similar to NUnit -- why didn't he used NUnit?), a framework for creating factory objects (a "factory factory" I'd call it).

Mind you, the code was actually remarkable, but what was the point?

Writing a better Core Library

At my current company, it always seemed like programmers writing useless amounts of code to replicate features already present in the .NET framework.

Among other things, they wrote:

• An active-directory framework for forms authentication in ASP.NET webforms -- inexplicable because ASP.NET has this function built-in.
• Hot-swappable themes and skins for websites -- also inexplicable, since the code was less functional than built-in ASP.NET themes and required 1000% more bloat.
• They inexplicably wrote their own typed data sets and data adapters. These objects provided less functionality than typed datasets which VS will autogenerate for you, while simultaneously requiring more boilerplate code than NHibernate domain objects.

Either they don't know the framework very well, or they think its notoriously inadequate.

There are only preciously few examples I can think of where the re-implemented library is better than the original (see Jane Street Core Library, C5 Generic Collections for .NET, a real currency class), but odds are, you won't write a better standard library.

Fai

Oh, and regarding the "better core library" comment, a C++ programmer once explained to me that the built-in string class is so frustratingly awful that its practically a rite of passage to write your own string class. – Fai

I be that guy had the time to do what he did. And, I bet he knew why things worked the way they did, and I bet he is much better at picking off-the-shelf things in the future. Sometimes, you need people that know how to do things from scratch. Whether that contributed to his bottom line or the company's is a different question. – Jago

It may not be fair to say "write it 1000x better than you ever could", without the implied ", given the time you have at your desk job". Could be the only reason it's 1000x better is that they had the time to devote, more than just boredom/time-filling coding – Total

Answer 5

As for the question about if you an architecture astronaut: If you are aware of the danger that puts you ahead of a lot of people. You don't want to go the way of your cow-orkers either, it sounds like some of them have become crusty old whiners.

Over-engineering is the result of a problem with prioritization that resulted in some part of the system getting too much attention. So the most apparent symptom of over-engineering would be that you can see all around other parts of the system that are hurting for lack of attention.

(There is also a tendency for over-engineering to expose the system to increased risks of bad design, because of increased complication and the amount of error-prone speculation involved in deciding what aspects to over-engineer, but as a comment points out, that doesn't automatically follow.)

Raskin

Having multiple places that need to change for a single 'change unit' is not a symptom of over engineering, but to tight coupling/bad design/underengineering – Whensoever

That's a good point, I was thinking of over-engineering as implying that parts of the system get too much attention, with the consequence that other parts of the system get starved for attention. When I see something that's overengineered there are usually other parts of it that are underengineered. – Raskin

Answer 6

For most in-house business applications, most of your code should be concerned with implementing business concerns, and not technical concerns unrelated to the business (like your "disconnected change tracking architecture"). The currently available frameworks are pretty mature and support most common use cases. If you're inventing new technology or (in the context of business-application development) just wrapping some other existing framework or library just for the sake of wrapping, you're probably doing it wrong. Ideally every piece of architecture you build should be traceable back to some business requirement. Keep it simple.

Uzzia

Answer 7

When a co-worker’s computer comes flying past your head because they’ve spent the last 6 hours trying and failing to make a freaking dialog box appear using your ridiculous framework, that’s a pretty concrete symptom right there.

Fleurdelis

well thats actually what they were telling me they are afraid of (they haven't worked with the codebase yet). It is an MDI application and uses some custom wrapping functions for launching child dialogs etc. – Interviewer

Answer 8

IMHO most of the comments you got about your application aren't really about over-engineering because over-engineering is not about technology. It's about architecture. New technologies can be learned and understood in a reasonable amount of time. Understanding an over-engineered application is usually much more difficult and sometimes even impossible. This makes the points 2, 4 and 5 invalid. The first point is not really valid because you obviously got paid for writing the application as it is and if it works you got no problem here.

This is my "quick test" to find out if an application tends to be over-engineered:

• Wrappers for "everything": Wrappers are useful but it's easy overdoing it. Check if you only wrap things that really need to be wrapped. (I basically wrapped my own wrapper once. I know what I'm talking about ;-) .)
• Reinventing the wheel: A classic. This is very common and you already mentioned it. Did you implement some functionality because you needed to your wanted to? What does your framework do what other available libraries don't?
• "Feels" over-engineered: This is the most important point but also the hardest point to see. Take a look at your code and look which parts feel overly complicated. Ask your self then if there is a easier way to implement it and why you didn't choose this way. If you got no good answer this part is probably over-engineered.

These are just quick tips which I use for my applications. They are not guaranteed to be the be-all and end-all of "over-engineering detection".

Miraflores

LOL at "wrapping your own wrapper"! But IMHO, wrappers usually make code more readable, so I believe it's OK if you overdo that. – Farrow

Sometimes, if you have time, you should reinvent the wheel. It is nice to know why something turns, not just that it turns. – Jago

Answer 9

Avoiding any use of YAGNI, DRY, and KISS come to mind in looking at things that are over-engineered. If there are many parts that seem to be partially completed and many parts of code that seem to have a, "What if this happens? What if that happens?" feel to it, that would be another point. Ignoring good principles of OO design or SOLID principles would be another to note. If you think you have written the perfect code, that would be another sign of trouble as it is extremely rare for anyone to write something that can't be improved in one way or another.

IMO, beware that some people may be overly critical of your work as part of any code base can involve people liking things a certain way, e.g. naming conventions on methods, tests and variables. That's just the way it is. Now, what you may have to figure out is how to handle people in siutations such as conflict or persuasion/influence, where there are tools that can help.

Colorist

would you agree that violating DRY all too obvious is a sign of "under"-engineering, a lack of design? – Interviewer

Sure, violating DRY is a sign of "under"-engineering, a lack of planning. Some of what I describe is why refactroring can be so useful at times. – Colorist

Answer 10

Plugins which provide intrinsic functionality to your app

Let's face it, pluggable architectures are just damn sexy and fun to write. However, this is another one of those areas where you need to ask yourself, "do I really need to do it this way?".

If you don't have a need for ad-hoc additions to your application, don't expect anyone to write third-party extensions, and the scope of your application fairly well-defined, you don't need a pluggable architecture.

You shouldn't write plugins to support intrinsic functionality in your app. Let's say you were writing a Paint program; you'd probably support plugins to save files in multiple formats, but you wouldn't need a pluggable undo manager or file browse dialog.

Fai

Answer 11

You're not an architecture astronaut. LINQ is pretty simple and basic and useful, for one. Same goes for .NET 3.5.

At the same time, you're the newbie on the team and going to get some kind of ribbing, even if they like what you did.

Take it all with a grain of salt. Just accept their criticism, nod, and have a beer with them afterwards.

If they ask you to change it, then your comment is "jeez .. I know i did it wrong, but it works and it's going to be too much trouble to change".

Klinger

Answer 12

On a more generic and high level approach,

I feel that when there's a gap between the complexity of the problem and
the complexity of the solution, then you have a clear case of overengineering. 

What are the ways to achieve that? Solve problems that you don't have, seeing the problem more complex that it really is, trying to forecast too much in the future, building too generic stuff, among others.

Tessatessellate

Answer 13

Try to evaluate if you have done things to minimize work over the expected life of the code. This includes maintenance as well as development.

In the code lifecycle, remember that the life of the code is not the same as the life of the application. It maybe better to write a quick prototype first, then re-engineer/refactor after better understanding of the domain. In this situation, the lifecycle is very short, so keep it simple.

In addition, different applications require different amouhts of engineering. Is this application going to cost lives if it fails? I.e. is it a controller for a mechanical heart, or for the space shuttle navigation rockets? Or is it a contact list for bored teens?

Klinger

Answer 14

half of it is old men (i'm in their league) sticking to the tried and true steam engine they've known. the other half is true, but not really telling you anything new.

other than that, Jeff Sternal's answer is stellar.

Loganiaceous

Answer 15

Did you implement your project within a reasonable time frame? Is it working right now? If so, you can probably relax a little bit, as one of the worst problems with over engineering is never actually getting anything useful done.

The guys you talked too may have some good suggestions, but it doesn't mean you need to take everything they say as gospel. For example, using a recent version of .NET on a fresh project does not strike me as anything to worry about. Are they really complaining about that?

Babblement

Agreed. Any complaints about using v3.5 and LINQ don't seem like complaints at all, they seem like admissions of not knowing or having worked with them before. Which is fine, but shouldn't be a complaint. – Triaxial

No, they were not complaining. It's more like Funka said. The application is in production since almost half a year now, users and product owner are happy. The schedule was met reasonably, considering my experience and circumstances (type of contract, visiting school etc.) – Interviewer

Answer 16

As someone who has been developing software professionally for 24 years, all I can say is that this issue - making the call on the level of abstraction required for a particular problem - is the hardest part of my daily design & programming work.

In the "old" days, under-engineering was rife, which led to all the structured methodologies and practices. Now, it seems we've gone the other way. There is no easy answer, and sometimes you will only know if you under or over engineered in hindsight ;)

I've had multiple hindsight experiences where I said to myself: "I wish I didn't complicate this so much", but also "If only I listened to the little voice in the back of my head, and made it a bit more generic here".

I've yet to distill absolute rules for making the call...

I suspect I'm prone to over-engineering, so I've made this picture my PC's wallpaper

Historiated

Goodness knows why this has 0 upvotes. It's great, and even 6 years later, I want to thank you. – Sciomancy

@Sciomancy Glad to help, dude. I should put that desktop background back up :D – Historiated

Answer 17

Kind of want to offer a different perspective, but I think this term should be struck out and replaced with a more appropriate description. It places a stigma on "engineering" which should be about simplicity and maintainability and practicality above all, when a lot of what is described as "over-engineering" has the precise opposite qualities (as though engineering too much, or taking it too seriously, is supposed to yield the most convoluted solutions). It is common, for example, to see a correlation between over-engineering and poor testing procedure (at least I have often seen these two go hand-in-hand), and what kind of excess in engineering performs the minimal amount of formal testing?

I say that coming from a standpoint where I often heard software managers and occasional dinosaurs who weren't so SE-savvy to throw this term around at anyone attempting to, say, reduce the maintenance costs in a system or propose more sound testing procedures and safety standards. It's too easy for the layman to think that any kind of serious focus on sound engineering, testing, and not producing code at the sprint-like snap-snap-snap pace they want is "over-engineering".

There are plenty who genuinely recognize the true smells of "over-engineering" but the term is at least arguably misleading.

As for the symptoms, besides the excellent ones already provided, to me it's a blindspot in being able to properly evaluate and reevaluate the work in front of you. Programmers can build worlds in their imagination, get obsessed and absorbed with towering concepts. While it's kind of a boring and generic answer, that can lead to a danger of no longer seeing the actual problems and priorities right under our nose. To conceptualize too deep is to often develop a blindspot as to what is right in front of us, to overcomplicate problems, to lose touch with reality and the actual user-end needs. To me the biggest symptoms to watch out for revolve around psychology.

All kinds of professions revolving around production are vulnerable to this basic trend where one loses sight of the actual and immediate problems to solve as result of getting too obsessed about concepts (ex: a film director). With programming in general, a way to combat it is to favor plainness, simplicity, practicality, coding sooner before conceptualizing something to the nth degree (buying us more time to reevaluate before we fall too deeply in love with any ideas). One can't go too far wrong embracing these ideals, but the key to me is to avoid developing that blindspot which prevents us from being able to properly evaluate our own work. An overengineered codebase doesn't come about overnight -- it takes a long amount of dedicated work in the wrong direction to yield it, and to me the biggest preventable issue with that is not foresight-related but often developing the proper hindsight too late.

Matelda