Is ORM still the "Vietnam of Computer Science"? [closed]

Question

I read this post last night, and I noticed it was from 2006. I could go either way on the ORM, database thing, but I was just wondering if everything bad Jeff said about ORM still applies even now considering the post is from 2006.

Sectarianism

Vietnam was the last reference point for the US when the article was written in 2004. Nowadays isn't it safe to say, "Iraq of Computer Science?" Or is that still uncool for another twenty years? – Annecorinne

@Daniel: yes, too soon. Vietnam is still the preferred metaphor for "intractable quagmire." – Lattice

I will say "Afganistan" of CS. It provides more colorful comparison. – Shaven

As of 9/28/2022, the original post by Ted Neward can no longer be found. However, the Internet Archive has recorded it for posterity. web.archive.org/web/20220823105749/http://blogs.tedneward.com/… – Arsonist

Answer 1

It's still true.

Even more than OO software, the database suffers if it isn't treated precisely the way intended. And it wasn't intended that you should interpose some abstraction layer in front of it.

I think of impermeable abstraction layers as trying to build a Lego castle with all the pieces closed up into a pillowcase. SQL is damn hard to do correctly. It doesn't share many patterns with procedural programming, and best practices for one can be the opposite for the other. You need to be able to grok every single item in a SQL statement, and have a pretty good idea what it's intended to do, and what it in fact does.

Lots of people seem to think that, like horseshoes, close is good enough - if the right answer pops out, that implies you're nearly there. In SQL, that's simply not true.

RoR and the ActiveRecord pattern have deservedly earned a reputation as dbms resource hogs for this reason. Optimized ActiveRecord design is more often than not suboptimal SQL design, because it encourages SQL statement decomposition.

Pore

actually, SQL is dead easy to do.. if you know what you're doing. That can be said about every other technology though, so if you think its hard, you should not try to force it using an ORM, you just need to make friends with a DBA (who'll think procedural coding is impossibly hard :) ) – Fed

"SQL is damn hard to do correctly. It doesn't share many patterns with procedural programming": erm, isn't that the point of ORM? i.e. for procedural programmers who, for whatever reason, want to avoid getting involved with SQL? I'd hope the tradeoffs are known and deemed to be acceptable. – Budworth

I know SQL, and yeah I don't really think it's that hard...(although it was hard to learn...once you've got it, it isn't that bad) – Sectarianism

One of the most popular (.NET) ORMs suffers from SELECT N+1 straight out of the box – Fries

There is rarely worse debugging and maintenance issues than having to look through some joker's chained-together lambda expressions which include joining, grouping, etc. When compared to debugging a stored procedure or even parameterized query it is a complete nightmare. – Bon

Yea - you're not a "developer" unless you know SQL – Scrivings

Answer 2

Yes.

Object-oriented is still object-oriented and Relational is still Set-oriented. Nothing has changed in these two paradigms in the past two years to make them work better together.

In many people's eyes, SQL is ugly, complex, and confusing. But trying to make an object-oriented interface to perform the same functionality is always uglier, more complex, and has a steeper learning curve.

In all programming, there is a tradeoff between flexibility and assumptions. Frameworks (such as Rails) try to solve the problem by being "opinionated." That is, they limit flexibility of either the relational or the object-oriented aspects of the problem, making assumptions about how data is structured, and what operations you can do with it. Naturally, simplifying the problem space makes the solution simpler as well.

In addition, it's frustrating to discover that an ORM framework is incomplete, so some ordinary operations in SQL have no solution in a given ORM. This is also a consequence of "opinionated" frameworks.

Lattice

"Naturally, simplifying the problem space makes the solution simpler as well." - Bill Karwin Gee Bill, thanks, I take comfort in that statement. :) – Sectarianism

But wait a minute, doesn't a framework like Hibernate, support a record-set type thing (i.e. Projection Queries?) And probably the reply I'm going to get at this point is, what's the point of using a framework when you have a projection-query that operates just like SQL considering the overhead. – Sectarianism

According to ohloh.net, Hibernate consists of over 800,000 lines of Java and XML code (not counting comments, or other Hibernate-related tools). Is this overhead really necessary, considering it's just as hard to use as SQL and doesn't really solve the problem any better? – Lattice

Answer 3

A lot of web 2.0 companies are working on key-value stores. And all these companies have to go through the same painfull process of making it work.

If ORM is the "vietnam of computer science" then building your own key-value store is probably the "Iraq of computer science" :-)

Agonic

Answer 4

I can only speak for my experience. I use DAL and DTO's, and I am still capable of performing pretty complex queries (joins and all), and also I am able to resort to SP's or custom SQL whenever I need. It made my life easier, my code consistent and my deadlines more attainable.

Philomenaphiloo

Just because you can do it, doesn't mean you should. You've ended up learning a whole nother paradigm, adding to the complexity, without replacing the other two. – Pore

@le dorfier: My 8+ years of many successful projects using this approach is really the only validation I need, I was just trying to offer a different perspective. I don't believe I added complexity, but reasonable people can disagree. – Dermatophyte

Yes, and I think RoR is a useful framework. I wish it could have the isam it was designed for, instead of being mismatched with RDBMSs. Then you could use it for optimized persistence, for a net complexity gain. (Sometimes complexity is worth it; but it doesn't help to pretend it's not there.) – Pore

I wouldn't use RoR, or even Hibernate, they simply don't scale. Believe me, I painstakingly profiled, made it thread safe, and I can run heavy processes with it - without manually writing a single line of SQL. – Dermatophyte

I wouldn't try to imply that it can't be done - it obviously has been done. But painstakingly profiling a database is IMHO best done in the database; and thread-safety simply doesn't come up. BTW, I upvoted you. Your model is much more viable than ORMs, IMHO. – Pore

@le dorfier: yes, by profiling I meant really looking the time it took to run the generated queries in the database itself, studying the execution plans, using parameters, etc. The multithreaded part is to allow many threads on client code to connect and run without interfering with each other. – Dermatophyte

Answer 5

I think starting from the assumption that Jeff's conclusions are correct is not necessarily good; having maintained stored procedure code as well as JDBC-based data layers, I can say that these caused maintenance problems galore, mostly related to the inability to understand what was going on at a higher level.

A database is necessarily low-level; it stores numbers and strings essentially. Business logic is high-level. This is why we have abstraction.

Personally, I think the Rails/ActiveRecord way is the best solution to having an object/domain model but also being able to take advantage of a relational database.

So: don't throw out ORM, but don't default to it either. It's a tool that solves certain problems. To ignore it would be ignorant and to always use it would be arrogant.

Inimical

Answer 6

Jeffs article links through to Ted Newards article. If you are into the details then you need to look there:

1. original - http://blogs.tedneward.com/post/the-vietnam-of-computer-science/

2. followup - http://blogs.tedneward.com/post/thoughts-on-vietnam-commentary/

Of Ted's original points I have it as:

• 1 was wrong (Identity)
• 2 of them solved (Partial objects and N + 1)
• 2 are debatable (Dual schema / shared schema).

Disclaimer: I'm the author of Ebean ORM so I'll reference that for the various 'solutions' to the issues raised.

Ted's original points (distilled because it's really wordy):

1. Partial Object problem.

Always solvable. Ebean ORM made partial objects fundemental to it's query language and all internals. JPQL didn't make this a priority to it's more a problem there unfortunately.

• Refer: http://ebean-orm.github.io/docs/query/partialobjects

2. N + 1 (Ted's Load time paradox)

Always solvable. Should have been written as 1 + N / batchSize but it is more interesting than that (per path, need to take into account SQL paging, avoid sql cartesian product). Some ORM's make a right mess of this unfortunately and that brings ORM in general into disrepute. Some ORM's are ok until you hit a level of complexity (like OneToMany inside OneToMany inside OneToMany).

• Refer: http://ebean-orm.github.io/docs/query/nplus1

Just to up the ante here ORM's can profile the object graph use and automatically optimise the query (only fetching what is needed, defining fetch paths to optimise for N + 1 etc).

• Refer: http://ebean-orm.github.io/docs/query/autotune

This automatic ORM query optimisation idea came out of the University of Texas (using Hibernate). It was included as part of Ebean ORM in 2008 so it's been around for a while now.

• Refer: https://en.wikipedia.org/wiki/AutoFetch

3. Identity

Ted cracks on about a mismatch on identity and concurrency. This point is misplaced as ORMs (well, all the ones I know) go about this aspect in exactly the same manor as the prior client/server tools and specifically ORM's are providing a SNAPSHOT view of part of the database to the application. There was never a problem here but ORM implementations could get themselves into strife with an over reliance on hashCode()/equals() for example.

4. Dual schema problem

This is debatable. If the organisation allows then the ORM can provide a DIFF/SQL script to the schema and that is run by FlywayDB/Liquibase etc. If organisations don't allow that this might still be an issue to some extent.

5. DB Refactoring / Shared schema

This is debatable. DB design/normalisation folks would argue that the DB design should get to 4NF and that means any refactoring should solely be additive (denormalisation, adding columns/tables) and not breaking changes. People who don't believe in normalisation will be going nuts worried about shared schema.

Weald

Answer 7

I think it does.

I think the last sentence is the most interesting of all: "I tend to err on the side of the database-as-model camp, because I think objects are overrated." Java, C++, and C# are certainly the dominant languages, but functional programming is making a comeback with F#, Scala, etc.

Tody

Comeback - they've never been a dominant model before, so 'breakthrough' might be correct :D – Lesh

Answer 8

This isn't my area of expertise, but I worked in Rails for about a year and I think ActiveRecord solved most of the DB Mapping problem. I realize it has a few issues, but I think it did a fantastic job.

I don't think his post took into account the possibility of the framework itself (in this case AcitveRecord/Rails) defining the database AND the Object Model, which--as far as I can tell--makes the problem go away.

Since this is the opposite of the first two responses (basically that the post is outdated) I feel like I'm probably not understanding something; If that's the case please correct me instead of just voting me down because I think there is an important point I'm missing.

Drummond

It doesn't make the problem go away, it gives the illusion the problem is solved but creating horrible performance and design limitations compared to using the full features of a database and SQL. Pure object databases (which is essentially what you are talking about) never caught on. – Oslo

++ Kendal. The design of ActiveRecord naturally leads to terrible performance bottlenecks (the downside of wonderfully easy-to-use abstraction layers). – Saldivar

ActiveRecord (the pattern, not the Rails implementation) is fine if you use it in the circumstances for which it's designed. But there are a bunch of other data access patterns in Martin Fowler's PoEAA book, which seem to be ignored by Rails! – Lattice

Interesting points, thank you. Performance is often critical in database & web apps. I'm not sure that an ActiveRecord type of implementation HAS to sacrifice performance, but from reports rails implementation problems, I'm guessing currently it's not too good. – Drummond

Yeah, hold on... I doubt active record is the problem so much as Rails. Ruby itself is known to be slow (1.9 helps a lot) and Rails is also slow. – Poirer

One of the neat advantages to any platform like this is that the platform can improve from under you. There is nothing in the conceptual design of ActiveRecord that REQUIRES slowness, I'm pretty sure it's just the implementation--same with Ruby, it happens to be like 100x slower than c/java, but it's getting better. I believe JRuby is now significantly faster than Ruby... And also, performance doesn't matter in every environment. Finally, it does perform better than you guys are implying. genetree.com is in Ruby on Rails (I helped write it) and it's public & pretty quick. – Drummond

'Pure object databases (which is essentially what you are talking about) never caught on' - mostly because of the already massive investment in RDBMSes. – Susurrous

Answer 9

IMHO, Monadic approaches like Scala's Slick and Quill largely bypass the aforementioned quagmire, offering more robust solutions to many of Ted's problems (also JOOQ deserves to be mentioned). While they're not perfect, they definitely kill the N+1 and Partial Object problems, mostly kill the Identity problem, and partially kill the Dual Schema problem. Forget about clumsy and verbose CriteriaBuilder queries (anyone read "Execution in the Kingdom of Nouns???"), Scala's monadic for-comprehensions give you a simple DSL in which to write criteria-queries:

case class Person(id: Int, name: String, age: Int)
case class Contact(personId: Int, phone: String)

val query = for {
    p <- query[Person] if(p.id == 999)
    c <- query[Contact] if(c.personId == p.id)
  } yield {
    (p.name, c.phone)
  }

This kind of syntax stays sane, even for more complex queries e.g. Ted's is-acceptable-spouse query:

case class Person(name:String, spouse:Option[PersonId]) {
    isThisAnAccpetableSpouse(person:Person) {...}
}

val query = for {
    p1 <- people
    p2 <- people if (
        p1.spouse.isEmpty && p2.spouse.isEmpty 
        && p1.isThisAnAccpetableSpouse(p2)
        && p1.isThisAnAccpetableSpouse(p1))
} yield (p1, p2)

All of this get's compiled into a single SELECT query and run against the database. Inserts, Updates, and Deletes are similar.

Spermicide

These are better called FRMs instead of ORMs. Functional Programming doesn't have the same paradigm disparity that Object Oriented Programming does... In fact, the Relational Paradigm is a subset of the Functional Paradigm. Hence why FRMs completely avoid the problems of ORMs. Honestly, the only true ORMs that are halfway decent use Function mechanics to realize some modicum of parity. – Brundisium

Answer 10

Disclosure: I'm the author of RDO.Net.

Yes. I believe existing ORMs are going in the wrong direction - they're trying to map relational data into arbitrary objects which IMHO is simply not possible. In the OO world, the arbitrary object is not serialization/deserialization friendly, because every object has an object reference (the address) that is local to the current process. It's OK when you're dealing with a single object, you will get into trouble when you're facing a complex object graph.

Hopefully this problem can be solved using a different approach, as in RDO.Net: instead of mapping relational data into arbitrary objects, we map the schema of the data as rich metadata (Model that contains columns, primary/foreign keys, validations, etc.), and expose concrete Db, DbTable, DbQuery and DataSet for data access. When performing database CRUD operations, you construct the SQL Abstract Syntax Tree (AST) explicitly using strongly typed OO language, as demonstrated in the following sample C# code, which creates a query and returns a DataSet for hierarchical data:

public async Task<DataSet<SalesOrderInfo>> GetSalesOrderInfoAsync(_Int32 salesOrderID, CancellationToken ct = default(CancellationToken))
{
    var result = CreateQuery((DbQueryBuilder builder, SalesOrderInfo _) =>
    {
        builder.From(SalesOrderHeader, out var o)
            .LeftJoin(Customer, o.FK_Customer, out var c)
            .LeftJoin(Address, o.FK_ShipToAddress, out var shipTo)
            .LeftJoin(Address, o.FK_BillToAddress, out var billTo)
            .AutoSelect()
            .AutoSelect(c, _.Customer)
            .AutoSelect(shipTo, _.ShipToAddress)
            .AutoSelect(billTo, _.BillToAddress)
            .Where(o.SalesOrderID == salesOrderID);
    });

    await result.CreateChildAsync(_ => _.SalesOrderDetails, (DbQueryBuilder builder, SalesOrderInfoDetail _) =>
    {
        builder.From(SalesOrderDetail, out var d)
            .LeftJoin(Product, d.FK_Product, out var p)
            .AutoSelect()
            .AutoSelect(p, _.Product)
            .OrderBy(d.SalesOrderDetailID);
    }, ct);

    return await result.ToDataSetAsync(ct);
}

Burweed