ColdFusion 2016 performance versus ColdFusion 9

Question

We are in the process of upgrading from ColdFusion 9 to ColdFusion 2016 and we have noticed an overall decrease of performance. We ran several simulations to give more insight. Below is a script that gives a good example of the performance decrease. The script builds a query and then creates a structure from the query.

<!--- Machine info --->
<cfset runtime = createObject("java", "java.lang.System")>
<cfset props = runtime.getProperties()> 
<cfset env = runtime.getenv()> 

<Cfoutput>

coldfusion: #SERVER.ColdFusion.ProductVersion# #SERVER.ColdFusion.ProductLevel#<br>

java.version: #props["java.version"]#<br>
java.vm.name: #props["java.vm.name"]#<br>
os.name: #props["os.name"]#<br>

PROCESSOR_IDENTIFIER: #env["PROCESSOR_IDENTIFIER"]#<br>
PROCESSOR_ARCHITECTURE: #env["PROCESSOR_ARCHITECTURE"]#<br>
NUMBER_OF_PROCESSORS: #env["NUMBER_OF_PROCESSORS"]#<br><Br>
</Cfoutput>

<!--- Create a query --->
<cfset myQuery = QueryNew("Name, Time, Advanced", "VarChar, Time, Bit")> 
<cfset testQuery = QueryNew("ColumnA,ColumnB,ColumnC,ColumnD,ColumnE,ColumnF,ColumnG,ColumnH,ColumnI,ColumnJ,ColumnK,ColumnL,ColumnM,ColumnN","VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar")>

<!--- Populate the query --->
<Cfloop from=1 to=300 index="x">
    <cfset QueryAddRow(testQuery, 1)> 
    <cfloop index="intLetter" from="#Asc('A')#" to="#Asc('N')#" step="1">
        <cfset temp = QuerySetCell(testQuery, "Column#chr(intLetter)#", "Row #x# column #intLetter#", x)> 
    </cfloop>
</cfloop>

<Cfset init = GetTickCount()>
<!--- Query to structure --->
<Cfset queryToStruct = structNEw()>
<cfloop query="testQuery">
<Cfset init2 = GetTickCount()>
    <cfset queryToStruct[testQuery.currentrow] = structNew()>
    <cfset queryToStruct[testQuery.currentrow]['ColumnA'] = structNew()>
    <cfloop list="#testQuery.columnList#" index="key">
        <cfset queryToStruct[testQuery.currentrow]['ColumnA'][testQuery[key][testQuery.currentrow]] = testQuery[key][testQuery.currentrow]>
    </cfloop>
    <cfoutput>#x#:#GetTickCount()-init2#<br></cfoutput>
</cfloop>

<cfoutput>-----------<br><b>#GetTickCount()-init#</b><br><br><Br></cfoutput>

<!---Cfdump var=#queryToStruct# --->

We have two servers with the exact same hardware configuration. One server is running on Windows 2008 / ColdFusion Server 9 Enterprise (Java version 1.6.0_14) and the other one is running on Windows 2016 / ColdFusion 2016 Standard (Java version 1.8.0_112). Both ColdFusion servers have the same Minimum JVM Heap Size (5024 MB) and Maximum JVM Heap Size (5048 MB).

The performance on the ColdFusion 9 server is more then 4x faster. Can someone give an explanation of why this is happening and how to solve this?

Update

To rule out any other process that would slow down ColdFusion I installed ColdFusion 9, ColdFusion 11 and ColdFusion 2016 all on the same virtual machine and all using the built in web server. Default installation settings. The outcome: ColdFusion 9 is fastest, followed closely by ColdFusion 11. ColdFusion 2016 is much slower.

Update 2 Made some changes to the script, so it is more clear what this script is doing.

Update 3 The results can be viewed here: http://136.144.177.152/test2.asp or http://136.144.177.152/test-toma.asp or http://136.144.177.152/test-ag.asp Note that the code is actually processed, so each time you load the page the results differ slightly.

Also i would like to point out i am not trying to optimize this code. I tried to make a very simple reproducible example. The sole purpose is to point out the difference in performance and find a reason and a solution.

Update 4 Did some extra testing and found the potential problem. For some reason the following code is very slow on coldfusion 2016 / Windows 2016 :

<cfset tmp = testQuery['ColumnA'][testQuery.currentrow]>

What i found very strange is that updating the query value is not slow. E.g.

<cfset testQuery['ColumnA'][testQuery.currentrow] = key>

All results can be found here: http://136.144.177.152/test5.asp or http://136.144.177.152/test6.asp. I also installed coldfusion 2016 on my laptop and found no performance issues. I also tried installing coldfusion 2016 on windows 2012 machine. Here i found the same performance issues.

Update 5 Based on Tomalak suggestion I removed the indexed access notation. This clearly is a performance issue on coldfusion 2016. Actual results can be found here http://136.144.177.152/bug-adobe.asp. I opened a bug at adobe for this issue here https://tracker.adobe.com/#/view/CF-4201966.

Testerman

ColdFusion 2009 is not a version. Do you mean ColdFusion 9? – Daw

As a first step, please get rid of the Evaluate() call. – Foliar

@Daw Yes, sorry i meant coldfusion 9 – Testerman

@Foliar Removed the evaluate function. – Testerman

@Testerman - Any change in results? – Monogamous

@Ageax No, no changes. – Testerman

There seems to be a problem with nested <cfloop> tags. When I put the inner <cfloop list="#testQuery.columnList#" index="key"> into a separate function and simply call that function in the <cfloop query="testQuery">, things speed up significantly on my CF2016. – Foliar

Also, your entire queryToStruct code doesn't make any sense, really. Please fix that code first, or explain what it should do. – Foliar

@Testerman I'm comparing CF10 to 2016. My virtualized OS/RAM/Processor configs are not the same, but both use Java 1.8x. The CF10 server is beefier than 2016, but the processing times are only different by ~1ms after compilation & caching. (On average: CF10 = 6, CF2016=7). What performance times are you experiencing? – Circumcision

@Foliar the code has no real function. The only purpose is to show a difference in performance between the different coldfusion versions. – Testerman

@JamesMoberg On average coldfusion 9 and 2011 run this script in 16ms or less. The coldfusion 2016 version runs this script between 32 - 47ms. Although this may seem very small, it is consistent and noticeable on larger scale. – Testerman

Sure, but your intention seems to be to convert a query to a struct, and your code... doesn't really do that. It's inefficient on all versions I've tested it on. I've written a sensible version of that piece of code and it runs plenty fast. – Foliar

@Foliar Could you elaborate a little more on you comment about removing the inner loop and creating a function. I tried your suggestion but i do not see any difference in performance. If inner looping is an issue on coldfusion 2016 this could be a reason why i am seeing worse performance on coldfusion 2016. – Testerman

Might be a red herring. Your original code seemed to have an issue there, your current code no longer has. (Your current code is still broken, though.) – Foliar

In addition to upgrading, will you also be refactoring any old code? There were quite a number of improvements between CF9 and CF2016 that should give you significant boosts in performance with newer code. – Drumfire

@Drumfire It shouldn't run any slower, though, even un-refactored. – Foliar

@Foliar Agreed. I think there are other things going on causing it to be slow. I was just pointing out that CF2016 is a good bit ahead of CF9 in terms of code efficiency. – Drumfire

Your code runs 2.5-3 times faster in CF2016 (~10 ms) compared to CF10 (~28 ms) for me on the same JVM (8.161), default settings. 3000 rows result in ~80 ms and ~250 ms. – Felder

@Felder Can you tell me on which OS you tested the code? – Testerman

@Testerman we were recently upgrading from CF8 (I know, I know) to 2016 but also compared it to Lucee. I found Lucee's performance to be amazing. There are some features of Adobe CF that it does not have, but overall there has been very little work for us to convert existing Adobe CF sites over to Lucee. If you are running test sand switching anyways, maybe consider trying some tests on Lucee? Sorry, just saw date of post and may be too late to the game. – Anglicism

Answer 1

I can't reproduce the problem, really. This runs quite fast on my ColdFusion 2016. I've created an optimized version of the QueryToStruct code, but other than that there's little difference.

I don't have a CF 9 server available, is this code also 4 times as fast on CF 9 when you test it?

<!--- Machine info --->
<cfset runtime = createObject("java", "java.lang.System")>
<cfset props = runtime.getProperties()>
<cfset env = runtime.getenv()>
<cfoutput>
ColdFusion: #SERVER.ColdFusion.ProductVersion# #SERVER.ColdFusion.ProductLevel#<br>
java.version: #props["java.version"]#<br>
java.vm.name: #props["java.vm.name"]#<br>
os.name: #props["os.name"]#<br>
PROCESSOR_IDENTIFIER: #env["PROCESSOR_IDENTIFIER"]#<br>
PROCESSOR_ARCHITECTURE: #env["PROCESSOR_ARCHITECTURE"]#<br>
NUMBER_OF_PROCESSORS: #env["NUMBER_OF_PROCESSORS"]#<br>
<br>
</cfoutput>

<!--- Create a query --->
<cfset ROWNUM = 2000>
<cfset testColumns = "ColumnA,ColumnB,ColumnC,ColumnD,ColumnE,ColumnF,ColumnG,ColumnH,ColumnI,ColumnJ,ColumnK,ColumnL,ColumnM,ColumnN">
<cfset testTypes = "VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar ,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar,VarChar">
<cfset testQuery = QueryNew(testColumns, testTypes)>

<!--- Populate the query --->
<cfloop from="1" to="#ROWNUM#" index="x">
    <cfset QueryAddRow(testQuery, 1)>
    <cfloop from="#Asc('A')#" to="#Asc('N')#" index="intLetter">
        <cfset QuerySetCell(testQuery, "Column#chr(intLetter)#", "#x#-#intLetter#", x)>
    </cfloop>
</cfloop>

<!--- Convert the query to a struct --->
<cfset init = GetTickCount()>
<cfset converted = QueryToStruct(testQuery, "ColumnA")>

<cfoutput>
<b>My version:</b> #StructCount(converted)# rows, #GetTickCount()-init# ms<br>
</cfoutput>

<!--- Convert the query to a struct --->
<cfset init = GetTickCount()>
<cfset converted = QueryToStructOP(testQuery)>

<cfoutput>
<b>OP version:</b> #StructCount(converted)# rows, #GetTickCount()-init# ms<br>
</cfoutput>

<!--- +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ --->
<cffunction name="QueryToStruct">
  <cfargument name="Query" type="query" required="yes">
  <cfargument name="IndexColumn" type="string" required="yes">

  <cfset var result = StructNew()>
  <cfset var rowTemplate = StructNew()>
  <cfset var key = "">
  <cfset var thisRow = "">

  <cfloop list="#Query.ColumnList#" index="key">
    <cfset rowTemplate[key] = "">
  </cfloop>

  <cfloop query="Query">
    <cfset thisRow = Duplicate(rowTemplate)>
    <cfset result[Query[IndexColumn][Query.CurrentRow]] = thisRow>
    <cfloop collection="#thisRow#" item="key">
      <cfset thisRow[key] = Query[key][Query.CurrentRow]>
    </cfloop>
  </cfloop>

  <cfreturn result>
</cffunction>

<cffunction name="QueryToStructOP">
  <cfargument name="Query" type="query" required="yes">

  <cfset var queryToStruct = StructNew()>
  <cfset var key = "">
  <cfset var index = "">

  <cfloop query="Query">
    <cfset index = Query['ColumnA'][Query.CurrentRow]>
    <cfset queryToStruct[index] = StructNew()>
    <cfloop list="#Query.ColumnList#" index="key">
      <cfset queryToStruct[index][Query[key][Query.CurrentRow]] = Query[key][Query.CurrentRow]>
    </cfloop>
  </cfloop>

  <cfreturn queryToStruct>
</cffunction>

Result (This is not even server hardware, and it's an old CPU on top of that):

java.version: 1.8.0_162
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
os.name: Windows 7
ColdFusion: 2016,0,05,308055 Developer
PROCESSOR_IDENTIFIER: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
PROCESSOR_ARCHITECTURE: AMD64
NUMBER_OF_PROCESSORS: 4

My version: 2000 rows, 78 ms
OP version: 2000 rows, 229 ms

Foliar

Comments are not for extended discussion; this conversation has been moved to chat. – Bosson

Answer 2

First, you're comparing CF 9 Enterprise to CF 2016 Standard. Shouldn't be a big deal for this example, but if you're regression testing your whole app on Standard, you'll see issues. My previous company migrated from 9 Ent to 2016 Ent and all we saw were performance improvements across the board. When you see bottlenecks, you should always consider refactoring. It's one reason why you upgrade.

The biggest problem is how you are converting a query to a struct. CF 2016 has much more advanced functionality. Compare your legacy process to this one that takes advantage of the member functions in query objects.

public array function arrayOfStructs(required query data) {
    var results = [];
    arguments.data.each(function(row) {
        arrayAppend(results, arguments.row);
    });
    return results;
}

The query member function each() references the contents of each row as a struct. No need to loop over each column, enter a new key and assign a value. Boom! Done. Fast as hell.

Do your upgrade. :)

Opossum

The member functions really are syntactic sugar, I would not expect them to perform significantly better than compiled CFML. Both approaches should compile into more or less the same Java code. – Foliar

For example: "The query member function each() references the contents of each row as a struct." That's a nice abstraction, but internally, .each() uses a loop to build a struct. The fact that you don't have to write the loop yourself doesn't make it disappear. ;) – Foliar

Although you may be right that my code can be optimized for coldfusion 2016. This is not really my question nor my intend (btw i do appreciate you contribution). My question is why coldfusion 2016 is so much slower running this code then coldfusion 2011 or coldfusion 9. I tried to make a very simple reproducible example. – Testerman

Adrian are you running CF 2016 on windows 2016 or another OS? – Testerman

@Testerman When we upgraded to CF 2016, we moved to new servers w/ the latest Windows server, JDK, etc. – Opossum

Answer 3

I hope it is allowed to make a recommendation that solves the performance and upgrade problem in a different (more radical) way that is easier to implement for (time-)economic reasons at best: just install and try Lucee, the free extremely performant alternative to the Adobe version of Coldfusion. We use Lucee with several instances in our company for a high-traffic web application in the statistics area, which requires a lot of computing power, and we are extremely satisfied. Lucee is open source and community driver: https://www.lucee.org/

Legere