Avoid heap error while looping in coldfusion

Question

I am trying to loop over a 2-D array whose dimensions are 12000 * 20 and I am continuously getting java.lang.OutOfMemoryError.

Initially I thought this might be because of heap size so I increased my heap size but I was still getting the same error. So I ran a garbage collector like this:

<cflock name="checkMemory" type="exclusive" timeout="1" throwontimeout="yes">
    <cfset objSystem = CreateObject( "java", "java.lang.System" )>
    <cfset objSystem.gc()>
</cflock>

and I dumped the free memory which was around 850MB:

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime()>
<cfset freeMemory = runtime.freeMemory()>
<cfdump var="#freeMemory#" label="free">

Here I am trying to create a XML variable and while looping I am getting heap error:

<cfxml variable="variables.XML">
  <cfoutput>
    <ROWS>
      <cfloop from="3" to="#arrayLen(local.array)#" index="i" step="1">
        <ROW>
          <cfloop from="1" to="#arrayLen(local.array[2])#" index="j" step="1">
            <#ucase(local.array[2][j])#>
              <![CDATA[#trim(local.array[i][j])#]]>
            </#ucase(local.array[2][j])#>
          </cfloop>
        </ROW>
      </cfloop>
    </ROWS>
  </cfoutput>
</cfxml> 

This is the stack trace:

java.lang.OutOfMemoryError at java.io.WinNTFileSystem.getBooleanAttributes(Native Method) at java.io.File.exists(File.java:733) at coldfusion.xml.XmlProcessor.getSourceURL(XmlProcessor.java:246) at coldfusion.xml.XmlProcessor.parse(XmlProcessor.java:155) at coldfusion.tagext.lang.XmlTag.doEndTag(XmlTag.java:85) at cffeeds2ecfc1003675922$funcDEMO1._factor8(C:\component\abc.cfc:1235) at cffeeds2ecfc1003675922$funcDEMO1.runFunction(C:\component\abc.cfc:1192) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:405) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2582) at cffeeds2ecfc1003675922$funcDEMO.runFunction(\component\abc.cfc:935) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:405) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:517) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:496) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:355) at coldfusion.filter.ComponentFilter.invoke(ComponentFilter.java:188) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:374) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.xml.rpc.CFCServlet.invoke(CFCServlet.java:139) at coldfusion.xml.rpc.CFCServlet.doPost(CFCServlet.java:290) at javax.servlet.http.HttpServlet.service(HttpServlet.java:760) at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorFilter.i(FusionReactorFilter.java:566) at com.intergral.fusionreactor.filter.FusionReactorFilter.c(FusionReactorFilter.java:258) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:164) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Is there any better solution to avoid looping like this or to resolve this error?

I created a cfm page locally and added some variables in it like this:

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime()>
<cfset freeMemory = runtime.freeMemory()>
<cfset totalMemory = runtime.totalMemory()>
<cfset maxMemory = runtime.maxMemory()>
<cfdump var="#freeMemory#" label="free">
<cfdump var="#totalMemory#" label="total">
<cfdump var="#maxMemory#" label="max">

Every time I was refreshing this page the free memory size was decreasing until I ran the GC. I am still trying to figure out why this is happening. Any suggestions in this regard?

Please help. Thanks in advance.

Webfooted

The relevant part of the code here would be the bit that you abbreviate as //Some operation. It's impossible to comment what one can do to make that code more memory-usage friendly without seeing it. – Jandy

So array2 isn't actually the sub-array of array, it's just a different area of the same length, defining all the XML nodes? It might help to show how you create your array variables to confirm this. I'd be tempted to just use UCase when you populate array2, thus reducing 12000 * 20 * 2 function calls. – Davinadavine

@Davinadavine its array[2] not array2. I have corrected it. – Webfooted

same thing still applies. The value of array[2][j] is always the same, every single time you call it on your 3..12000 iterations. – Davinadavine

@Davinadavine I am calling it every time to create same node in different row node. And array[2] is also dynamic. So I have to loop. Can this be done in some better way? – Webfooted

Loop once just over array[2], setting all the values to UCase, before your main loop. Then you remove the need to call UCase twice on every one of the thousands of iterations of the subsequent loops – Davinadavine

I have modified my code but still getting heap error. – Webfooted

I'd perhaps build the string first, then convert it to XML, rather than use CFXML, which I've had memory issues with in the past. What are you doing with this XML once you've created it? Also agree with Duncan: you don't need to ucase() the same string multiple times. Possibly also use StringBuilder, rather than String for this. – Jandy

@AdamCameron This XML will be used for reporting purpose and some DB inserts. I have modified my code and removed the multiple ucase. I am checking this using stringBuilder. Thanks – Webfooted

The problem seems to be that you are trying to deal with too much data. Look at where that data is coming from and see if it can be reduced. If not, look for a way to break it into chunks and process each chunk separately. – Matroclinous

So to be clear, do you ever actually need the XML to be a CFML XML object, or will it be used by a separate system (so it can just stay being a string)? – Jandy

Yes, I need to create a CFML XML object for other system. – Webfooted

What happens if you move the CFOUTPUT to the outside of your loops so you aren't calling it 240,00 separate times. When building large strings, I prefer to dump string fragments into a 1 dimensional array & then use ArrayToList(Array, "") at the end. Then perform CFXML with the final, single generated string versus doing it all within the CFXML block. – Asir

@JamesMoberg I have modified my code and moved cfoutput. – Webfooted

increased your heap size to how much exactly? are you using 64-bit? – Restaurateur

@Beginner Did you try using an array to build the string outside of the CFXML function? This will help identify whether the issue is with the XML generation versus the CFXML function. (I'm guessing it's w/CFXML.) – Asir

@JamesMoberg I have added my answer. – Webfooted

Answer 1

After suggestions from Adam Cameron, Dan Bracuk and James Moberg, I have done the following and I am not getting the heap error now. I suspect that it was because of CFXML tag but not sure.

Converting all element of array[2] to uppercase:

<cfloop from="1" to="#arrayLen(local.array[2])#" index="i" step="1">
  <cfset local.array[2][i] = ucase(local.array[2][i])>
</cfloop>

Then using cfsavecontent to generate the xml string:

<cfsavecontent variable="local.xmlString">
  <?xml version="1.0" encoding="UTF-8"?>
  <ROWS>
    <cfoutput>
      <cfloop from="3" to="#arrayLen(local.array)#" index="local.currentRow" step="1">
        <ROW>
          <cfloop from="1" to="#arrayLen(local.array[2])#" index="local.currentColumn" step="1">
            <#local.array[2][local.currentColumn]#>
              <![CDATA[#trim(local.array[local.currentRow][local.currentColumn])#]]>
            </#local.array[2][local.currentColumn]#>
          </cfloop>
        </ROW>
      </cfloop>
    </cfoutput>
  </ROWS>
</cfsavecontent>
<cfset local.xml = xmlParse(local.xmlString)>

Webfooted