How can I avoid having two instances of a very large matrix at the same time when loading it into a solver?

Question

I am using both Cplex and Gurobi for an LP program whose inequality constraint matrix A can become truly large -- around 5 to 10GB. When I want to use one of those solvers, I have to create a separate struct with all the problem constraints. This means that I have the matrix A in my workspace, and the matrix A in my solver struct at the same time. Even if I clear it in my Workspace as fast as possible, there is still a time when both exist and my RAM is overloaded.

I am asking if there is some clever method to deliver the matrix A into the model without both existing at the same time. The only thing I can think of right now is delivering it in small chunks...

Manipulator

Depends in the code, but you need to show the code. – Vehement

Answer 1

MATLAB using copy-on-write, or lazy copying. This means that, as long as you don't modify one of the copies, all copies of a matrix share the same data:

A = randn(10000);
B = A; % does not take up extra memory
myfunc(B);

function myfunc(matrix)
   C = matrix; % does not take up extra memory.

For reference, see for example on Loren's blog and Undocumented Matlab.

Roxyroy