Allocating more than 4GB memory in a 64 bit system

Question

I'm running this code, compiled on 64 bits vc++ 2005, on Windows Server 2008 R2 with 32GB. There is an access violation inside the for loop.

#include <iostream>
using namespace std;


int main(int argc, char* argv[])
{   
    double *x = new double[536870912];

    cout << "memory allocated" << endl;

    for(long int i = 0; i < 536870912; i++)
    {   
        cout << i << endl;
        x[i] = 0;
    }

    delete [] x;
    return 0;
}

So if there is no exception in new double[536870912], why am I getting an access violation when doing an assignment over a particular array position?

Another point worth mentioning is that this program was succesfully tested on another computer.

Self

Over-committing could well be the culprit. – Mcalpine

One problem is that (I think) long int is 32 bits on 64-bit Windows, so the loop will never terminate. You should change the type of i to size_t to be sure it's big enough for any array index. I've no idea whether that's the only problem, though. – Smorgasbord

The thing is that it prints out over a hundred numbers and then there is an access violation, so it seems it is not reaching the limit. – Self

Just because the computer has sufficient memory, doesn't mean it can find a free 4GB consecutive block in there. – Sosthenna

@MikeSeymour Even so, a little over 500 million shouldn't be a problem even for a 32-bit integer. – Nosology

True, but then wouldn't the allocation fail? – Cointon

@Mike Seymour The loop iteration size is much smaller than 2^32 though. – Dewyeyed

@JoachimPileborg: True; I lost count of the digits and thought there were 5G of them. That's not the culprit then. – Smorgasbord

@KirkBackus true, but if new[] cannot allocate the requested amount then it should throw an exception at the allocation point. – Mcavoy

536870912 * sizeof(double) yields 0 for me. Looks like a wrap around inside operator new. – Hospital

@Joe Runde The allocation will fail and returns null – Sosthenna

@KirkBackus new doesn't retun null unless using the nothrow version which isn't the case here? – Fehr

@aaronman not here it isn't. This is the nothrow syntax: double *x = new (std::nothrow) double [536870912]. – Mcavoy

@KirkBackus I have checked that it does not return null – Self

You may have compiled on 64-bit windows, but the compiler you are using is in 64-bit mode, right? – Clementineclementis

What is "the particular array position"? What compiler have you used? There is nothing wrong with the program. You have the live repro; attach a debugger and investigate. – Jessi

If the allocation fails and you are not using the no throwing version, it does NOT return null. It doesn't return anything. An exception is thrown. Checking for null does nothing. – Clementineclementis

@NeilKirk Yes, the target platform is x64 – Self

@JamesMcNellis Visual C++ 2005, and it fails on iteration 768 – Self

Did you try rebooting the windows machine? Sounds like a joke, I know... but... windows... you never know. – Janus

@Janus sorry but that makes no sense at all – Fehr

@user4272 Rather than the array index, I think it would be more interesting to know the culprit address (ie. &x[i]). Wouldn't surprise me if it was on a page boundary. – Mcalpine

@Mcalpine Iterations and addresses: 764-0000000000F17F60 765-0000000000F17F68 766-0000000000F17F70 767-0000000000F17F78 768-0000000000F17F80 – Self

@stijn: ever heard of overcommiting memory allocations? Memory fragmentation? Kernel in an unstable state? I know Macs may need a reboot every week or so if you need a large contiguous block of memory, otherwise the allocation fails. – Janus

Are you compiling in debug mode or release? – Erective

@FlorisVelleman I have tried both. In debug it reaches the 768 iteration and in release it reaches the 486 iteration. – Self

@user4272: is the system running under a virtual machine that might do memory overcommitment? That means you would detect failures only when actually writing to the memory. – Janus

For what it's worth, this program works fine in VS2012 on Win8 x64 – Northcutt

@Janus yes to all. And I seriously doubt it has anything to do with this. And even if it were, it's not exactly a proper solution, nor probably the solution the OP is looking for. – Fehr

@user4272 Hmm not a page boundary so here goes my theory. – Mcalpine

What is the smallest value you can allocate before it crashes? 4GB+1? – Clementineclementis

It might be interesting to know that it works just fine if you pull out the constant into a variable (at least for me). My compiler also warns about a constant overflow when just using the constant directly, so I suspect that the number being used isn't the number you expect. – Sass

@NeilKirk Yes, no problem when using 536870911 – Self

@Fehr The code contains no errors, so the issue must therefore be in the system. How is asking about known problems about the system in question not helping? – Janus

what happens if you malloc the memory (this one will return null for failure) – Clementineclementis

Answer 1

It is probably one of the following problem:

• long int is 32-bits: that mean your maximum value is 2147483647, and sizeof(double)*536870912 >= 2147483647. (I don't really know if that has sense. It probably depend on how the compiller work)
• Your allocation is failing.

I suggest you to test the following code:

#include<conio.h>
#include <iostream>
using namespace std;

#define MYTYPE unsigned long long


int main(int argc, char* argv[])
{   
    // Test compiling mode
    if (sizeof(void*) == 8) cout << "Compiling 64-bits" << endl;
    else cout << "Compiling 32-bits" << endl;

    // Test the size of mytype
    cout << "Sizeof:" << sizeof(MYTYPE) << endl;
    MYTYPE len;

    // Get the number of <<doubles>> to allocate
    cout << "How many doubles do you want?" << endl;
    cin >> len;
    double *x = new (std::nothrow) double[len];
    // Test allocation
    if (NULL==x)
    {
        cout << "unable to allocate" << endl;
        return 0;
    }
    cout << "memory allocated" << endl;

    // Set all values to 0
    for(MYTYPE i = 0; i < len; i++)
    {   
        if (i%100000==0) cout << i << endl;
        x[i] = 0;
    }

    // Wait before release, to test memory usage
    cout << "Press <Enter> key to continue...";
    getch();

    // Free memory.
    delete [] x;

}

Editing: Using this code, i just achieved allocate a single block of 9GB.

Bambara