Workaround for python MemoryError

Question

How can I change this function to make it more efficient? I keep getting MemoryError

def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate(sequences):
        results[i, sequence] = 1.
    return results

I call the function here:

x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)

Train and Test data are IMDB dataset for sentiment analysis, i.e.

(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

EDIT: I am running this on 64 bit Ubuntu system with 4 GB RAM.

Here is the Traceback:

Traceback (most recent call last):

  File "/home/uttam/PycharmProjects/IMDB/imdb.py", line 29, in <module>
    x_test = vectorize_sequences(test_data)
  File "/home/uttam/PycharmProjects/IMDB/imdb.py", line 20, in vectorize_sequences
    results = np.zeros((len(sequences), dimension))
MemoryError

Brimstone

Looks like 2x 763 MB of data which is not gigantic. Please post the full error message including the traceback showing the line where it happened. Please also post the details of the hardware and OS where you're running this. – Land

Basically you have two options: use less memory or make more memory available. – Schuster

@JohnZwinck I have edited the question accordingly. Thanks – Brimstone

A related question: #68422910 – Guadalajara

Answer 1

Your array appears to be 10k x 10k which is 100 million elements of 64 bits each (because the default dtype is float64). So that's 800 million bytes, aka 763 megabytes.

If you use float32 it will cut the memory usage in half:

np.zeros((len(sequences), dimension), dtype=np.float32)

Or if you only care about 0 and 1, this will cut it by 88%:

np.zeros((len(sequences), dimension), dtype=np.int8)

Land