Combining SpatialPointsDataFrame with SpatialPolygonsDataFrame error: maximum returned dense matrix size exceeded

Question

I am trying to combine a SpatialPointsDataFrame (grid) of 1000x1000m squares over a SpatialPolygonsDataFrame (info) to aggregate all the information of the points within each grid square.

I tried the code:

combined <- intersect(info, grid)

But I recive this error:

Error in RGEOSBinPredFunc(spgeom1, spgeom2, byid, func) : 
  rgeos_binpredfunc_prepared: maximum returned dense matrix size exceeded

Is there anotherway to do what I want or to resolve the error?

Zeena

Answer 1

The error indicates you are maxing out your memory. One solution is to break up you dataset and do the intersect in chunks. The new SF package makes this a bit easier to do using dplyr verbs. Add a column to define your chunks and then try the following:

combined <- info %>%
    group_by(chuncks) %>%
    do(sf::st_intersection(., grid))

To speed the process up you could also try splitting your dataset into a list of chuncks and then applying the st_intersection function in parallel. This is much faster.

#import packages
library(foreach)
library(doParallel)

#setup parallel backend to use 8 processors
cl<-makeCluster(4)
registerDoParallel(cl)

tmp <- split(info, info$chunks)

# run using foreach
by.chunk <- foreach(df = tmp) %dopar% {
    sf::st_intersection(df, grid)
}

# combine list of data.frames 
combined <- ldply(by.chunck

Halt