Cannot fix the lack of memory problem in running "pvargmm"

Asked 5/12, 2021 at 9:21 Answered 23/12, 2021 at 17:31

Solved r memory memory-management out-of-memory vector-auto-regression

My computer uses a CPT of Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz 2.59 GHz. Also my RAM memory size is 16 GB. When I run the following panel VAR model "pvargmm" in R,

library(imputeTS)
library("panelvar")
data1=data.frame(na.remove(cbind(Country,   Date,   x1, x2, x3, x4, x5, x6, x7, x8, x9, x10,    x11,    x12,    x13,    x14,x15,x16,x17,x18)))
                                                            
colnames(data1)<-cbind("Country",   "Date", "x1",   "x2",   "x3",   "x4",   "x5",   "x6",   "x7",   "x8",   "x9",   "x10",  "x11",  "x12",  "x13",  "x14","x15","x16","x17","x18")
                                                            
                                                            
regp=pvargmm(dependent_vars = c("x13","x2","x3","x4","x5","x6"),lags = 1,                                                           
             exog_vars = c("x14"),                                                          
             data = data1,steps= c("mstep"),                                                            
             panel_identifier = c("Country", "Date"))

I always get the following error:

Error in h(simpleError(msg, call)) : 
  error in evaluating the argument 'current' in selecting a method for function 'all.equal': cannot allocate vector of size 7.1 Gb

So I tried using only two dependent variables to see whether the memory can afford instead of six dependent variables I had earlier.

Then still I had the memory error but in different form as follows:

Error in .dense2C(from) :                                                                                                                                                                            
  Cholmod error 'out of memory' at file ../Core/cholmod_memory.c, line 146

But I currently use the following codes trying to boost the memory:

options(java.parameters = "- Xmx800000000000000m")
memory.limit(size=8e+14)

My windows is 64 bit and my R program is also 64 bit as well.

The data is balanced with 2060 number of rows with no missing values.

The snippet of the first 50 rows using dput(data1) are as follows:

    > dput(data1[1:50,])
structure(list(Country = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), Date = c(48, 
49, 52, 53, 54, 57, 59, 60, 64, 65, 69, 71, 86, 87, 88, 92, 101, 
102, 105, 106, 110, 113, 118, 119, 121, 123, 124, 125, 126, 127, 
129, 132, 133, 136, 137, 143, 144, 148, 149, 151, 152, 155, 156, 
157, 158, 161, 162, 166, 167, 168), x1 = c(0.014748522, 
0.118574701, 0.014776643, 0.110949861, 0.01481079, 0.118697229, 
0.109259581, 0.106920507, 0.09964718, 0.107359397, 0.100214624, 
0.101336456, 0.084556183, 0.109388135, 0.049318414, 0.083084846, 
0.101614654, 0.09898533, 0.08605765, 0.099262524, 0.097317145, 
0.094441761, 0.088059271, 0.101287244, 0.102545664, 0.106297825, 
0.097040955, 0.080330986, 0.103339081, 0.108313506, 0.100936735, 
0.10794291, 0.11167398, 0.111364648, 0.108089542, 0.110835368, 
0.112419189, 0.110474815, 0.112116887, 0.122428299, 0.114857692, 
0.115030436, 0.119601122, 0.114017072, 0.114926991, 0.113645471, 
0.117205805, 0.115805775, 0.11617135, 0.114326404), x2 = c(0.044647275, 
0.053976585, 0.030403218, 0.044558117, 0.063132462, 0.103456438, 
0.117170791, 0.104951921, 0.108145525, 0.107693444, 0.096528502, 
0.095931022, 0.083300776, 0.080563349, 0.076819818, 0.084028311, 
0.095892312, 0.096190825, 0.091091159, 0.090343147, 0.096242416, 
0.085306606, 0.085667078, 0.09251297, 0.105269247, 0.095251763, 
0.093446551, 0.096549008, 0.100387759, 0.101508899, 0.100509418, 
0.107830747, 0.109448071, 0.110830736, 0.109078427, 0.109318996, 
0.112848661, 0.110987973, 0.112196608, 0.115601933, 0.114478704, 
0.116686745, 0.116382225, 0.113006561, 0.109417021, 0.114979708, 
0.115397391, 0.115777083, 0.114273074, 0.111343996), x3 = c(25, 
25, 41.67, 75, 88.89, 93.52, 93.52, 93.52, 93.52, 93.52, 93.52, 
93.52, 90.74, 90.74, 90.74, 90.74, 90.74, 88.89, 88.89, 88.89, 
88.89, 88.89, 88.89, 92.59, 92.59, 92.59, 92.59, 92.59, 92.59, 
92.59, 92.59, 90.74, 90.74, 90.74, 90.74, 88.89, 87.96, 87.96, 
87.96, 87.96, 87.96, 87.96, 87.96, 87.96, 87.96, 87.96, 87.96, 
87.96, 87.96, 87.96), x4 = c(0, 0, 0, 0, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), 
    x5 = c(4.815325122, 4.815325122, 4.815325122, 
    4.815325122, 4.815325122, 4.815325122, 4.815325122, 4.815325122, 
    4.815325122, 4.815325122, 4.815325122, 4.815325122, 4.815325122, 
    4.815325122, 4.815325122, 4.815325122, 4.815325122, 4.815325122, 
    4.815325122, 4.815325122, 6.041347309, 6.041347309, 6.041347309, 
    6.041347309, 6.041347309, 6.041347309, 6.041347309, 6.041347309, 
    6.041347309, 6.041347309, 6.041347309, 6.041347309, 6.041347309, 
    6.041347309, 6.041347309, 6.041347309, 6.041347309, 6.041347309, 
    6.041347309, 6.041347309, 6.041347309, 6.041347309, 6.041347309, 
    6.041347309, 6.041347309, 6.041347309, 6.041347309, 6.041347309, 
    6.041347309, 6.041347309), x6 = c(0.7935, 
    0.7303, 0.5763, 0.5331, 0.4907, 0.3064, 0.2461, 0.1939, 0.1127, 
    0.096, 0.0012, -0.0282, -0.2368, -0.2497, -0.2622, -0.3073, 
    -0.4152, -0.425, -0.4503, -0.461, -0.5089, -0.5376, -0.5856, 
    -0.5956, -0.6147, -0.6337, -0.6429, -0.652, -0.6779, -0.6863, 
    -0.7033, -0.7285, -0.7366, -0.7596, -0.7673, -0.8152, -0.8226, 
    -0.8511, -0.8582, -0.8817, -0.8897, -0.913, -0.9206, -0.9285, 
    -0.9366, -0.9632, -0.9714, -1.0053, -1.0137, -1.0223), x7 = c(38, 
    38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 
    38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 
    38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 
    38, 38, 38, 38), X8 = c(-4.397966662, -6.304929628, 
    0.488928104, -6.304929628, 2.54486109, -3.296545249, 1.344450099, 
    3.782659735, -0.844822382, 4.83150399, -6.304929628, 2.159834672, 
    1.420876501, -3.354324242, 3.589037795, 1.061780955, 4.228123326, 
    -0.404162634, -5.056291726, 0.010801841, -5.328349718, -1.493660218, 
    -0.696633142, -4.105707617, -0.871840445, 5.29044444, -1.962123959, 
    0.586428005, 1.138495764, 1.753597336, 0.275856688, 2.375667683, 
    3.884202996, 1.723158621, -1.047778386, -2.310359726, 0.175022741, 
    -4.057753192, 1.331212028, -4.328358106, 2.086407315, -1.432959593, 
    -0.337455739, -1.618003031, -3.500966569, -0.620899578, -3.649420293, 
    -0.459085095, 2.257504544, 0.745875601), X9 = c(-4.302658422, 
    -6.110280589, 0.490125308, -6.110280589, 2.577519125, -3.242801379, 
    1.353528468, 3.855112975, -0.841263786, 4.950123801, -6.110280589, 
    2.183327935, 1.431018931, -3.298690566, 3.654221238, 1.067437852, 
    4.318781661, -0.403346996, -4.930588828, 0.010802424, -5.188881247, 
    -1.482560447, -0.694212278, -4.022565186, -0.868050937, 5.432889579, 
    -1.942999592, 0.58815086, 1.145001292, 1.769063124, 0.276237523, 
    2.404111465, 3.960624404, 1.738090643, -1.04230831, -2.28387527, 
    0.175175995, -3.976528721, 1.340112104, -4.236021695, 2.108324957, 
    -1.422741592, -0.336886997, -1.604983674, -3.440391694, -0.61897598, 
    -3.583631679, -0.45803291, 2.283179015, 0.748664182), X10 = c(0.022036057, 
    0.022099114, 0.022148854, 0.022295818, 0.022296321, 0.022417636, 
    0.022468635, 0.022471382, 0.022464479, 0.022474524, 0.022565, 
    0.022556508, 0.022628762, 0.022632952, 0.022636849, 0.022625484, 
    0.022663127, 0.022660331, 0.022713486, 0.022710519, 0.022745041, 
    0.022848741, 0.022858749, 0.022866118, 0.022865227, 0.022874749, 
    0.022874749, 0.022874749, 0.022874749, 0.022874749, 0.022873025, 
    0.022861229, 0.022866133, 0.022853027, 0.022850894, 0.022853874, 
    0.022850921, 0.022855289, 0.022853114, 0.022862262, 0.022861413, 
    0.022849419, 0.022846619, 0.022845453, 0.022850036, 0.022871213, 
    0.022874749, 0.022860246, 0.022859786, 0.022857052), x11 = c(0.02205167, 
    0.022114713, 0.022164428, 0.022311364, 0.022311864, 0.022433137, 
    0.022484114, 0.022486855, 0.022479932, 0.022489972, 0.022580409, 
    0.022571904, 0.022644075, 0.022648261, 0.022652155, 0.022640772, 
    0.022678364, 0.022675565, 0.022728696, 0.022725727, 0.022760221, 
    0.022863891, 0.022873875, 0.02288124, 0.022880342, 0.022889387, 
    0.022889387, 0.022889387, 0.022889387, 0.022889387, 0.022888096, 
    0.022876286, 0.022881185, 0.022868066, 0.02286593, 0.022868884, 
    0.022865929, 0.022870278, 0.0228681, 0.022877231, 0.022876379, 
    0.022864371, 0.022861568, 0.022860399, 0.022864979, 0.022886138, 
    0.022889387, 0.022875151, 0.022874688, 0.022871951), x12 = c(0.021513181, 
    0.021571753, 0.021617452, 0.02174688, 0.021747569, 0.021882247, 
    0.021932113, 0.021935407, 0.021929198, 0.021940171, 0.022036504, 
    0.022028441, 0.022112581, 0.02211688, 0.022121171, 0.022110325, 
    0.022152497, 0.022149788, 0.022207397, 0.022204502, 0.022237638, 
    0.022350023, 0.022361011, 0.022368394, 0.022367831, 0.022392916, 
    0.022392916, 0.022392916, 0.022385136, 0.022383687, 0.022381105, 
    0.022369664, 0.022375024, 0.022362253, 0.02236023, 0.022365686, 
    0.022362796, 0.022367793, 0.022365675, 0.022375336, 0.022374587, 
    0.022363052, 0.022360332, 0.022359293, 0.022363957, 0.022387616, 
    0.022392877, 0.022377085, 0.02237674, 0.022374056), x13 = c(0.021528877, 
    0.021587435, 0.021633108, 0.021762508, 0.021763194, 0.021897824, 
    0.021947669, 0.021950955, 0.021944726, 0.021955694, 0.022051985, 
    0.022043909, 0.022127962, 0.022132257, 0.022136544, 0.02212568, 
    0.022167799, 0.022165088, 0.022222671, 0.022219773, 0.022252881, 
    0.022365232, 0.022376196, 0.022383574, 0.022383005, 0.022407741, 
    0.022407741, 0.022407741, 0.022400273, 0.022398821, 0.022396232, 
    0.022384778, 0.022390134, 0.022377348, 0.022375323, 0.022380752, 
    0.02237786, 0.022382837, 0.022380717, 0.022390361, 0.022389608, 
    0.02237806, 0.022375337, 0.022374295, 0.022378955, 0.022402595, 
    0.022407741, 0.022392044, 0.022391696, 0.022389009), x14 = c(355.7064977, 
    355.7064977, 355.7064977, 355.7064977, 355.7064977, 355.7064977, 
    355.7064977, 366.871849, 366.871849, 366.871849, 366.871849, 
    366.871849, 436.6764361, 436.6764361, 436.6764361, 436.6764361, 
    343.7874609, 343.7874609, 343.7874609, 343.7874609, 343.7874609, 
    343.7874609, 343.7874609, 343.7874609, 351.4579307, 351.4579307, 
    351.4579307, 351.4579307, 351.4579307, 351.4579307, 351.4579307, 
    351.4579307, 351.4579307, 351.4579307, 351.4579307, 313.8276295, 
    313.8276295, 313.8276295, 313.8276295, 313.8276295, 313.8276295, 
    313.8276295, 313.8276295, 313.8276295, 313.8276295, 299.7095158, 
    299.7095158, 299.7095158, 299.7095158, 299.7095158), x15 = c(13, 
    13, 13, 13, 13, 13, 13, -1.5, -1.5, -1.5, -1.5, -1.5, -1.5, 
    -1.5, -1.5, -1.5, -1.5, -1.5, -1.5, -1.5, -1.5, -1.5, -1.5, 
    -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, 
    -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, 
    -5.5, -5.5, -5.5, -5.5, -5.5, -5.5, -5.5), x16 = c(2, 2, 
    2, 2, 2, 2, 2, 3.3, 3.3, 3.3, 3.3, 3.3, 1.5, 1.5, 1.5, 1.5, 
    1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 2.2, 2.2, 2.2, 2.2, 2.2, 
    2.2, 2.2, 2.2, 2.2, 2.2, 2.2, 2.2, 1.9, 1.9, 1.9, 1.9, 1.9, 
    1.9, 1.9, 1.9, 1.9, 1.9, 2.7, 2.7, 2.7, 2.7, 2.7), x17 = c(53.9, 
    75.47, 75.91, 75.91, 72, 61, 57.08, 57.06, 46.7, 43.35, 40.11, 
    43.83, 33.04, 35.28, 32.61, 27.99, 25.66, 25.81, 27.57, 27.57, 
    33.47, 31.77, 31.78, 30.43, 27.68, 27.94, 29.43, 28.08, 32.19, 
    29.52, 28, 24.84, 24.32, 24.74, 25.44, 22.99, 22.65, 22.28, 
    22.13, 21.51, 22.54, 22.37, 22.03, 23.27, 24.47, 26.12, 26.57, 
    31.46, 28.81, 29.71), x18 = c(13.95348837, 40.01855288, 
    -8.199298585, 0.711368726, -5.820797907, -4.61297889, -12.9081477, 
    6.574523721, 3.227232538, -7.173447537, -1.787463271, 14.88859764, 
    19.84040624, 6.779661017, -7.568027211, -8.319685555, -4.396423249, 
    0.58456742, 6.819062379, 0, -0.594000594, -9.538724374, -8.494097322, 
    -4.247954688, -3.284416492, 0.939306358, 5.33285612, -4.587155963, 
    17.95529498, -8.294501398, 0.864553314, 1.553556827, -2.093397746, 
    -4.256965944, 2.829426031, -3.240740741, -1.478903871, -7.282563462, 
    -0.673249551, 0.74941452, 4.788470479, -0.754214729, -1.519892713, 
    5.628688153, 5.156854319, -1.098068913, 1.722817764, 2.308943089, 
    -8.423394787, 3.123915307)), row.names = c(NA, 50L), class = "data.frame")

If I show the first 50 rows of the raw data itself with data1[1:50,], it shows as below:

Country Date    x1  x2  x3  x4  x5  x6  x7  x8  x9  x10 x11 x12 x13 x14 x15 x16 x17 x18
1   48  0.01474852  0.04464728  25  0   4.815325    0.7935  38  -4.39796666 -4.30265842 0.02203606  0.02205167  0.02151318  0.02152888  355.7065    13  2   53.9    13.9534884
1   49  0.1185747   0.05397659  25  0   4.815325    0.7303  38  -6.30492963 -6.11028059 0.02209911  0.02211471  0.02157175  0.02158743  355.7065    13  2   75.47   40.0185529
1   52  0.01477664  0.03040322  41.67   0   4.815325    0.5763  38  0.4889281   0.49012531  0.02214885  0.02216443  0.02161745  0.02163311  355.7065    13  2   75.91   -8.1992986
1   53  0.11094986  0.04455812  75  0   4.815325    0.5331  38  -6.30492963 -6.11028059 0.02229582  0.02231136  0.02174688  0.02176251  355.7065    13  2   75.91   0.7113687
1   54  0.01481079  0.06313246  88.89   1   4.815325    0.4907  38  2.54486109  2.57751912  0.02229632  0.02231186  0.02174757  0.02176319  355.7065    13  2   72  -5.8207979
1   57  0.11869723  0.10345644  93.52   1   4.815325    0.3064  38  -3.29654525 -3.24280138 0.02241764  0.02243314  0.02188225  0.02189782  355.7065    13  2   61  -4.6129789
1   59  0.10925958  0.11717079  93.52   1   4.815325    0.2461  38  1.3444501   1.35352847  0.02246864  0.02248411  0.02193211  0.02194767  355.7065    13  2   57.08   -12.9081477
1   60  0.10692051  0.10495192  93.52   1   4.815325    0.1939  38  3.78265974  3.85511297  0.02247138  0.02248686  0.02193541  0.02195096  366.8718    -1.5    3.3 57.06   6.5745237
1   64  0.09964718  0.10814553  93.52   1   4.815325    0.1127  38  -0.84482238 -0.84126379 0.02246448  0.02247993  0.0219292   0.02194473  366.8718    -1.5    3.3 46.7    3.2272325
1   65  0.1073594   0.10769344  93.52   1   4.815325    0.096   38  4.83150399  4.9501238   0.02247452  0.02248997  0.02194017  0.02195569  366.8718    -1.5    3.3 43.35   -7.1734475
1   69  0.10021462  0.0965285   93.52   1   4.815325    0.0012  38  -6.30492963 -6.11028059 0.022565    0.02258041  0.0220365   0.02205198  366.8718    -1.5    3.3 40.11   -1.7874633
1   71  0.10133646  0.09593102  93.52   1   4.815325    -0.0282 38  2.15983467  2.18332793  0.02255651  0.0225719   0.02202844  0.02204391  366.8718    -1.5    3.3 43.83   14.8885976
1   86  0.08455618  0.08330078  90.74   1   4.815325    -0.2368 38  1.4208765   1.43101893  0.02262876  0.02264407  0.02211258  0.02212796  436.6764    -1.5    1.5 33.04   19.8404062
1   87  0.10938813  0.08056335  90.74   1   4.815325    -0.2497 38  -3.35432424 -3.29869057 0.02263295  0.02264826  0.02211688  0.02213226  436.6764    -1.5    1.5 35.28   6.779661
1   88  0.04931841  0.07681982  90.74   1   4.815325    -0.2622 38  3.58903779  3.65422124  0.02263685  0.02265216  0.02212117  0.02213654  436.6764    -1.5    1.5 32.61   -7.5680272
1   92  0.08308485  0.08402831  90.74   1   4.815325    -0.3073 38  1.06178095  1.06743785  0.02262548  0.02264077  0.02211033  0.02212568  436.6764    -1.5    1.5 27.99   -8.3196856
1   101 0.10161465  0.09589231  90.74   1   4.815325    -0.4152 38  4.22812333  4.31878166  0.02266313  0.02267836  0.0221525   0.0221678   343.7875    -1.5    1.5 25.66   -4.3964232
1   102 0.09898533  0.09619082  88.89   1   4.815325    -0.425  38  -0.40416263 -0.403347   0.02266033  0.02267557  0.02214979  0.02216509  343.7875    -1.5    1.5 25.81   0.5845674
1   105 0.08605765  0.09109116  88.89   1   4.815325    -0.4503 38  -5.05629173 -4.93058883 0.02271349  0.0227287   0.0222074   0.02222267  343.7875    -1.5    1.5 27.57   6.8190624
1   106 0.09926252  0.09034315  88.89   1   4.815325    -0.461  38  0.01080184  0.01080242  0.02271052  0.02272573  0.0222045   0.02221977  343.7875    -1.5    1.5 27.57   0
1   110 0.09731714  0.09624242  88.89   1   6.041347    -0.5089 38  -5.32834972 -5.18888125 0.02274504  0.02276022  0.02223764  0.02225288  343.7875    -1.5    1.5 33.47   -0.5940006
1   113 0.09444176  0.08530661  88.89   1   6.041347    -0.5376 38  -1.49366022 -1.48256045 0.02284874  0.02286389  0.02235002  0.02236523  343.7875    -1.5    1.5 31.77   -9.5387244
1   118 0.08805927  0.08566708  88.89   1   6.041347    -0.5856 38  -0.69663314 -0.69421228 0.02285875  0.02287387  0.02236101  0.0223762   343.7875    -1.5    1.5 31.78   -8.4940973
1   119 0.10128724  0.09251297  92.59   1   6.041347    -0.5956 38  -4.10570762 -4.02256519 0.02286612  0.02288124  0.02236839  0.02238357  343.7875    -5.5    2.2 30.43   -4.2479547
1   121 0.10254566  0.10526925  92.59   1   6.041347    -0.6147 38  -0.87184045 -0.86805094 0.02286523  0.02288034  0.02236783  0.02238301  351.4579    -5.5    2.2 27.68   -3.2844165
1   123 0.10629782  0.09525176  92.59   1   6.041347    -0.6337 38  5.29044444  5.43288958  0.02287475  0.02288939  0.02239292  0.02240774  351.4579    -5.5    2.2 27.94   0.9393064
1   124 0.09704095  0.09344655  92.59   1   6.041347    -0.6429 38  -1.96212396 -1.94299959 0.02287475  0.02288939  0.02239292  0.02240774  351.4579    -5.5    2.2 29.43   5.3328561
1   125 0.08033099  0.09654901  92.59   1   6.041347    -0.652  38  0.58642801  0.58815086  0.02287475  0.02288939  0.02239292  0.02240774  351.4579    -5.5    2.2 28.08   -4.587156
1   126 0.10333908  0.10038776  92.59   1   6.041347    -0.6779 38  1.13849576  1.14500129  0.02287475  0.02288939  0.02238514  0.02240027  351.4579    -5.5    2.2 32.19   17.955295
1   127 0.10831351  0.1015089   92.59   1   6.041347    -0.6863 38  1.75359734  1.76906312  0.02287475  0.02288939  0.02238369  0.02239882  351.4579    -5.5    2.2 29.52   -8.2945014
1   129 0.10093673  0.10050942  92.59   1   6.041347    -0.7033 38  0.27585669  0.27623752  0.02287303  0.0228881   0.0223811   0.02239623  351.4579    -5.5    2.2 28  0.8645533
1   132 0.10794291  0.10783075  90.74   1   6.041347    -0.7285 38  2.37566768  2.40411147  0.02286123  0.02287629  0.02236966  0.02238478  351.4579    -5.5    2.2 24.84   1.5535568
1   133 0.11167398  0.10944807  90.74   1   6.041347    -0.7366 38  3.884203    3.9606244   0.02286613  0.02288118  0.02237502  0.02239013  351.4579    -5.5    2.2 24.32   -2.0933977
1   136 0.11136465  0.11083074  90.74   1   6.041347    -0.7596 38  1.72315862  1.73809064  0.02285303  0.02286807  0.02236225  0.02237735  351.4579    -5.5    2.2 24.74   -4.2569659
1   137 0.10808954  0.10907843  90.74   1   6.041347    -0.7673 38  -1.04777839 -1.04230831 0.02285089  0.02286593  0.02236023  0.02237532  351.4579    -5.5    2.2 25.44   2.829426
1   143 0.11083537  0.109319    88.89   1   6.041347    -0.8152 38  -2.31035973 -2.28387527 0.02285387  0.02286888  0.02236569  0.02238075  313.8276    -5.5    1.9 22.99   -3.2407407
1   144 0.11241919  0.11284866  87.96   1   6.041347    -0.8226 38  0.17502274  0.175176    0.02285092  0.02286593  0.0223628   0.02237786  313.8276    -5.5    1.9 22.65   -1.4789039
1   148 0.11047482  0.11098797  87.96   1   6.041347    -0.8511 38  -4.05775319 -3.97652872 0.02285529  0.02287028  0.02236779  0.02238284  313.8276    -5.5    1.9 22.28   -7.2825635
1   149 0.11211689  0.11219661  87.96   1   6.041347    -0.8582 38  1.33121203  1.3401121   0.02285311  0.0228681   0.02236568  0.02238072  313.8276    -5.5    1.9 22.13   -0.6732496
1   151 0.1224283   0.11560193  87.96   1   6.041347    -0.8817 38  -4.32835811 -4.23602169 0.02286226  0.02287723  0.02237534  0.02239036  313.8276    -5.5    1.9 21.51   0.7494145
1   152 0.11485769  0.1144787   87.96   1   6.041347    -0.8897 38  2.08640732  2.10832496  0.02286141  0.02287638  0.02237459  0.02238961  313.8276    -5.5    1.9 22.54   4.7884705
1   155 0.11503044  0.11668674  87.96   1   6.041347    -0.913  38  -1.43295959 -1.42274159 0.02284942  0.02286437  0.02236305  0.02237806  313.8276    -5.5    1.9 22.37   -0.7542147
1   156 0.11960112  0.11638223  87.96   1   6.041347    -0.9206 38  -0.33745574 -0.336887   0.02284662  0.02286157  0.02236033  0.02237534  313.8276    -5.5    1.9 22.03   -1.5198927
1   157 0.11401707  0.11300656  87.96   1   6.041347    -0.9285 38  -1.61800303 -1.60498367 0.02284545  0.0228604   0.02235929  0.02237429  313.8276    -5.5    1.9 23.27   5.6286882
1   158 0.11492699  0.10941702  87.96   1   6.041347    -0.9366 38  -3.50096657 -3.44039169 0.02285004  0.02286498  0.02236396  0.02237895  313.8276    -5.5    1.9 24.47   5.1568543
1   161 0.11364547  0.11497971  87.96   1   6.041347    -0.9632 38  -0.62089958 -0.61897598 0.02287121  0.02288614  0.02238762  0.0224026   299.7095    -5.5    2.7 26.12   -1.0980689
1   162 0.1172058   0.11539739  87.96   1   6.041347    -0.9714 38  -3.64942029 -3.58363168 0.02287475  0.02288939  0.02239288  0.02240774  299.7095    -5.5    2.7 26.57   1.7228178
1   166 0.11580577  0.11577708  87.96   1   6.041347    -1.0053 38  -0.45908509 -0.45803291 0.02286025  0.02287515  0.02237709  0.02239204  299.7095    -5.5    2.7 31.46   2.3089431
1   167 0.11617135  0.11427307  87.96   1   6.041347    -1.0137 38  2.25750454  2.28317901  0.02285979  0.02287469  0.02237674  0.0223917   299.7095    -5.5    2.7 28.81   -8.4233948
1   168 0.1143264   0.111344    87.96   1   6.041347    -1.0223 38  0.7458756   0.74866418  0.02285705  0.02287195  0.02237406  0.02238901  299.7095    -5.5    2.7 29.71   3.1239153

May I get help on fixing this error please?

Reynalda answered 5/12, 2021 at 9:21 Comment(18)

data.frame(cbind( has a nasty habit of turning values that are expected to be numeric to character ... which would then make the size of the model design matrix explode. To rule this out can you add your example data to your question with dput(data)` please. – Labradorite 9/12, 2021 at 10:54

@user20650: the output is too long to put from dput(data). Does not even fit one output screen in R either. – Reynalda 9/12, 2021 at 10:56

@user20650: Thank you. I have edited my question above accordingly. The na.remove command produces no errors. – Reynalda 9/12, 2021 at 11:4

Thanks. okay the values are all numeric so you can ignore my suggestion – Labradorite 9/12, 2021 at 11:5

@user20650: I added both the dput[data1[1:50,]) and data1[1:50,] for you and others' reference. – Reynalda 9/12, 2021 at 11:10

It seems like a poorly written implementation, a deep dive might be necessary to fix the issues. Even without memory issues, it's running insanely slow with the 50 rows provided. I've noticed it does a lot of large matrix inverses and as.matrix calls, which can also cause issues if translating sparse matrices to numeric. Alternative to rewriting parts of the code, you might want to try increasing the swap memory on your machine, might work as a slow workaround in some cases.. – Plosion 10/12, 2021 at 3:53

For example, instead of min(abs(eigen(sum_Lambda)$values)) < tol check that takes ages just to find the smallest value, one can use RSpectra or similar packages. Then they call Lambda_inv_vec <- solve(as.matrix(sum_Lambda_vec)), which is already a massive sparse matrix, and solve or ginv it.. So it seems a deeper dive into the code might be needed to make it more efficient – Plosion 10/12, 2021 at 4:26

@runr: This is a "pvargmm" function (from "panelvar" package) where I simply inputted the corresponding values following their syntax. So there is no way to have good or bad writing over it. You also talk about the RSpectra package and other eigen value codes. However, my main objective is to run the panel vector autoregression using "pvargmm" I'm afraid. – Reynalda 10/12, 2021 at 5:25

Yes, I commented about the way the panelvar package is written, sorry If I wasn't clear about it. My point was that it's a package issue, hence fixing the memory issue might not be as quick and simple (except for maybe trying increasing the swap memory in your machine). If you follow through the code and see exactly where it breaks, possible workarounds might be to reduce the parameter space a bit... For example, even with 50 rows data, the Lambda_inv_vec is a 47k x 47k matrix -- maybe reducing the search grid size or something would help a lot. This also needs a deeper dive.. – Plosion 10/12, 2021 at 15:5

@runr: Thank you. Then may I know a deeper dive you are mentioning? – Reynalda 10/12, 2021 at 15:21

If one library doesn't work you should probably try another. plm is a good one for panel data and has GMM estimators. fixest is famously fast, I'm not sure if it does gmm though. – Coelacanth 10/12, 2021 at 19:26

@dash2: Does this mean I cannot use "pvargmm" at all? The point here is to bring the vector autoregression framework. The simple VAR seems to work but due to the panel data structure, I am trying to make the panel VAR (pvargmm) work instead which is the point of my question. – Reynalda 11/12, 2021 at 2:13

Hi @Eric, Could you tell me how much the file size of your data? Thus, I can replicate the problem and probably I can help you to optimize the code in the package. – Guan 11/12, 2021 at 4:5

@AhmadZaenal: 403 KB, 20 columns, and 2060 rows. Balanced panel data. – Reynalda 11/12, 2021 at 4:28

@Eric: the point is, fixing package code is hard. (Maybe Ahmad Zanael will help.) If you can't do that, you may have to seek alternatives. Are you sure plm doesn't have the estimator you need? – Coelacanth 11/12, 2021 at 19:3

@dash2: plm is working fine. However, I am in need of panel vector autoregression not panel regression at this moment I'm afraid. plm has nothing to do with panel vector autoregression unfortunately. – Reynalda 12/12, 2021 at 5:6

I don't understand. If this package cannot even take care of 2060 x 20 matrix of panel data with memory limit, I don't think this package is practical at all for general users unfortunately. Even normal research takes care of ten thousands of observations or so. – Reynalda 13/12, 2021 at 17:46

@Eric, how many periods do you have for each country? Are you aware of the way panel GMM estimates are actually computed? – Considerable 15/12, 2021 at 0:33

As I tried to allude in my comment, this behavior is a feature and not a bug. In dynamic panel GMM the most popular procedure is the Arellano-Bond where the first difference of the dependent variable in t-1 is instrumented by all the observations of the dependent variable up to t-2. Therefore the size of the instrument matrix grows quite rapidly: it is of order $T^3$.

The procedure you are using is an extension of this idea to panel VAR, which includes many dependent variables, weakly exogenous variables and contemporary variables which makes this issue even worse. Much worse in fact.

To learn more about the details consult the companion paper to the package:

Sigmund, M., Ferstl, R. (2017) Panel Vector Autoregression in R with the Package panelvar

Especially equations (4) - (11)

The solution is to limit the maximal lag of the instrument using the options:

max_instr_dependent_vars and max_instr_predet_vars

This decreases size of the instrument matrix accordingly. From the point of view of efficiency the answer to the question what is the best number of lags does not have a general answer. Any number of lags yields a consistent result. I would discourage setting minimum lags i.e.

min_instr_dependent_vars and min_instr_predet_vars as the most recent observations are most highly correlated with the instrumented variable. Throwing them out should worsen the relative efficiency of the estimate.

Considerable answered 23/12, 2021 at 17:31 Comment(11)

Thank you. I will try. Given the pvargmm form I used above, what figures do you suggest to input for those four categories you provided? – Reynalda 25/12, 2021 at 15:34

@Eric, as I explained I don't think you should set minimum lags. The optimal instrument set is also unknowable. So the maximum of lag number is up to you. You should use Ahmad Zaenal's version with the matrix multiplication in C++. I have only 3.7 GB RAM. I tried with max lag = 3 for both predetermined and endogenous, adapting Roger-123's code to 20 countries ad 103 observations each and the size of the vector that could not be allocated was only 1.5 Gb so I think this should work for you. – Considerable 25/12, 2021 at 16:6

Without the minimum lag set, I get the following error: Error in solve(t(sum_S_ZX_a %x% Diagonal(nof_dependent_vars)) %*% (Lambda_inv_vec) %*% : Lapack dgecon(): system computationally singular, reciprocal condition number = 0 – Reynalda 25/12, 2021 at 16:24

@Reynalda Are you trying this with your data or with Roger-123's? If its with his data then it is not surprising. Independent random variables are not correlated and cannot be instruments for each other – Considerable 25/12, 2021 at 17:3

Yes I am trying with my own data. The snippet of my own data is as above, both in dput and first multiple rows. – Reynalda 25/12, 2021 at 17:4

@Reynalda X12 and X13 are practically identical. At least in the sample you provide. This may be causing the problem. I.e. X is almost singular – Considerable 25/12, 2021 at 17:28

Sorry the question had a typo on that part in the formula. I have edited now. So the exogenous variable should read as 'exog_vars = c("x14")'. Even with this case, I see the same error. – Reynalda 26/12, 2021 at 5:45

Also, I wonder how I can provide an impulse function analysis over this and visually show the impulse impact over this Panel VAR please. – Reynalda 26/12, 2021 at 10:36

I have found a function called "girf" within the "panelvar" package which could do the impulse function for pvargmm. However, the error message I get is "Error in 1:(nrow(Phi) * lags) : argument of length 0". I wonder how I can fix this error please. I use the following which works for me: lag=2 and max_instr_dependent_vars=3, max_instr_predet_vars = 3, min_instr_dependent_vars = 1L, min_instr_predet_vars = 1L. – Reynalda 27/12, 2021 at 16:19

Using my pvargmm function itself, I also see these warnings as well: "In pvargmm(dependent_vars = c("x13","x2","x3","x4","x5","x6"), :The matrix D_e is singular, therefore the general inverse is used”. – Reynalda 27/12, 2021 at 16:20

Hello???? Are you there???? – Reynalda 31/12, 2021 at 14:2

The problem happened because there is a lot of huge matrix multiplication with this operation %*%. I have faced a similar problem when creating an R package. Thus, I apply a similar approach to solve this problem by changing that operation with the Rcpp code in some part of panelvar package.

The modified code was uploaded to my GitHub repository here ==> https://github.com/zaenalium/panelvar.

To use that modified package please run this code:

devtools::install_github('zaenalium/panelvar')

That code has been tested and compared to the original version, the result is identical, also it reduces a lot of memory consumption.

Let me know if there is still any problem. Thanks.

Note: if you are using Windows OS, please install Rtools first.

Guan answered 14/12, 2021 at 3:5 Comment(3)

Thank you. The technical side error seems not to appear now. However, the main error about the size still persists as "Error: cannot allocate vector of size 9.5 Gb". – Reynalda 14/12, 2021 at 13:14

Well, regarding that problem I think there's a lot of work to do to optimize that package. I just fixed a few parts of it. If you are serious want to use that package we can collaborate and invest our time creating a new version of panelvar with more efficient code. – Guan 14/12, 2021 at 16:15

I think waiting for the new package would be better. – Reynalda 19/12, 2021 at 15:57

Not an answer, but this might help someone else answer this. I coded this to re-create a data.frame of the size @Eric is working with.

#create example dataset
#OP said data was 20 x 2060
#Creating sample with 5 countries and 412 dates 
#(not sure of original number of distinct dates and countries, so just picked some numbers)

Country <- rep(1:5, 412)
Date <-   as.integer(0:2059/5)

xdata_matrix<-matrix(data = runif(2060*18, min = -100, max = 100),
                    nrow = 2060,
                    ncol = 18)
colnames(xdata_matrix)<-  paste0("x", 1:18)   
data1<- data.frame(Country, Date, xdata_matrix)
rm(Country, Date, xdata_matrix)

Parament answered 14/12, 2021 at 0:24 Comment(0)