which is the best criteria for choosing between ets() and auto.arima() functions in R?

Question

I am using ets() and auto.arima() functions from forecast package to predict future values in R. Which criteria should be used to choose the best model between these two?

Following is the accuracy output from ets (data.ets) and auto.arima (data.ar).

> accuracy(data.ets)
   ME      RMSE       MAE       MPE      MAPE      MASE 
0.6995941 4.1325246 3.2634246 0.5402465 2.7777897 0.5573740 

> accuracy(data.ar)
    ME       RMSE        MAE        MPE       MAPE       MASE 
-0.8215465  4.3640818  3.1070931 -0.7404200  2.5783128  0.5306735 

and the AIC of each model are as follows

> ETSfit$aic
[1] 613.8103
> ARIMAfit$aic
[1] 422.5597

Following is the fitted model of both ets and auto.arima

> ETSfit 
ETS(A,N,A) 

Call:
 ets(y = data.ts) 

Smoothing parameters:
alpha = 0.5449 
gamma = 1e-04 

Initial states:
l = 95.8994 
s=6.3817 -3.1792 6.8525 3.218 -3.4445 -1.2408
       -4.5852 0.4434 1.7133 0.8123 -1.28 -5.6914

sigma:  4.1325

 AIC     AICc      BIC 
613.8103 620.1740 647.3326

> ARIMAfit
Series: data.ts 
ARIMA(1,1,1)(0,1,1)[12]                    

Coefficients:
     ar1      ma1     sma1
  0.3808  -0.7757  -0.7276
s.e.  0.1679   0.1104   0.2675

sigma^2 estimated as 22.68:  log likelihood=-207.28
AIC=422.56   AICc=423.19   BIC=431.44

Kindly help.

Vinic

Answer 1

You are showing in-sample accuracy measures which are hard to compare without knowing how many parameters are in each model. Also, the AIC values are not comparable between these model classes.

The simplest approach is to use a test set that is not used for model selection or estimation, and then compare accuracy of the forecasts on the test set.

A more sophisticated version of that is to use time series cross-validation, as described at http://otexts.com/fpp/2/5/.

Nedneda

Thanks for your prompt reply. I went through your website, which is very good. Now, I have added the fitted model for each ets and auto.arima, if it helps – Vinic

(+1) for linking an awesome resource – Radiotelegraph

Answer 2

You might consider using a simple average of both, but you should base this decision on out of sample performance.

I just read an article today by an author of the forecast package. He ran his models for about 3,000 series in a forecasting competition and found he got the best results when taking a simple average of both ets() and auto.arima().

article

After I dug up the link, I realized the author of that article has already answered your question above!

Sulk