Multiple Linear Regression in Power BI

Question

Suppose I have a set of returns and I want to compute its beta values versus different market indices. Let's use the following set of data in a table named Returns for the sake of having a concrete example:

  Date       Equity  Duration  Credit  Manager
-----------------------------------------------
01/31/2017   2.907%   0.226%   1.240%   1.78%
02/28/2017   2.513%   0.493%   1.120%   3.88%
03/31/2017   1.346%  -0.046%  -0.250%   0.13%
04/30/2017   1.612%   0.695%   0.620%   1.04%
05/31/2017   2.209%   0.653%   0.480%   1.40%
06/30/2017   0.796%  -0.162%   0.350%   0.63%
07/31/2017   2.733%   0.167%   0.830%   2.06%
08/31/2017   0.401%   1.083%  -0.670%   0.29%
09/30/2017   1.880%  -0.857%   1.430%   2.04%
10/31/2017   2.151%  -0.121%   0.510%   2.33%
11/30/2017   2.020%  -0.137%  -0.020%   3.06%
12/31/2017   1.454%   0.309%   0.230%   1.28%

Now in Excel, I can just use the LINEST function to get the beta values:

= LINEST(Returns[Manager], Returns[[Equity]:[Credit]], TRUE, TRUE)

It spits out an array that looks like this:

0.077250253 -0.184974002  0.961578127 -0.001063971
0.707796954  0.60202895   0.540811546  0.008257129
0.50202386   0.009166729  #N/A         #N/A
2.688342242  8            #N/A         #N/A
0.000677695  0.000672231  #N/A         #N/A

The betas are in the top row and using them gives me the following linear estimate:

Manager = 0.962 * Equity - 0.185 * Duration + 0.077 * Credit - 0.001

The question is how can I get these values in Power BI using DAX (preferably without having to write a custom R script)?

---

For simple linear regression against one column, I can go back to the mathematical definition and write a least squares implementation similar to the one given in this post.

However, when more columns become involved (I need to be able to do up to 12 columns, but not always the same number), this gets messy really quickly and I'm hoping there's a better way.

Thermidor

Answer 1

The essence:

DAX is not the way to go. Use Home > Edit Queries and then Transform > Run R Script. Insert the following R snippet to run a regression analysis using all available variables in a table:

model <- lm(Manager ~ . , dataset)
df<- data.frame(coef(model))
names(df)[names(df)=="coef.model."] <- "coefficients"
df['variables'] <- row.names(df)

Edit Manager to any of the other available variable names to change the dependent variable.

---

The details:

Good question! Why Microsoft has not introduced more flexible solutions is beyond my understanding. But at the time being, you won't be able to find very good approaches without using R in Power BI.

My suggested approach will therefore ignore your request regarding:

The question is how can I get these values in Power BI using DAX (preferably without having to write a custom R script)?

My answer will however meet your requirements regarding:

A good answer should generalize to more than 3 columns (probably by working on an unpivoted data table with the indices as values rather than column headers).

Here we go:

---

I'm on a system using comma as a decimal separator, so I'm going to be using the following as the data source (If you copy the numbers directly into Power BI, the column separation will not be maintained. If you first paste it into Excel, copy it again and THEN paste it into Power BI the columns will be fine):

Date    Equity  Duration    Credit  Manager
31.01.2017  2,907   0,226   1,24    1,78
28.02.2017  2,513   0,493   1,12    3,88
31.03.2017  1,346   -0,046  -0,25   0,13
30.04.2017  1,612   0,695   0,62    1,04
31.05.2017  2,209   0,653   0,48    1,4
30.06.2017  0,796   -0,162  0,35    0,63
31.07.2017  2,733   0,167   0,83    2,06
31.08.2017  0,401   1,083   -0,67   0,29
30.09.2017  1,88    -0,857  1,43    2,04
31.10.2017  2,151   -0,121  0,51    2,33
30.11.2017  2,02    -0,137  -0,02   3,06
31.12.2017  1,454   0,309   0,23    1,28

Starting from scratch in Power BI (for reproducibility purposes) I'm inserting the data using Enter Data:

Now, go to Edit Queries > Edit Queries and check that you have this:

In order to maintain flexibility with regards to the number of columns to include in your analysis, I find it is best to remove the Date Column. This will not have an impact on your regression results. Simply right-click the Date column and select Remove:

Notice that this will add a new step under Query Settings > Applied Steps>:

And this is where you are going to be able to edit the few lines of R code we're going to use. Now, go to Transform > Run R Script to open this window:

Notice the line # 'dataset' holds the input data for this script. Thankfully, your question is only about ONE input table, so things aren't going to get too complicated (for multiple input tables check out this post). The dataset variable is a variable of the form data.frame in R and is a good (the only..) starting point for further analysis.

Insert the following script:

model <- lm(Manager ~ . , dataset)
df<- data.frame(coef(model))
names(df)[names(df)=="coef.model."] <- "coefficients"
df['variables'] <- row.names(df)

Click OK, and if all goes well you should end up with this:

Click Table, and you'll get this:

Under Applied Steps you'll se that a Run R Script step has been inserted. Click the star (gear ?) on the right to edit it, or click on df to format the output table.

This is it! For the Edit Queries part at least.

Click Home > Close & Apply to get back to Power BI Report section and verfiy that you have a new table under Visualizations > Fields:

Insert a Table or Matrix and activate Coefficients and Variables to get this:

I hope this is what you were looking for!

---

Now for some details about the R script:

As long as it's possible, I would avoid using numerous different R libraries. This way you'll reduce the risk of dependency issues.

The function lm() handles the regression analysis. The key to obtain the required flexibilty with regards to the number of explanatory variables lies in the Manager ~ . , dataset part. This simply says to run a regression analysis on the Manager variable in the dataframe dataset, and use all remaining columns ~ . as explanatory variables. The coef(model) part extracts the coefficient values from the estimated model. The result is a dataframe with the variable names as row names. The last line simply adds these names to the dataframe itself.

Magical

This looks pretty good. Ideally, the R script could be responsive to slicers and filters on a report page (since I can't reasonably pre-compute all the possible combinations I want betas for), but I don't think it's possible to use R as part of a measure at this time (and it's beyond the scope of my question). – Thermidor

Couldn't agree more on how these things should be in Power BI. But to my knowledge, we're just not there yet. Regarding their pace on adding new functionality, it might not last long until we can do what you're describing though. But right now, the one thing Power BI excels at, is visualizing results from analysis made elsewhere. When it comes to pre-computing betas you would like to see results for, I would suggest that you take a look at Python and the statsmodels package. You'll get up to speed really quick. I think R and Python is much easier to deal with than DAX for example. – Magical

So if you're able to handle DAX in Power BI, Python should be easy. Google Anaconda and start from there. Spyder or Visual Studio should do the trick as a good starter IDE. Then you could have a look at something I posted recently about running regressions on subsets of a table here: #48624571 The only thing you need is Python and a textfile. Then you can run any regression you want, and even store the results in another text file. – Magical

And even visualize the results in Power BI if that is still your favorite toolkit. By now I've made it sound more complicated than it is, I guess... But you should really have a look if you're into these things. – Magical

Did you have any luck with implementing this suggestion? – Magical

Answer 2

As there is no equivalent or handy replacement for LINEST function in Power BI (I'm sure you've done enough research before posting the question), any attempts would mean rewriting the whole function in Power Query / M, which is already not that "simple" for the case of simple linear regression, not to mention multiple variables.

Rather than (re)inventing the wheel, it's inevitably much easier (one-liner code..) to do it with R script in Power BI.

It's not a bad option given that I have no prior R experience. After a few searches and trial-and-error, I'm able to come up with this:

# 'dataset' holds the input data for this script
# install.packages("broom") # uncomment to install if package does not exist
library(broom)

model <- lm(Manager ~ Equity + Duration + Credit, dataset)
model <- tidy(model)

lm is the built-in linear model function from R, and the tidy function comes with the broom package, which tidies up the output and output a data frame for Power BI.

With the columns term and estimate, this should be sufficient to calculate the estimate you want.

The M Query for your reference:

let
    Source = Csv.Document(File.Contents("returns.csv"),[Delimiter=",", Columns=5, Encoding=1252, QuoteStyle=QuoteStyle.None]),
    #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
    #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Date", type text}, {"Equity", Percentage.Type}, {"Duration", Percentage.Type}, {"Credit", Percentage.Type}, {"Manager", Percentage.Type}}),
    #"Run R Script" = R.Execute("# 'dataset' holds the input data for this script#(lf)# install.packages(""broom"")#(lf)library(broom)#(lf)#(lf)model <- lm(Manager ~ Equity + Duration + Credit, dataset)#(lf)model <- tidy(model)",[dataset=#"Changed Type"]),
    #"""model""" = #"Run R Script"{[Name="model"]}[Value]
in
    #"""model"""

Kynthia

Answer 3

I've figured out how to do this for three variables specifically but this approach doesn't scale up or down to more or fewer variables at all.

Regression = 
VAR ShortNames =
    SELECTCOLUMNS (
        Returns,
        "A", [Equity],
        "D", [Duration],
        "C", [Credit],
        "Y", [Manager]
    )
VAR n = COUNTROWS ( ShortNames )

VAR A = SUMX ( ShortNames, [A] )
VAR D = SUMX ( ShortNames, [D] )
VAR C = SUMX ( ShortNames, [C] )
VAR Y = SUMX ( ShortNames, [Y] )

VAR AA = SUMX ( ShortNames, [A] * [A] ) - A * A / n
VAR DD = SUMX ( ShortNames, [D] * [D] ) - D * D / n
VAR CC = SUMX ( ShortNames, [C] * [C] ) - C * C / n

VAR AD = SUMX ( ShortNames, [A] * [D] ) - A * D / n
VAR AC = SUMX ( ShortNames, [A] * [C] ) - A * C / n
VAR DC = SUMX ( ShortNames, [D] * [C] ) - D * C / n

VAR AY = SUMX ( ShortNames, [A] * [Y] ) - A * Y / n
VAR DY = SUMX ( ShortNames, [D] * [Y] ) - D * Y / n
VAR CY = SUMX ( ShortNames, [C] * [Y] ) - C * Y / n

VAR BetaA =
    DIVIDE (
        AY*DC*DC - AD*CY*DC - AY*CC*DD + AC*CY*DD + AD*CC*DY - AC*DC*DY,
        AD*CC*AD - AC*DC*AD - AD*AC*DC + AA*DC*DC + AC*AC*DD - AA*CC*DD
    )
VAR BetaD =
    DIVIDE (
        AY*CC*AD - AC*CY*AD - AY*AC*DC + AA*CY*DC + AC*AC*DY - AA*CC*DY,
        AD*CC*AD - AC*DC*AD - AD*AC*DC + AA*DC*DC + AC*AC*DD - AA*CC*DD
    )
VAR BetaC =
    DIVIDE (
      - AY*DC*AD + AD*CY*AD + AY*AC*DD - AA*CY*DD - AD*AC*DY + AA*DC*DY,
        AD*CC*AD - AC*DC*AD - AD*AC*DC + AA*DC*DC + AC*AC*DD - AA*CC*DD
    )
VAR Intercept =
    AVERAGEX ( ShortNames, [Y] )
        - AVERAGEX ( ShortNames, [A] ) * BetaA
        - AVERAGEX ( ShortNames, [D] ) * BetaD
        - AVERAGEX ( ShortNames, [C] ) * BetaC
RETURN
        { BetaA, BetaD, BetaC, Intercept }

This is a calculated table that returns the regression coefficients specified:

These numbers match the output from LINEST for the data provided.

Note: The LINEST values I quoted in the question are slightly different from theses as they were calculated from unrounded returns rather than the rounded returns provided in the question.

---

I referenced this document for the calculation set up and Mathematica to solve the system:

Thermidor