Reshaping data.frame from wide to long format

Question

I have some trouble to convert my data.frame from a wide table to a long table. At the moment it looks like this:

Code Country        1950    1951    1952    1953    1954
AFG  Afghanistan    20,249  21,352  22,532  23,557  24,555
ALB  Albania        8,097   8,986   10,058  11,123  12,246

Now I would like to transform this data.frame into a long data.frame. Something like this:

Code Country        Year    Value
AFG  Afghanistan    1950    20,249
AFG  Afghanistan    1951    21,352
AFG  Afghanistan    1952    22,532
AFG  Afghanistan    1953    23,557
AFG  Afghanistan    1954    24,555
ALB  Albania        1950    8,097
ALB  Albania        1951    8,986
ALB  Albania        1952    10,058
ALB  Albania        1953    11,123
ALB  Albania        1954    12,246

I have looked at and already tried using the melt() and the reshape() functions as some people were suggesting in similar questions. However, so far I only get messy results.

If it is possible I would like to do it with the reshape() function since it looks a little bit nicer to handle.

Spinoza

Don't know if that was the problem, but the functions in the reshape package are melt and cast (and recast.) – Lillalillard

And the reshape package has been superseded by reshape2. – Arni

And now reshape2 has been superseded by tidyr. – Norvil

And now tidyr's gather and spread have been replaced by pivot_* functions. – Statued

Answer 1

reshape() takes a while to get used to, just as melt/cast. Here is a solution with reshape, assuming your data frame is called d:

reshape(d, 
        direction = "long",
        varying = list(names(d)[3:7]),
        v.names = "Value",
        idvar = c("Code", "Country"),
        timevar = "Year",
        times = 1950:1954)

Spherulite

Answer 2

Two alternative solutions:

1) With data.table:

You can use the melt function:

library(data.table)
long <- melt(setDT(wide), id.vars = c("Code","Country"), variable.name = "year")

which gives:

> long
    Code     Country year  value
 1:  AFG Afghanistan 1950 20,249
 2:  ALB     Albania 1950  8,097
 3:  AFG Afghanistan 1951 21,352
 4:  ALB     Albania 1951  8,986
 5:  AFG Afghanistan 1952 22,532
 6:  ALB     Albania 1952 10,058
 7:  AFG Afghanistan 1953 23,557
 8:  ALB     Albania 1953 11,123
 9:  AFG Afghanistan 1954 24,555
10:  ALB     Albania 1954 12,246

Some alternative notations:

melt(setDT(wide), id.vars = 1:2, variable.name = "year")
melt(setDT(wide), measure.vars = 3:7, variable.name = "year")
melt(setDT(wide), measure.vars = as.character(1950:1954), variable.name = "year")

2) With tidyr:

Use pivot_longer():

library(tidyr)

long <- wide %>% 
  pivot_longer(
    cols = `1950`:`1954`, 
    names_to = "year",
    values_to = "value"
)

Note:

• names_to and values_to default to "name" and "value", respectively, so you could write this extra-succinctly as wide %>% pivot_longer(`1950`:`1954`).
• The cols argument uses the highly flexible tidyselect DSL, so you can select the same columns using a negative selection (!c(Code, Country)), a selection helper(starts_with("19"); matches("^\\d{4}$")), numeric indices (3:7), and more.
• tidyr::pivot_longer() is the successor to tidyr::gather() and reshape2::melt(), which are no longer under development.

Transforming values

Another problem with the data is that the values will be read by R as character-values (as a result of the , in the numbers). You can repair with gsub and as.numeric, either before reshaping:

long$value <- as.numeric(gsub(",", "", long$value))

Or during reshaping, with data.table or tidyr:

# data.table
long <- melt(setDT(wide),
             id.vars = c("Code","Country"),
             variable.name = "year")[, value := as.numeric(gsub(",", "", value))]

# tidyr
long <- wide %>%
  pivot_longer(
    cols = `1950`:`1954`, 
    names_to = "year",
    values_to = "value",
    values_transform = ~ as.numeric(gsub(",", "", .x))
  )

---

Data:

wide <- read.table(text="Code Country        1950    1951    1952    1953    1954
AFG  Afghanistan    20,249  21,352  22,532  23,557  24,555
ALB  Albania        8,097   8,986   10,058  11,123  12,246", header=TRUE, check.names=FALSE)

Embolden

great answer, just one more tiny reminder : do not put any variables other than id andtime in your data frame, melt could not tell what you want to do in this case. – Drunken

@JasonGoal Could you elaborate on that? As I'm interpreting you comment, it shouldn't be a problem. Just specify both the id.vars and the measure.vars. – Embolden

,then that's good for me, don't know id.vars and the measure.vars can be specified in the first alternative,sorry for the mess, its my fault. – Drunken

Sorry to necro this post - could someone explain to me why 3 works? I've tested it and it works, but I don't understand what dplyr is doing when it sees -c(var1, var2)... – Hanfurd

@ReputableMisnomer When tidyr sees -c(var1, var2) it omits these variables when transforming the data from wide to long format. – Embolden

Ah wow - what a neat trick. Truly a time saver. Was so resistant to move to the tidyverse() yet it surprises me every day... – Hanfurd

According to the tidyverse blog gather is now retired and as been replaced by pivot_longer. They state: "New pivot_longer() and pivot_wider() provide modern alternatives to spread() and gather(). They have been carefully redesigned to be easier to learn and remember, and include many new features. spread() and gather() won’t go away, but they’ve been retired which means that they’re no longer under active development." – Botswana

Answer 3

With tidyr_1.0.0, another option is pivot_longer

library(tidyr)
pivot_longer(df1, -c(Code, Country), values_to = "Value", names_to = "Year")
# A tibble: 10 x 4
#   Code  Country     Year  Value 
#   <fct> <fct>       <chr> <fct> 
# 1 AFG   Afghanistan 1950  20,249
# 2 AFG   Afghanistan 1951  21,352
# 3 AFG   Afghanistan 1952  22,532
# 4 AFG   Afghanistan 1953  23,557
# 5 AFG   Afghanistan 1954  24,555
# 6 ALB   Albania     1950  8,097 
# 7 ALB   Albania     1951  8,986 
# 8 ALB   Albania     1952  10,058
# 9 ALB   Albania     1953  11,123
#10 ALB   Albania     1954  12,246

data

df1 <- structure(list(Code = structure(1:2, .Label = c("AFG", "ALB"), class = "factor"), 
    Country = structure(1:2, .Label = c("Afghanistan", "Albania"
    ), class = "factor"), `1950` = structure(1:2, .Label = c("20,249", 
    "8,097"), class = "factor"), `1951` = structure(1:2, .Label = c("21,352", 
    "8,986"), class = "factor"), `1952` = structure(2:1, .Label = c("10,058", 
    "22,532"), class = "factor"), `1953` = structure(2:1, .Label = c("11,123", 
    "23,557"), class = "factor"), `1954` = structure(2:1, .Label = c("12,246", 
    "24,555"), class = "factor")), class = "data.frame", row.names = c(NA, 
-2L))

Tetralogy

This needs more upvotes. According to the Tidyverse Blog gather is being retired and pivot_longer is now the correct way to accomplish this. – Botswana

@EvanRosica only until they decide to change the function again :p – Gerda

What does -c(...) do? It looks like deleting a column but it is not. Can you please explain the syntaxis? – Whitted

Answer 4

Using reshape package:

#data
x <- read.table(textConnection(
"Code Country        1950    1951    1952    1953    1954
AFG  Afghanistan    20,249  21,352  22,532  23,557  24,555
ALB  Albania        8,097   8,986   10,058  11,123  12,246"), header=TRUE)

library(reshape)

x2 <- melt(x, id = c("Code", "Country"), variable_name = "Year")
x2[,"Year"] <- as.numeric(gsub("X", "" , x2[,"Year"]))

Copy

Answer 5

Since this answer is tagged with r-faq, I felt it would be useful to share another alternative from base R: stack.

Note, however, that stack does not work with factors--it only works if is.vector is TRUE, and from the documentation for is.vector, we find that:

is.vector returns TRUE if x is a vector of the specified mode having no attributes other than names. It returns FALSE otherwise.

I'm using the sample data from @Jaap's answer, where the values in the year columns are factors.

Here's the stack approach:

cbind(wide[1:2], stack(lapply(wide[-c(1, 2)], as.character)))
##    Code     Country values  ind
## 1   AFG Afghanistan 20,249 1950
## 2   ALB     Albania  8,097 1950
## 3   AFG Afghanistan 21,352 1951
## 4   ALB     Albania  8,986 1951
## 5   AFG Afghanistan 22,532 1952
## 6   ALB     Albania 10,058 1952
## 7   AFG Afghanistan 23,557 1953
## 8   ALB     Albania 11,123 1953
## 9   AFG Afghanistan 24,555 1954
## 10  ALB     Albania 12,246 1954

Crummy

You've saved package developers everywhere 🥳 – Microscopy

Answer 6

Here is another example showing the use of gather from tidyr. You can select the columns to gather either by removing them individually (as I do here), or by including the years you want explicitly.

Note that, to handle the commas (and X's added if check.names = FALSE is not set), I am also using dplyr's mutate with parse_number from readr to convert the text values back to numbers. These are all part of the tidyverse and so can be loaded together with library(tidyverse)

wide %>%
  gather(Year, Value, -Code, -Country) %>%
  mutate(Year = parse_number(Year)
         , Value = parse_number(Value))

Returns:

   Code     Country Year Value
1   AFG Afghanistan 1950 20249
2   ALB     Albania 1950  8097
3   AFG Afghanistan 1951 21352
4   ALB     Albania 1951  8986
5   AFG Afghanistan 1952 22532
6   ALB     Albania 1952 10058
7   AFG Afghanistan 1953 23557
8   ALB     Albania 1953 11123
9   AFG Afghanistan 1954 24555
10  ALB     Albania 1954 12246

Crepe

Answer 7

Here's a sqldf solution:

sqldf("Select Code, Country, '1950' As Year, `1950` As Value From wide
        Union All
       Select Code, Country, '1951' As Year, `1951` As Value From wide
        Union All
       Select Code, Country, '1952' As Year, `1952` As Value From wide
        Union All
       Select Code, Country, '1953' As Year, `1953` As Value From wide
        Union All
       Select Code, Country, '1954' As Year, `1954` As Value From wide;")

To make the query without typing in everything, you can use the following:

^{Thanks to G. Grothendieck for implementing it.}

ValCol <- tail(names(wide), -2)

s <- sprintf("Select Code, Country, '%s' As Year, `%s` As Value from wide", ValCol, ValCol)
mquery <- paste(s, collapse = "\n Union All\n")

cat(mquery) #just to show the query
 #> Select Code, Country, '1950' As Year, `1950` As Value from wide
 #>  Union All
 #> Select Code, Country, '1951' As Year, `1951` As Value from wide
 #>  Union All
 #> Select Code, Country, '1952' As Year, `1952` As Value from wide
 #>  Union All
 #> Select Code, Country, '1953' As Year, `1953` As Value from wide
 #>  Union All
 #> Select Code, Country, '1954' As Year, `1954` As Value from wide

sqldf(mquery)

 #>    Code     Country Year  Value
 #> 1   AFG Afghanistan 1950 20,249
 #> 2   ALB     Albania 1950  8,097
 #> 3   AFG Afghanistan 1951 21,352
 #> 4   ALB     Albania 1951  8,986
 #> 5   AFG Afghanistan 1952 22,532
 #> 6   ALB     Albania 1952 10,058
 #> 7   AFG Afghanistan 1953 23,557
 #> 8   ALB     Albania 1953 11,123
 #> 9   AFG Afghanistan 1954 24,555
 #> 10  ALB     Albania 1954 12,246

Unfortunately, I don't think that PIVOT and UNPIVOT would work for R SQLite. If you want to write up your query in a more sophisticated manner, you can also take a look at these posts:

• Using sprintf writing up sql queries

• Pass variables to sqldf

Francinefrancis

Answer 8

You can also use the cdata package, which uses the concept of (transformation) control table:

# data
wide <- read.table(text="Code Country        1950    1951    1952    1953    1954
AFG  Afghanistan    20,249  21,352  22,532  23,557  24,555
ALB  Albania        8,097   8,986   10,058  11,123  12,246", header=TRUE, check.names=FALSE)

library(cdata)
# build control table
drec <- data.frame(
    Year=as.character(1950:1954),
    Value=as.character(1950:1954),
    stringsAsFactors=FALSE
)
drec <- cdata::rowrecs_to_blocks_spec(drec, recordKeys=c("Code", "Country"))

# apply control table
cdata::layout_by(drec, wide)

I am currently exploring that package and find it quite accessible. It is designed for much more complicated transformations and includes the backtransformation. There is a tutorial available.

Cotangent

Answer 9

Here's two options in base R (use x=unlist(df) instead of x=c(m) when the input is a dataframe and not a matrix):

> m=matrix(sample(1:100,6),3,dimnames=list(2021:2023,c("male","female")))
> m
     male female
2021   89     42
2022   39     96
2023   26     40
> cbind(expand.grid(dimnames(m)),x=c(m))
  Var1   Var2  x
1 2021   male 89
2 2022   male 39
3 2023   male 26
4 2021 female 42
5 2022 female 96
6 2023 female 40
> data.frame(row=rownames(m),col=colnames(m)[col(m)],x=c(m))
   row    col  x
1 2021   male 89
2 2022   male 39
3 2023   male 26
4 2021 female 42
5 2022 female 96
6 2023 female 40

A third option is to use as.table followed by as.data.frame, but it converts the row and column names to factors, and if your input is a dataframe then you have to convert it to a matrix first:

> as.data.frame(as.table(m))
  Var1   Var2 Freq
1 2021   male   89
2 2022   male   39
3 2023   male   26
4 2021 female   42
5 2022 female   96
6 2023 female   40
> as.data.frame(as.table(m))|>sapply(class)
     Var1      Var2      Freq
 "factor"  "factor" "integer"
> d=as.data.frame(m)
> as.data.frame(as.table(d))
Error in h(simpleError(msg, call)) :
  error in evaluating the argument 'x' in selecting a method for function 'as.data.frame': cannot coerce to a table
> as.data.frame(as.table(as.matrix(d)))
  Var1   Var2 Freq
1 2021   male   89
2 2022   male   39
3 2023   male   26
4 2021 female   42
5 2022 female   96
6 2023 female   40

A fourth option is to use stack, but it converts the rownames and column names to factors, and the column names get converted to an Rle factor when the input is a matrix (but not when the input is a dataframe):

> stack(m)
DataFrame with 6 rows and 3 columns
       row    col     value
  <factor>  <Rle> <integer>
1     2021   male        89
2     2022   male        39
3     2023   male        26
4     2021 female        42
5     2022 female        96
6     2023 female        40

When the input for stack is a dataframe, the rownames don't get included as a column so you have to cbind them:

> d=as.data.frame(m);cbind(row=rownames(d),stack(d))
   row values    ind
1 2021     89   male
2 2022     39   male
3 2023     26   male
4 2021     42 female
5 2022     96 female
6 2023     40 female

Withal