Drop data frame columns by name

Question

I have a number of columns that I would like to remove from a data frame. I know that we can delete them individually using something like:

df$x <- NULL

But I was hoping to do this with fewer commands.

Also, I know that I could drop columns using integer indexing like this:

df <- df[ -c(1, 3:6, 12) ]

But I am concerned that the relative position of my variables may change.

Given how powerful R is, I figured there might be a better way than dropping each column one by one.

Kearse

Can someone explain to me why R doesn't have something simple like df#drop(var_name), and instead, we need to do these complicated work-arounds? – Verleneverlie

@Verleneverlie The 'subset()' function in R is about as parsimonious as the 'drop()' function in Python, except you don't need to specify the axis argument... I agree that it's annoying that there can't be just one, ultimate, easy keyword/syntax implemented across the board for something so basic as dropping a column. – Dwaynedweck

Answer 1

There's also the subset command, useful if you know which columns you want:

df <- data.frame(a = 1:10, b = 2:11, c = 3:12)
df <- subset(df, select = c(a, c))

UPDATED after comment by @hadley: To drop columns a,c you could do:

df <- subset(df, select = -c(a, c))

Behah

I really wish the R subset function had an option like "allbut = FALSE", which "inverts" the selection when set to TRUE, i.e. retains all columns except those in the select list. – Behah

@prasad, see @joris answer below. A subset without any subset criteria is a bit of overkill. Try simply: df[c("a", "c")] – Adder

@JD I knew that, but I like the syntactic convenience of the subset command where you don't need to put quotes around the column names -- I guess I don't mind typing a few extra characters just to avoid quoting names :) – Behah

oh that's a good point. I hadn't thought about the quote issue. – Adder

Note that you shouldn't use subset inside other functions. – Pym

@mac #12850641 – Pym

subset(df, select = -c(b)) @spacetyper – Outlaw

I was using this, but I am getting an error now: Error: unexpected '=' in "DT <- subset(DT, select=" Did something change? – Unsteel

You need to pass an argument for subset. e.g.: df <- subset(df, subset = TRUE, select = -c(a, c)) – Vizor

Answer 2

You can use a simple list of names :

DF <- data.frame(
  x=1:10,
  y=10:1,
  z=rep(5,10),
  a=11:20
)
drops <- c("x","z")
DF[ , !(names(DF) %in% drops)]

Or, alternatively, you can make a list of those to keep and refer to them by name :

keeps <- c("y", "a")
DF[keeps]

EDIT : For those still not acquainted with the drop argument of the indexing function, if you want to keep one column as a data frame, you do:

keeps <- "y"
DF[ , keeps, drop = FALSE]

drop=TRUE (or not mentioning it) will drop unnecessary dimensions, and hence return a vector with the values of column y.

Naivete

the subset function works better as it won't convert a data frame with one column into a vector – Fury

@Fury check the argument drop=FALSE of the indexing function. – Naivete

Shouldn't that be DF[,keeps] instead of DF[keeps] ? – Menashem

@Menashem No. It can, but then you have to add drop=FALSE to keep R from converting your data frame to a vector if you only select a single column. Don't forget that data frames are lists, so list selection (one-dimensional like I did) works perfectly well and always returns a list. Or a data frame in this case, which is why I prefer to use it. – Naivete

@AjayOhri Yes, it would. Without a comma, you use the "list" way of selecting, which means that even when you extract a single column, you still get a data frame returned. If you use the "matrix" way, as you do, you should be aware that if you only select a single column, you get a vector instead of a data frame. To avoid that, you need to add drop=FALSE. As explained in my answer, and in the comment right above yours... – Naivete

Why not use DF[!(names(DF) %in% drops)]? By what has been said in these comments, it should be equivalent to using drop=FALSE, which is probably what we want anyway. – Outer

Answer 3

within(df, rm(x))

is probably easiest, or for multiple variables:

within(df, rm(x, y))

Or if you're dealing with data.tables (per How do you delete a column by name in data.table?):

dt[, x := NULL]   # Deletes column x by reference instantly.

dt[, !"x"]   # Selects all but x into a new data.table.

or for multiple variables

dt[, c("x","y") := NULL]

dt[, !c("x", "y")]

Mamoun

within(df, rm(x)) is by far the cleanest solution. Given that this is a possibility, every other answer seems unnecessarily complicated by an order of magnitude. – Badmouth

Note that within(df, rm(x)) will not work if there are duplicate columns named x in df. – Barre

@Barre to clarify, it removes neither but seems to change the data's values. One has bigger problems if this is the case, but here's an example: df <- data.frame(x = 1, y = 2); names(df) <- c("x", "x"); within(df, rm(x)) returns data.frame(x = 2, x = 2). – Mamoun

@MilesErickson Problem is that you rely on a function within() which is powerful but also uses NSE. The note on the help page states clearly that for programming sufficient care should be used. – Naivete

@MilesErickson How often would one encounter a dataframe with duplicate names in it? – Felucca

@Felucca df<-data.frame(x=1:3,y=4:6,x=7:9) is valid, but the last column becomes x.1. I've only seen duplicate names in lists. To get it in a data frame, you need to pass the check.names = FALSE argument. – Outer

Two other major benefits of within: You do not need to pass a drop=FALSE argument and rm will warn you if the column that you are attempting to delete is missing. [ is not so kind. – Outer

If column names are "weird", for example with spaces etc. you can also use within(DF,rm(list=c("x","z"))) – Inversely

Answer 4

You could use %in% like this:

df[, !(colnames(df) %in% c("x","bar","foo"))]

Cahier

Am I missing something, or is this effectively the same solution as the first part of Joris' answer? DF[ , !(names(DF) %in% drops)] – Beachcomber

@DanielFletcher: it's the same. Look at the timestamps on the answers. We answered at the same time... 5 years ago. :) – Cahier

Nutty. identical(post_time_1, post_time_2) [1] TRUE =D – Beachcomber

Why not drop the comma? I see no reason why df[!(colnames(df) %in% c("x","bar","foo"))] would not be equivalent. – Outer

why does it return FALSE/TRUE vector when I run on data.table data frame. Why? – Timely

@J.Mini Kind of safety. x[names(x)] works on any object that has subsetting and defined names. x[, colnames(x)] works only on objects that have two dimensions and defined column names, which is typically just data.frame and matrix. If the object doesn't have two dimensions, it explicitly throws error. – Isadoraisadore

Answer 5

list(NULL) also works:

dat <- mtcars
colnames(dat)
# [1] "mpg"  "cyl"  "disp" "hp"   "drat" "wt"   "qsec" "vs"   "am"   "gear"
# [11] "carb"
dat[,c("mpg","cyl","wt")] <- list(NULL)
colnames(dat)
# [1] "disp" "hp"   "drat" "qsec" "vs"   "am"   "gear" "carb"

Cephalad

Brilliant! This extends the NULL assignment to a single column in a natural way, and (seemingly) avoids copying (although I don't know what happens under the hood so it may be no more efficient in memory usage ... but seems to me clearly more efficient syntactically.) – Bandaid

You do not need list(NULL), NULL is sufficient. e.g: dat[,4]=NULL – Sulphurous

OP's question was how to delete multiple columns. dat[,4:5] <- NULL won't work. That is where list(NULL) comes in. It works for 1 or more columns. – Cephalad

This also doesn't work when trying to remove a duplicated column name. – Barre

@Barre Works fine for me. Either give a label if you want to remove the first of the columns with the same name or give indices for each column you want to remove. If you have an example where it doesn't work I'd be interested to see it. Perhaps post it as a new question? – Cephalad

This syntax will also work for data.table as well. dat[,c("mpg","cyl","wt")] <- NULL – Mistaken

Answer 6

There is a potentially more powerful strategy based on the fact that grep() will return a numeric vector. If you have a long list of variables as I do in one of my dataset, some variables that end in ".A" and others that end in ".B" and you only want the ones that end in ".A" (along with all the variables that don't match either pattern, do this:

dfrm2 <- dfrm[ , -grep("\\.B$", names(dfrm)) ]

For the case at hand, using Joris Meys example, it might not be as compact, but it would be:

DF <- DF[, -grep( paste("^",drops,"$", sep="", collapse="|"), names(DF) )]

Elianore

If we define drops in the first place as paste0("^", drop_cols, "$"), this becomes much nicer (read: more compact) with sapply: DF[ , -sapply(drops, grep, names(DF))] – Barre

Answer 7

If you want remove the columns by reference and avoid the internal copying associated with data.frames then you can use the data.table package and the function :=

You can pass a character vector names to the left hand side of the := operator, and NULL as the RHS.

library(data.table)

df <- data.frame(a=1:10, b=1:10, c=1:10, d=1:10)
DT <- data.table(df)
# or more simply  DT <- data.table(a=1:10, b=1:10, c=1:10, d=1:10) #

DT[, c('a','b') := NULL]

If you want to predefine the names as as character vector outside the call to [, wrap the name of the object in () or {} to force the LHS to be evaluated in the calling scope not as a name within the scope of DT.

del <- c('a','b')
DT <- data.table(a=1:10, b=1:10, c=1:10, d=1:10)
DT[, (del) := NULL]
DT <-  <- data.table(a=1:10, b=1:10, c=1:10, d=1:10)
DT[, {del} := NULL]
# force or `c` would also work.   

You can also use set, which avoids the overhead of [.data.table, and also works for data.frames!

df <- data.frame(a=1:10, b=1:10, c=1:10, d=1:10)
DT <- data.table(df)

# drop `a` from df (no copying involved)

set(df, j = 'a', value = NULL)
# drop `b` from DT (no copying involved)
set(DT, j = 'b', value = NULL)

Shortcut

For some reason when using datatables, object size does not change in Rstudio's environmental windows. With dataframes, it does so immediately. – Branca

Answer 8

Another dplyr answer. Use select(-column).

If your variables have some common naming structure, you might try starts_with(). For example

library(dplyr)
df <- data.frame(var1 = rnorm(5), var2 = rnorm(5), var3 = rnorm (5), 
                 var4 = rnorm(5), char1 = rnorm(5), char2 = rnorm(5))
df
#        var2      char1        var4       var3       char2       var1
#1 -0.4629512 -0.3595079 -0.04763169  0.6398194  0.70996579 0.75879754
#2  0.5489027  0.1572841 -1.65313658 -1.3228020 -1.42785427 0.31168919
#3 -0.1707694 -0.9036500  0.47583030 -0.6636173  0.02116066 0.03983268

df1 <- df %>% select(-starts_with("char"))

df1
#        var2        var4       var3       var1
#1 -0.4629512 -0.04763169  0.6398194 0.75879754
#2  0.5489027 -1.65313658 -1.3228020 0.31168919
#3 -0.1707694  0.47583030 -0.6636173 0.03983268

If you want to drop a sequence of variables in the data frame, you can use :. For example if you wanted to drop var2, var3, and all variables in between, you'd just be left with var1:

df2 <- df1 %>% select(-c(var2:var3) )  
df2
#        var1
#1 0.75879754
#2 0.31168919
#3 0.03983268

Demolition

Not to forget about all the other opportunities that come with select(), such as contains() or matches(), which also accepts regex. – Pickel

Answer 9

Dplyr Solution

I doubt this will get much attention down here, but if you have a list of columns that you want to remove, and you want to do it in a dplyr chain I use one_of() in the select clause:

Here is a simple, reproducable example:

undesired <- c('mpg', 'cyl', 'hp')

mtcars <- mtcars %>%
  select(-one_of(undesired))

Documentation can be found by running ?one_of or here:

http://genomicsclass.github.io/book/pages/dplyr_tutorial.html

Hyacinthhyacintha

Answer 10

Another possibility:

df <- df[, setdiff(names(df), c("a", "c"))]

or

df <- df[, grep('^(a|c)$', names(df), invert=TRUE)]

Uranium

Too bad that this is not upvoted more because use of setdiff is the optimal especially in the case of a very large number of columns. – Argentiferous

Another angle on this: df <- df[ , -which(grepl('a|c', names(df)))] – Panthea

Answer 11

Out of interest, this flags up one of R's weird multiple syntax inconsistencies. For example given a two-column data frame:

df <- data.frame(x=1, y=2)

This gives a data frame

subset(df, select=-y)

but this gives a vector

df[,-2]

This is all explained in ?[ but it's not exactly expected behaviour. Well at least not to me...

Jerol

Answer 12

DF <- data.frame(
  x=1:10,
  y=10:1,
  z=rep(5,10),
  a=11:20
)
DF

Output:

    x  y z  a
1   1 10 5 11
2   2  9 5 12
3   3  8 5 13
4   4  7 5 14
5   5  6 5 15
6   6  5 5 16
7   7  4 5 17
8   8  3 5 18
9   9  2 5 19
10 10  1 5 20

---

DF[c("a","x")] <- list(NULL)

Output:

        y z
    1  10 5
    2   9 5
    3   8 5
    4   7 5
    5   6 5
    6   5 5
    7   4 5
    8   3 5    
    9   2 5
    10  1 5

Diamante

Answer 13

Here is a dplyr way to go about it:

#df[ -c(1,3:6, 12) ]  # original
df.cut <- df %>% select(-col.to.drop.1, -col.to.drop.2, ..., -col.to.drop.6)  # with dplyr::select()

I like this because it's intuitive to read & understand without annotation and robust to columns changing position within the data frame. It also follows the vectorized idiom using - to remove elements.

Jinn

Adding to this that (1) user want replace original df (2) magrittr has %<>% operator to replace input object it could be simplified to df %<>% select(-col.to.drop.1, -col.to.drop.2, ..., -col.to.drop.6) – Eldon

If you have a long list of columns to drop, with dplyr, it might be easier to group them and put only one minus: df.cut <- df %>% select(-c(col.to.drop.1, col.to.drop.2, ..., col.to.drop.n)) – Drive

Answer 14

I keep thinking there must be a better idiom, but for subtraction of columns by name, I tend to do the following:

df <- data.frame(a=1:10, b=1:10, c=1:10, d=1:10)

# return everything except a and c
df <- df[,-match(c("a","c"),names(df))]
df

Adder

Not a good idea to negate match - df[,-match(c("e","f"),names(df))] – Hound

.@JDLong - What if I want to drop column where the column name starts with -? – Setter

Answer 15

There's a function called dropNamed() in Bernd Bischl's BBmisc package that does exactly this.

BBmisc::dropNamed(df, "x")

The advantage is that it avoids repeating the data frame argument and thus is suitable for piping in magrittr (just like the dplyr approaches):

df %>% BBmisc::dropNamed("x")

Rosaleerosaleen

Answer 16

Beyond select(-one_of(drop_col_names)) demonstrated in earlier answers, there are a couple other dplyr options for dropping columns using select() that do not involve defining all the specific column names (using the dplyr starwars sample data for some variety in column names):

library(dplyr)
starwars %>% 
  select(-(name:mass)) %>%        # the range of columns from 'name' to 'mass'
  select(-contains('color')) %>%  # any column name that contains 'color'
  select(-starts_with('bi')) %>%  # any column name that starts with 'bi'
  select(-ends_with('er')) %>%    # any column name that ends with 'er'
  select(-matches('^f.+s$')) %>%  # any column name matching the regex pattern
  select_if(~!is.list(.)) %>%     # not by column name but by data type
  head(2)

# A tibble: 2 x 2
homeworld species
  <chr>     <chr>  
1 Tatooine  Human  
2 Tatooine  Droid 

If you need to drop a column that may or may not exist in the data frame, here's a slight twist using select_if() that unlike using one_of() will not throw an Unknown columns: warning if the column name does not exist. In this example 'bad_column' is not a column in the data frame:

starwars %>% 
  select_if(!names(.) %in% c('height', 'mass', 'bad_column'))

Succession

Answer 17

Another solution if you don't want to use @hadley's above: If "COLUMN_NAME" is the name of the column you want to drop:

df[,-which(names(df) == "COLUMN_NAME")]

Gingivitis

(1) Problem is to drop multiple columns at once. (2) It won't work if COLUMN_NAME is not in df (check yourself: df<-data.frame(a=1,b=2)). (3) df[,names(df) != "COLUMN_NAME"] is simpler and don't suffer from (2) – Eldon

Can you give some more information about this answer? – Hootman

Answer 18

Provide the data frame and a string of comma separated names to remove:

remove_features <- function(df, features) {
  rem_vec <- unlist(strsplit(features, ', '))
  res <- df[,!(names(df) %in% rem_vec)]
  return(res)
}

Usage:

remove_features(iris, "Sepal.Length, Petal.Width")

Airsickness

Answer 19

There are a lot of ways you can do...

Option-1:

df[ , -which(names(df) %in% c("name1","name2"))]

Option-2:

df[!names(df) %in% c("name1", "name2")]

Option-3:

subset(df, select=-c(name1,name2))

Buiron

Answer 20

Drop and delete columns by columns name in data frame.

A <- df[ , c("Name","Name1","Name2","Name3")]

Kenric

Answer 21

Find the index of the columns you want to drop using which. Give these indexes a negative sign (*-1). Then subset on those values, which will remove them from the dataframe. This is an example.

DF <- data.frame(one=c('a','b'), two=c('c', 'd'), three=c('e', 'f'), four=c('g', 'h'))
DF
#  one two three four
#1   a   d     f    i
#2   b   e     g    j

DF[which(names(DF) %in% c('two','three')) *-1]
#  one four
#1   a    g
#2   b    h

Yahiya

Answer 22

If you have a large data.frame and are low on memory use [ . . . . or rm and within to remove columns of a data.frame, as subset is currently (R 3.6.2) using more memory - beside the hint of the manual to use subset interactively.

getData <- function() {
  n <- 1e7
  set.seed(7)
  data.frame(a = runif(n), b = runif(n), c = runif(n), d = runif(n))
}

DF <- getData()
tt <- sum(.Internal(gc(FALSE, TRUE, TRUE))[13:14])
DF <- DF[setdiff(names(DF), c("a", "c"))] ##
#DF <- DF[!(names(DF) %in% c("a", "c"))] #Alternative
#DF <- DF[-match(c("a","c"),names(DF))]  #Alternative
sum(.Internal(gc(FALSE, FALSE, TRUE))[13:14]) - tt
#0.1 MB are used

DF <- getData()
tt <- sum(.Internal(gc(FALSE, TRUE, TRUE))[13:14])
DF <- subset(DF, select = -c(a, c)) ##
sum(.Internal(gc(FALSE, FALSE, TRUE))[13:14]) - tt
#357 MB are used

DF <- getData()
tt <- sum(.Internal(gc(FALSE, TRUE, TRUE))[13:14])
DF <- within(DF, rm(a, c)) ##
sum(.Internal(gc(FALSE, FALSE, TRUE))[13:14]) - tt
#0.1 MB are used

DF <- getData()
tt <- sum(.Internal(gc(FALSE, TRUE, TRUE))[13:14])
DF[c("a", "c")]  <- NULL ##
sum(.Internal(gc(FALSE, FALSE, TRUE))[13:14]) - tt
#0.1 MB are used

Thicket

Answer 23

Another option using the function fselect from the collapse package. Here is a reproducible example:

DF <- data.frame(
  x=1:10,
  y=10:1,
  z=rep(5,10),
  a=11:20
)

library(collapse)
fselect(DF, -z)
#>     x  y  a
#> 1   1 10 11
#> 2   2  9 12
#> 3   3  8 13
#> 4   4  7 14
#> 5   5  6 15
#> 6   6  5 16
#> 7   7  4 17
#> 8   8  3 18
#> 9   9  2 19
#> 10 10  1 20

^{Created on 2022-08-26 with reprex v2.0.2}

Excoriation

Answer 24

Another data.table option which hasn't been posted yet is using the special verb .SD, which stands for subset of data. Together with the .SDcols argument you can select/drop columns by name or index.

require(data.table)
# data
dt = data.table(
  A = LETTERS[1:5],
  B = 1:5,
  C = rep(TRUE, 5)
)
# delete B
dt[ , .SD, .SDcols =! 'B' ]
# delete all matches (i.e. all columns)
cols = grep('[A-Z]+', names(dt), value = TRUE)
dt[ , .SD, .SDcols =! cols ]

A summary of all the options for such a task in data.table can be found here

Childe

Answer 25

df <- data.frame(
+   a=1:5,
+   b=6:10,
+   c=rep(22,5),
+   d=round(runif(5)*100, 2),
+   e=round(runif(5)*100, 2),
+   f=round(runif(5)*100, 2),
+   g=round(runif(5)*100, 2),
+   h=round(runif(5)*100, 2)
+ )
> df
  a  b  c     d     e     f     g     h
1 1  6 22 76.31 39.96 66.62 72.75 73.14
2 2  7 22 53.41 94.85 96.02 97.31 85.32
3 3  8 22 98.29 38.95 12.61 29.67 88.45
4 4  9 22 20.04 53.53 83.07 77.50 94.99
5 5 10 22  5.67  0.42 15.07 59.75 31.21

> # remove cols: d g h
> newDf <- df[, c(1:3, 5), drop=TRUE]
> newDf
  a  b  c     e
1 1  6 22 39.96
2 2  7 22 94.85
3 3  8 22 38.95
4 4  9 22 53.53
5 5 10 22  0.42

Carbolize