R DBI ODBC error: nanodbc/nanodbc.cpp:3110: 07009: [Microsoft][ODBC Driver 13 for SQL Server]Invalid Descriptor Index

Question

I continue to read the DBI/ODBC is faster than RODBC, so I tried as follows:

require(DBI);require(odbc)
con <- DBI::dbConnect(odbc::odbc(), dsn = 'SQLSERVER1', database = 'AcumaticaDB')

I can make a successful connection to the DSN, but the following query:

rs <- dbGetQuery(con, "SELECT * FROM inventoryitem")
dbFetch(rs)

gives me the following error:

Error in result_fetch(res@ptr, n, ...) : nanodbc/nanodbc.cpp:3110: 07009: [Microsoft][ODBC Driver 13 for SQL Server]Invalid Descriptor Index

What am I doing wrong ? Please, no RODBC solutions. Thanks!

Disorientate

Leaving out dbFetch(), does rs come through as a dataframe? – Macswan

From the source and man page, dbGetQuery() ... calls 'dbSendQuery()', then 'dbFetch()', ensuring that the result is always free-d by 'dbClearResult()'. – Distributive

rs <- dbSendQuery(con, "SELECT * FROM InventoryItem") : this works and creates rs as class <odbcresult> (it's not a dataframe) dbFetch(rs) : gives me the error: "Error in result_fetch(res@ptr, n, ...) : nanodbc/nanodbc.cpp:3110: 07009: [Microsoft][ODBC Driver 13 for SQL Server]Invalid Descriptor Index" – Disorientate

I have the same problem as you have and came to the conclusion that it has to be a bug. I wound up returning to RODBC instead which while a bit slower atleast works. – Electrograph

This is a known issue, see here. – Chlorous

Answer 1

I appreciate that this question was asked a long time ago, but I've managed to find a workaround. The answers above got me most of the way there. The problem I had was with columns of type nvarchar that had a CHARACTER_MAXIMUM_LENGTH in the schema table of -1, which I understand means they are the maximum length possible.

My solution was to lookup the relevant table in the INFORMATION_SCHEMA.COLUMNS table and then rearrange my fields appropriately:

require(DBI);require(odbc)
library(tidyverse)
con <- DBI::dbConnect(odbc::odbc(), dsn = 'SQLSERVER1', database = 'AcumaticaDB')

column.types <- dbGetQuery(con, "SELECT COLUMN_NAME, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='inventoryitem'")

ct <- column.types %>%
  mutate(cml = case_when(
    is.na(CHARACTER_MAXIMUM_LENGTH) ~ 10,
    CHARACTER_MAXIMUM_LENGTH == -1 ~ 100000,
    TRUE ~ as.double(CHARACTER_MAXIMUM_LENGTH)
    )
  ) %>%
  arrange(cml) %>%
  pull(COLUMN_NAME)

fields <- paste(ct, collapse=", ")
query <- paste("SELECT", fields, "FROM inventoryitems")

tbl(con, sql(query)) %>% head(5)

Petes

This worked beautifully. You sir are a god among mere mortals. Thank you! – Brozak

Answer 2

I have also been struggling with this issue for several months. However, I have come across a solution that may help you as well.

In a nutshell, the issue occurs when certain text columns do not appear after integer/numeric columns. When the columns are not aligned properly in the query, an error of invalid index is thrown and your connection may freeze. The issue then is, how do I know what to put at the end of my query?

To determine this, one could typically examine a column using class() or typeof(). To examine such information from the database, you can use a query such as:

dbColumnInfo(dbSendQuery(con, "SELECT * from schema.table")) # You may not require the schema part...

This will return a table with a type field for every column in the data-set of interest. You can then use this table as an index to sort the select() statement. My particular difficulty is that the type field in the table was all numbers! However, I noticed that every column with a negative number, when placed at the end of the select statement, fixed my query and I could pull the whole table just fine. For example, my full solution:

# Create my index of column types (ref to the current order)
index <- dbColumnInfo(dbSendQuery(con, "SELECT * from schema.table"))
index$type <- as.integer(index$type) # B/c they are + and - numbers!

# Create the ref to the table
mySQLTbl <- tbl(con, in_schema("schema", "tablename"))

# Use the select statement to put all the + numbered columns first!
mySQLTbl %>%
  select(c(which(index$type>=0),
                 which(index$type<0)))

As for reason for why this occurs, I am not sure and I do not have the data access privileges to dig much deeper in my use-case

Boff

Thanks for this workaround. I've read that we have to downgrade to 1.1.6 by devtools::install_github("r-dbi/[email protected]"). Here is the issue on github : github/r-dbi/odbc #309: Error "Invalid descriptor index" when collecting a table with a blob column in the middle. This downgrading to 1.1.6 works. – Capers

Answer 3

There is a workaround:

Reorder your SELECT statements such that longer datatypes (typically strings) are last.

If you have a complex query that is generated by dbplyr itself, then get the SQL query directly via show_query(). Copy-paste and modify the first SELECT statement such that long datatypes are last in the list. It should then work.

EDIT: in many cases it is possible to reorder the fields by adding this to the query:

%>% select(var1, var2, textvar1, textvar2)

Electromechanical

So, @mzuba, how (generically) do we know that the underlying DBMS believes a field to be "large"? The documentation does not indicate a fixed cutoff, so it is not only specific to SQL Server, but also perhaps instance-specific (meaning what is "large" on one server may not be large on another). – Distributive

I dont't know if we can find out which fields are 'large'. This is a workaround and not a solution. – Electromechanical

Answer 4

rs <- dbGetQuery(con, "SELECT * FROM inventoryitem")
dbFetch(rs)

If inventoryitem table contains mix of long data/variable-length columns (eg. VARBINARY, VARCHAR) and columns of simple types (eg. INT), you can not query them in arbitrary order via ODBC.

Applications should make sure to place long data columns at the end of the select list.

Long data is retrieved from database using ODBC API call SQLGetData and it has to be retrieved after the other data in the row has been fetched.

These are known and documented ODBC restrictions

To retrieve long data from a column, an application first calls SQLFetchScroll or SQLFetch to move to a row and fetch the data for bound columns. The application then calls SQLGetData.

See https://learn.microsoft.com/en-us/sql/odbc/reference/develop-app/getting-long-data

Chamonix

Does this infect non-Windows ODBC as well? Seems to me it's an abject breakage of the "S" in SQL. – Distributive

Okay, that's a myth (to which I succumbed), the "S" is for structured not standard, but that doesn't change (in my mind) my reservation about needing to know that much detail about a table's storage/structure. – Distributive

Answer 5

In case someone else comes across this SO question....

I also was having a problem with the using the SELECT * FROM table statement with a setup like this:

library(odbc)
library(DBI)

con <- DBI::dbConnect(odbc::odbc(),
                      Driver   = "SQL Server",
                      Server   = "CorporateServer",
                      Database = "DB_of_Interest",
                      Port     = 1433)

sql <- "SELECT TOP (10) *
  FROM [DB_of_Interest].[dbo].[tb_of_interest]"

dbGetQuery(con, sql)

Which generated this error:

Error in result_fetch(res@ptr, n) : 
  nanodbc/nanodbc.cpp:3069: 07009: [Microsoft][ODBC SQL Server Driver]Invalid Descriptor Index 
Warning message:
In dbClearResult(rs) : Result already cleared

My Solution

After looking at RStudio -- Connect to a Database, I looked to see what other drivers might be available to me:

sort(unique(odbcListDrivers()[[1]]))

 [1] "Amazon Redshift (x64)"                                 
 [2] "Hortonworks Hive ODBC Driver"                          
 [3] "Hortonworks Spark ODBC Driver"                         
 [4] "Microsoft Access Driver (*.mdb, *.accdb)"              
 [5] "Microsoft Access Text Driver (*.txt, *.csv)"           
 [6] "Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)"
 [7] "MySQL ODBC 5.3 ANSI Driver"                            
 [8] "MySQL ODBC 5.3 Unicode Driver"                         
 [9] "ODBC Driver 17 for SQL Server"                         
[10] "PostgreSQL ANSI(x64)"                                  
[11] "PostgreSQL Unicode(x64)"                               
[12] "Simba ODBC Driver for Google BigQuery"                 
[13] "SQL Server"                                            
[14] "SQL Server Native Client 11.0"      

And #9 caught my eye: ODBC Driver 17 for SQL Server. I went into the Windows Control Panel, and set up a new ODBC Data Source using that particular driver (in my case, I named it "Buyer Central 2" and I have Integrated Windows authentication, so no username/password), and then revised my R code to be:

con2 <- DBI::dbConnect(odbc::odbc(),
                       dsn = "Buyer Central 2")

dbGetQuery(con2, sql) 

Now the sql statement with the glob (*) works for me.

Alternative that doesn't work (for me)

I thought and tried to use the other ODBC driver without the formal setup in the data source administrator:

con3 <- DBI::dbConnect(odbc::odbc(),
                      Driver   = "ODBC Driver 17 for SQL Server",
                      Server   = "CorporateServer",
                      Database = "DB_of_Interest",
                      Port     = 1433)

But that line fails for me with the following:

Error: nanodbc/nanodbc.cpp:1021: 01S00: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user ''.  [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute 

I searched for a reason why it works when set up in admin tools, but not as above, and found comments related to SQL Server 2014 not working with this driver due to SSL issues, but I checked my version and I'm using SQL Server 2016... so not that. Curious why it doesn't work this way, but the above solution does work.

Weidar

Answer 6

I certainly encountered this problem recently. Here is my solution. Basically you have to reorder columns based on the column information fetched from the database first. Columns could mix with positive and negative types. So sorting them with positive first, then negative will do the trick.

It works perfectly with my data when having "Invalid Descriptor Index" issue. Please let me know whether it works for you too.

sqlFetchData <- function(connection, database, schema, table, nobs = 'All') {

  #'wrap function to fetch data from SQL Server
  #
  #@ connection: an established odbc connection
  #@ database: database name
  #@ schema: a schema under the main database
  #@ table: the name of the data table to be fetched. 
  #@ nobs: number of observation to be fetched. Either 'All' or an integer number. 
  #        The default value is 'All'. It also supports the input of 'all', 'ALL' and
  #        etc. . 

  if (is.character(nobs)) {
    if (toupper(nobs) == 'ALL') {
      obs_text <- 'select'
    } else {
      stop("nobs could either be 'ALL' or a scalar integer number")
    }
  } else {
    if (is.integer(nobs) && length(nobs) == 1) {
      obs_text <- paste('select top ', nobs, sep = '')
    } else {
      stop("nobs could either be 'ALL' or a scalar integer number")
    }
  }

  initial_sql <- paste("select * from ", database, '.', schema, ".", table, 
                       sep = '')
  dbquery <- dbSendQuery(connection, initial_sql)
  cols <- dbColumnInfo(dbquery) 
  dbClearResult(dbquery)

  #' sort the rows by query type due to error message:
  #' Invalid Descriptor Index 

  colInfo <- cols
  colInfo$type <- as.integer(colInfo$type)
  cols_neg <- colInfo[which(colInfo$type < 0), ]
  cols_neg <- cols_neg[order(cols_neg[, 2]), ]
  cols_pos <- colInfo[which(colInfo$type >= 0), ]
  cols_pos <- cols_pos[order(cols_pos[, 2]), ]
  cols <- rbind(cols_pos, cols_neg)

  add_comma <- "c(cols$name[1], paste(',', cols$name[-1L], sep = ''))"

  sql1 <- paste(c(obs_text, eval(parse(text = add_comma))),
                collapse = ' ', sep = '')
  data_sql <- paste(sql1, ' from ', database, '.', schema, '.', table, 
                    sep = '')

  dataFetch <- dbGetQuery(connection, data_sql)[, colInfo$name]
  return(dataFetch)
}

According

Answer 7

ODBC/DBI convert character variable data type in the database into 'ntext' while making the connection. So, you need to convert your character variables (say x) in the SQL string in R as CONVERT(varchar(100),x). Then dbGetQuery function should work.

Phenylketonuria

Answer 8

I recently faced the same problem. The short and the easiest solution is to find the column info first and place the column at the end with max values(suppose varchar(max) and or varbinary(max)). Try the example below.

With Error

library(DBI)
library(ODBC)
myquery<- dbGetQuery(con,"SELECT * FROM MYTABLE")

Error in Result_fetch....:  Invalid Descriptor Index

Solution

dbcolumnInfo(dbSendWuery(con,"SELECT * FROM MYTABLE")

As you can see the datatype for DateTimeSampled is varchar(max). Place this data column at the end while retrieving the columns of MYTABLE using following query.

myquery<- dbGetQuery(con,"SELECT [PROCHI],[hb_extract],
[QuantityValue],[QuantityUnit],[Interpretation],
[LabNumber],[LocalClinicalCodeValue],[DateTimeSampled]
FROM MYTABLE")

Enjoy SQL with R

Gemology

Answer 9

I got this error as a result of trying to load in a timestamp variable. Try removing any timestamp variables from your query.

Try the below or similar. Let me know what works and I'll update my post.

require(DBI);require(odbc)
con <- DBI::dbConnect(odbc::odbc(), dsn = 'SQLSERVER1', database = 'AcumaticaDB')

column.types = DBI::dbGetQuery( 
    con, 
    'SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = "inventoryitem"' 
))

sql = paste(c(
        'select ', 
        paste(column.types$COLUMN_NAME[column.types$DATA_TYPE != 'timestamp'], collapse = ', '), 
        ' from inventoryitem'
     ),
    collapse = ''
)

dbFetch(dbGetQuery(con, sql))

Thermograph

Answer 10

This error can also occur if you have a field of type geometry. You should SELECT AS and use a spatial method to convert to WKT before pulling into a data.table. See below, where you know that Shape is a field of type geometry. By using the spatial method .STAsText() it converts it to Well-Known Text (WKT) format, which R can work with more easily.

library(sf)
library(tidyverse)

query <- "
SELECT company, revenue, address, Shape.STAsText() AS ShapeWKT
FROM businesses_table
"

results <- dbFetch(dbSendQuery(con, query)) %>%
  filter(!is.na(ShapeWKT)) %>%
  mutate(Shape = st_as_sfc(ShapeWKT)) %>%
  select(-ShapeWKT) %>%
  st_as_sf()

I built a more complex function that allows you to specify a db connection, table name, a field exception (defaults to return all fields EXCEPT those in exception field), a WHERE clause to filter your query, and specify a spatial field name, since it varies from table to table (i.e., could be Shape could be Geo, etc.).

spatial_query <- function(con, tablename, 
                          exception = "", 
                          wherefilter = "", 
                          spatialfieldname = "Shape",
                          spatial = TRUE){
  ## Get last element (table name) from explicit table name
  tmp <- strsplit(tablename, "[.]")[[1]]
  tablename_short <- tmp[length(tmp)]
  
  fields <- dbListFields(con, tablename_short)
  
  if(spatial){
    fields_to_grab <- fields[!(fields %in% c(exception, spatialfieldname))]
    ## TODO adjust query to be responsive to Shape
    qstring <- paste0('SELECT ', paste(fields_to_grab, collapse = ", "), ', ', spatialfieldname, '.STAsText() AS ShapeWKT FROM ', tablename, ' ', wherefilter)
    cat(paste("Sending following SQL statement:", qstring, "\n", sep = "\n"))
    ret <- dbFetch(dbSendQuery(con, qstring)) %>%
      filter(!is.na(ShapeWKT)) %>%
      mutate(Shape = st_as_sfc(ShapeWKT)) %>%
      select(-ShapeWKT) %>%
      st_as_sf()
  }
  
  else{
    fields_to_grab <- fields[!(fields %in% exception)]
    qstring <- paste0('SELECT ', paste(fields_to_grab, collapse = ", "),' FROM ', tablename, ' ', wherefilter)
    cat(paste("Sending following SQL statement:", qstring, "\n", sep = "\n"))
    ret <- dbFetch(dbSendQuery(con, qstring))
  }
  
  return(ret)
}

Messeigneurs

Answer 11

Maybe I missed someone else adding this, but I had this exact same error, read through this string and tried several elaborate things like reordering columns, and didn't know how to solve it for several months. Then, I found out that I just had to upgrade to the most recent SQL Server Driver for my Windows 10 and change the driver in my connection string, and the problem was solved.

Viveca

Answer 12

Today I came across the exact same problem but for me it turned out to be the types of my table that were causing it.

I had an id INT and 3 other columnheaders with VARCHAR(MAX) and one more with VARCHAR(50).

For some reason it didn't like it that one was VARCHAR(50) so I changed it from 50 to MAX and that fixed it for me. So maybe you also had one or more set to MAX instead of something else.

Shanghai