what is a mysql buffered cursor w.r.t python mysql connector
Asked Answered
A

1

19

Can someone please give an example to understand this?

After executing a query, a MySQLCursorBuffered cursor fetches the entire result set from the server and buffers the rows. For queries executed using a buffered cursor, row-fetching methods such as fetchone() return rows from the set of buffered rows. For nonbuffered cursors, rows are not fetched from the server until a row-fetching method is called. In this case, you must be sure to fetch all rows of the result set before executing any other statements on the same connection, or an InternalError (Unread result found) exception will be raised.

Thanks

Andra answered 11/10, 2017 at 7:13 Comment(0)
E
17

I can think of two ways these two types of Cursors are different.

The first way is that if you execute a query using a buffered cursor, you can get the number of rows returned by checking MySQLCursorBuffered.rowcount. However, the rowcount attribute of an unbuffered cursor returns -1 right after the execute method is called. This, basically, means that the entire result set has not yet been fetched from the server. Furthermore, the rowcount attribute of an unbuffered cursor increases as you fetch rows from it, while the rowcount attribute of a buffered cursor remains the same, as you fetch rows from it.

The following snippet code tries to illustrate the points made above:

import mysql.connector


conn = mysql.connector.connect(database='db',
                               user='username',
                               password='pass',
                               host='localhost',
                               port=3306)

buffered_cursor = conn.cursor(buffered=True)
unbuffered_cursor = conn.cursor(buffered=False)

create_query = """
drop table if exists people;
create table if not exists people (
    personid int(10) unsigned auto_increment,
    firstname varchar(255),
    lastname varchar(255),
    primary key (personid)
);
insert into people (firstname, lastname)
values ('Jon', 'Bon Jovi'),
('David', 'Bryan'),
('Tico', 'Torres'),
('Phil', 'Xenidis'),
('Hugh', 'McDonald')
"""

# Create and populate a table
results = buffered_cursor.execute(create_query, multi=True)
conn.commit()

buffered_cursor.execute("select * from people")
print("Row count from a buffer cursor:", buffered_cursor.rowcount)
unbuffered_cursor.execute("select * from people")
print("Row count from an unbuffered cursor:", unbuffered_cursor.rowcount)

print()
print("Fetching rows from a buffered cursor: ")

while True:
    try:
        row = next(buffered_cursor)
        print("Row:", row)
        print("Row count:", buffered_cursor.rowcount)
    except StopIteration:
        break

print()
print("Fetching rows from an unbuffered cursor: ")

while True:
    try:
        row = next(unbuffered_cursor)
        print("Row:", row)
        print("Row count:", unbuffered_cursor.rowcount)
    except StopIteration:
        break

The above snippet should return something like the following:

Row count from a buffered reader:  5
Row count from an unbuffered reader:  -1

Fetching rows from a buffered cursor:
Row: (1, 'Jon', 'Bon Jovi')
Row count: 5
Row: (2, 'David', 'Bryan')
Row count: 5
Row: (3, 'Tico', 'Torres')
Row count: 5
Row: (4, 'Phil', 'Xenidis')
Row count: 5
Row: (5, 'Hugh', 'McDonald')
Row: 5

Fetching rows from an unbuffered cursor:
Row: (1, 'Jon', 'Bon Jovi')
Row count: 1
Row: (2, 'David', 'Bryan')
Row count: 2
Row: (3, 'Tico', 'Torres')
Row count: 3
Row: (4, 'Phil', 'Xenidis')
Row count: 4
Row: (5, 'Hugh', 'McDonald')
Row count: 5

As you can see, the rowcount attribute for the unbuffered cursor starts at -1 and increases as we loop through the result it generates. This is not the case with the buffered cursor.

The second way to tell the difference is by paying attention to which of the two (under the same connection) executes first. If you start with executing an unbuffered cursor whose rows have not been fully fetched and then try to execute a query with the buffered cursor, an InternalError exception will be raised, and you will be asked to consume or ditch what is returned by the unbuffered cursor. Below is an illustration:

import mysql.connector


conn = mysql.connector.connect(database='db',
                               user='username',
                               password='pass',
                               host='localhost',
                               port=3306)

buffered_cursor = conn.cursor(buffered=True)
unbuffered_cursor = conn.cursor(buffered=False)

create_query = """
drop table if exists people;
create table if not exists people (
    personid int(10) unsigned auto_increment,
    firstname varchar(255),
    lastname varchar(255),
    primary key (personid)
);
insert into people (firstname, lastname)
values ('Jon', 'Bon Jovi'),
('David', 'Bryan'),
('Tico', 'Torres'),
('Phil', 'Xenidis'),
('Hugh', 'McDonald')
"""

# Create and populate a table
results = buffered_cursor.execute(create_query, multi=True)
conn.commit()

unbuffered_cursor.execute("select * from people")
unbuffered_cursor.fetchone()
buffered_cursor.execute("select * from people")

The snippet above will raise a InternalError exception with a message indicating that there is some unread result. What it is basically saying is that the result returned by the unbuffered cursor needs to be fully consumed before you can execute another query with any cursor under the same connection. If you change unbuffered_cursor.fetchone() with unbuffered_cursor.fetchall(), the error will disappear.

There are other less obvious differences, such as memory consumption. Buffered cursor will likely consume more memory since they may fetch the result set from the server and buffer the rows.

I hope this proves useful.

Enisle answered 27/11, 2017 at 1:24 Comment(2)
You allude to this, but I feel like one of the biggest concerns I have about buffered connections is the memory consumption, and so I think it might be helpful to know to what extent the data is buffered. From my experience a buffer does not imply that all data is stored in memory, but in this case, it kind of sounds like the buffer could contain all data - like it is a dynamically sized buffer.Gustavo
@Gustavo apologies about the confusion. I did not mean to insinuate that all the records are fetched and buffered. While that may still be the case (when the result set is small enough), it is safer/clearer to say that batches of the records are fetched and buffered.Enisle

© 2022 - 2024 — McMap. All rights reserved.