Salesforce Data into Python (simple-Salesforce or salesforce-bulk)

Question

I'm trying to load Salesforce data into a Python dataframe, so we can do all of our manipulations there. simple_salesforce worked with the caveat that we reached the 2,000 limit:

from simple_salesforce import Salesforce as s
eatpies = sf.query('Select Id from Case')
attrs = ['Id']
records = eatpies['records']

data = {}

for rec in records:
    for k in attrs:
        data.setdefault(k, []).append(rec[k])

dframe = pd.DataFrame(data)

print(dframe)

Supposedly, salesforce-bulk (https://pypi.python.org/pypi/salesforce-bulk/1.0.7) is able to bypass this limit, but I can't get further than this:

job = bulk.create_query_job("Case", contentType='CSV')
batch = bulk.query('select Id, type from Case')

TypeError                                 Traceback (most recent call last)
<ipython-input-13-076e14bf245d> in <module>()
----> 1 batch = bulk.query('select Id, type from Case')

TypeError: query() missing 1 required positional argument: 'soql'

Please help, thanks! If the solution can be done in simple-Salesforce to over-come the Salesforce limit, that would be great, but I couldn't find any solutions via Google.

Sackville

Answer 1

Change the line

eatpies = sf.query('Select Id from Case')

to the following:

eatpies = sf.query_all('Select Id from Case')

The method query_all a convenience wrapper around query(...) and query_more(...).

From the docs:

If, due to an especially large result, Salesforce adds a nextRecordsUrl to your query result, such as "nextRecordsUrl" : "/services/data/v26.0/query/01gD0000002HU6KIAW-2000", you can pull the additional results with either the ID or the full URL (if using the full URL, you must pass 'True' as your second argument)

sf.query_more("01gD0000002HU6KIAW-2000")
sf.query_more("/services/data/v26.0/query/01gD0000002HU6KIAW-2000", True)

You also have access to Bulk API through simple-salesforce. For example, replace

eatpies = sf.query('Select Id from Case')
attrs = ['Id']
records = eatpies['records']

with the following:

eatpies = sf.bulk.Case.query('Select Id from Case')
attrs = ['Id']
records = eatpies

More information about using Bulk API: https://github.com/simple-salesforce/simple-salesforce#using-bulk

Polysynthetic

Answer 2

sf.query_all("select count(Id) from visitors where CreatedDate >= 2017-12-01T00:00:00.000+0000 and CreatedDate < 2019-01-01T00:00:00.000+0000", True)

Medorra