Get Tweepy search results as JSON

Question

I would like to have the search result from Twitter using Tweepy as JSON. I saw here that I should add a class to Tweepy code to make this feature works.

But when I look at Tweepy code, this is what I get :

class JSONParser(Parser):

    payload_format = 'json'

    def __init__(self):
        self.json_lib = import_simplejson()

    def parse(self, method, payload):
        try:
            json = self.json_lib.loads(payload)
        except Exception, e:
            raise TweepError('Failed to parse JSON payload: %s' % e)

        needsCursors = method.parameters.has_key('cursor')
        if needsCursors and isinstance(json, dict) and 'previous_cursor' in json and 'next_cursor' in json:
            cursors = json['previous_cursor'], json['next_cursor']
            return json, cursors
        else:
            return json

    def parse_error(self, payload):
        error = self.json_lib.loads(payload)
        if error.has_key('error'):
            return error['error']
        else:
            return error['errors']

So I am not obliged to hack its code since the fucntion already exists.

This is how my code looks:

from tweepy.parsers import JSONParser
for tweet in tweepy.Cursor(api.search,
                       q=hashtag,
                       include_entities=True,
                       rpp=100,
                       parser=tweepy.parsers.JSONParser()
                       ).items(limit):

This is the error I get :

   print (json.dumps(tweet))
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <tweepy.models.Status object at 0xb6df2fcc> is not JSON serializable

What should I undrestand from this error ? How can I fix it ?

Teryn

A better (and simpler) approach can be found in this answer to a similar question – Plateau

Answer 1

If you use Cursor like this

import json
api = tweepy.API(auth)
max_tweets=100
query='Ipython'
searched_tweets = [status._json for status in tweepy.Cursor(api.search,  q=query).items(max_tweets)]
json_strings = [json.dumps(json_obj) for json_obj in searched_tweets]  

searched_tweets is list of JSON objects, while json_strings is a list of JSON strings

Nildanile

status._json is actually a dict – Ossa

status._json is a private member and thus should not be used. – Milamilady

In Tweepy 4.5 I get AttributeError: 'API' object has no attribute 'search' but I do see e.g. api.search_30_day() – Dosia

Answer 2

If you can work without Cursor, you could use JSONParser. But if you can handle paging you can do it like this:

>>> api = tweepy.API(auth, parser=tweepy.parsers.JSONParser())

Make sure you change rpp to count because rpp is obsolete in Twitter Search API

>>> results = api.search(q="IPython", count=100)

You will get the results in their raw format. Meaning you will get a dict with two keys

>>> results.keys()
[u'search_metadata', u'statuses']

You can get your search results from "statuses" value.

>>> results["statuses"]
[{u'contributors': None,
  u'coordinates': None,
  u'created_at': u'Wed Oct 15 03:36:08 +0000 2014',
  ....

Duke

Answer 3

As of Tweepy 4.0 (Twitter API 2), you can specify the Result type when initiating the client

client = tweepy.Client(bearer_token, return_type=dict)
response = client.search_recent_tweets(search_text)
isinstance(response, dict) # True

Beth