Twitter stream API gives JSONDecodeError("Expecting value", s, err.value) from None

Question

I am using the stream API of Twitter (through tweepy) to collect tweets matching certain criteria, but when I use json.loads() to parse the created jsonl file I get this following error:

File "twitter_time_series.py", line 19, in <module> 
    tweet = json.loads(line)

File "C:\Program Files\Anaconda3\lib\json\__init__.py", line 319, in loads 
    return _default_decoder.decode(s)

File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 339, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 357, in raw_decode 
    raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

I cannot determine why, as with other types of jsonl files (like when getting twitter timelines etc), this does not occur. Anyone that can help me? thanks!

I am using the basic for stream tweets:

auth = get_twitter_auth() 

twitter_stream = Stream(auth, CustomListener(query_fname)) 

twitter_stream.filter(track=['curiosity'], async=True)

and for loading json:

fname = sys.argv[1] 

with open(fname, "r") as f: 

    for line in f:
      tweet = json.loads(line)

I am using python 3.5 and tweepy version 3.3.0 and this is one line of the json file:

{"created_at":"Mon Dec 26 16:03:06 +0000 2016",

"id":813414846033170432,

"id_str":"813414846033170432",

"text":"Battle proper and at greater risk https:\/\/t.co\/W6U9Irgst9","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e",

"truncated":false,

"in_reply_to_status_id":null,

"in_reply_to_status_id_str":null,

"in_reply_to_user_id":null,

"in_reply_to_user_id_str":null,

"in_reply_to_screen_name":null,

"user":{"id":544570972,"id_str":"544570972","name":"Fay Moore","screen_name":"MooreFay","location":"Maryland","url":"http:\/\/faymoore.wordpress.com","description":"Author, writer for hire, crazy woman in a crazy world","protected":false,"verified":false,"followers_count":482,"friends_count":322,"listed_count":22,"favourites_count":92,
"statuses_count":17326,"created_at":"Tue Apr 03 20:30:46 +0000 2012","utc_offset":-18000,"time_zone":"Eastern Time (US & Canada)","geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/3252060596\/e06263f9a226ca0e3f83915812c331e6_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/3252060596\/e06263f9a226ca0e3f83915812c331e6_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/544570972\/1360842914","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},

"geo":null,

"coordinates":null,

"place":null,

"contributors":null,

"is_quote_status":false,

"retweet_count":0,

"favorite_count":0,

"entities":{"hashtags":[],"urls":[{"url":"https:\/\/t.co\/W6U9Irgst9","expanded_url":"http:\/\/a.msn.com\/01\/en-us\/BBxzSWF?ocid=st","display_url":"a.msn.com\/01\/en-us\/BBxzS\u2026","indices":[81,104]}],"user_mentions":[],"symbols"[]},

"favorited":false,

"retweeted":false,

"possibly_sensitive":false,

"filter_level":"low",

"lang":"en",

"timestamp_ms":"1482768186468"}

Pianette

We need the relevant code - a minimal reproducible example. – Entity

Also please add the tag for the programming language (python) and the relevant libraries (tweepy). – Entity

Answer 1

When writing to file, in the Listener class, you can try:

def on_data(self, data):
    with open('filename.json', 'a', newline='\n') as f:
        f.write(data)

I also faced this problem on Windows machines. That's because Windows uses CR LF line ending while Linux uses LF.

If you try to read a single tweet, I don't think it would give an error as you can see here:

json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)  

This is more likely a bug in the json python package rather than the TwitterAPI.

See: The modern way: use newline=''

More on line endings: On Wikipedia, as always.

Maximin

Answer 2

Inside the for loop, you can try:

  tweet = json.loads(line.decode())

or remove the for loop and try:

  tweets = json.load(f)

Fiend

Answer 3

I met the same problem just now.

I tried to use the json analyzer to test the json string and found there are some wrong formats.

Such as, False should be "False", None should be "None".

I mean first you should check whether the format of the json string is correct, and one method is to test it on a website that specifically converts json format.

Hope the answer is a little helpful for you.

Tolyl