Yahoo Finance V7 API now requiring cookies? (Python)

Question

url = 'https://query2.finance.yahoo.com/v7/finance/quote?symbols=TSLA&fields=regularMarketPreviousClose&region=US&lang=en-US'
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
}
data = requests.get(url,headers=headers)
prepost_data = data.json()

It seems recently, Yahoo Finance changed their V7 API to require cookies for every request. Running the code above, I get the Invalid Crumb error

{"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Crumb"}}}

This issue seems to also be known in this Github repo: https://github.com/joshuaulrich/quantmod/issues/382

They seem to have a patch that works: https://github.com/joshuaulrich/quantmod/pull/383/commits

But the code is all written in R... Anyone know how to translate this to Python?

Briannabrianne

Running your code I get normal Json response {'quoteResponse': {'result': [{'language': 'en-US', 'region': 'US', 'quoteType': 'EQUITY', ... – Oglesby

@AndrejKesely Yeah it seems they fixed it overnight. I guess my question now is should I be including cookies in my headers? If so, what is the best way to do that? – Briannabrianne

Have you found a fix for this? I use an application that updates stock prices using that API, and it's been broken for a few weeks now. – Haro

Answer 1

This worked for me.

1. Make HTTP GET call to URL https://fc.yahoo.com. Although this call result in a 404 error, we just need it to extract set-cookie from response headers which is then used in the subsequent calls
2. Now make an HTTP GET call to the URL https://query2.finance.yahoo.com/v1/test/getcrumb, by including the obtained cookie from the previous response headers. This call will retrieve the crumb value.
3. Replace [crumb-value] in the following URL and make a HTTP GET call with cookie https://query2.finance.yahoo.com/v7/finance/quote?symbols=TSLA&crumb=[crumb-value]
4. Cache the cookie value and crumb value to skip first two steps going forward

Construct

Thanks, I was able to do this as well. – Briannabrianne

set-cookie isn't in the response headers anymore for step #1. Can anyone else verify? – Latanya

@Latanya For me, the "set-cookie" is still present in the response headers. – Construct

Answer 2

Here's a sample Python code based on the answer from @mahindar to get the cookie and crumb:

import requests


def get_yahoo_cookie():
    cookie = None

    user_agent_key = "User-Agent"
    user_agent_value = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

    headers = {user_agent_key: user_agent_value}
    response = requests.get(
        "https://fc.yahoo.com", headers=headers, allow_redirects=True
    )

    if not response.cookies:
        raise Exception("Failed to obtain Yahoo auth cookie.")

    cookie = list(response.cookies)[0]

    return cookie


def get_yahoo_crumb(cookie):
    crumb = None

    user_agent_key = "User-Agent"
    user_agent_value = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

    headers = {user_agent_key: user_agent_value}

    crumb_response = requests.get(
        "https://query1.finance.yahoo.com/v1/test/getcrumb",
        headers=headers,
        cookies={cookie.name: cookie.value},
        allow_redirects=True,
    )
    crumb = crumb_response.text

    if crumb is None:
        raise Exception("Failed to retrieve Yahoo crumb.")

    return crumb


# Usage
cookie = get_yahoo_cookie()
crumb = get_yahoo_crumb(cookie)

Lambrequin

Answer 3

I did in php, the code is:

<?php

/* 1 - Get cookie */
//https://mcmap.net/q/1240218/-yahoo-finance-v7-api-now-requiring-cookies-python
$url_yahoo = "https://fc.yahoo.com";
$yahoo_headers = get_headers($url_yahoo, true);
//print_r($yahoo_headers);
$cookie_name = 'Set-Cookie';

/* 2 - Get crumb , setting cookie */
$url_yahoo2 = "https://query2.finance.yahoo.com/v1/test/getcrumb";
$c = curl_init($url_yahoo2);
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_COOKIE, $yahoo_headers[$cookie_name]);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$crumb = curl_exec($c);
curl_close($c);
//echo "<BR>Crumb:" . $crumb;

/* 3 - Get quotes with crumb, setting cookie. Using sample tickets*/
$tickets_list = "AAPL,TSLA";
$url_cotacao = "https://query2.finance.yahoo.com/v7/finance/quote?symbols=" . $tickets_list . "&crumb=" . $crumb;
$c = curl_init($url_cotacao);
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_COOKIE, $yahoo_headers[$cookie_name]);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$data_quote = curl_exec($c);
curl_close($c);


/* 4 - Get data from yahoo */
$resJson_decode = json_decode($data_quote, false);
if (!$resJson_decode->quoteResponse->result) {
    $resultado = "Ticket dont exists in yahoo!";
} else {
    foreach ($resJson_decode->quoteResponse->result as $ticket_result){
        echo "<BR>Ticket:" . $ticket_result->symbol;
        echo "<BR>Price:" . $ticket_result->regularMarketPrice;
    }
    
}

Sessoms

But the question is for Python. – Theologue

Answer 4

For completion let me add a JavaScript version. I was researching this problem quite a bit and I am astonished how simple the solution finally is. Alternatively, cookie and crumb can also be obtained and extracted from finance.yahoo.com

const API = 'https://query2.finance.yahoo.com'

async function getCredentials() {
  // get the A3 cookie
  const { headers } = await fetch('https://fc.yahoo.com')
  const cookie = headers.get('set-cookie')
  // now get the crumb
  const url = new URL('/v1/test/getcrumb', API)
  const request = new Request(url)
  request.headers.set('cookie', cookie)
  const response = await fetch(request)
  const crumb = await response.text()
  return { cookie, crumb } 
}

async function quote(symbols, cookie, crumb) {
  const url = new URL('v7/finance/quote', API)
  url.searchParams.set('symbols', symbols.join(','))
  url.searchParams.set('crumb', crumb)
  const request = new Request(url)
  request.headers.set('cookie', cookie)
  const response = await fetch(request)
  const {quoteResponse} = await response.json()
  return quoteResponse?.result || []
}

// main
const { cookie, crumb } = await getCredentials()
const quotes = await quote(['GOOG', 'TSLA'], cookie, crumb)
if (quotes.length) {
  for (const quote of quotes) {
    console.log(`${quote.symbol} price is ${quote.currency} ${quote.regularMarketPrice}`)
  }
}

Pallet

Could you please explain how OP would use your code in their Python script? – Invitatory

It's possible to call node from python, the code provided above is for Node.js. For use in Python just adapt the code. It turns out that Pyhton requests and responses have a cookies property which is convenient. Python version follows below: – Pallet

Thanks for writing an answer... and another answer. Usually it's best to answer in a way OP can use the code in the answer as a drop-in replacement. Explanations are a must for great answers. See How do I write a good answer? – Invitatory

Answer 5

import requests

apiBase = 'https://query2.finance.yahoo.com'
headers = { 
  "User-Agent": 
  "Mozilla/5.0 (Windows NT 6.1; Win64; x64)"
}

def getCredentials(cookieUrl='https://fc.yahoo.com', crumbUrl=apiBase+'/v1/test/getcrumb'):
  cookie = requests.get(cookieUrl).cookies
  crumb = requests.get(url=crumbUrl, cookies=cookie, headers=headers).text
  return {'cookie': cookie, 'crumb': crumb}

def quote(symbols, credentials):
  url = apiBase + '/v7/finance/quote'
  params = {'symbols': ','.join(symbols), 'crumb': credentials['crumb']}
  response = requests.get(url, params=params, cookies=credentials['cookie'], headers=headers)
  quotes = response.json()['quoteResponse']['result']
  return quotes

credentials = getCredentials()
quotes = quote(['GOOG', 'TSLA'], credentials)
if quotes:
  for quote in quotes:
    print(f"{quote['symbol']} price is {quote['currency']} {quote['regularMarketPrice']}")

# GOOG price is USD 121.08
# TSLA price is USD 256.24

I used https://www.codeconvert.ai/javascript-to-python-converter plus some modifications

Pallet

Answer 6

Based on the other answers (a big thank you to the contributors!) I was able to update my Excel VBA code which now works again (question is: how long...).

Here is the complete solution which also includes how to retrieve the cookie from the header as http.getResponseHeader "set-cookie" doesn't yield anything in VBA (question from @HBCondo). The cookie and crumb are requested only once and stored in global variables. No check for validity has been added so far although the cookie expires after a certain time. ParseJson is a function in the VBA-JSON project available on Github.

Public crumb As String
Public cookie As String

Public Sub YahooGetCrumb()
    Dim http As MSXML2.XMLHTTP60
    Dim strHeader As String
    Dim strFields() As String
    
    Set http = New MSXML2.XMLHTTP60
    
    http.Open "GET", "https://fc.yahoo.com"
    http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
    http.send
    
    strHeader = http.getAllResponseHeaders
    strFields = Split(strHeader, vbCrLf)
    cookie = Trim(Split(Split(strFields(5), ";")(0), ":")(1)) & "; " & Trim(Split(Split(strFields(6), ";")(0), ":")(1))

    http.Open "GET", "https://query2.finance.yahoo.com/v1/test/getcrumb"
    http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
    http.setRequestHeader "Cookie", cookie
    http.send
    
    crumb = http.responseText
    
End Sub

Public Function GetYahooData(sSymbol As String) As String
    Dim http As MSXML2.XMLHTTP60
    
    If crumb = "" Then
        YahooGetCrumb
    End If
    
    Set http = New MSXML2.XMLHTTP60
    
    http.Open "GET", "https://query2.finance.yahoo.com/v7/finance/quote?symbols=" & sSymbol & "&crumb=" & crumb, False
    http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
    http.send
    
    GetYahooData = http.responseText
    
End Function

Public Sub test()
    Dim JSON As Object

    responseText = GetYahooData("AAPL")
    
    Set JSON = ParseJson(responseText)
end Sub    

Cue

Answer 7

PowerShell Usage Example :

    # ► Create $session Object
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$Stock = "MSFT" 

    # ► Call 1st Url ( Gain some headers and cookies stuff ) - using try & catch to prevent 404 error ( as mentioned above )
try { Invoke-WebRequest -UseBasicParsing -Uri "https://fc.yahoo.com/" -SessionVariable session } catch {
  $null}        
        

    # ► Call 2nd Url ( Generate Crumb ) 
$crumb = Invoke-WebRequest -UseBasicParsing -Uri "https://query2.finance.yahoo.com/v1/test/getcrumb" -WebSession $session


    # ► Call 3rd Url ( Get Yahoo's Data ) 
$URL = $("https://query2.finance.yahoo.com/v10/finance/quoteSummary/" + $Stock + "?modules=price&crumb=" + $crumb)
$ResponseText = Invoke-WebRequest -UseBasicParsing -Uri $URL -WebSession $session

    
    # ► Print Result
$ResponseText = $ResponseText.ToString()
$ResponseText 

Fernandez

Answer 8

I did this for VBA macro:

Public Function GetStockData() As String
    Dim qurl As String
    Dim cookie As String
    Dim crumb As String
    Dim req As Object
    Dim cookieurl As String

    cookieurl = "https://fc.yahoo.com" 'This page needs to return a cookie, query1.finance.yahoo does not return cookie.
    
    Set req = CreateObject("WinHttp.WinHttpRequest.5.1")
    
    'get cookie
    With req
        .Open "GET", cookieurl, False
        .setRequestHeader "REFERER", cookieurl
        .setRequestHeader "User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
        .Send
        
        cookie = .getResponseHeader("Set-Cookie") 'gets cookie and saves it.
    End With
   
    'get crumb
    With req
        .Open "GET", "https://query2.finance.yahoo.com/v1/test/getcrumb", False 'gets crumb, which must be attached to all quote calls
        .setRequestHeader "REFERER", cookieurl
        .setRequestHeader "User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
        .setRequestHeader "Cookie", cookie 'applies cookie to header, currently working without this, but probably good practice to use it
        .Send
        
        crumb = .responseText 'saves crumb
    End With
    
    qurl = "http://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL"
  
    qurl = qurl & "&crumb=" & crumb 'add crumb info from GetCookie sub
    

    'get data
    With req
        .Open "GET", qurl, False
        'header is needed but not specific, could probbaly remove a couple lines without creating a problem
        .setRequestHeader "REFERER", cookieurl
        .setRequestHeader "User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
        .setRequestHeader "Cookie", cookie
        .Send
        GetStockData = .responseText 'get data and return
    End With
End Function

Lesalesak

Answer 9

Delphi code to parse a Yahoo price series v8:

test url: https://query2.finance.yahoo.com/v8/finance/chart/BRFS3.SA?period1=946857600&period2=1646265600&interval=1d&events=history&includeAdjustedClose=true

This server doesn't require cookie at this time.

If function returns true, series bars are returned in a variable fBars:TList < TSeriesBar >

        function ParseYahooJSON(const aText: String):boolean;  
        var
           JSonValue,aJV:TJSonValue;
           aJATime,aJAOpen,aJAClose,aJAHigh,aJALow,aJAVolume,aJAAdjClose:TJSonArray; //arrays
           aB:TSeriesBar;   // f.e. your own bar object
           i,CountItems:integer;
           aDt:TDatetime;

        begin
           Result := false;   // no parse
           JSonValue := TJSonObject.ParseJSONValue(aText);
           try               // any error here will throw an exception
             aJATime      := JsonValue.GetValue<TJSonArray>('chart.result[0].timestamp');
             if not Assigned(aJATime) then exit;
             CountItems   := aJATime.Count;                    // use time array count
             if CountItems=0 then exit;                        // no items ??
             //get access to JSon arrays
             aJAOpen      := JsonValue.GetValue<TJSonArray>('chart.result[0].indicators.quote[0].open');
             aJAClose     := JsonValue.GetValue<TJSonArray>('chart.result[0].indicators.quote[0].close');
             aJAHigh      := JsonValue.GetValue<TJSonArray>('chart.result[0].indicators.quote[0].high');
             aJALow       := JsonValue.GetValue<TJSonArray>('chart.result[0].indicators.quote[0].low');
             aJAVolume    := JsonValue.GetValue<TJSonArray>('chart.result[0].indicators.quote[0].volume');
             aJAAdjClose  := JsonValue.GetValue<TJSonArray>('chart.result[0].indicators.adjclose[0].adjclose');
           except
             exit;              //w/ false
           end;

           for i := 0 to CountItems-1 do
             begin
               aB := TSeriesBar.Create;  // create a custom bar object
               {$IFDEF AUTOREFCOUNT} aB.__ObjAddRef;  {$ENDIF AUTOREFCOUNT}
               try
                 aJV := aJATime.Items[i];                       //copy fields
                 aDt := UnixToDateTime(aJV.GetValue<integer>);
                 aB.Dt := aDt;
                 aJV := aJAOpen.Items[i];    aB.Open  := aJV.GetValue<Single>;
                 aJV := aJAClose.Items[i];   aB.Close := aJV.GetValue<Single>;
                 aJV := aJAHigh.Items[i];    aB.Max   := aJV.GetValue<Single>;
                 aJV := aJALow.Items[i];     aB.Min   := aJV.GetValue<Single>;
                 aJV := aJAVolume.Items[i];  aB.Vol   := aJV.GetValue<Single>;
                 //
                 aJV := aJAAdjClose.Items[i]; aB.AdjClose := aJV.GetValue<Single>;
               except
                 continue; //??  ignore error and go to next line
               end;

               fBars.Add(aB);   // if here, bar parsed ok... insert ..
             end;
           JsonValue.Free;

           Result := (fBars.Count>0);  // ok if at least 1 bar received ( clear fBars before calling this )
        end;

Flagon