YQL query service replacement now that Yahoo shut it down

Question

So now that Yahoo shut down query.yahooapis.com as the following message indicates, does anyone know of a free replacement?

"Important EOL Notice: As of Thursday, Jan. 3, 2019, the YQL service at query.yahooapis.com will be retired. This will impact users of datatables.org as well as developers who creates features using this YQL service. To continue using our free Yahoo Weather APIs, use https://weather-ydn-yql.media.yahoo.com/forecastrss as your new API endpoint. Contact [email protected] for credentials to onboard to this free Yahoo Weather API service. Other YQL based services that use query.yahooapis.com will no longer operate."

Need to replace "//query.yahooapis.com/v1/public/yql?q=" for my rss scraper to work.

function yql(a, b) {
        return (
          "**//query.yahooapis.com/v1/public/yql?q=**" +
          encodeURIComponent(
            "select * from " +
              b +
              ' where url="' +
              a +
              '" limit ' +
              params.feedcount
          ) +
          "&format=json"
        );
      }

Loats

My small app is affected by this too, apparently the whole YQL thing is going out-of-service. – Drooff

Answer 1

I found this and it worked great for me. https://api.rss2json.com There's a free layer and it's way more straightforward than YQL was for RSS to JSONP conversion.

Hyaena

This only supports RSS, correct? not HTML? – Excitation

Answer 2

I build CloudQuery which is able to turn most websites to API and it has a simple to use web interface to create the API. And it is open sourced on github

Transcendent

the site give this error :"Internal server error" – Cottonade

Answer 3

Here is a possible solution for you.

a) You need some kind of proxy to allow loading content with ajax from different sources. It is advisable to whitelist and add CORS Headers etc. to prevent exploiting your proxy. Create for example a php-file on one of your servers with this functionality:

$valid_url_regex = '/.*(rss|feed|atom).*/';
$url = $_GET['url'];
if ( !preg_match( $valid_url_regex, $url ) ) exit;

$feeds = file_get_contents($url);
//this is some workaround to get special namespaces into the json
$feeds = str_replace("<content:encoded>","<contentEncoded>",$feeds);
$feeds = str_replace("</content:encoded>","</contentEncoded>",$feeds);
$feeds = str_replace("<media:content ","<mediaContent ",$feeds);
$feeds = str_replace("</media:content>","</mediaContent>",$feeds);

$simpleXml = simplexml_load_string($feeds, "SimpleXMLElement", LIBXML_NOCDATA);//this is for CDATA
$json = json_encode($simpleXml);
header("Access-Control-Allow-Origin: http://yourdomainnamehere");
header('Access-Control-Allow-Credentials: true');
header('Access-Control-Max-Age: 86400'); 
print $json;

b) Perform an async ajax-call to the proxy-script and handle the data:

function loadRss(url)
{
    $.ajax({
        url: 'yourserverurl/rssproxy.php?url='+url,
        type: 'GET',          
        success: function(response) {
            handleResponse(JSON.parse(response));
        }
    });
}


function handleResponse(response) { 
    var entries; 

    if(response.entry) //ATOM
        entries = response.entry;
    else if(response.channel.item) //RSS 1/2
        entries = response.channel.item;

    var feedTitle="";

    if(response.title)
        feedTitle = response.title;
    else if(response.channel.title)
        feedTitle = response.channel.title;

    //iterate all news entries
    $.each(entries, function (i, e) {
            console.log("Entry #"+i);
            console.log(e);
            //access the data as necessary like e.content, e.summary, e.contentEncoded etc....
    }
    );

}

I changed my google rss api a few years ago to YQL, now i had to do it again today, took a few hours, but this time you wont be dependent on some 3rd-party vendor and hopefully you can use your new reader code until rss vanishes in preference of mankind for the famous filter bubble ;)

Above code is just a hint and of course you will have to invest some time if you want to map the response to the generalized YQL Structure. I didnt go that way and accessed the properties of the response as necessary.

Uptown

Thanks for the help, I will take a look and see if I can implement your solution. – Loats