Twitter website doesn't have open graph tags?

Question

I'm trying to get url previews (for websites that support them) to show up in a project I'm working on. I recently noticed that twitter urls don't have open graph meta tags anymore. I was expecting og:title, og:description and so on, which if I remember correctly used to exist for all twitter links.

E.g. if I see the page source for this link: twitter.com/DalaiLama/status/1274998376338124800

I don't see og metadata apart from og:site_name. I also don't see any twitter:title or respective content. What am I missing?

Update: so it turns out view source doesn't show og:title, but I do see it under Chrome's "inspect" menu. Does that mean the JS actually has it but not the HTML (also it only shows the og:title and not other fields)? Is that expected?

Profess

Did you find a solution for this? I'm facing the same issue and not sure how to get a tweet's metadata – Disperse

Answer 1

Twitter uses client-side-rendering (CSR) to generate HTML in the browser

Viewing the source directly will not show any of the relevant <meta> tags or actual page HTML content, because it is all dynamically generated on the client's browser in React using JavaScript (i.e. CSR: Client-side rendering). In fact, the HTML source will have a stub containing "We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?". This can be verified by opening up developer tools and peeking at the "Elements" tab during page load/render or downloading the page without JavaScript emulation.

However, to improve Search Engine Optimization (SEO) for various prominent web-crawlers, Twitter will instead return server-side-rendered (SSR) HTML content (which does contain the <meta> tags). This enables crawlers to not have to emulate JavaScript to view the page, and only crawl raw HTML content. Twitter recognizes crawlers based on the supplied User-Agent HTTP Header. Server-side-rendering is generally a more expensive operation than offloading the HTML rendering onto the client, which may be a reason why Twitter opts for client-side-rendering as the default behavior.

Bypassing the User-Agent whitelist to receive server-side-rendered (SSR) HTML

Various prominent web-crawlers are whitelisted by Twitter to receive server-side-rendered HTML. By spoofing the User-Agent HTTP Header in your own request, you can bypass the whitelist and receive server-side-rendered HTML containing the relevant <meta> tags (whether or not this is recommended is a totally different subject matter). For programmatic HTTP requests, check for support for changing the User-Agent HTTP Header in your HTTP library - most non-trivial libraries support this functionality.

whatismybrowser.com has a list of well known web-crawler User-Agent headers; some of these web crawlers are whitelisted (but not necessarily all). At the time of writing, here are some working user agents:

• Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
• Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
• facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
• Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)

Presume

Thanks for the detailed answer, now I know what to look to do an Ajax call – Styria

Answer 2

It looks like twitter is allowing the facebook crawler to view their open graph tags. If you can set your user agent similar to what is described in the Troubleshooting section on the facebook crawler site, the full set of tags appears.

$ curl -s --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "https://
twitter.com/sharifshameem/status/1284095222939451393" | grep -i 'og:'
    <meta  property="og:type" content="video">
    <meta  property="og:url" content="https://twitter.com/sharifshameem/status/1284095222939451393">
    <meta  property="og:title" content="Sharif Shameem on Twitter">
    <meta  property="og:image" content="https://pbs.twimg.com/ext_tw_video_thumb/1284094287383166977/pu/img/LsArMNT3djA7xg53.jpg">
    <meta  property="og:description" content="“I just built a *functioning* React app by describing what I wanted to GPT-3. &#10;&#10;I&#39;m still in awe. https://someurl”">
    <meta  property="og:site_name" content="Twitter">
    <meta  property="og:video:url" content="https://twitter.com/i/videos/1284095222939451393?embed_source=facebook">
    <meta  property="og:video:secure_url" content="https://twitter.com/i/videos/1284095222939451393?embed_source=facebook">
    <meta  property="og:video:type" content="text/html">
    <meta  property="og:video:width" content="1200">
    <meta  property="og:video:height" content="696">

Without specifying the user agent:

$ curl -s "https://twitter.com/sharifshameem/status/1284095222939451393" | grep -i 'og:'
  <meta property="og:site_name" content="Twitter" />

Alley

Answer 3

I couldn't get this to work with just parsing the http response. Registering as a twitter API user and calling their labs endpoints was useful. You can then decode the resulting json.

Profess