Facebook scraper doesn't load dynamic meta-tags

Asked 15/2, 2013 at 16:8 Answered 15/2, 2013 at 17:27

Solved html facebook web-scraping meta-tags scraper

I am creating the HTML meta-tags dynamically using the function below (GWT). It takes 1 second to have this on the DOM. It is working fine except for Facebook. When I share a link from my web, the scraper gets the meta-tags that are in the HTML: none. How can I fix this?

/**
* Include the HTML attributes: title, description and keywords (meta tags)
*/
private void createHTMLheader(MyClass thing) {

    String title=thing.getTitle();
    String description=thing.getDescription();

    Document.get().setTitle(title);

    MetaElement metaDesc = Document.get().createMetaElement();
    metaDesc.setName("description");
    metaDesc.setContent(description);
    NodeList<Element> nodes = Document.get().getElementsByTagName("head");
    nodes.getItem(0).appendChild(metaDesc);
}

This is the resulting HEAD on the DOM. The title aaaa and meta-description has been loaded dynamically. (Thanks CBroe for the tip). In the "view source" functionality, these dynamic tags are not displayed (only on developer tools - view dom).

<head>
    <title>aaaa</title>
    <meta content="text/html; charset=utf-8" http-equiv="content-type">
    <meta name="description" content="My description">

    <script language="javascript" type="text/javascript" src="dialective/dialective.nocache.js"></script><script defer="defer">dialective.onInjectionDone('dialective')</script>

</head>

The original HTML doesn't have a TITLE or META-DESCRIPTION tags.

Peppard answered 15/2, 2013 at 16:8 Comment(6)

Can you show the HTML this actually generates? – Slacker 15/2, 2013 at 16:9

Thanks. Just added the <head> – Peppard 15/2, 2013 at 16:22

Are you trying to add HTML tags with a Java app or something? The Facebook scraper isn't going to run your Java code. It'll see exactly what your web server is sending. – Slacker 15/2, 2013 at 17:12

Exactly. The code below is run on client-side (a GWT presenter). So there is no way to have this done? Thanks – Peppard 15/2, 2013 at 17:20

Nope. Generate the meta tags on the server. – Slacker 15/2, 2013 at 17:23

can you tell me how you solved this problem – Oldworld 29/12, 2015 at 14:43

The Facebook scraper can only see <meta> tags included in the original HTML response from the server. It's not "smart enough" to run any JavaScript code, Flash plugins, Java applets, or anything else that a full-fledged browser might run.

You'll need to generate these <meta> tags on the server using a server side framework.

Also, Facebook provides a handy testing tool to make sure your page exposes the appropriate meta tags. You might need to add OpenGraph tags as well, such as og:title and og:description.

Slacker answered 15/2, 2013 at 17:27 Comment(2)

Hi Mike, the point you made about the Facebook scraper only seeing the orig' HTML is what I've come to find as I'm having same issue as OP. However I figured what if I loaded AddThis using $.getScript after I'd made changes to the DOM. In my theory that way FB would only arrive to scrape with the new DOM. But is this not possible? I fear that when AddThis fires and FB share is clicked the URL is passed to FB & then it does an independent scrape of the share URL which doesn't have the javascript edited meta tags. Would you have any thoughts on this? Daniel. – Redundant 22/9, 2014 at 11:20

Thanks! Makes sense. What if the page is generated dynamically, like a WordPress activity post, would this then not be possible? – Intracranial 12/9, 2023 at 23:3

-2

For some reason, the meta description has not been loaded with the function above...

That’s because you are only creating a new MetaElement – but you do nothing with it.

You have to append it to the document, more specific to its head element.

Something like

getHead().appendChild(metaDesc)

is missing.

Offcenter answered 15/2, 2013 at 16:27 Comment(0)

Recommended topics

Hot tags