Using AJAX, How to Make Your Site Crawlable

Why Ajax?

AJAX is used to load the contents into the page faster with providing a superior user experience. It happens so as the data exchanges with a server and the web page reflects the changed data without reloading. So AJAX is gaining popularity among the developers.

Problems with AJAX usage:

However, it has its own limitation for not allowing Google and other search engines to crawl an AJAX based website. As a result of which AJAX can damage the SEO of the site, because the browser address bar does not change with AJAX load. You need to generate SEO friendly URL dynamically.

Solution:

Here I discuss how to balance out both AJAX as well as SEO. That means, the site can still be crawlable with AJAX on.

Step-1: Load the AJAX content with the change in browser address bar:

• We need to manage the address bar using client side code and add ‘#’ at the end of the URL, before the AJAX page name or parameters.

1. If about-us page loads in AJAX, the URL will be: https://www.mysite.com/#about-us
2. NOTE: the hash fragments are never sent to server

• After refreshing the page, we can get the hash fragments using client side code and avail the AJAX contents

Step2: Change the URL format to crawl by the crawlers

• ‘Googlebot’ still cannot crawl the content of ‘about-us’ page, because the crawler cannot get the content using client side scripting.

1. Solution: it can be solved by adding an  ‘!’ after ‘#’
2. Example: https://www.mysite.com/#!about-us

• The hash fragment (#!) works as a signal to Googlebot, to use an alternative URL (Ugly URL) for creating a snapshot of the page including the AJAX-loaded contents.

Step3: Avail the AJAX content to the crawler

• Pretty URL: The URLs loaded on the client browser that loads dynamic content.

1. Example: https://www.mysite.com/#!about-us

• Ugly URL: The alternate URLs of Pretty URLs which is used by ‘Googlebot’ to get the content of the page “? _escaped_fragment_=” in place of “”#!”

1. Example: https://www.mysite.com/?_escaped_fragment_=about-us

• Important: There should be same page or content while accessing both the URLs.

Step4: Avail the crawlable URL format

• Always have the Pretty URL (URL with “#!“) link on all <a href> attribute to load the AJAX content.

• Add Pretty URLs to the XML sitemap.

How it works?

• ‘Googlebot’ spider finds the Pretty URL “https://www.mysite.com/#!about-us” from the sitemap or from the webpage.

• It would fetch the Pretty URL as “https://www.mysite.com/?_escaped_fragment_=about-us“

• The indexer sorts every word and stores the resulting index of words in Google database.

• The query processor returns Pretty URLs to the user on the search results as  “https://www.mysite.com/#!about-us“

Final Tidbits:

• A URL may contain at most one hash fragment

• Do not include links such as https://www.mysite.com/?_escaped_fragment_=about-us in the website sitemap; ‘Googlebot’ does not follow links that contain “_escaped_fragment_”

• When “? _escaped_fragment_=about-us”is activated, you can only provide the “snapshot” of the page.

• The crawler escapes certain characters in the fragment during the transformation. To retrieve the original fragment, make sure to un-escape all %XX characters in the fragment; specifically, %26 should become &, %20 should become a space, %23 should become #, and %25 should become %, and so on.

• Test the crawl-ability of your app: see what the crawler sees with https://support.google.com/webmasters/answer/158587?hl=en