Business Affiliate ProgramsDiscountsPersonalsAdvertisingShopping



Getting the website index in Google each engine can be easy if you know the technique to adopt. If you are looking to get your site indexed for the first time by Google, there are a boat load of tutorials about that online. Two hours MINIMUM and 48 hours MAX is all it takes to index a client’s new site using the four methods below:
  1. Once the site/blog has been completely setup with articles and all that, install and activate the XML Sitemap with which will automatically create a sitemap.xml file and then ping Google, Bing, Ask and Yahoo. At the same time, I also make use of wordpress ping features to alert other directories/sites about the existence of my new site.
  2. Proceed to tweet several posts/pages from the new site. Over the last few months, I’ve noticed that Google tends to crawl my tweets quite well as does a particular RSS directory.
  3. Finally and most importantly, I update the Dofollow Blogs folder of RSS reader and hope to God that they’ve got something interesting about which I can leave a few comments. The trick here is to make sure that Google loves these dofollow blogs and crawls them constantly, that way your new site’s link will be detected, crawled and hopefully indexed.
  4. Submit your site to Google via the Google Webmasters Tool 

The three tips above do ESSENTIALLY two things. They get Google to crawl your sites and then index them. If the crawling part is made tedious for Google, chances of your site not been indexed are HIGH and that’s why James Adams is going to show you how you can make the crawling process easier for the search engines so that they keep coming back frequently.

Here are 9 ways of getting your website crawled faster.
1. Optimize Your URLs: Rather than having your site create URLs with redundant information in each one, try sending the repeated information to a cookie and making each URL more unique. This means that anything in the page URL that does not affect the content of the page should be removed. Because crawlers don’t have to deal with this redundant information, your site should get crawled much faster because there will be much less duplicated content.

Quick Tips:
  1. If you’re using wordpress, set your permalink structure to either /%postname%/%post_id%/ or /%post_id%/%postname%/. These structures are amongst the best for every wordpress blog.
  2. To avoid the duplicate content penalty, be sure to make use of the More Tag and to also apply nofollow to EVERY tag link on your site. Using All In One SEO, which forms an integral part of the Top 16 Most Essential WordPress Plug-ins, be sure to activate noindex for Categories, Archives and Tag Archives.

2. Control What Gets Crawled: Use a robots.txt file to tell crawlers to skip content that they are unable to index or that you do not want indexed. This will reduce the time it takes for the search engines to scour your site. You could also use the robots meta tag to tell crawlers whether they should be crawling a page or not.
Quick Tip: If your site is wordpress based and wordpress was installed in the root directory ie and not, you can make use of this robot.txt.

3. Build More Backlinks With Relevant Anchor Text: Google (and Bing/Yahoo!) is known to place more weight on sites that it considers more important based on the number of quality links the site has. By working on your SEO, you should see spiders crawling yourself more frequently as you build links from related high quality websites, particularly competitors.
Quick Tip: Go crazy commenting on forums and blogs while leaving your links behind ( either in your forum signature or the URL field of the blog where you’re commenting). Dofollow, Keyword Luv and Comment Luv enabled blogs work best .

WARNING: Make sure not to use the same anchor text everywhere as Google might think you’re trying to spam them.
Also make sure not to leave your link within your blog comments as you would be tagged a spammer in no time and once Akismet get’s wind of it, your backlink creation days will be hampered.

4. Optimize Your Server: If your site is extremely slow to load (or the pages are massive) then Google may not crawl your site as often or as deep as it could. Log into Google Webmaster Tools (create a free account if you don’t have one) and see if Google has limited its activities with your site because of slow response time. Spend some time optimizing your server for speed so that robots (like Googlebot) can access your content quickly and reliably.

Quick Tip: The speed of your site depends heavily on two factors – The quality of the server on which your site is hosted and the use of external images and JavaScript.
  1. Make sure your host has a reputation for quality in every angle such as Hostgator does.
  2. Try as much as possible to host about 95% of your images and JavaScript on your site’s domain or a reliable CDN provider. You can even create your own CDN using a subdomain ;-).
  3. Even though the server on which your site is hosted runs pretty fast, you are strongly advised to use cache plug-ins to minimize the load on the server especially if you’re using shared hosting. Check out my step by step guide to configuring W3 Total Cache.
5. Simplify the Structure of Your Site: By streamlining the paths around your site you can speed up the time crawlers take to index it. Make sure to externalize common files such as CSS and JS and ideally use a Content Delivery Network (CDN) to host them so that spiders and users can download them in parallel along side the actual page. Be sure to use a logical tree directory structure with short names that is not more than a few folders deep.

6. Pay Close Attention to Competitors: By looking at a site in the same niche as yours that gets listed higher in search results, you can get some clues as to how to configure your site. Look out for how they structure their site, how frequently they update it, etc.

7. Improve Your Internal Link Structure: Often website owners get so busy working on content that they forget that the internals of the site impact how often it is crawled by the search engines. Every page should have at least one link to another page, ideally many more. You want the links to be similar in context and with decent anchor text so that they are more effective.

Quick Tip: Whenever you are writing creating a new post/page for your site, be sure to add links to your past articles which are totally relevant to the new post just like I’ve done in this post, if you’ve cared to notice.

8. Add New Content Frequently: One of the easiest ways to get your site crawled faster and deeper is to update it frequently with news, reviews of products, commentary, etc. There’s always something you can write about, even if you’re in a boring niche. Just make sure you do it!

Warning: In as much as adding fresh content to your site on a regular basis will keep the crawlers coming back ever so frequently, you ought to be carefully not to water down the quality of your posts ’cause if you do, you risk losing the respect of your loyal readers and prospective new readers.

9. Promote Your Best Content on Social Networks: Get your links on Facebook, Twitter, and social bookmarking sites like StumbleUpon, Reddit, and Digg. If you can use these sites to drive traffic to your site, links will follow and the crawlers won’t be far behind!

Search engines are the lifeblood of most websites. Follow these 9 ways for getting your website crawled faster and you will get better results, more timely visitors, and more money in your pocket.
Have you got tips of your own on how we all could get our sites crawled faster? Please be kind enough as to share with us below. Cheers.