当前位置: 动力学知识库 > 问答 > 编程问答 >

robots.txt - google is not indexing my site

问题描述:

Something is seriously wrong with my website. It has been 3 months since launch but google never indexed my site pages properly.

Till last week there were more than 7k pages in my site but google hardly crawled 300 pages in 3 months. But from last one week, google completely stopped indexing my pages and now there are only 87 pages indexed in total. I have not used any black hat SEO techniques and I dint even get any message in webmaster for manual action.

One more observation, google is giving following message for most of the pages which are even crawlable as per robot file (I checked it using webmaster robots.txt check).

“A description for this result is not available because of this site’s robots.txt – learn more."

Following is the robots file, can somebody please help me identify if there is any problem in this..

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file

#User-agent: NinjaBot

#Allow: /

Sitemap: http://socktail.com/sitemap.xml

User-agent: *

Disallow: /giftcard/

Disallow: /fancybox/

Disallow: /notify/

Disallow: /cms/

Disallow: /feed/

Disallow: /landing/

Disallow: /checkout/

Disallow: /user_settings/

Disallow: /product/

Disallow: /search*/

Disallow: /cart/

Disallow: /order/

Disallow: /shop/

Disallow: /colorsby/

Disallow: /settings/

Disallow: /admin/

Disallow: /fancybox

Disallow: /pages/sellers-service-agreement

Disallow: /login*

Disallow: /signup*

Disallow: /shopby/shop‎/

Disallow: /shopby/all/‎*

Disallow: /giftguide/list/

Disallow: /colorsby/

Disallow: /shopby/shopby/

Disallow: /shopby/things/

Disallow: /shopby/giftguide/

Disallow: /shopby/login*

Disallow: /shop$

Disallow: /user/*/lists

Disallow: /user/*/lists/

Disallow: /user/*/wants

Disallow: /user/*/wants/

Disallow: /user/*/owns

Disallow: /user/*/owns/

Disallow: /user/*/following

Disallow: /user/*/following/

Disallow: /user/*/followers

Disallow: /user/*/followers/

Disallow: /user/*/settings

Disallow: /user/*/settings/

Disallow: /user/send_noty_mails/

Disallow: /*?

Disallow: /bookmarklet/

Disallow: /socktail.com

Disallow: /www.mudradecor.com

Disallow: /site/

Disallow: /settings

Disallow: /login#sidebar

Disallow: /login#navigation

Disallow: /login#header

Disallow: /login#content

Disallow: /lang

Disallow: /gifts

Disallow: /uploaded/

Disallow: /pages/about

Disallow: /pages/fancy-for-business

Disallow: /www.dumysite.com

网友答案:

Robots.txt is only a recommendation to bots visiting your site, such as Googles search bot and also others. It is only a recommendation, some search engines bots don't follow it.

A closer inspection of your robots.txt file shows disallow for a lot of sub folders on your site. This obviously is bad because Google a good search engine will listen to your rules and won't index those pages.

This link has some information, but maybe complex. Moz robots txt information

This link for a robots.txt generator can help you create a better one: Robots.txt generator

Note... the only things you need to disallow is pages you don't want indexing such as admin and private pages. If you got an estore then some parts may need keeping private.

From there I recommend pinging your site. Do a search on Google for ping website and should find several options. This will help robots visit your site and see the change in robots.txt file and all should be well.

网友答案:

https://www.google.com/search?hl=en&safe=off&q=site%3Asocktail.com&btnG=Search -> (About 11,600 results)?

Your robots.txt seems to be ok.

But a bit slow, try http://www.webpagetest.org

分享给朋友:
您可能感兴趣的文章:
随机阅读: