当前位置: 动力学知识库 > 问答 > 编程问答 >

php - Robots.txt priority question

问题描述:

If I have these lines in robots.txt:

Disallow /folder/

Allow /folder/filename.php

Will the filename.php be allowed then?

Which order does google prioritize the lines?

What will happen here for example?:

Allow /

Disallow /

I am mainly referring to Google here as a search engine.

Thanks

网友答案:

For your first example, yes, the file /folder/filename.php will be allowed, as Google will prioritize the most specific rule. See here or check Wikipedia's article, for example, for more details.

Furthermore, Google has a detailed documentation on how it uses robots.txt over here.

Edit : For the second example, the Allow will be prioritized as per the Wikipedia quote Byron Whitlock posted in his answer.

网友答案:

It depends on the search engine:

While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that Allow patterns with equal or more characters in the directive path win over a matching Disallow pattern. Bing uses the Allow or Disallow directive which is the most specific.

In order to be compatible to all robots, if one wants to allow single files inside an otherwise disallowed directory, it is necessary to place the Allow directive(s) first, followed by the Disallow.

http://en.wikipedia.org/wiki/Robots_exclusion_standard

分享给朋友:
您可能感兴趣的文章:
随机阅读: