2010/05/05

Avoid Error 404 page found by search engine

It is quite embarassing to me that the Error 404 Page of my website can be found by Google. On checking the error 404 html file, I noticed that I have not included the appropriate meta tag in header to tell search engine not to touch on that file. The meta tag should look like:



After 1 - 2 months, the Error 404 Page should not be found again.

Another approach is to place the Error 404 html file in a special directory and use robots.txt to disallow web spider to crawl on that directory which looks like:

[file robots.txt]

User-agent: *
Disallow: /cgi-bin/
Disallow :/404-file/

No comments: