Most of the common users or readers use various available search engines to search out the piece of information they needed. But how this information is given by search engines? Where from they've gathered these information? Basically most of these search engines maintain their own database of information. These database includes the sites available within the webworld which fundamentally take care of the depth web pages information for each available sites. Visit http://www.seekingalpha.com/user/41042325/comments/ to study where to see about it. Essentially search engine do some back ground work-by using programs to get data and maintain the database. They make collection of accumulated data and then present it publicly or at-times for private use.
In this article we will discuss about those people which loiter in the global internet setting or we will about net robots which move around in netspace. We are going to learn
What its all about and what function they serve??
Pros and cons of using these entities.
How we can keep our pages away from robots??
Differences between the common spiders and programs.
Within the following percentage we shall divide the whole research work under the following two sections :
I. Search Motor Index : Robots.txt.
II. Search Engine Programs : Meta-tags Explained.
I. Search Engine Index : Robots.txt
What is robots.txt file??
An internet robot is a system or se software that visits sites regularly and automatically and get through the webs hypertext construction by bringing a record, and recursively finding all of the papers which are referenced. Often site owners do not want each of their site pages to be crawled from the web robots. Because of this they can exclude few of their pages being crawled by the programs by using some common agents. Therefore the majority of the robots abide by the Robots Exclusion Standard, some demands to eliminates robots behavior.
Robot Exclusion Standard is just a protocol utilized by the site administrator to manage the action of the spiders. It will search for a file named robots.txt in the root domain of the site (http://www.anydomain.com/robots.txt) when search engine robots come to a site. It is a plain-text file which implements Robots Exclusion Protocols by allowing or disallowing specific files within the directories of files. Site supervisor can disallow use of cgi, temporary or individual directories by indicating software user agent names. Clicking internet www.seekingalpha.com/ perhaps provides aids you might give to your girlfriend.
The structure of the robot.txt report is very simple. It contains two field : user-agent and one or more disallow field.
What's User-agent??
This really is the technical name for an development methods in the worldwide network environment and used to say the particular search-engine software inside the file.
For instance :
User-agent: googlebot
We can also utilize the wildcard character * to establish all robots :
User-agent: *
Means all of the robots are allowed to come to see.
What's Disallow??
In-the file 2nd field is known as the disallow: These lines guide the programs, to which file should be crawled or which shouldn't be. As an example to avoid installing email.htm the syntax can be:
Disallow: email.htm
Reduce crawling through sites the syntax may be:
Disallow: /cgi-bin/
White Space and Comments :
Using # at the beginning of any line-in the robots.txt file will be considered as remarks only and applying # at the beginning of the robots.txt such as the following case include people which website to be crawled.
# robots.txt for www.anydomain.com
Access Details for robots.txt :
1) User-agent: *
Disallow:
The asterisk (*) inside the User-agent field is denoting all robots are invited. As nothing is disallowed so all spiders are free-to get through.
2) User-agent: *
Disallow: /cgi-bin/
Disallow: /temp/
Disallow: /private/
All robots are permitted to get through the all records except the temp, cgi-bin and private record.
3) User-agent: dangerbot
Disallow: /
Dangerbot is not allowed to get through the directories. / stands for all sites.
4) User-agent: dangerbot
Disallow: /
User-agent: *
Disallow: /temp/
The blank line indicates beginning of new User-agent records. Except dangerbot all one other spiders are permitted to crawl through all the directories except temp directories.
5) User-agent: dangerbot
Disallow: /links/listing.html
User-agent: *
Disallow: /email.html/
Dangerbot is not allowed for the list page of links index usually all the robots are allowed for all websites except installing email.html page.
6) User-agent: abcbot
Disallow: /*.gif$
To eliminate all files from a specific file type (e.g. Be taught further on a related site by visiting www.seekingalpha.com/user/41042325/comments. .gif ) we are going to use the above robots.txt entry.
7) User-agent: abcbot
Disallow: /*?
To reduce net crawler from running active pages we are going to utilize the above robots.txt entry.
Observe : Disallow area may end with $ to indicate the end of-the name and may include * to follow any series of figures.
Eg : Within the image files to exclude all gif files but allowing others from moving
User-agent: Googlebot-Image
Disallow: /*.gif$
Disadvantages of robots.txt :
Problem with Disallow field:
Disallow: /css/ /cgi-bin/ /images/
Different index may read the above subject in different way. Some will study /css//cgi-bin//images/ and will disregard the areas and may possibly only con-sider either /images/ or /css/ ignoring others.
The proper syntax ought to be :
Disallow: /css/
Disallow: /cgi-bin/
Disallow: /images/
All Files listing:
Revealing each and every file name within a service is mostly used mistake
Disallow: /ab/cdef.html
Disallow: /ab/ghij.html
Disallow: /ab/klmn.html
Disallow: /op/qrst.html
Disallow: /op/uvwx.html
Above piece could be written as:
Disallow: /ab/
Disallow: /op/
A trailing slash means a great deal that's a listing is offlimits.
Capitalization:
USER-AGENT: REDBOT
DISALLOW:
Though fields are not case sensitive but the datas like sites, filenames are case sensitive.
Conflicting syntax:
User-agent: *
Disallow: /
#
User-agent: Redbot
Disallow:
What'll happen?? Redbot is allowed to investigate anything but will this permission override the disallow area or disallow will override the allow permission.
II. Search Engine Robots: Meta-tag Explained:
What's robot meta tag??
Besides robots.txt search-engine can also be having still another methods to crawl through web-pages. This is the META-TAG which shows net spider to index a full page and follow links on it, which might be more helpful in some instances, as it can be utilized on page-by-page basis. It's also helpful incase you dont have the prerequisite permission to access the computers root directory to control robots.txt document. Identify more on the affiliated encyclopedia - Click here: seekingalpha.com/user/41042325/comments.
We used to put this tag within the header portion of html.
Structure of the Robots Meta-tag :
Within the HTML file it's put in the TOP section.
html
Mind
META NAME=robots CONTENT=index,follow
META NAME=description CONTENT=Welcome to.
titletitle
Mind
Human body
Robots Meta Tag options :
There are four possibilities that can be used in the CONTENT portion of the Meta Robots. These are index, noindex, follow, nofollow.
This label letting se spiders to index a specific site and can follow all the link living onto it. can change index,follow with noindex,nofollow if site admin doesnt want any pages to be listed or any url to be followed then.
According to the requirements, the robots can be used by site admin within the following different choices :
META NAME=robots CONTENT=index,follow"> Index this page, follow links from this page.
META NAME=robots CONTENT =noindex,follow"> Dont index this page but follow link from this page.
META NAME=robots CONTENT =index,nofollow"> Index this page but dont follow links from this page
META NAME=robots CONTENT =noindex,nofollow"> Dont index this page, dont follow links from this page..
文章定位：
