How crawler works

• Web crawlers work by starting at a seed, or list of known URLs, reviewing and then categorizing the webpages. 
• Before each page is reviewed, the web crawler looks at the webpage's robots.txt file, which specifies the rules for bots that access the website. These rules define which pages can be crawled and the links that can be followed.
• To get to the next webpage, the crawler finds and follows hyperlinks that appear. Which hyperlink the crawler follows depends on defined policies that make it more selective about what order the crawler should follow. 
• For example, how many pages link to that page, the number of page views, and brand authority.