The spiders crawl the URLs systematically. at the same time, they confer with the robots.txt file to check whether they are permitted to crawl any certain URL. I'd be remiss if I didn't give a shout-out to several of https://ariabookmarks.com/story2950754/seo-ranking-software-for-dummies