Dignity nobility quality condition highness standing course caste status superiority precedence manner distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robotic regarding which pages aren't to get crawled. To be a internet search engine crawler may well maintain https://www.youtube.com/watch?v=RSlwOZLlUGg