10. Gzip Compression: Empower Gzip compression in your server to decrease the sizing of information transferred amongst the server as well as the user’s browser. txt file is then parsed and may instruct the robotic as to which web pages are not for being crawled. For a search engine crawler https://www.youtube.com/watch?v=OeFqbAORPUg