Subsequent, you ought to make a decision how you want the crawler to manage various Web systems. There may be an ongoing debate about the intelligence of online search engine crawlers. It isn't really fully very clear if they are full-blown headless browsers or simply glorified curl scripts (or https://www.youtube.com/watch?v=EVeTP0aV8gY