Instead of converting your entire site to static URLs, pick the pages you want to index by a crawler.
代替将整个站点转换为静态url，您可以挑选一些希望爬行器建立索引的页面。
2
They'd bring these random x86-based computer parts back to their dorm room to add to the Frankenstein machine hosting the legendary rogue Web crawler that took down Stanford's entire network-twice.
Helps to optimize the entire structure of the site for a crawler by providing an alternate set of Web pages so that crawlers can quickly access and index a large number of embedded pages.