Verisign gets patent for triggering web crawling based on registry data

.Com registry awarded patent for using registry data to trigger web crawling.

With Verisign analyzing how to profit from its patent portfolio, competitors should pay close attention to the company’s new patents.

Today Verisign was awarded U.S. Patent 8,433,700 (pdf) for “Method and system for triggering web crawling based on registry data”.

The patent describes ways in which a registry can provide data to other companies, such as search engines, that may help them determine when to crawl a domain name’s content.

For example, a registry can provide information about new domain registrations, registrar transfers, and deletions that can be used by search engines to initiate crawling. A domain transfer may indicate a change in ownership and a site’s use, and thus a need to recrawl the domain.

The service could also use resolution request data to indicate active web sites that should be crawled. In one example explained in the patent, increased traffic to a blog could also suggest new comments are being left on a blog post. This could warrant speedier recrawling.

In Verisign’s case with .com, it operates a thin registry so it’s zone file doesn’t include much information about each domain. The patent allows that additional data resources could be included to provide even more triggers:

Other data sources include data received from registrars such as information that registrant has changed hands, that a registrant has changed hosting companies, data received from hosting companies related to changes to the hosted websites such as file uploading data. These data feeds may be free and publicly available or may be purchased and then integrated into the systems described herein. Therefore, the quality of the data used to trigger crawls may be improved by supplementing the data discussed herein, e.g., changes to the registry database, with additional data useful to the entity performing the crawling.