Indexing aids at corporate websites: the use of robots.txt and META tags

作者:

Highlights:

摘要

Sixty corporate websites selected from the Fortune Global 500 companies were examined in 2000 and again in 2001 to see if they provided support for automatic indexing. In particular, use of the robots.txt and Meta tags for “keywords” and “description” was examined. Slightly fewer than half of the sites provided one or both of these aids. Among sites providing indexing aids there was a clear under-representation of Asian sites. Nearly 80% of the sites used Java, suggesting a reasonable level of technical sophistication among website creators. About one-third of the sites used cookies, raising the possibility that repeat visitors might find the navigation of the site customized to their needs. Overall an increase in the use of indexing aids, especially Meta tags, represents one way in which web robots could index sites more quickly and thus improve overall index coverage of the web.

论文关键词:Web indexing,Document access,Corporate web sites

论文评审过程:Received 15 May 2000, Accepted 17 May 2001, Available online 27 November 2001.

论文官网地址:https://doi.org/10.1016/S0306-4573(01)00039-5