再保险:关于robots . txt的问题

星期二,2015年12月8日,棺材,克里斯写道::我们很久以前不允许选择的索引:cve.mitre.org web站点。至少这个决定只是部分:资源约束?CVE蹒跚学步的年时,搜索引擎:索引器非常资源密集型。“决定”是基于垃圾借口,甚至。=)的人跑两个站点的时间跑CVE冠冕,和集中看到登录其中一个(attrition.org,因为1998-10-07),搜索引擎没有资源密集型。减员人员讨论过这个问题,没有阻止任何内容的机器人。txt因为搜索引擎垃圾邮件是礼物,但不沉重。对于那些感兴趣的是互联网历史上…迫使~ /home/admin/util/list.更多美元过滤器72.14.203.104 forced.attrition.org images.search.yahoo.com casualgamer.org myspace.com stumbleupon.com f-mai。gif f-bak。gif f-att。gif thefiles。gif panopta.com divinelanguage.com forced ~$ grep -i google /home/admin/util/list.* /home/admin/util/list.bot:googlebot.com /home/admin/util/list.bot:Feedfetcher-Google /home/admin/util/list.filter-old:google.com /home/admin/util/list.filter-old:google.co.jp/search /home/admin/util/list.filter-old:google.de /home/admin/util/list.filter-old:google.fr /home/admin/util/list.filter-old:google.co.uk forced ~$ "list.filter-old" is from 2003-08-25. The limited set of Google domains should be very telling, given the year and traffic generated. We actually *stopped* filtering Google at some point, while ignoring Yahoo early on. Why? Because they were simply not hammering sites and causing any undue burden, to a random desktop machine bought at the local computer store. Those were "ignore displaying those entries in our log parser", not "block them from reaching our web server" via iptables. That was Attrition when it was run on a ~ $500 box bought in 1998 and hosted on a consumer link, compared to MITRE's resources and CVE contract money from the government at the time. So to be clear, MITRE's answer in 2015, is based on people forgetting what it was like in 1997 - 1999. That said, after Kurt's mail in December of 2015... in the last ~ 30 - 60 days, I noticed that MITRE finally changed that. Google is now indexing and caching the CVE pages. Thank you, as a long-time taxpayer funding MITRE's projects, including CVE, to the tune of $1,487,334,000 in MITRE income last year. Good to see you making these small changes to help the industry. : We are currently re-examining this policy and will keep the Board : posted. Except... you didn't. Just like you didn't ask us about the 3k+ RESERVED fiasco that got several of us talking about this morning, figuring out how we'd handle it. When NVD spoke up, we all collectively said "hell yeah!" The fact that NVD called you out, and has since said they will be 'ignoring' those IDs, is also very significant in CVE history. This is the first *real* break that NVD has had from CVE ever. There have been other breaks the last year+, but they were more pedantic and favored NVD over MITRE/CVE, based on the time of entries becoming public (e.g. NVD published before MITRE did). Brian