User-agent: * Allow: / User-agent: Mediapartners-Google Allow: /wiki/ Allow: / User-agent: Googlebot Allow: /wiki/ Allow: / User-Agent: bingbot Crawl-delay: 30 Disallow: /cache/ Disallow: /docs/ Disallow: /extentions/ Disallow: /maintenance/ User-Agent: AhrefsBot Crawl-Delay: 30 #Baiduspider User-agent: Baiduspider Disallow: / #Yandex User-agent: Yandex Disallow: / # Semrush User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: /