olexsmir.xyz/static/robots.txt(view raw)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
# https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt # https://codeberg.org/bbbhltz/pages/src/branch/master/robots.txt Sitemap: https://olexsmir.xyz/sitemap.xml User-agent: AcademicBotRTU User-agent: AddSearchBot User-agent: adidxbot User-agent: Adsbot User-agent: AdsBot-Google User-agent: AhrefsBot User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: aiHitBot User-agent: AmazonBuyForMe User-agent: atlassian-bot User-agent: amazon-kendra User-agent: Amazonbot User-agent: Andibot User-agent: Anomura User-agent: anthropic-ai User-agent: Applebot User-agent: Applebot-Extended User-agent: AspiegelBot User-agent: Awario User-agent: bedrockbot User-agent: bigsur.ai User-agent: BLEXBot User-agent: BrandVerity/1.0 User-agent: Bravebot User-agent: Brightbot 1.0 User-agent: BuddyBot User-agent: Bytespider User-agent: CCBot User-agent: ChatGPT Agent User-agent: ChatGPT-User User-agent: CheckMarkNetwork/1.0 User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: ClaudeBot User-agent: Cloudflare-AutoRAG User-agent: CloudVertexBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Cotoyogi User-agent: Crawlspace User-agent: Datenbank Crawler User-agent: DeepSeekBot User-agent: Devin User-agent: Diffbot User-agent: dotbot User-agent: DuckAssistBot User-agent: Echobot Bot User-agent: EchoboxBot User-agent: FacebookBot User-agent: facebookexternalhit User-agent: Factset_spyderbot User-agent: FirecrawlAgent User-agent: FriendlyCrawler User-agent: Gemini-Deep-Research User-agent: GenAI User-agent: Google-CloudVertexBot User-agent: Google-Extended User-agent: Google-Firebase User-agent: Google-NotebookLM User-agent: GoogleAgent-Mariner User-agent: GoogleOther User-agent: GoogleOther-Image User-agent: GoogleOther-Video User-agent: GPTBot User-agent: iaskspider/2.0 User-agent: IbouBot User-agent: ICC-Crawler User-agent: ImagesiftBot User-agent: img2dataset User-agent: imgproxy User-agent: ISSCyberRiskCrawler User-agent: Kangaroo Bot User-agent: KlaviyoAIBot User-agent: LinerBot User-agent: Linguee Bot User-agent: Mediapartners-Google User-agent: meta-externalagent User-agent: Meta-ExternalAgent User-agent: meta-externalfetcher User-agent: Meta-ExternalFetcher User-agent: meta-webindexer User-agent: MistralAI-User User-agent: MistralAI-User/1.0 User-agent: MJ12bot User-agent: MyCentralAIScraperBot User-agent: netEstate Imprint Crawler User-agent: NotebookLM User-agent: NovaAct User-agent: NPBot User-agent: OAI-SearchBot User-agent: omgili User-agent: omgilibot User-agent: OpenAI User-agent: Operator User-agent: PanguBot User-agent: Panscient User-agent: panscient.com User-agent: peer39_crawler User-agent: peer39_crawler/1.0 User-agent: Perplexity-User User-agent: PerplexityBot User-agent: PetalBot User-agent: PhindBot User-agent: PiplBot User-agent: Poseidon Research Crawler User-agent: QualifiedBot User-agent: QuillBot User-agent: quillbot.com User-agent: rogerbot User-agent: SBIntuitionsBot User-agent: Scrapy User-agent: SemrushBot User-agent: SemrushBot-FT User-agent: SemrushBot-OCOB User-agent: SemrushBot-SA User-agent: SemrushBot-SWA User-agent: ShapBot User-agent: Sidetrade indexer bot User-agent: SlySearch User-agent: TerraCotta User-agent: Thinkbot User-agent: TikTokSpider User-agent: Timpibot User-agent: TurnitinBot User-agent: ubermetrics User-agent: VelenPublicWebCrawler User-agent: WARDBot User-agent: Webzio-Extended User-agent: wpbot User-agent: YaK User-agent: Yandex User-agent: YandexAdditional User-agent: YandexAdditionalBot User-agent: YouBot User-agent: ZoominfoBot Disallow: / User-agent: * Disallow: |