Forum Moderators: open
robots.txt: no
IP: 47.128.various
UA: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36;TikTokSpider;ttspider-feedback@tiktok.com
UA: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)
to grab robots.txt with the user-agent Go-http-client/2.0 for some reasonI wish the next passing botrunner would explain the reason, because 99 times out of 100, giving a different UA just for robots.txt can only be called counterproductive. On my site, a handful of factors--one of them being bad_agent--nets you a generic
User-Agent: *
Disallow: /where a truthful UA might have won permission to enter.