Crawling Settings for Help Center (Allow AI | Bot Access)
Help Center pages currently block AI agents and other automated crawlers, causing errors when tools attempt to fetch or index help content.
It would be helpful to have configurable crawling settings - such as allow-list for specific bots, toggling public indexing, or adjusting robots.txt behavior - so teams can enable AI tools, search engines, or internal automation to access their Help Center pages as needed.
- Post type
- 💡 New feature
Log in to comment and vote
Comments1
aimr
Jan 5
Yes it's very annoying especially on your website because I can't have Claude go and research your documents