You can find this section under Security > Potential Crawlers in the IRP Admin left navigation menu.
The PotentialCrawlers.aspx page lists any IP addresses that may be Crawlers according to the values of the Application Settings that are described below in the section called Crawler Application Settings. The main grid shows all of the IP addresses that have gone over the value of the 'Crawler Page Views Per Minute Threshold' Application Setting:
From here, you can either ignore the IP addresses, delete them or add as them as Crawlers (and also exclude them from the Insights functionality).
All of the sessions belonging to the IP address that was flagged as a Potential Crawler will be visible on the PotentialCrawlerActivity.aspx page. To view this page, click the View Activity button as shown in the screen capture above:
You can then click the Page Views button for each session to give insight into whether or not the Potential Crawler is in fact a Crawler:
Crawler Application Settings
The Application Setting 'Crawler Page Views Per Minute Threshold' (in the 'Security' section) considers any customer who views this many pages or more per minute as a Potential Crawler. This value is 60 by default. For example, a value of 60 means 60 page views per minute or one view every second. Ensure that this value is high enough to avoid false positives, such as customers opening several tabs at once. Any customers viewing the site at a rate equal to or faster than the provided value will either be flagged as a Potential Crawler or automatically added as a Crawler depending on the value of the Application Setting called 'Automatically Exclude Suspected Crawlers From Insights' (also in the 'Security' section).
Note that you can also use the 'Automatically Exclude Potential Crawlers' button at the top of the PotentialCrawlers.aspx page to fulfil the same function as the 'Automatically Exclude Suspected Crawlers From Insights' Application Setting. You can also use the 'Check For Potential Crawlers' button at the top of the PotentialCrawlers.aspx page to perform a check right away.
When the 'Automatically Exclude Suspected Crawlers From Insights' Application Setting is enabled, any IP addresses that are detected viewing pages above the provided "Crawler Page Views Per Minute Threshold" Application Setting value will automatically be added as Crawlers and excluded from Insights without having to be reviewed on the PotentialCrawlers.aspx page. When disabled, any suspected crawlers will be listed on the PotentialCrawlers.aspx page and will need to be reviewed and manually added to the list of Crawlers.
