Findologic uses the following tools to detect bot requests:
First, we use a growing blacklist of bot addresses. This blacklist also contains the addresses of Google, Bing and other benign bots. These are automatically excluded from billing.
In addition, conspicuous search queries are automatically excluded from billing. Among other things, the queries are searched for the following patterns:
The number of individual search terms differs greatly from the remaining queries.
The same query is often sent within a very short time.
Individual IP addresses cause a large number of search queries.
What can be done if there are nevertheless more bot requests, which are not recognised as such by Findologic?
As a first step, we recommend our customers to adapt the robots.txt of the shop. More information can be found here.
If adjusting the robots.txt did not help, you can request the IP addresses of the requests at support@findologic.com. With this information, the addresses of the bots can be blocked. We really only recommend blocking IP addresses if the attacks always come from the same addresses and these are clearly recognisable as bots.
Services such as Cloudflare have been shown to provide reliable protection against bot requests.
By integrating our search with direct integration, bot requests are reliably prevented. Since the search and the results are completely integrated using an external JavaScript with direct integration, there are hardly any bot queries with direct integration.