There are several ways of protecting your test/dev environment.
- Password protection: In this case you will need to provide Nosto support with the environment credentials and they can set them up to the crawler. This way, the crawler is not being denied access anymore and can freely enter the store.
- If the firewall is based on IP rules: there is a solution, but it's not scalable and it eludes the purpose of IP protection. Currently our application servers are hosted in AWS US-EAST-1 region, so you can whitelist that. Please find more info here:
In case you need to add for instance a specific rule for bots similarly to search engine spiders, Nosto’s crawler uses following crawler header. This is especially needed in case you need to whitelist Nosto’s crawler in order to allow it to access directly your product pages.
Mozilla/5.0 (compatible; NostoCrawlerBot/1.0; +http://my.nosto.com/tagging)