There are several ways of protecting your test/dev environment.
Password protection, Nosto's crawler supports HTTP Basic authentication. In this case you will need to provide Nosto support with the environment credentials and they can set them up to the crawler. This way, the crawler is not being denied access anymore and can freely enter the store.
IP rules based whitelisting. Nosto's crawler requests will come from IP address ranges listed at https://api.nosto.com/meta under the property "crawler". The IP addresses are not expected to be shifting constantly, but can change from time to time.
In case you need to add for instance a specific rule for bots similarly to search engine spiders, Nosto’s crawler uses following crawler header. This is especially needed in case you need to whitelist Nosto’s crawler in order to allow it to access directly your product pages.
Mozilla/5.0 (compatible; NostoCrawlerBot/1.0; +http://my.nosto.com/tagging)