nrobots doesn't handle entries like /?/ properly

[sirjimjones]

nrobots doesn't handle entries like "Disallow: /?/" properly.

[sjdirect]

Looks like NRobots is converting the "Disallow: /?/" into "Disallow: /". The last checkin to that lib fixed a similar issue but obviously this presents other issues.

[sjdirect]

Even though this bug still exists in the nrobots lib, abot gives a workaround to this and similar issues with robots.txt preventing the crawl. See sjdirect/abot@9bd3d7d
for details. There is now an isIgnoreRobotsDotTextIfRootDisallowedEnabled config value that if set to true will ignore the robots.txt file for when the root uri of the crawl is disallowed.