Read an unavailable robots.txt as the standard does, not as a refusal - #32

Read an unavailable robots.txt as the standard does, not as a refusal#32

Badri-Rajendran merged 3 commits into

patch_docs_truth_upBadri-Rajendran/PolicyPal:patch_docs_truth_upfrom

feature_robots_unavailableBadri-Rajendran/PolicyPal:feature_robots_unavailableCopy head branch name to clipboard