Support for multiple sections

[Jeavon]

I have been thinking about how to extend this useful package to support multiple sections, for example:

User-agent: PowerMapper
Allow: /

User-agent: SemrushBot
Allow: /

User-Agent: *
Disallow: /

I appreciate it's a relatively niche requirement so was thinking about adding app setting Umbraco.Robots.Advanced which could contain JSON such as [{ua:'PowerMapper',allow:'/'},{ua:'SemrushBot',allow:'/'},{ua:'*',disallow:'/'}]
When Umbraco.Robots.Advanced has a value, Umbraco.Robots.UserAgent, Umbraco.Robots.Disallow & Umbraco.Robots.Allow would be ignored. Umbraco.Robots.Sitemaps would continue to be used.

Just an idea, maybe there is a better approach... 🤔

[callumbwhyte]

Thanks @Jeavon, I've been dreading someone bringing this up...! 😂

There's certainly value in supporting this, and it's something I've thought about. Keeping the configuration nice and simple has always been the sticking point - this will get so much easier with V9 thankfully.

What about picking up the app settings by convention: Umbraco.Robots.{UserAgent}.{Property} e.g. Umbraco.Robots.PowerMapper.Allow. You could then keep the existing Umbraco.Robots.Allow etc for the catch-all * useragent.

Perhaps making this configurable in code would make sense as an initial step, won't be too hard...

[Jeavon]

@callumbwhyte Sorry! I also thought about app settings with a naming convention, the order is important and I'm not sure if the order of app settings is maintained when iterating through in C#?

[callumbwhyte]

@Jeavon When you say "order is important", do you mean the order the definitions are rendered in the robots.txt or the order they are configured in?

If it's the former, I assume the only requirement is the catch-all * to come last? In which case, we just need to ensure that is added last when we build the configuration object.

If it's the latter and order of config matters, care to share why?

[Jeavon]

@callumbwhyte there is a convention for bots that they evaluate the sections in order and stop when they hit a match, I can't think of a scenario when it matters apart from as you say that the catchall is last, so sounds good to me!