Adjust docs tooling for integration plugins

[karenzone]

Related:

• New snmp-integration docs not being picked up for Logstash Reference (LSR) #106
• Add incomplete integration plugins to metadata.json as skip logstash#16174

Change to processing logic for integration plugins.

We have a situation in which we are moving an integration plugin into Technical Preview before we bundle it with Logstash by default. This means that we need the ability to generate integration plugin docs while the plugin status is default-plugins: false.

We've prepared for this change by adding the two impacted plugins (logstash-integration-internal and logstash-integration-zeromq) to plugins-metadata.json, and setting their values to default-plugins: false and "skip-list": true.

.

[karenzone]

@jsvd @mashhurs Here's the proposed fix to docs-tools. Please check it out.

@robbavey Will you please take a look at this to ensure that we're achieving your goals and not violating your design intent in #40?

In order to avoid overwriting individual plugins for releases that
are capable of running integration plugins, only write doc
entries for integration plugins that are marked as default.

[jsvd]

LGTM

[jsvd]

if we merge this then we'll have to backport elastic/logstash#16174 (review) to all release threads still producing builds

[robbavey]

@karenzone Questions from Slack...

What does this do for integration plugins that are in development, when we still want to preserve the originals?
Will this overwrite the existing SNMP docs for the plugins that are bundled at the moment, and for older versions of Logstash
with the integration docs?
I'm also assuming that even if we had an additional "tech preview" or "beta" state, we can’t have both integration and stand alone versions of docs co-exist for the same version?

[karenzone]

What does this do for integration plugins that are in development, when we still want to preserve the originals?
Will this overwrite the existing SNMP docs for the plugins that are bundled at the moment, and for older versions of Logstash
with the integration docs?
I'm also assuming that even if we had an additional "tech preview" or "beta" state, we can’t have both integration and stand alone versions of docs co-exist for the same version?

Once again, this is a case where the "plugin-stack version mismatch" bites us. For now, I am managing this manually by allowing in the new SNMP integration page v4.0.0 at Technical Preview in, and keeping the branch-appropriate versions of the stand-alone input-snmp and input-snmptrap. Obviously this is less than ideal, but it's the best I've come up with so far.

Is there a way that we can make our plugins and our docs-tooling branch aware beyond gemlock files?