Document consuming the Python implementation

[kieran-ryan]

🤔 What's the problem you've observed?

Require appropriate documentation for consuming the Python implementation appropriately to make it easy to adopt and ensure is used in the most effective manner.

✨ Do you have a proposal for making it better?

Once the package has been published, to define an optimal implementation while replacing respective code within the Cucumber Messages Python package tests and update the respective documentation in the compatibility kit accordingly.

📚 Any additional context?

No response

[luke-hill]

Ruby example is here

# spec/my_compatibility_checks_spec.rb
require 'cucumber/compatibility_kit'

describe Cucumber::CompatibilityKit do
  let(:cucumber_command) { 'bundle exec cucumber --publish-quiet --profile none --format message' }

  # Don't run the retry or skipped CCK Examples (For whatever reason)
  examples = Cucumber::CompatibilityKit.gherkin.reject { |example| example == 'retry' || example == 'skipped' }

  examples.each do |example_name|
    describe "'#{example_name}' example" do
      include_examples 'cucumber compatibility kit' do
        let(:example) { example_name }
        # You will need to specify the relative support code and cck paths
        let(:messages) { `#{cucumber_command} --require #{support_code_path} #{cck_path}` }
      end
    end
  end
end

In Pseudo code what we do is essentially.

1. Determine the "name" of all examples. This should lean on the .gherkin method inside your CCK.
2. Define your cucumber execution command (NB: This will be v diff to normal ones, because you'll need to provide an abstract location for your tests AND an abstract location for your support/step defs)
   2. a NB: The above ones likely need defining lower down the loop because they will change each time with each test
3. Define your executed and dumped message command (Wrapping the cucumber command and the support/step def code)

Now we can move onto our shared test suite. This is implemented inside cucumber-ruby

RSpec.shared_examples 'cucumber compatibility kit' do
  include CCK::Helpers

  let(:cck_path) { CCK::Examples.feature_code_for(example) }

  let(:parsed_original) { parse_ndjson_file("#{cck_path}/#{example}.ndjson") }
  let(:parsed_generated) { parse_ndjson(messages) }

  let(:original_messages_types) { parsed_original.map { |msg| message_type(msg) } }
  let(:generated_messages_types) { parsed_generated.map { |msg| message_type(msg) } }

  it 'generates valid message types' do
    expect(generated_messages_types).to contain_exactly(*original_messages_types)
  end

  it 'generates valid message structure' do
    comparator = CCK::MessagesComparator.new(parsed_generated, parsed_original)

    expect(comparator.errors).to be_empty, "There were comparison errors: #{comparator.errors}"
  end

  it 'ensures a consistent `testRunStartedId` across the entire test run' do
    test_run_started_id = parsed_generated.detect { |msg| message_type(msg) == :test_run_started }.test_run_started.id
    ids = parsed_generated.filter_map do |msg|
      # These two types of message are the only ones containing the testRunStartedId attribute
      msg.send(message_type(msg)).test_run_started_id if %i[test_case test_run_finished].include?(message_type(msg))
    end

    expect(ids).to all eq(test_run_started_id)
  end
end

Here in pseudo code we do the following

1. cck_path is already dynamic, so we're good here, it's used previously
2. Define two NDJSON objects to be compared. A Source (Golden), one and a generated one from your tests
   2. a you'll notice the parsed_generated will read the messages output from stdout directly feeding it into the parse_ndjson method (Which takes either a file or a live input)
3. I have 2 further sub-method which just scans the message types (As sort of a "mini-test"), this is up to you dependent on how many tests you want to write for your CCK tests proper.
4. Until v18, we had 2 tests. One checking top level key, one checking nested keys (NB: You never check values). However now we have 3 tests, a third one which checks the value of the test_run_started_id message that is propagated marries up throughout the entire run. NB: This is only valid in 2 child messages, which I document

[mpkorstanje]

Once the package has been published, to define an optimal implementation while replacing respective code within the Cucumber Messages Python package tests and update the respective documentation in the compatibility kit accordingly.

No. We can not test messages against the CCK. That creates a circular dependency. Additionally, this would only tests if we can round trip messages. A potential solution to ensure round-trips work would be to have a few interesting handcrafted examples in a testdata folder (cucumber/messages#110).

[luke-hill]

We have a little bit of this left in the ruby ecosystem. We removed most of the circular dependencies in html_formatter. The only remaining one is in the messages serialization tests

We use the entire CCK (Albeit v15 at the time), and check that

1. The file exists? (duh, kind of a pointless test, and likely semi-covered in the CCK)
2. Unwrapping then re-wrapping it generates the same file (Which maybe has "some" value, but we don't need to test it 15+ times, testing it once or twice is probably sufficient).