[WIP] Port SharQ to Python 3, update msgpack handling, and add Redis Cluster compatibility

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Original prompt

Goal: Open a pull request that merges the migration branch into master with the title: "Port SharQ to Python 3, update msgpack handling, and add Redis Cluster compatibility".

Background:

• Repository: flowdacity/flowdacity-queue
• The migration branch contains updates to make the SharQ codebase Python 3 compatible, improve msgpack payload serialization/deserialization to handle legacy pre-Py3 payloads, and add Redis Cluster support via redis-py-cluster.

What to include in the PR:

1. Summary (PR description):

   • Short summary: "Migrate the SharQ codebase to Python 3, fix msgpack bytes/string handling for legacy payloads, and add Redis Cluster compatibility. Update packaging and tests accordingly."
   • Detailed bullet list of key changes:
     • Update setup.py: set python_requires to ">3.5" and add redis-py-cluster dependency.
     • utils.py: add/modify deserialize_payload to correctly handle older msgpack payloads that include surrounding quotes from py2-to-py3 migration; serialize_payload uses msgpack.packb(use_bin_type=True).
     • queue.py: import and support RedisCluster (redis-py-cluster) in addition to StrictRedis; update code paths as necessary for cluster vs non-cluster configs; maintain existing APIs (enqueue, dequeue, finish, requeue, metrics, clear_queue, interval).
     • Tests: ensure tests/test_func.py and other tests run under Python 3 and validate msgpack handling, metrics, enqueue/dequeue behavior, requeue/finish semantics, and clear_queue/purge behavior.
     • Lua scripts: metrics.lua, requeue.lua, finish.lua, interval.lua remain and are used by the Python code (no functional changes expected).
     • Documentation: README and Makefile updated if needed to reflect Python 3 support and test instructions.

2. Testing performed / verification steps to include in the PR description for reviewers:

   • Run unit/functional tests: make test (or python -m tests.test_func, python -m tests.test_queue).
   • Verify serialization: enqueue a job with a payload and ensure deserialize_payload returns the original dict when reading back from Redis (including handling legacy msgpack payloads with surrounding quotes).
   • Verify Redis cluster mode: test with a redis-cluster test environment or mocked cluster to ensure connections via rediscluster.RedisCluster work and Lua scripts continue functioning.
   • Verify metrics sliding window behavior and counters recorded correctly (see metrics.lua and tests).

3. Merge notes / compatibility considerations for reviewers:

   • This is a breaking change for users still on Python 2 — call out that consumers must upgrade to Python 3 (>=3.5).
   • Verify any consumers/producers that write payloads into Redis using older SharQ versions still deserialize correctly (the utils.py change handles surrounding quotes, but reviewers should validate with real data where possible).
   • Ensure deployment and CI are updated to use Python 3 interpreters and include redis-py-cluster in environment.

4. Suggested PR metadata (labels / reviewers):

   • Labels: migration, python3, redis-cluster, breaking-change
   • Reviewers: core maintainers familiar with SharQ and Redis deployments (add repo-maintainers or specific usernames if known)

Request to the code agent:

• Create a pull request titled exactly: "Port SharQ to Python 3, update msgpack handling, and add Redis Cluster compatibility" that merges branch "migration" into base branch "master" for repository flowdacity/flowdacity-queue.
• Use the PR description based on the summary and testing/notes above.
• Do not assign an assignee.
• Add the suggested labels and reviewers metadata where possible.

If additional information is needed (e.g., exact reviewer GitHub usernames or a different base branch), ask in the PR comments rather than blocking the PR creation.

This pull request was created as a result of the following prompt from Copilot chat.

Goal: Open a pull request that merges the migration branch into master with the title: "Port SharQ to Python 3, update msgpack handling, and add Redis Cluster compatibility".

Background:

• Repository: flowdacity/flowdacity-queue
• The migration branch contains updates to make the SharQ codebase Python 3 compatible, improve msgpack payload serialization/deserialization to handle legacy pre-Py3 payloads, and add Redis Cluster support via redis-py-cluster.

What to include in the PR:

1. Summary (PR description):

   • Short summary: "Migrate the SharQ codebase to Python 3, fix msgpack bytes/string handling for legacy payloads, and add Redis Cluster compatibility. Update packaging and tests accordingly."
   • Detailed bullet list of key changes:
     • Update setup.py: set python_requires to ">3.5" and add redis-py-cluster dependency.
     • utils.py: add/modify deserialize_payload to correctly handle older msgpack payloads that include surrounding quotes from py2-to-py3 migration; serialize_payload uses msgpack.packb(use_bin_type=True).
     • queue.py: import and support RedisCluster (redis-py-cluster) in addition to StrictRedis; update code paths as necessary for cluster vs non-cluster configs; maintain existing APIs (enqueue, dequeue, finish, requeue, metrics, clear_queue, interval).
     • Tests: ensure tests/test_func.py and other tests run under Python 3 and validate msgpack handling, metrics, enqueue/dequeue behavior, requeue/finish semantics, and clear_queue/purge behavior.
     • Lua scripts: metrics.lua, requeue.lua, finish.lua, interval.lua remain and are used by the Python code (no functional changes expected).
     • Documentation: README and Makefile updated if needed to reflect Python 3 support and test instructions.

2. Testing performed / verification steps to include in the PR description for reviewers:

   • Run unit/functional tests: make test (or python -m tests.test_func, python -m tests.test_queue).
   • Verify serialization: enqueue a job with a payload and ensure deserialize_payload returns the original dict when reading back from Redis (including handling legacy msgpack payloads with surrounding quotes).
   • Verify Redis cluster mode: test with a redis-cluster test environment or mocked cluster to ensure connections via rediscluster.RedisCluster work and Lua scripts continue functioning.
   • Verify metrics sliding window behavior and counters recorded correctly (see metrics.lua and tests).

3. Merge notes / compatibility considerations for reviewers:

   • This is a breaking change for users still on Python 2 — call out that consumers must upgrade to Python 3 (>=3.5).
   • Verify any consumers/producers that write payloads into Redis using older SharQ versions still deserialize correctly (the utils.py change handles surrounding quotes, but reviewers should validate with real data where possible).
   • Ensure deployment and CI are updated to use Python 3 interpreters and include redis-py-cluster in environment.

4. Suggested PR metadata (labels / reviewers):

   • Labels: migration, python3, redis-cluster, breaking-change
   • Reviewers: core maintainers familiar with SharQ and Redis deployments (add repo-maintainers or specific usernames if known)

Request to the code agent:

• Create a pull request titled exactly: "Port SharQ to Python 3, update msgpack handling, and add Redis Cluster compatibility" that merges branch "migration" into base branch "master" for repository flowdacity/flowdacity-queue.
• Use the PR description based on the summary and testing/notes above.
• Do not assign an assignee.
• Add the suggested labels and reviewers metadata where possible.

If additional information is needed (e.g., exact reviewer GitHub usernames or a different base branch), ask in the PR comments rather than blocking the PR creation.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.