[Feat] Support automatic startup of `Yuanrong` for `transfer_queue.init()`

[dpj135]

Description

Support automatic startup of Yuanrong for transfer_queue.init().

Plan

• Add automatic startup of etcd and yuanrong-datasystem with retries.
• Add Yuanrong to options of test_e2e_lifecycle.
• Add add TQ_TEST_BACKEND=Yuanrong pytest tests/e2e/test_e2e_lifecycle_consistency.py to github ci actions.

[tianyi-ge]

Current steps to start a yuanrong perftest

For details, refer to README_PERFTEST.md

1. download transferqueue：TransferQueue
2. download yuanrong datasystem：
   1. 3/30-py311-arm
   2. 3/30-py310-arm
3. download etcd：
   1. arm

Head node scripts

export HEAD_NODE_IP=10.170.27.237
export WORKER_NODE_IP=10.170.27.158
etcd --name etcd-single \
     --data-dir /tmp/etcd-data \
     --listen-client-urls http://${HEAD_NODE_IP}:2379 \
     --advertise-client-urls http://${HEAD_NODE_IP}:2379 \
     --listen-peer-urls http://${HEAD_NODE_IP}:2390 \
     --initial-advertise-peer-urls http://${HEAD_NODE_IP}:2390 \
     --initial-cluster etcd-single=http://${HEAD_NODE_IP}:2390 &> /dev/null &

cd TransferQueue
pip uninstall TransferQueue -y
pip install -e .
ASCEND_RT_VISIBLE_DEVICES=0 ray start --head \
     --resources='{"node:'${HEAD_NODE_IP}'":1}'

export DS_H2D_MEMCPY_POLICY=direct
export DS_D2H_MEMCPY_POLICY=direct
export SHARED_MEMORY_SIZE_MB=65536
echo $((${SHARED_MEMORY_SIZE_MB}/2)) > /proc/sys/vm/nr_hugepages
rm -rf datasystem/
dscli start --timeout 600 \
     -w --worker_address ${HEAD_NODE_IP}:31501 \
     --etcd_address ${HEAD_NODE_IP}:2379 \
     --arena_per_tenant 1 \
     --max_log_size 1024 \
     --max_log_file_num 10 \
     --node_timeout_s 600 \
     --node_dead_timeout_s 1800 \
     --enable_fallocate false \
     --enable_worker_worker_batch_get true \
     --shared_memory_populate true \
     --shared_memory_size_mb ${SHARED_MEMORY_SIZE_MB} \
     --remote_h2d_device_ids "0" \
     --enable_huge_tlb true

# wait until worker-node datasystem is ready
sleep 30
cd TransferQueue/scripts/performance_test
sed -i '/^BACKENDS=/s/"SimpleStorage" "Yuanrong" "MooncakeStore"/"Yuanrong"/' run_perf_test.sh
sed -i '/# ---- Ray baseline ----/i\exit' run_perf_test.sh
DEVICE=npu bash run_perf_test.sh

Worker node scripts

export HEAD_NODE_IP=10.170.27.237
export WORKER_NODE_IP=10.170.27.158
cd TransferQueue
pip uninstall TransferQueue -y
pip install -e .
ASCEND_RT_VISIBLE_DEVICES=0 ray start --address="${HEAD_NODE_IP}:6379" \
     --resources='{"node:'${WORKER_NODE_IP}'":1}'

export DS_H2D_MEMCPY_POLICY=direct
export DS_D2H_MEMCPY_POLICY=direct
export SHARED_MEMORY_SIZE_MB=65536
echo $((${SHARED_MEMORY_SIZE_MB}/2)) > /proc/sys/vm/nr_hugepages
rm -rf datasystem/
dscli start --timeout 600 \
     -w --worker_address ${WORKER_NODE_IP}:31501 \
     --etcd_address ${HEAD_NODE_IP}:2379 \
     --arena_per_tenant 1 \
     --max_log_size 1024 \
     --max_log_file_num 10 \
     --node_timeout_s 600 \
     --node_dead_timeout_s 1800 \
     --enable_fallocate false \
     --enable_worker_worker_batch_get true \
     --shared_memory_populate true \
     --shared_memory_size_mb ${SHARED_MEMORY_SIZE_MB} \
     --remote_h2d_device_ids "0" \
     --enable_huge_tlb true

check csv under TransferQueue/scripts/performance_test/results

[tianyi-ge]

Remaining tasks:

1. now only support single-node (each tq.init creates an etcd process)
2. host field is not necessary, because config is designed to be identical for all clients. host is different on different nodes
3. refactor to avoid too long functions