From 4e85f09ac406b8683d7daf2ef3b395a7765d016c Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Tue, 1 Oct 2024 03:56:09 +0200 Subject: [PATCH 1/7] Write first Software Architecture Draft --- SoftwareArchitecture.md | 80 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 SoftwareArchitecture.md diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md new file mode 100644 index 0000000..b39485e --- /dev/null +++ b/SoftwareArchitecture.md @@ -0,0 +1,80 @@ +# Software Architecture + +This chart is a high-level overview of the software architecture of the project. + +_It is a work in progress and will be updated as the project progresses._ + +```mermaid +flowchart TB + subgraph D["Data & Analytics"] + S["S3-Storage"] + + DS[" + Data Scrapers + Active Matches, HLTV-Stream, Metadata + "] + DS -- Match Data --> S + DS -- Add to Validate Queue --> MBV + DS -- Publish Events --> MBPS + + H["Housekeeper"] + H == Scan for unprocessed old Matches === S + H == Compress and Archive old Matches === S + H -- Add to Validate Queue --> MBV + + subgraph MB["Message Broker"] + MBV["Validate Queue"] + MBI["Ingest Queue (Exchange)"] + MBPS["Pub/Sub"] + end + + V["Validator & Parser"] + MBV -- Get from Validate Queue --> V + S -- Match Data --> V + V -- Parsed Match Data --> S + V -- Add to Parser Queue --> MBI + + subgraph ingest["Ingest Workers"] + IP["DBMS 1 Ingest Worker"] + IC["DBMS 2 Ingest Worker"] + end + + subgraph databases["Databases"] + DBP["DBMS 1"] + DBC["DBMS 2"] + end + + MBI -- Get from Ingest Queue --> IC + MBI -- Get from Ingest Queue --> IP + S -- Parsed Match Data --> IP + S -- Parsed Match Data --> IC + IC -- Ingest Data --> DBC + IP -- Ingest Data --> DBP + end + + subgraph T["Telemetry & Metrics"] + %% Not designed yet + P["Prometheus"] + G["Grafana"] + P -- Metrics --> G + end + + subgraph B["Backend"] + %% Not designed yet + end + DBC -- Application Data --> B + DBP -- Application Data --> B + + subgraph AP["Admin Panel"] + %% Not designed yet + end + B -- Application Data --> AP + + subgraph F["Public Frontend"] + %% Not designed yet + end + B -- Application Data --> F + + SU["Live Event Subscribers"] + MBPS == Subscribe ==>SU +``` From b372caaf3c629f88ce32b7836ead3009e63249e1 Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Wed, 2 Oct 2024 00:06:14 +0200 Subject: [PATCH 2/7] Update Software Architecture Draft --- SoftwareArchitecture.md | 96 +++++++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 42 deletions(-) diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md index b39485e..a694258 100644 --- a/SoftwareArchitecture.md +++ b/SoftwareArchitecture.md @@ -4,9 +4,13 @@ This chart is a high-level overview of the software architecture of the project. _It is a work in progress and will be updated as the project progresses._ +# Data & Analytics + +## Overview ```mermaid flowchart TB subgraph D["Data & Analytics"] + direction TB S["S3-Storage"] DS[" @@ -14,67 +18,75 @@ flowchart TB Active Matches, HLTV-Stream, Metadata "] DS -- Match Data --> S - DS -- Add to Validate Queue --> MBV - DS -- Publish Events --> MBPS + DS -- Add to Validate Queue --> RMQV + DS -- Publish Events --> RMQPS H["Housekeeper"] H == Scan for unprocessed old Matches === S H == Compress and Archive old Matches === S - H -- Add to Validate Queue --> MBV + H -- Add to Validate Queue --> RMQV - subgraph MB["Message Broker"] - MBV["Validate Queue"] - MBI["Ingest Queue (Exchange)"] - MBPS["Pub/Sub"] + subgraph RMQ["RabbitMQ"] + RMQV["Validate Queue"] + RMQI["Ingest Queue (Exchange)"] + RMQPS["Pub/Sub"] end V["Validator & Parser"] - MBV -- Get from Validate Queue --> V + RMQV -- Get from Validate Queue --> V S -- Match Data --> V V -- Parsed Match Data --> S - V -- Add to Parser Queue --> MBI + V -- Add to Parser Queue --> RMQI - subgraph ingest["Ingest Workers"] - IP["DBMS 1 Ingest Worker"] - IC["DBMS 2 Ingest Worker"] + subgraph I["Ingest Workers"] + direction TB + I1["DBMS 1 Ingest Worker"] + I2["DBMS 2 Ingest Worker"] end + RMQI -- Get from Ingest Queue --> I + S -- Parsed Match Data --> I - subgraph databases["Databases"] - DBP["DBMS 1"] - DBC["DBMS 2"] + subgraph DB["Databases"] + direction TB + DB1["DBMS 1"] + DB2["DBMS 2"] end - - MBI -- Get from Ingest Queue --> IC - MBI -- Get from Ingest Queue --> IP - S -- Parsed Match Data --> IP - S -- Parsed Match Data --> IC - IC -- Ingest Data --> DBC - IP -- Ingest Data --> DBP - end - - subgraph T["Telemetry & Metrics"] - %% Not designed yet - P["Prometheus"] - G["Grafana"] - P -- Metrics --> G + I -- Ingest Data --> DB end - subgraph B["Backend"] - %% Not designed yet + E["External"] + DB == Application Data ==> E + D == Telemetry & Metrics ==> E + RMQPS == Game Events ==> E +``` + +## Data Scrapers + +### HLTV-Spectator +```mermaid +flowchart TB + subgraph M["Match Manager"] + direction TB + DB["Database"] + MM["Match Manager"] + MM === DB end - DBC -- Application Data --> B - DBP -- Application Data --> B - subgraph AP["Admin Panel"] - %% Not designed yet + subgraph SB["Spectator Bots"] + direction TB + SB1["Spectator Bot 1"] + SB2["Spectator Bot 2"] end - B -- Application Data --> AP - - subgraph F["Public Frontend"] - %% Not designed yet + + subgraph RMQ["RabbitMQ"] + direction BT + EQ["Error Queue"] + SQ["Spectate Queue"] + SQ -- NACK --> EQ end - B -- Application Data --> F - SU["Live Event Subscribers"] - MBPS == Subscribe ==>SU + M -- Matches to Spectate --> RMQ + RMQ -- Match to Spectate --> SB + SB -- ACK / NACK --> RMQ + RMQ -- Failed Matches --> M ``` From 55f768ae9b74dd1916c2c073f7c54fe0d0e40e74 Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Tue, 8 Oct 2024 06:51:04 +0200 Subject: [PATCH 3/7] Add User Ingest Chart --- SoftwareArchitecture.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md index a694258..e641c28 100644 --- a/SoftwareArchitecture.md +++ b/SoftwareArchitecture.md @@ -90,3 +90,22 @@ flowchart TB SB -- ACK / NACK --> RMQ RMQ -- Failed Matches --> M ``` + +### User-Ingest Server +```mermaid +flowchart TB + U["User"] + subgraph UI["User-Ingest Server"] + UIA["User-Ingest API"] + VR["Valve Replay Server"] + UIA -- Match Metadata (Salts) --> VR + VR -- Match Data --> UIA + end + U == Match Metadata (Salts) ==> UIA + + S["S3-Storage"] + UIA -- Match Data ---> S + + RMQV["Validate Queue"] + UIA -- Match Data ---> RMQV +``` From 1b9b2d16c0d3fb22c0107240ddcb6b163fbab0df Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Tue, 8 Oct 2024 07:03:17 +0200 Subject: [PATCH 4/7] Update User Ingest Chart --- SoftwareArchitecture.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md index e641c28..9233513 100644 --- a/SoftwareArchitecture.md +++ b/SoftwareArchitecture.md @@ -95,14 +95,14 @@ flowchart TB ```mermaid flowchart TB U["User"] - subgraph UI["User-Ingest Server"] - UIA["User-Ingest API"] - VR["Valve Replay Server"] - UIA -- Match Metadata (Salts) --> VR - VR -- Match Data --> UIA - end + UIA["User-Ingest API"] + U == Match Metadata (Salts) ==> UIA + VR["Valve Replay Server"] + UIA -- Match Metadata (Salts) --> VR + VR -- Match Data --> UIA + S["S3-Storage"] UIA -- Match Data ---> S From 6b81e6b6dcc6f6db99a5369103c7df4723801843 Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Tue, 8 Oct 2024 08:15:57 +0200 Subject: [PATCH 5/7] Update HLTV-Spectator Chart --- SoftwareArchitecture.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md index 9233513..ce9170b 100644 --- a/SoftwareArchitecture.md +++ b/SoftwareArchitecture.md @@ -64,9 +64,9 @@ flowchart TB ### HLTV-Spectator ```mermaid -flowchart TB +flowchart BT subgraph M["Match Manager"] - direction TB + direction LR DB["Database"] MM["Match Manager"] MM === DB @@ -78,17 +78,15 @@ flowchart TB SB2["Spectator Bot 2"] end - subgraph RMQ["RabbitMQ"] + subgraph E["External"] direction BT - EQ["Error Queue"] SQ["Spectate Queue"] - SQ -- NACK --> EQ end - M -- Matches to Spectate --> RMQ - RMQ -- Match to Spectate --> SB - SB -- ACK / NACK --> RMQ - RMQ -- Failed Matches --> M + M -- Matches to Spectate --> SQ + SQ -- Match to Spectate --> SB + SB -- ACK / NACK --> SQ + SQ -- Failed Matches --> M ``` ### User-Ingest Server @@ -103,9 +101,11 @@ flowchart TB UIA -- Match Metadata (Salts) --> VR VR -- Match Data --> UIA - S["S3-Storage"] + + subgraph E["External"] + S["S3-Storage"] + RMQV["Validate Queue"] + end UIA -- Match Data ---> S - - RMQV["Validate Queue"] UIA -- Match Data ---> RMQV ``` From 62b68723bd09afbe04db43e53f1f06d5fbe07624 Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Wed, 9 Oct 2024 12:18:44 +0200 Subject: [PATCH 6/7] Merge Validator and Parser and Ingest Workers --- SoftwareArchitecture.md | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md index ce9170b..9f34824 100644 --- a/SoftwareArchitecture.md +++ b/SoftwareArchitecture.md @@ -18,40 +18,29 @@ flowchart TB Active Matches, HLTV-Stream, Metadata "] DS -- Match Data --> S - DS -- Add to Validate Queue --> RMQV + DS -- Add to Ingest Queue --> RMQI DS -- Publish Events --> RMQPS H["Housekeeper"] - H == Scan for unprocessed old Matches === S - H == Compress and Archive old Matches === S - H -- Add to Validate Queue --> RMQV + H == Scan for unprocessed old Matches
Compress and Archive old Matches === S + H -- Add to Ingest Queue --> RMQI subgraph RMQ["RabbitMQ"] - RMQV["Validate Queue"] - RMQI["Ingest Queue (Exchange)"] + RMQI["Ingest Queue"] RMQPS["Pub/Sub"] end - V["Validator & Parser"] - RMQV -- Get from Validate Queue --> V + V["Validator & Parser & Ingestor"] + RMQI -- Get from Ingest Queue --> V S -- Match Data --> V V -- Parsed Match Data --> S - V -- Add to Parser Queue --> RMQI - - subgraph I["Ingest Workers"] - direction TB - I1["DBMS 1 Ingest Worker"] - I2["DBMS 2 Ingest Worker"] - end - RMQI -- Get from Ingest Queue --> I - S -- Parsed Match Data --> I subgraph DB["Databases"] direction TB DB1["DBMS 1"] DB2["DBMS 2"] end - I -- Ingest Data --> DB + V -- Ingest Data --> DB end E["External"] @@ -104,8 +93,8 @@ flowchart TB subgraph E["External"] S["S3-Storage"] - RMQV["Validate Queue"] + RMQI["Ingest Queue"] end UIA -- Match Data ---> S - UIA -- Match Data ---> RMQV + UIA -- Match Data ---> RMQI ``` From 374cf0a6734df1c956425b30193cf8c237858208 Mon Sep 17 00:00:00 2001 From: Manuel Raimann Date: Wed, 9 Oct 2024 12:29:47 +0200 Subject: [PATCH 7/7] Put Databases in DB Shape --- SoftwareArchitecture.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SoftwareArchitecture.md b/SoftwareArchitecture.md index 9f34824..069b816 100644 --- a/SoftwareArchitecture.md +++ b/SoftwareArchitecture.md @@ -37,8 +37,8 @@ flowchart TB subgraph DB["Databases"] direction TB - DB1["DBMS 1"] - DB2["DBMS 2"] + DB1[("DBMS 1")] + DB2[("DBMS 2")] end V -- Ingest Data --> DB end