diff --git a/.gitignore b/.gitignore index cf99758..7bf5fba 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,4 @@ /target -Justfile \ No newline at end of file +Justfile +/commit.sh +/GO.sh diff --git a/README.md b/README.md index b370cdb..17eb7d4 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,34 @@ Hulykvs is a simple key-value store service implemented in Rust. It uses cockroachdb as the backend and provides a simple http api for storing and retrieving key-value pairs. -## API +## API v2 +Create a key-value pair api + +```POST /api2/{workspace}/{namespace}/{key}``` +Stores request payload as the value for the given key in the given namespace. Existing keys will be overwritten. Returs 204 (NoContent) on sucesss. + + +```GET /api2/{workspace}/{namespace}/{key}``` +Retrieves the value for the given key in the given namespace. Returns 404 if the key does not exist. + + +```DELETE /api2/{workspace}/{namespace}/{key}``` +Deletes the key-value pair for the given key in the given namespace. Returns 404 if the key does not exist, 204 (NoContent) on success, 404 if the key does not exist. + + +```GET /api2/{workspace}/{namespace}?[prefix=]``` +Retrieves all key-value pairs in the given namespace. Optionally, a prefix can be provided to filter the results. The following structure is returned: +```json +{ + "workspace": "workspace", + "namespace": "namespace", + "count": 3, + "keys": ["key1", "key2", "keyN"] +} +``` +## API (old) +workspace = "defaultspace" + Create a key-value pair ```POST /api/{namespace}/{key}``` @@ -20,14 +47,14 @@ Deletes the key-value pair for the given key in the given namespace. Returns 404 Retrieves all key-value pairs in the given namespace. Optionally, a prefix can be provided to filter the results. The following structure is returned: ```json { - "namespace": "namespace", + "namespace": "namespace", "count": 3, "keys": ["key1", "key2", "keyN"] } ``` - + ## Running -Pre-build docker images is available at: hardcoreeng/service_hulykvs:{tag}. +Pre-build docker images is available at: hardcoreeng/service_hulykvs:{tag}. You can use the following command to run the image locally: ```bash diff --git a/commmit.sh b/commmit.sh new file mode 100755 index 0000000..e04b6e3 --- /dev/null +++ b/commmit.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +clear + +#git checkout -b feature/workspace-support +git add . +git commit -m "Add API v2 with workspace" + +exit + + +git push origin feature/workspace-support \ No newline at end of file diff --git a/hulykvs/hulykvs b/hulykvs/hulykvs new file mode 120000 index 0000000..20a8ba0 --- /dev/null +++ b/hulykvs/hulykvs @@ -0,0 +1 @@ +/home/work/huly.io/hulykvs/target/release/hulykvs \ No newline at end of file diff --git a/hulykvs_server/etc/migrations/V2__add_workspace.sql b/hulykvs_server/etc/migrations/V2__add_workspace.sql new file mode 100644 index 0000000..9866f4b --- /dev/null +++ b/hulykvs_server/etc/migrations/V2__add_workspace.sql @@ -0,0 +1,19 @@ +-- 1. Create new +CREATE TABLE kvs_new ( + workspace TEXT NOT NULL, + namespace TEXT NOT NULL, + key TEXT NOT NULL, + md5 BYTES NOT NULL, + value BYTES NOT NULL, + PRIMARY KEY (workspace, namespace, key) +); + +-- 2. Copy +INSERT INTO kvs_new (workspace, namespace, key, md5, value) +SELECT 'defaultspace', namespace, key, md5, value FROM kvs; + +-- 3. Del +DROP TABLE kvs; + +-- 4. Rename +ALTER TABLE kvs_new RENAME TO kvs; diff --git a/hulykvs_server/src/handlers.rs b/hulykvs_server/src/handlers.rs index 340e86c..728769c 100644 --- a/hulykvs_server/src/handlers.rs +++ b/hulykvs_server/src/handlers.rs @@ -39,10 +39,10 @@ pub async fn get( let connection = pool.get().await?; let statement = r#" - select value from kvs where namespace=$1 and key=$2 + select value from kvs where workspace=$1 and namespace=$2 and key=$3 "#; - let result = connection.query(statement, &[&nsstr, &keystr]).await?; + let result = connection.query(statement, &[&"defaultspace", &nsstr, &keystr]).await?; let response = match result.as_slice() { [] => HttpResponse::NotFound().finish(), @@ -76,16 +76,16 @@ pub async fn post( let md5 = md5::compute(&body); let statement = r#" - insert into kvs(namespace, key, md5, value) - values($1, $2, $3, $4) - on conflict(namespace, key) + insert into kvs(workspace, namespace, key, md5, value) + values($1, $2, $3, $4, $5) + on conflict(workspace, workspace, namespace, key) do update set md5=excluded.md5, value=excluded.value "#; connection - .execute(statement, &[&nsstr, &keystr, &&md5[..], &&body[..]]) + .execute(statement, &[&"defaultspace", &nsstr, &keystr, &&md5[..], &&body[..]]) .await?; Ok(HttpResponse::NoContent().finish()) @@ -111,10 +111,10 @@ pub async fn delete( let connection = pool.get().await?; let statement = r#" - delete from kvs where namespace=$1 and key=$2 + delete from kvs where workspace=$1 and namespace=$2 and key=$3 "#; - let response = match connection.execute(statement, &[&nsstr, &keystr]).await? { + let response = match connection.execute(statement, &[&"defaultspace", &nsstr, &keystr]).await? { 1 => HttpResponse::NoContent(), 0 => HttpResponse::NotFound(), _ => panic!("multiple rows deleted, unique constraint is probably violated"), @@ -157,16 +157,16 @@ pub async fn list( let response = if let Some(prefix) = &query.prefix { let pattern = format!("{}%", prefix); let statement = r#" - select key from kvs where namespace=$1 and key like $2 + select key from kvs where workspace=$1 and namespace=$2 and key like $3 "#; - connection.query(statement, &[&nsstr, &pattern]).await? + connection.query(statement, &[&"defaultspace",&nsstr, &pattern]).await? } else { let statement = r#" - select key from kvs where namespace=$1 + select key from kvs where workspace=$1 and namespace=$2 "#; - connection.query(statement, &[&nsstr]).await? + connection.query(statement, &[&"defaultspace",&nsstr]).await? }; let count = response.len(); diff --git a/hulykvs_server/src/handlers_v2.rs b/hulykvs_server/src/handlers_v2.rs new file mode 100644 index 0000000..a647703 --- /dev/null +++ b/hulykvs_server/src/handlers_v2.rs @@ -0,0 +1,205 @@ +// +// Copyright © 2025 Hardcore Engineering Inc. +// +// Licensed under the Eclipse Public License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. You may +// obtain a copy of the License at https://www.eclipse.org/legal/epl-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// +// See the License for the specific language governing permissions and +// limitations under the License. +// + +use actix_web::{ + HttpResponse, error, + web::{self, Data, Json, Query}, +}; +use serde::{Deserialize, Serialize}; +use tracing::{error, trace}; + +use super::Pool; + +type BucketPath = web::Path<(String, String)>; +type ObjectPath = web::Path<(String, String, String)>; + +pub async fn get( + path: ObjectPath, + pool: Data, +) -> Result { + let (workspace, namespace, key) = path.into_inner(); + trace!(workspace, namespace, key, "get request"); + + let wsstr = workspace.as_str(); + let nsstr = namespace.as_str(); + let keystr = key.as_str(); + + async move || -> anyhow::Result { + let connection = pool.get().await?; + + let statement = r#" + select value from kvs where workspace=$1 and namespace=$2 and key=$3 + "#; + + let result = connection.query(statement, &[&wsstr, &nsstr, &keystr]).await?; + + let response = match result.as_slice() { + [] => HttpResponse::NotFound().finish(), + [found] => HttpResponse::Ok().body(found.get::<_, Vec>("value")), + _ => panic!("multiple rows found, unique constraint is probably violated"), + }; + + Ok(response) + }() + .await + .map_err(|error| { + error!(op = "get", workspace, namespace, key, ?error, "internal error"); + error::ErrorInternalServerError("") + }) +} + + + +pub async fn post( + path: ObjectPath, + pool: Data, + body: web::Bytes, +) -> Result { + let (workspace, namespace, key) = path.into_inner(); + trace!(workspace, namespace, key, "post request"); + + let wsstr = workspace.as_str(); + let nsstr = namespace.as_str(); + let keystr = key.as_str(); + + async move || -> anyhow::Result { + let connection = pool.get().await?; + + let md5 = md5::compute(&body); + + let statement = r#" + INSERT INTO kvs(workspace, namespace, key, md5, value) + VALUES($1, $2, $3, $4, $5) + ON CONFLICT (workspace, namespace, key) + DO UPDATE SET + md5 = excluded.md5, + value = excluded.value + "#; + + connection + .execute(statement, &[&wsstr, &nsstr, &keystr, &&md5[..], &&body[..]]) + .await?; + + Ok(HttpResponse::NoContent().finish()) + }() + .await + .map_err(|error| { + error!(op = "upsert", workspace, namespace, key, ?error, "internal error"); + error::ErrorInternalServerError("") + }) +} + + + +pub async fn delete( + path: ObjectPath, + pool: Data, +) -> Result { + let (workspace, namespace, key) = path.into_inner(); + trace!(workspace, namespace, key, "delete request"); + + let wsstr = workspace.as_str(); + let nsstr = namespace.as_str(); + let keystr = key.as_str(); + + async move || -> anyhow::Result { + let connection = pool.get().await?; + + let statement = r#" + DELETE FROM kvs WHERE workspace=$1 AND namespace=$2 AND key=$3 + "#; + + let response = match connection.execute(statement, &[&wsstr, &nsstr, &keystr]).await? { + 1 => HttpResponse::NoContent(), + 0 => HttpResponse::NotFound(), + _ => panic!("multiple rows deleted, unique constraint is probably violated"), + }; + + Ok(response.into()) + }() + .await + .map_err(|error| { + error!(op = "delete", workspace, namespace, key, ?error, "internal error"); + error::ErrorInternalServerError("") + }) +} + + +#[derive(Deserialize)] +pub struct ListInfo { + prefix: Option, +} + +#[derive(Serialize)] +pub struct ListResponse { + workspace: String, + namespace: String, + count: usize, + keys: Vec, +} + +pub async fn list( + path: BucketPath, + pool: Data, + query: Query, +) -> Result, actix_web::error::Error> { + let (workspace, namespace) = path.into_inner(); + trace!(workspace, namespace, prefix = ?query.prefix, "list request"); + + let wsstr = workspace.as_str(); + let nsstr = namespace.as_str(); + + async move || -> anyhow::Result> { + let connection = pool.get().await?; + + let response = if let Some(prefix) = &query.prefix { + let pattern = format!("{}%", prefix); + let statement = r#" + select key from kvs where workspace=$1 and namespace=$2 and key like $3 + "#; + + connection.query(statement, &[&wsstr, &nsstr, &pattern]).await? + } else { + let statement = r#" + select key from kvs where workspace=$1 and namespace=$2 + "#; + + connection.query(statement, &[&wsstr, &nsstr]).await? + }; + + let count = response.len(); + + let keys = response.into_iter().map(|row| row.get(0)).collect(); + +/* + let mut keys = Vec::new(); + for row in response { + keys.push(row.get::<_, String>(0)); + } +*/ + + Ok(Json(ListResponse { + keys, + count, + namespace: nsstr.to_owned(), + workspace: wsstr.to_owned(), + })) + }() + .await + .map_err(|error| { + error!(op = "list", workspace, namespace, ?error, "internal error"); + error::ErrorInternalServerError("") + }) +} diff --git a/hulykvs_server/src/main.rs b/hulykvs_server/src/main.rs index 74424d0..6b71f9b 100644 --- a/hulykvs_server/src/main.rs +++ b/hulykvs_server/src/main.rs @@ -29,6 +29,7 @@ use tracing::info; mod config; mod handlers; +mod handlers_v2; mod token; use config::CONFIG; @@ -147,6 +148,14 @@ async fn main() -> anyhow::Result<()> { .route("/{bucket}/{id}", web::post().to(handlers::post)) .route("/{bucket}/{id}", web::delete().to(handlers::delete)), ) + .service( + web::scope("/api2") + .wrap(middleware::from_fn(interceptor)) + .route("/{workspace}/{bucket}", web::get().to(handlers_v2::list)) + .route("/{workspace}/{bucket}/{id}", web::get().to(handlers_v2::get)) + .route("/{workspace}/{bucket}/{id}", web::post().to(handlers_v2::post)) + .route("/{workspace}/{bucket}/{id}", web::delete().to(handlers_v2::delete)), + ) .route("/status", web::get().to(async || "ok")) }) .bind(socket)? diff --git a/scripts/claims.json b/scripts/claims.json new file mode 100644 index 0000000..59b9b83 --- /dev/null +++ b/scripts/claims.json @@ -0,0 +1,4 @@ +{ + "account": "lleo", + "extra": {} +} \ No newline at end of file diff --git a/scripts/db_view.sh b/scripts/db_view.sh new file mode 100755 index 0000000..f9cae58 --- /dev/null +++ b/scripts/db_view.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +clear + +CN="postgresql://root@huly.local:26257/defaultdb?sslmode=disable" + +#psql "$CN" -c "SELECT * FROM hulykvs.kvs;" +psql "$CN" -c "SELECT workspace,namespace,key,convert_from(value,'UTF8') AS value, encode(md5,'hex') AS md5 FROM hulykvs.kvs ORDER BY namespace, key;" diff --git a/scripts/shut.sh b/scripts/shut.sh new file mode 100755 index 0000000..f774760 --- /dev/null +++ b/scripts/shut.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +clear + +NS="TESTS" +KEY="keylleo2" +VALUE="{\"name\": \"John Fox\", \"penis\": \"$(( RANDOM % 20 + 5 ))\"}" +TOKEN=$(./token.sh lleo) + +# read all (GET) +echo +echo -n "📥 GET /api/${NS} = " +curl -s -X GET "http://localhost:8094/api/${NS}" -H "Authorization: Bearer $TOKEN" + +# write (POST) +curl -s -o /dev/null -w "✅ Stored key '%s' in namespace '%s' → HTTP %s\n" \ + -X POST "http://localhost:8094/api/$NS/$KEY" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d "$VALUE" \ + --write-out "📥 POST(%{http_code}) /api/$NS/$KEY = $VALUE\n" + +# read (GET) +echo +echo -n "📥 GET /api/$NS/$KEY = " +curl -s -X GET "http://localhost:8094/api/$NS/$KEY" -H "Authorization: Bearer $TOKEN" +# | jq . + +# read all (GET) +echo +echo -n "📥 GET /api/${NS} = " +curl -s -X GET "http://localhost:8094/api/${NS}" -H "Authorization: Bearer $TOKEN" diff --git a/scripts/shut_v2.sh b/scripts/shut_v2.sh new file mode 100755 index 0000000..03f0bf4 --- /dev/null +++ b/scripts/shut_v2.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +clear + +WS="Huone" +NS="TESTS" +KEY="AnyKey" +VALUE="{\"name\": \"Pavel\", \"penis\": \"$(( RANDOM % 20 + 5 ))\"}" +TOKEN=$(./token.sh lleo) + +# read all (GET) +echo +echo -n "📥 GET /api2/${WS}/${NS} = " +curl -s -X GET "http://localhost:8094/api2/${WS}/${NS}" -H "Authorization: Bearer $TOKEN" + +# write (POST) +curl -s -o /dev/null -w "✅ Stored key '%s' in namespace '%s' → HTTP %s\n" \ + -X POST "http://localhost:8094/api2/${WS}/$NS/$KEY" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d "$VALUE" \ + --write-out "📥 POST(%{http_code}) /api2/${WS}/$NS/$KEY = $VALUE\n" + +# read (GET) +echo +echo -n "📥 GET /api2/${WS}/$NS/$KEY = " +curl -s -X GET "http://localhost:8094/api2/${WS}/$NS/$KEY" -H "Authorization: Bearer $TOKEN" +# | jq . + +# read all (GET) +echo +echo -n "📥 GET /api2/${WS}/${NS} = " +curl -s -X GET "http://localhost:8094/api2/${WS}/${NS}" -H "Authorization: Bearer $TOKEN" + +# read all ?prefix=keyl (GET) +echo +echo -n "📥 GET /api2/${WS}/${NS}?prefix=keyl = " +curl -s -X GET "http://localhost:8094/api2/${WS}/${NS}?prefix=keyl" -H "Authorization: Bearer $TOKEN" diff --git a/scripts/token.sh b/scripts/token.sh new file mode 100755 index 0000000..1973f64 --- /dev/null +++ b/scripts/token.sh @@ -0,0 +1,13 @@ +#!/bin/bash + +CONFIG_PATH="../hulykvs_server/src/config/default.toml" +SECRET=$(grep '^token_secret' "$CONFIG_PATH" | sed -E 's/.*=\s*"(.*)"/\1/') # " + +if [ -z "$SECRET" ]; then + echo "❌No token_secret in $CONFIG_PATH" + exit 1 +fi + +TOKEN=$(echo -n "${SECRET}" | jwt -alg HS256 -key - -sign claims.json) + +echo "$TOKEN"