Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f9b2027
Changed writing test from "skipped" to "failed"
CalebDepatie Aug 6, 2025
f4338ed
Fixed writing bug
CalebDepatie Aug 7, 2025
0d8446a
Removed direct_parallel_writes setting not supported on Linux
CalebDepatie Aug 7, 2025
a9e9c00
Merge pull request #48 from riverosorg/feature/writing
CalebDepatie Aug 11, 2025
12d06b4
Added an optional default query argument per #49
CalebDepatie Aug 12, 2025
13ea141
Updated testing action
CalebDepatie Aug 12, 2025
15771df
Improved error handling
CalebDepatie Aug 12, 2025
b835aa8
Repaired broken tests
CalebDepatie Aug 18, 2025
998df79
Additional safety checks, logs, and code reformatting
CalebDepatie Sep 1, 2025
8a84e29
Added license to utilities.cpp
CalebDepatie Sep 1, 2025
77eab40
add missing <optional> herader to parser.hpp
CalebDepatie Sep 1, 2025
3b035ac
Readded freebsd specific code
CalebDepatie Oct 30, 2025
c2d066f
Log to file and console
CalebDepatie Oct 30, 2025
68568b6
Limited action permissions
CalebDepatie Oct 30, 2025
5d944ef
Fixed #36
CalebDepatie Nov 11, 2025
3a65e55
Improved backup system and fixed #38
CalebDepatie Nov 11, 2025
1b62261
Added a string splitting util function
CalebDepatie Nov 11, 2025
b4ef299
Working folder tags; Resolved #51
CalebDepatie Nov 11, 2025
c697566
Resolved test runner issue with creating log files
CalebDepatie Nov 11, 2025
39cf828
Merge remote-tracking branch 'origin/staging/v0.2.3' into feature/fol…
CalebDepatie Nov 11, 2025
5301fa7
Refactored utilities.cpp to share code
CalebDepatie Nov 11, 2025
a89e5ac
Merge pull request #53 from riverosorg/feature/folder_tagging
CalebDepatie Nov 11, 2025
c2d6658
Added clang-format
CalebDepatie Nov 11, 2025
f8602e7
Fixed make format command
CalebDepatie Nov 11, 2025
44ea936
Version bump
CalebDepatie Nov 11, 2025
c031f78
clang-format version bump
CalebDepatie Nov 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .clang-format
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# SPDX-FileCopyrightText: 2025 Caleb Depatie
#
# SPDX-License-Identifier: 0BSD

BasedOnStyle: LLVM
IndentWidth: 4

ColumnLimit: 100

# Force pointers to the type for C++.
DerivePointerAlignment: false
PointerAlignment: Left

BreakStringLiterals: true
BreakBeforeBraces: Allman
25 changes: 25 additions & 0 deletions .github/workflows/format.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# SPDX-FileCopyrightText: 2025 Caleb Depatie
#
# SPDX-License-Identifier: 0BSD

name: Format Checker
permissions:
contents: read

on: [push]
jobs:
formatting-check:
name: Clang Formatting Check
runs-on: ubuntu-latest
strategy:
matrix:
path:
- 'filesystem/src'
- 'filesystem/tests'
steps:
- uses: actions/checkout@v4
- name: Run clang-format style check
uses: jidicula/clang-format-action@v4.16.0
with:
clang-format-version: '19'
check-path: ${{ matrix.path }}
2 changes: 2 additions & 0 deletions .github/workflows/reuse.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
# SPDX-License-Identifier: CC0-1.0

name: REUSE Compliance Check
permissions:
contents: read

on: [ push, pull_request ]

Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,17 @@
# SPDX-License-Identifier: 0BSD

name: Testing Suite
permissions:
contents: read

on: [ push, pull_request ]
on: [ push ]

jobs:
unit_test:
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- uses: cachix/install-nix-action@v18
with:
nix_path: nixpkgs=channel:nixos-unstable
Expand Down
5 changes: 4 additions & 1 deletion docs/lakefs.rst.in
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Initialize a LakeFS instance
SYNOPSIS
========

| lakefs [*OPTION*]... *mount_point*
| lakefs [*OPTION*]... *mount_point* [*default_query*]


DESCRIPTION
Expand Down Expand Up @@ -59,6 +59,9 @@ OPTIONS
mount_point
Folder to mount the LakeFS instance under.

default_query
Optional argument to specify the initial default query of the mount.

-f
Run program in foreground rather than as a daemon.

Expand Down
2 changes: 2 additions & 0 deletions filesystem/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ sources = [
'src/backups.cpp',
'src/config.cpp',
'src/control.cpp',
'src/utilities.cpp',
'src/query_lang/ast.cpp',
'src/query_lang/parser.cpp'
]
Expand Down Expand Up @@ -66,6 +67,7 @@ executable(

tests_dict = {
'SQLite' : ['tests/vendors/sqlite.cpp', sources],
'Utility Functions' : ['tests/utilities.cpp', sources],
'Parsing' : ['tests/parsing.cpp', sources],
'Config Reading' : ['tests/config.cpp', sources],
'SQL generation' : ['tests/query_generation.cpp', sources],
Expand Down
116 changes: 63 additions & 53 deletions filesystem/src/backups.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,32 @@
#include "backups.hpp"
#include "db.hpp"

#include <algorithm>
#include <filesystem>
#include <optional>

#include <pthread.h>
#include <sys/signal.h>
#include <time.h>

#include <spdlog/spdlog.h>
#include <vector>

// better way to pass these in? handler args
uint32_t _number_backups;
std::string _backup_dir;

static auto handle_backup(sigval val) -> void;

auto create_backup_timer(std::chrono::seconds interval, uint32_t number_backups, std::string backup_path) -> void {
spdlog::info("Setting up backups to keep {0} copies and run every {1} hours",
number_backups,
std::chrono::duration_cast<std::chrono::hours>(interval).count());
auto create_backup_timer(std::chrono::seconds interval, uint32_t number_backups,
std::string backup_path) -> bool
{
spdlog::info("Setting up backups to keep {0} copies and run every {1} hours", number_backups,
std::chrono::duration_cast<std::chrono::hours>(interval).count());

_number_backups = number_backups;
_backup_dir = backup_path;

// Create a C timer
// Create a C timer
sigevent event;

pthread_attr_t backup_thread_attr;
Expand All @@ -40,9 +42,10 @@ auto create_backup_timer(std::chrono::seconds interval, uint32_t number_backups,
event.sigev_notify_attributes = &backup_thread_attr;

timer_t timer_id;
if (timer_create(CLOCK_MONOTONIC, &event, &timer_id) == -1) {
if (timer_create(CLOCK_MONOTONIC, &event, &timer_id) == -1)
{
spdlog::critical("Could not create timer! {0}", strerror(errno));
// todo: pass up
return false;
}

itimerspec timer_spec = {};
Expand All @@ -51,19 +54,23 @@ auto create_backup_timer(std::chrono::seconds interval, uint32_t number_backups,

spdlog::debug("Making timer for {0} seconds", timer_spec.it_interval.tv_sec);

//start timer
if (timer_settime(timer_id, 0, &timer_spec, nullptr) == -1) {
// start timer
if (timer_settime(timer_id, 0, &timer_spec, nullptr) == -1)
{
spdlog::critical("Could not start timer! {0}", strerror(errno));
// todo: pass up
return false;
}

return true;
}

static auto handle_backup(sigval val) -> void {
static auto handle_backup(sigval val) -> void
{
spdlog::info("Starting Backup...");

const auto now = time(0);
const auto current_time = *std::localtime(&now);
char buf [128];
char buf[128];

strftime(buf, sizeof(buf), "%Y-%m-%d.%X", &current_time);

Expand All @@ -77,60 +84,63 @@ static auto handle_backup(sigval val) -> void {
// delete files if needed..
auto dir_iter = std::filesystem::directory_iterator(_backup_dir);

int backup_count = std::count_if(
begin(dir_iter),
end(dir_iter),
[](auto& entry) {
const auto filename = entry.path().filename();
int backup_count = std::count_if(begin(dir_iter), end(dir_iter),
[](auto& entry)
{
const auto filename = entry.path().filename();

return entry.is_regular_file() && (filename.string().ends_with(".backup.db"));
}
);
return entry.is_regular_file() &&
(filename.string().ends_with(".backup.db"));
});

if (backup_count > _number_backups) {
if (backup_count > _number_backups)
{
spdlog::info("Removing oldest backup");

dir_iter = std::filesystem::directory_iterator(_backup_dir);
auto oldest_entry = std::make_optional<std::filesystem::directory_entry>();
std::tm oldest_entry_date;

for (auto entry : dir_iter) {

if (entry.is_regular_file()) {
// Derives the time from the file name
const auto get_time = [](const std::filesystem::path entry) -> time_t
{
const auto file_stem = entry.stem();

// Get time from filename
const auto new_entry_name = entry.path().stem();
spdlog::debug("Looking at file {0} {1}", entry.c_str(), entry.stem().c_str());

// TODO: if this is removed, the backup fails!
spdlog::debug("Looking at file {0} {1}", entry.path().c_str(), entry.path().stem().c_str());
std::tm file_date;
strptime(file_stem.c_str(), "%Y-%m-%d.%X", &file_date);

std::tm new_entry_date;
strptime(new_entry_name.c_str(), "%Y-%m-%d.%X", &new_entry_date);
return mktime(&file_date);
};

if (!oldest_entry.has_value()) {
oldest_entry = entry;
oldest_entry_date = new_entry_date;
continue;
}
dir_iter = std::filesystem::directory_iterator(_backup_dir);

if (difftime(mktime(&new_entry_date), mktime(&oldest_entry_date)) < 0) {
oldest_entry = entry;
oldest_entry_date = new_entry_date;
}
// Placing the iterator into a vector so its more straightforward to
// work with
std::vector<std::filesystem::path> files{};
for (auto entry : dir_iter)
{
const auto filename = entry.path().filename();

if (entry.is_regular_file() && (filename.string().ends_with(".backup.db")))
{
files.push_back(entry.path());
}
}

if (oldest_entry.has_value()) {
if (std::filesystem::remove(oldest_entry->path())) {
spdlog::info("Removed file at {0}", oldest_entry->path().c_str());

} else {
spdlog::error("Could not remove file at {0}", oldest_entry->path().c_str());
std::sort(files.begin(), files.end(),
[get_time](const std::filesystem::path& entry_a,
const std::filesystem::path& entry_b) -> bool
{ return get_time(entry_a) < get_time(entry_b); });

// remove files
for (int i = 0; i < (backup_count - _number_backups); i++)
{
if (std::filesystem::remove(files[i]))
{
spdlog::info("Removed file at {0}", files[i].c_str());
}
else
{
spdlog::error("Could not remove file at {0}", files[i].c_str());
}

} else {
spdlog::error("Could not remove entry, no file found");
}
}

Expand Down
5 changes: 3 additions & 2 deletions filesystem/src/backups.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
#pragma once

#include <chrono>
#include <string>
#include <cstdint>
#include <string>

auto create_backup_timer(std::chrono::seconds interval, uint32_t number_backups, std::string backuppath) -> void;
auto create_backup_timer(std::chrono::seconds interval, uint32_t number_backups,
std::string backuppath) -> bool;
17 changes: 9 additions & 8 deletions filesystem/src/command_interface.h
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@
#pragma once

#ifdef __cplusplus
extern "C" {
extern "C"
{
#endif

// Command Interface
// Command Interface

#define LAKE_SOCKET_PATH "/tmp/lakefs.sock"

Expand All @@ -18,13 +19,13 @@ extern "C" {
#define LAKE_REMOVE_TAG 4
#define LAKE_RELINK_FILE 5

typedef struct __attribute__((packed)) lake_command_t {
int command;
int size;
char data[];
} lake_command_t;
typedef struct __attribute__((packed)) lake_command_t
{
int command;
int size;
char data[];
} lake_command_t;

#ifdef __cplusplus
}
#endif

Loading
Loading