dir-sync

High-performance, type-safe directory synchronization tool utilizing Python MP/MT, Pydantic validation, and structured logging.

Key Features

• Hybrid Concurrency Architecture:
  • Uses multiprocessing for CPU-intensive diff calculation.
  • Uses multithreading for I/O-bound directory scanning, file transfer and deletion operations.
• Type Safety & Validation: Configuration management is handled via Pydantic, ensuring strict schema validation and runtime type checking.
• Enterprise-Grade Observability: Replaces standard outputs with structured Logging for detailed auditing and easier debugging.
• Full CRUD Synchronization: Handles Copy, Update, Directory Creation, and Deletion.
• Dry-run: Using a special flag, it is possible to simulate synchronisation operations instead of actually performing them.

Tech Stack

• Language: Python 3.11+
• Validation: Pydantic
• Concurrency: multiprocessing, threading, concurrent.futures
• Logging: Native Python logging module
• Testing unittest framework

Installation & Usage

1. Open the Notebook in Google Colab: Open In Colab

2. To execute the tool:

   • Instantiate the main class with your source/destination paths.
   • Execute plan generation (a list of pending operations: Copy, Update, Delete, Mkdir)
   • Invoke the execution method to apply the Action Plan.

Disclaimer

This tool performs destructive actions (file deletion) to ensure source-destination mirroring. Use with caution.

License

Distributed under the MIT License. See LICENSE for more information.