v0.3.0: Unique indexes, by-key helpers, JS array migration, and benchmark suite

README.md

@@ -15,6 +15,8 @@ It is not a SQL database or persistence layer. ColQL is for data you already wan
 - Lazy queries with filtering, projection, aggregation, streaming, limit, and offset
 - Object predicates plus tuple-style `where(column, operator, value)`
 - Explicit equality indexes and sorted numeric indexes for hot predicates
+- Unique indexes for stable ID lookups and duplicate-key protection
+- JS Array migration helpers such as `fromRows`, `firstWhere`, `countWhere`, and `exists`
 - Mutable tables with `updateMany` and `deleteMany`
 - Runtime validation with structured `ColQLError` codes
 - Binary serialization for table data
@@ -48,6 +50,7 @@ users.insertMany([
 
 users.createIndex("status");
 users.createSortedIndex("age");
+users.createUniqueIndex("id");
 
 const activeAdults = users
   .where({
@@ -64,6 +67,8 @@ const result = users.updateMany(
 
 console.log(activeAdults);
 console.log(result.affectedRows);
+
+console.log(users.findBy("id", 1));
 ```
 
 ## Performance Snapshot
@@ -88,6 +93,7 @@ npm run test:large
 ```
 
 For benchmark scripts and interpretation notes, see [Performance and Benchmarks](./docs/doc/13-performance-and-benchmarks.md).
+For JS Array comparisons, run `npm run benchmark:array-comparison`; results are local guidance, not universal promises.
 
 ## When To Use ColQL
 
@@ -98,12 +104,14 @@ Use ColQL when:
 - filters and aggregations should avoid intermediate arrays
 - a TypeScript schema can describe your columns
 - explicit indexes are acceptable for hot equality or range predicates
+- stable identity can be modeled with an explicit ID column and unique index
 - runtime validation matters because data may come from untyped sources
 
 Avoid ColQL when:
 
 - you need durable storage, transactions, joins, or SQL
 - row indexes must be stable external identifiers
+- a small/simple JavaScript array is already clear and fast enough
 - every query requires arbitrary sorting or grouping
 - you need concurrent writers or multi-process coordination
 - you want automatic indexes, compound indexes, or query planning across tables
@@ -132,6 +140,7 @@ Recommended reading:
 - [Querying](./docs/doc/04-querying.md)
 - [Equality Indexes](./docs/doc/06-indexing.md)
 - [Sorted Indexes](./docs/doc/07-sorted-indexes.md)
+- [Unique Indexes](./docs/doc/17-unique-indexes.md)
 - [Mutations](./docs/doc/08-mutations.md)
 - [Serialization](./docs/doc/11-serialization.md)
 - [Memory Model](./docs/doc/12-memory-model.md)
@@ -159,6 +168,8 @@ users.deleteMany({ status: "archived" });
 
 users.createIndex("id");
 users.createSortedIndex("age");
+users.createUniqueIndex("id");
+users.findBy("id", 123);
 
 const buffer = users.serialize();
 const restored = table.deserialize(buffer);
@@ -195,6 +206,7 @@ npm run benchmark:range
 npm run benchmark:optimizer
 npm run benchmark:serialization
 npm run benchmark:delete
+npm run benchmark:array-comparison
 ```
 
 ## Status
@@ -203,4 +215,4 @@ ColQL v0.2.x aims to keep the public API reasonably stable, but breaking changes
 
 ## Limitations
 
-ColQL intentionally does not include SQL parsing, joins, transactions, concurrency control, automatic indexes, compound indexes, or durable storage. Indexes are derived performance structures; query results must be the same whether ColQL uses an index or a full scan.
+ColQL intentionally does not include SQL parsing, joins, transactions, concurrency control, automatic indexes, compound indexes, or durable storage. Equality and sorted indexes are derived performance structures; query results must be the same whether ColQL uses an index or a full scan. Unique indexes are derived too, but they also enforce uniqueness while present and are not serialized.

benchmarks/array-comparison.mjs

@@ -0,0 +1,241 @@
+import os from "node:os";
+import { column, fromRows, table } from "../dist/index.mjs";
+
+const DEFAULT_SIZES = [1_000, 100_000, 1_000_000];
+const RUNS = 3;
+
+const sizes = process.env.COLQL_ARRAY_BENCH_SIZES
+  ? process.env.COLQL_ARRAY_BENCH_SIZES.split(",").map((value) => Number.parseInt(value.trim(), 10))
+  : DEFAULT_SIZES;
+const jsonOutput = process.argv.includes("--json");
+
+for (const size of sizes) {
+  if (!Number.isInteger(size) || size < 1) {
+    throw new Error(`Invalid benchmark size: ${String(size)}`);
+  }
+}
+
+const schema = {
+  id: column.uint32(),
+  age: column.uint8(),
+  score: column.uint32(),
+  status: column.dictionary(["active", "passive", "archived"]),
+  active: column.boolean(),
+};
+
+function mb(bytes) {
+  return bytes / 1024 / 1024;
+}
+
+function memoryTotal() {
+  const usage = process.memoryUsage();
+  return usage.heapUsed + usage.arrayBuffers;
+}
+
+function forceGc() {
+  if (global.gc) {
+    global.gc();
+  }
+}
+
+function createRows(rowCount) {
+  return Array.from({ length: rowCount }, (_unused, id) => ({
+    id,
+    age: (id * 7) % 100,
+    score: (id * 13) % 10_000,
+    status: id % 3 === 0 ? "active" : id % 3 === 1 ? "passive" : "archived",
+    active: id % 2 === 0,
+  }));
+}
+
+function createTable(rows, indexes = "none") {
+  const users = fromRows(schema, rows);
+  if (indexes === "equality") {
+    users.createIndex("id").createIndex("status");
+  }
+  if (indexes === "sorted") {
+    users.createSortedIndex("age");
+  }
+  if (indexes === "unique") {
+    users.createUniqueIndex("id");
+  }
+  return users;
+}
+
+function time(fn) {
+  const start = performance.now();
+  const result = fn();
+  return { duration: performance.now() - start, result };
+}
+
+function average(values) {
+  return values.reduce((total, value) => total + value, 0) / values.length;
+}
+
+function median(values) {
+  const sorted = [...values].sort((left, right) => left - right);
+  return sorted[Math.floor(sorted.length / 2)];
+}
+
+function measureMemory(label, factory) {
+  forceGc();
+  const before = memoryTotal();
+  const value = factory();
+  forceGc();
+  const after = memoryTotal();
+  return { label, ms: 0, memoryMB: Math.max(0, mb(after - before)), value };
+}
+
+function runWorkloads(rows) {
+  const rowCount = rows.length;
+  const targetId = rowCount - 10;
+  const broadAge = 10;
+  const selectiveAge = 95;
+  const results = [];
+
+  const memoryArray = measureMemory("memory: JS object array", () => createRows(rowCount));
+  results.push({ label: memoryArray.label, ms: 0, memoryMB: memoryArray.memoryMB });
+  memoryArray.value.length = 0;
+
+  const memoryColql = measureMemory("memory: ColQL scan table", () => createTable(rows));
+  results.push({ label: memoryColql.label, ms: 0, memoryMB: memoryColql.memoryMB });
+
+  const buildFromRows = time(() => createTable(rows).rowCount);
+  results.push({ label: "fromRows / insertMany", ms: buildFromRows.duration });
+  if (buildFromRows.result !== rowCount) throw new Error("fromRows sanity check failed.");
+
+  const scan = createTable(rows);
+  const equality = createTable(rows, "equality");
+  const sorted = createTable(rows, "sorted");
+  const unique = createTable(rows, "unique");
+
+  const workloads = [
+    { label: "filter/count: JS array", fn: () => rows.filter((row) => row.status === "active" && row.age >= 18).length },
+    { label: "filter/count: ColQL scan", fn: () => scan.where({ status: "active", age: { gte: 18 } }).count() },
+    { label: "filter/count: ColQL equality index", fn: () => equality.where({ status: "active", age: { gte: 18 } }).count() },
+    [
+      "projection+limit: JS array",
+      () =>
+        rows
+          .filter((row) => row.status === "active" && row.age >= 18)
+          .slice(0, 100)
+          .map((row) => ({ id: row.id, age: row.age })).length,
+    ],
+    [
+      "projection+limit: ColQL",
+      () => scan.where({ status: "active", age: { gte: 18 } }).select(["id", "age"]).limit(100).toArray().length,
+    ],
+    { label: "find by id: JS array", fn: () => rows.find((row) => row.id === targetId)?.id },
+    { label: "find by id: ColQL equality index", fn: () => equality.where("id", "=", targetId).first()?.id },
+    { label: "unique lookup findBy", fn: () => unique.findBy("id", targetId)?.id },
+    { label: "exists: JS array", fn: () => rows.some((row) => row.id === targetId) },
+    { label: "exists: ColQL helper", fn: () => equality.exists("id", "=", targetId) },
+    { label: "countWhere: JS array", fn: () => rows.filter((row) => row.status === "passive").length },
+    { label: "countWhere: ColQL helper", fn: () => equality.countWhere({ status: "passive" }) },
+    { label: "range query: JS array", fn: () => rows.filter((row) => row.age >= selectiveAge).length },
+    { label: "range query: ColQL scan", fn: () => scan.where("age", ">=", selectiveAge).count() },
+    { label: "range query: ColQL sorted index", fn: () => sorted.where("age", ">=", selectiveAge).count() },
+    { label: "broad scan: JS array", fn: () => rows.filter((row) => row.age >= broadAge).length },
+    { label: "broad scan: ColQL", fn: () => sorted.where("age", ">=", broadAge).count() },
+    { label: "callback filter(fn): JS array", fn: () => rows.filter((row) => row.active && row.score % 7 === 0).length },
+    { label: "callback filter(fn): ColQL", fn: () => scan.filter((row) => row.active && row.score % 7 === 0).count() },
+    [
+      "update by predicate: JS array",
+      () => rows.map((row) => (row.status === "archived" ? { ...row, status: "passive" } : row)).filter((row) => row.status === "passive").length,
+    ],
+    {
+      label: "update by predicate: ColQL",
+      prepare: () => createTable(rows),
+      fn: (users) => {
+        users.updateMany({ status: "archived" }, { status: "passive" });
+        return users.where("status", "=", "passive").count();
+      },
+    },
+    [
+      "delete by predicate: JS array",
+      () => rows.filter((row) => row.age < selectiveAge).length,
+    ],
+    {
+      label: "delete by predicate: ColQL",
+      prepare: () => createTable(rows),
+      fn: (users) => {
+        users.deleteMany({ age: { gte: selectiveAge } });
+        return users.rowCount;
+      },
+    },
+    {
+      label: "updateBy: unique index",
+      prepare: () => createTable(rows, "unique"),
+      fn: (users) => {
+        const result = users.updateBy("id", targetId, { score: 123 });
+        return `${result.affectedRows}:${users.findBy("id", targetId)?.score}`;
+      },
+    },
+    {
+      label: "deleteBy: unique index",
+      prepare: () => createTable(rows, "unique"),
+      fn: (users) => {
+        const result = users.deleteBy("id", targetId);
+        return `${result.affectedRows}:${users.findBy("id", targetId) === undefined}`;
+      },
+    },
+  ];
+
+  const expected = new Map();
+  for (const workload of workloads) {
+    const normalizedWorkload = Array.isArray(workload)
+      ? { label: workload[0], fn: workload[1] }
+      : workload;
+    const runs = Array.from({ length: RUNS }, () => {
+      const input = normalizedWorkload.prepare?.();
+      return time(() => normalizedWorkload.fn(input));
+    });
+    const normalized = runs.map((run) => (run.result === undefined ? "undefined" : JSON.stringify(run.result)));
+    if (new Set(normalized).size !== 1) {
+      throw new Error(`Benchmark sanity check failed for ${normalizedWorkload.label}.`);
+    }
+
+    const group = normalizedWorkload.label.replace(/: (JS array|ColQL.*|unique index)$/, "");
+    const value = normalized[0];
+    if (expected.has(group) && expected.get(group) !== value) {
+      throw new Error(`Array and ColQL results differ for ${group}.`);
+    }
+    expected.set(group, value);
+
+    results.push({ label: normalizedWorkload.label, ms: average(runs.map((run) => run.duration)), medianMs: median(runs.map((run) => run.duration)) });
+  }
+
+  return results;
+}
+
+function printHuman(allResults) {
+  console.log("JS Array vs ColQL comparison benchmark");
+  console.log(`Node ${process.version} on ${process.platform} ${process.arch}`);
+  console.log(`CPU: ${os.cpus()[0]?.model ?? "unknown"} (${os.cpus().length} logical cores)`);
+  console.log("Caveats: local machine only; results vary with Node version, CPU, memory pressure, data distribution, selectivity, and workload shape.");
+  console.log("These numbers are not CI requirements or universal guarantees.\n");
+
+  for (const group of allResults) {
+    console.log(`${group.rows.toLocaleString()} rows, average over ${RUNS} runs`);
+    console.log("workload                                      avg ms    median ms   memory MB");
+    console.log("--------------------------------------------------------------------------");
+    for (const result of group.results) {
+      const ms = result.ms.toFixed(3).padStart(8);
+      const medianMs = (result.medianMs ?? result.ms).toFixed(3).padStart(9);
+      const memory = result.memoryMB === undefined ? "".padStart(9) : result.memoryMB.toFixed(2).padStart(9);
+      console.log(`${result.label.padEnd(44)} ${ms} ${medianMs} ${memory}`);
+    }
+    console.log("");
+  }
+}
+
+const allResults = sizes.map((rows) => {
+  const sourceRows = createRows(rows);
+  return { rows, results: runWorkloads(sourceRows) };
+});
+
+if (jsonOutput) {
+  console.log(JSON.stringify({ env: { node: process.version, platform: process.platform, arch: process.arch, cpu: os.cpus()[0]?.model }, results: allResults }, null, 2));
+} else {
+  printHuman(allResults);
+}

docs/doc/06-indexing.md

@@ -1,6 +1,6 @@
 # Equality Indexes
 
-Equality indexes are optional derived performance structures for selective equality and membership queries. A query must return the same result whether ColQL uses an index or a full scan.
+Equality indexes are optional derived performance structures for selective equality and membership queries. A query must return the same result whether ColQL uses an index or a full scan. Unique indexes are separate integrity indexes; see [Unique Indexes](./17-unique-indexes.md).
 
 ```ts
 users.createIndex("id");

docs/doc/08-mutations.md

@@ -14,6 +14,9 @@ users.deleteMany(predicate);
 
 users.updateWhere(column, operator, value, partialRow);
 users.deleteWhere(column, operator, value);
+
+users.updateBy(uniqueColumn, value, partialRow);
+users.deleteBy(uniqueColumn, value);
 ```
 
 New mutation APIs return:
@@ -111,13 +114,16 @@ ColQL applies mutation safety rules internally:
 - matching row indexes are snapshotted before predicate mutation
 - update input is validated before writing to storage
 - predicate updates are all-or-nothing for validation
+- unique-index violations are checked before writing and keep bulk updates all-or-nothing
 - predicate deletes delete matched row indexes from highest to lowest
 - no-match predicate update/delete returns `{ affectedRows: 0 }`
 - nonzero update/delete mutations mark existing indexes dirty
 - incremental index maintenance is not attempted
 
 Dirty indexes are rebuilt before an indexed query uses them, so index dirtiness affects rebuild cost, not query correctness.
 
+Unique indexes are stricter than equality and sorted indexes. They enforce uniqueness for indexed columns and reject duplicate-producing inserts or updates with `COLQL_DUPLICATE_KEY`. Deletes free unique keys for reuse.
+
 Snapshotting matters when an update changes the predicate column:
 
 ```ts

docs/doc/10-error-handling.md

@@ -61,6 +61,10 @@ Index errors:
 - `COLQL_SORTED_INDEX_EXISTS`
 - `COLQL_SORTED_INDEX_NOT_FOUND`
 - `COLQL_SORTED_INDEX_UNSUPPORTED_COLUMN`
+- `COLQL_DUPLICATE_KEY`
+- `COLQL_UNIQUE_INDEX_EXISTS`
+- `COLQL_UNIQUE_INDEX_NOT_FOUND`
+- `COLQL_UNIQUE_INDEX_UNSUPPORTED`
 
 Serialization errors:
 

docs/doc/11-serialization.md

@@ -26,16 +26,18 @@ Indexes are not serialized:
 
 - equality indexes
 - sorted indexes
+- unique indexes
 
-They are derived performance data and can be rebuilt after deserialization. Recreating indexes after deserialization affects performance only, not query correctness.
+They are derived data and can be rebuilt after deserialization. Recreating equality and sorted indexes affects performance only, not query correctness. Recreating unique indexes also restores uniqueness enforcement.
 
 ```ts
 const restored = table.deserialize(buffer);
 restored.createIndex("id");
 restored.createSortedIndex("age");
+restored.createUniqueIndex("id");
 ```
 
-`restored.indexes()` and `restored.sortedIndexes()` are empty until indexes are recreated.
+`restored.indexes()`, `restored.sortedIndexes()`, and `restored.uniqueIndexes()` are empty until indexes are recreated.
 
 ## After Mutations and Deletes