[Bug]: load_safetensors changes empty tensors to float32 on main #809

What happened?

On current main, fastembed.common.safetensors_utils.load_safetensors() returns float32 for every empty tensor, even when the file declares I64, BOOL, F16, or another supported dtype. Nonempty tensors retain their declared dtype.

This is a loader-level compatibility check on a tiny local file, not an observed model-inference failure. The published fastembed==0.9.0 wheel does not contain this new module; I am reporting the current main implementation, not a released-version regression.

Minimal reproduction

Run from a current main checkout with NumPy and the reference safetensors package installed:

from pathlib import Path
from tempfile import TemporaryDirectory

import numpy as np
from safetensors.numpy import load_file, save_file
from fastembed.common.safetensors_utils import load_safetensors

with TemporaryDirectory() as directory:
    path = Path(directory) / "empty.safetensors"
    save_file({"indices": np.empty((2, 0, 3), dtype=np.int64)}, path)
    print("reference:", load_file(path)["indices"].dtype)
    for mmap in (False, True):
        actual = load_safetensors(path, mmap=mmap)["indices"]
        print(f"fastembed mmap={mmap}: {actual.dtype}, shape={actual.shape}")

Output:

reference: int64
fastembed mmap=False: float32, shape=(2, 0, 3)
fastembed mmap=True: float32, shape=(2, 0, 3)

Expected: preserve int64 and the shape in both modes. Empty tensors are valid in the safetensors format.

Diagnosis and proposed scope

The end == start branch returns np.zeros(shape, dtype=np.float32) before resolving the header dtype. The existing intentional BF16-to-float32 conversion can stay as it is, but other supported dtypes should be preserved for empty tensors too. Empty arrays do not need a memory map.

Would you accept a small fix limited to dtype resolution for this branch and regression tests for empty integer/float tensors in both modes? I would like to implement it as Ra1nhy once the scope is confirmed.

Validation

• Current main file Git blob: bf2c9b27b770dc03db48ab5d2ba6bbe8a7f1290f, checked on 2026-10-13.
• Against files generated and read by the reference safetensors.numpy implementation: 16 dtype checks fail (eight supported non-F32 dtypes, both loading modes); six controls pass (empty F32 and nonempty I64/F64 in both modes).
• Existing tests/test_static_embeddings.py -k test_load_safetensors: 2 passed. Existing common and image-transform tests: 18 passed.
• No product code changed, no model weights downloaded, no inference or full model test suite run.
• Checked matching issues and PRs, including merged new: run minishlab potion models with numpy instead of onnx #805 and new: make memory-mapping of static embeddings optional #808; no matching empty-dtype fix found.

Environment: Python 3.12.14, macOS arm64, NumPy 2.5.4, safetensors 0.8.0, editable current main source (package metadata 0.9.0).