Add DM-Era support

[sven-ola]

Hey... I now have tested the --era switch in vitro as well as on a live system. It may be ready for prime time, so here is the pull RQ. I add my test scripts here, just in case you want to give it a try. If necessary, I can provide SSH access to a live test system (one of those little things depicted with the last pull RQ).

era-test.sh

#!/bin/sh

case $(id -u) in 0|root);;*)
	echo "This tests need root" >&2
	exit 1
;;esac

if ! vgs vg0>/dev/null;then
	echo "You need an LVS volume group called vg0" >&2
	exit 1
fi

PE=32
PE_SIZE=$(( 4 * 1024 * 1024 ))
ERA_SIZE=$(( 4096 * 512 ))

era=true

case ${1} in c|create)
	lvcreate --yes --extents 1 --name era vg0
	dm=$(readlink -f /dev/mapper/vg0-era)
	era_size=$(cat /sys/class/block/${dm##*/}/size)
	dd if=/dev/zero of=/dev/mapper/vg0-era bs=${PE_SIZE} count=1 status=none

	lvcreate --yes --extents ${PE} --name test vg0
	dm=$(readlink -f /dev/mapper/vg0-test)
	lv_size=$(cat /sys/class/block/${dm##*/}/size)

	if false;then
		dd if=/dev/zero bs=${PE_SIZE} count=${PE} status=none|tr "\000" "\377"|dd of=/dev/mapper/vg0-test bs=${PE_SIZE} status=none
	elif false;then
		dd if=/dev/zero bs=${PE_SIZE} count=${PE} status=none|dd of=/dev/mapper/vg0-test bs=${PE_SIZE} status=none
	elif false;then
		i=0
		while [ ${i} -lt $(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) ];do
			printf "magic sot %05d\n" ${i}|dd of=/dev/mapper/vg0-test bs=${ERA_SIZE} seek=${i} count=1 conv=notrunc status=none
			i=$(( ${i} + 1 ))
		done
	elif true;then
		blkdiscard /dev/mapper/vg0-test
	fi

	if ${era};then
		dmsetup create test-meta   --table "0 ${era_size} linear /dev/mapper/vg0-era 0"
		dmsetup create test-era    --table "0 ${lv_size} era /dev/mapper/test-meta /dev/mapper/vg0-test 4096"
		dmsetup create test-access --table "0 ${era_size} linear /dev/mapper/test-meta 0"
	fi
;;d|dump)
	era_dump /dev/mapper/test-access
;;t|take)
	dmsetup message test-era 0 drop_metadata_snap 2>/dev/null
	dmsetup message test-era 0 take_metadata_snap
	set -- $(dmsetup status test-era)
	echo "era now ${6}"
;;poke)
	if [ 0 -le ${2} ] 2>/dev/null && [ ${2} -lt $(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) ];then
		printf "% 5d\n" ${2}|dd of=/dev/mapper/test-era bs=${ERA_SIZE} seek=${2} count=1 conv=notrunc status=none
	else
		echo "Number 0..$(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) missing." >&2
		exit 1
	fi
;;Poke)
	if [ 0 -le ${2} ] 2>/dev/null && [ ${2} -lt $(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) ];then
		printf "% 5d\n" ${2}|dd of=/dev/mapper/vg0-test bs=${ERA_SIZE} seek=${2} count=1 conv=notrunc status=none
	else
		echo "Number 0..$(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) missing." >&2
		exit 1
	fi
;;peek)
	if [ 0 -le ${2} ] 2>/dev/null && [ ${2} -lt $(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) ];then
		dd if=/dev/mapper/test-era bs=${ERA_SIZE} skip=${2} count=1 conv=notrunc status=none|hexdump -C|head -n1
	else
		echo "Number 0..$(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) missing." >&2
		exit 1
	fi
;;Peek)
	if [ 0 -le ${2} ] 2>/dev/null && [ ${2} -lt $(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) ];then
		dd if=/dev/mapper/vg0-test bs=${ERA_SIZE} skip=${2} count=1 conv=notrunc status=none|hexdump -C|head -n1
	else
		echo "Number 0..$(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) missing." >&2
		exit 1
	fi
;;w|written)
	since=${2}
	set -- $(dmsetup status test-era)
	case ${since} in "")
		since=$(( ${6} - 1 ))
	;;esac
	era_invalidate --written-since ${since} /dev/mapper/test-access
;;m|md5|M|MD5)
	i=0
	old=
	test -f ${0%.*}.md5 && mv ${0%.*}.md5 ${0%.*}.old
	while [ ${i} -lt $(( ${PE} * ${PE_SIZE} / ${ERA_SIZE} )) ];do
		test -f ${0%.*}.old && old=$(sed -n "$(( ${i} + 1 ))p" ${0%.*}.old)
		case ${1} in M|MD5)
			md5=$(dd if=/dev/mapper/vg0-test bs=${ERA_SIZE} skip=${i} count=1 status=none|md5sum|sed 's,^\(......\).*,\1,')
		;;*)
			md5=$(dd if=/dev/mapper/test-era bs=${ERA_SIZE} skip=${i} count=1 status=none|md5sum|sed 's,^\(......\).*,\1,')
		;;esac
		num="$(printf "%2d" ${i})"
		case ${old} in "")
			echo "${num}: ${md5}"
		;;${md5})
			echo "${num}: ${md5}  ${old}"
		;;*)
			echo "${num}: ${md5}!=${old}"
		;;esac
		echo "${md5}" >> ${0%.*}.md5
		i=$(( ${i} + 1 ))
	done
	rm -f ${0%.*}.old
;;s|snap)
	lvcreate --extents $(( ${PE} / 8 )) --snapshot --name test-snap /dev/mapper/vg0-test
;;u|unsnap)
	lvremove --yes vg0/test-snap
;;r|remove)
	if ${era};then
		test -b /dev/mapper/test-access && dmsetup remove test-access
		test -b /dev/mapper/test-era && dmsetup remove test-era
		test -b /dev/mapper/test-meta && dmsetup remove test-meta
	fi

	test -b /dev/mapper/vg0-test && lvremove --yes vg0/test
	test -b /dev/mapper/vg0-era && lvremove --yes vg0/era

	rm -f ${0%.*}.md5
;;i|is)
	dmsetup status test-era
;;esac

digest-test.sh

#!/bin/bash

case $(id -u) in 0|root);;*)
	echo "This tests need root" >&2
	exit 1
;;esac

if ! vgs vg0>/dev/null;then
	echo "You need an LVS volume group called vg0" >&2
	exit 1
fi

${0%/*}/era-test.sh r
${0%/*}/era-test.sh c

${0%/*}/era-test.sh w | tee era.xml
time blocksync-fast --make-digest --src /dev/mapper/test-era --digest test.digest
time blocksync-fast --make-digest --src /dev/mapper/test-era --digest test-era.digest --era era.xml
dd if=test.digest skip=1 status=none|md5sum
dd if=test-era.digest skip=1 status=none|md5sum

mkfs.ext4 -F /dev/mapper/test-era
${0%/*}/era-test.sh t
${0%/*}/era-test.sh w | tee era.xml

time blocksync-fast --make-digest --src /dev/mapper/test-era --digest test.digest
time blocksync-fast --make-digest --src /dev/mapper/test-era --digest test-era.digest --era era.xml

dd if=test.digest skip=1 status=none|md5sum
dd if=test-era.digest skip=1 status=none|md5sum

delta-test.sh

#!/bin/bash

case $(id -u) in 0|root);;*)
	echo "This tests need root" >&2
	exit 1
;;esac

if ! vgs vg0>/dev/null;then
	echo "You need an LVS volume group called vg0" >&2
	exit 1
fi

case ${1} in ad-std)
	dev=$(readlink -f /dev/mapper/test-era)
	size=$(cat /sys/class/block/${dev##*/}/size)
	truncate --size=$(( 512 * ${size} )) test.img
	time blocksync-fast --apply-delta --dst=test.img --delta test.delta
	md5sum /dev/mapper/test-era test.img
;;ad-era)
	dev=$(readlink -f /dev/mapper/test-era)
	size=$(cat /sys/class/block/${dev##*/}/size)
	truncate --size=$(( 512 * ${size} )) test-era.img
	time blocksync-fast --apply-delta --dst=test-era.img --delta test-era.delta
	md5sum /dev/mapper/test-era test-era.img
;;md-std)
	time blocksync-fast --make-delta --src /dev/mapper/test-era --digest test.digest --delta test.delta --force
;;md-era)
	${0%/*}/era-test.sh t
	${0%/*}/era-test.sh w | tee era.xml
	time blocksync-fast --make-delta --src /dev/mapper/test-era --digest test-era.digest --delta test-era.delta --force --era era.xml
;;"")
	${0%/*}/era-test.sh r
	${0%/*}/era-test.sh c

	${0%/*}/era-test.sh w | tee era.xml
	time blocksync-fast --make-digest --src /dev/mapper/test-era --digest test.digest
	time blocksync-fast --make-digest --src /dev/mapper/test-era --digest test-era.digest --era era.xml
	dd if=test.digest skip=1 status=none|md5sum
	dd if=test-era.digest skip=1 status=none|md5sum

	mkfs.ext4 -F /dev/mapper/test-era
	${0%/*}/era-test.sh t
	${0%/*}/era-test.sh w | tee era.xml

	time blocksync-fast --make-delta --src /dev/mapper/test-era --digest test.digest --delta test.delta --force
	time blocksync-fast --make-delta --src /dev/mapper/test-era --digest test-era.digest --delta test-era.delta --force --era era.xml
	dd if=test.digest skip=1 status=none|md5sum
	dd if=test-era.digest skip=1 status=none|md5sum
	dd if=test.delta skip=1 status=none|md5sum
	dd if=test-era.delta skip=1 status=none|md5sum

	rm -f test.img
	rm -f test-era.img
;;*)
	echo "After initial test with '${0}' (no args), you may use" >&2
	echo "'${0} ad-std', then change something in /dev/mapper/test-era" >&2
	echo "then '${0} md-std'. Same with ad-era, chg, ad-std..." >&2
;;esac

[sven-ola]

As a side note: this creates sparse files on the backup destination as well as with the digest on the backup source. This is OK and expected. But I may further need to overthink / test this. There is still a difference between a block with zeroes and a gap which make me nervous somehow.

[nethappen]

That looks great – I can see you’ve put a lot of work into it. I’m currently modernizing my backup server, so I’ll definitely be testing your solution as well. I’ll take a closer look at it soon. Nice work!