ltfs_ordered_copy is really slow

[rezo552]

Hi,

Im still trying to understand what is the best way to retrieve files in batch from LTFS. The ltfs_ordered_copy takes ages to retrieve couple of hundreds of megs and it seems the drive is constantly seeking. Are HP drives supported at all? Thanks

Z.

After 40 minutes, I was only able to retrieve 17 files (cca. 20 megs each)

root@pop-os:/home/rezo# ./ltfs_ordered_copy /ltfs/n64/ /home/rezo/ -r
Tape order aware copy for LTFS
Destination /home/rezo/ is not LTFS
On disk sources: 1058
Source tapes: 0
Copying on 1058 disk files with 8 threads
File copy from disk is on going: 17/1058^C^Y^Z

[piste2750]

I believe this doesn't depend on the drive but depends on the version of LTFS.

Which LTFS version are you using on /ltfs?

From the log of ltfs_ordered_copy, it doesn't detect specified files is not on the LTFS.

On disk sources: 1058
Source tapes: 0
Copying on 1058 disk files with 8 threads
File copy from disk is on going: 17/1058

The LTFS you are using needs to return a correct data against VEAs, ltfs.volumeUUID, ltfs.partition and ltfs.startblock. (In Linux, user. prefix is required like user.ltfs.volumeUUID)

Once ltfs_ordered_copy capture those info, it tries to reorder the source files by ltfs.startblock. As a result, seek might be minimized.

The code below is your reference.

ltfs/src/utils/ltfs_ordered_copy

Lines 61 to 78 in 42c0871

	def eval(self): #access to the extended attributes present in some operating systems/filesystems by xattr
	try:
	self.vuuid = xattr.get(self.src, self.vea_pre + 'ltfs.volumeUUID')
	except Exception as e:
	self.vuuid = ''
	return (self.vuuid, self.part, self.start)
	try:
	self.part = xattr.get(self.src, self.vea_pre + 'ltfs.partition')
	start_str = xattr.get(self.src, self.vea_pre + 'ltfs.startblock')
	self.start = int(start_str)
	self.size = os.path.getsize(self.src)
	except Exception as e:
	self.logger.error('Failed to get attribute of "{0}": {1}'.format(self.src, str(str(e))))
	self.part = b'a'
	self.start = 0
	return (self.vuuid, self.part, self.start)

[rezo552]

Im using this version:
rezo@pop-os:~$ ltfs -V
LTFS version 2.5.0.0 (Prelim).
LTFS Format Specification version 2.4.0.

Does it matter if the drive was NOT written with ltfs_order_copy?

[rezo552]

If Im copying to the tape it seems to detect LTFS

Tape order aware copy for LTFS
Destination /ltfs is LTFS
On disk sources: 15082
Source tapes: 0
Copying on 15082 disk files with 1 threads
File copy from disk is on going: 630/15082

[piste2750]

I would like to check the LTFS returns VEA correctly.

Could you try the commands below against a source file ?

Linux:

attr -g ltfs.volumeUUID [srcfile]
attr -g ltfs.partition [srcfile]
attr -g ltfs.ltfs.startblock [srcfile]

Mac:

xattr -p ltfs.volumeUUID [srcfile]
xattr -p ltfs.partition [srcfile]
xattr -p ltfs.ltfs.startblock [srcfile]

[rezo552]

now I got it working

root@pop-os:/home/rezo# find /ltfs -type f | ./ltfs_ordered_copy -t /smbnasusb --keep-tree=/ltfs
Tape order aware copy for LTFS
Destination /smbnasusb is not LTFS

On disk sources: 0
Source tapes: 1
Processing tape b2ce8374-7e34-41a7-bf1a-818171a69993, Partition b
Sorting 15083 files on b2ce8374-7e34-41a7-bf1a-818171a69993 (b)
Copying files on b2ce8374-7e34-41a7-bf1a-818171a69993 (b)
File copy from tape is on going: 15083/15083
Copied 15083 files.

diff.txt

The issue was with the xattr python library, some functions are renamed

[piste2750]

Thank you for your investigation. And it is interesting...

I confirmed the functions you pointed out was obsoleting from v0.4. So it is a time to with to new functions as you fixed. I will do that in the near future.

[Nooby1]

for me on python 3.9, without this change the script just breaks and will not run

python3-xattr is already the newest version (0.9.7-1+b1).

module "xattr" has no attribute "get"

[piste2750]

Thank you for your investigation. And it is interesting...

I confirmed the functions you pointed out was obsoleting from v0.4. So it is a time to with to new functions as you fixed. I will do that in the near future.

@rezo552 , @Nooby1

I tried to work this today but I realize something funny. From the document of pyxattr module, getxattr() and setxattr() is obsoleting from v0.4.. -> See https://pyxattr.k1024.org/module.html#module-xattr

The ltfs_ordered_copy requires the pyxattr module and it is described here.

Could you confirm you are using the pyxattr module? I would like to note that there is another module xattr to handle extended attribute.

• pyxattr module -> https://pypi.org/project/pyxattr/
• xattr module -> https://pypi.org/project/xattr/

[Nooby1]

I was using the xattr module, not pyxattr, the script seems to work with some small changes with xattr however

[ia2115]

can you provide the edit verion of the script please? on freebsd its just broken nomatter what i try :/