Can not read .mf4 file

[meidland]

Hi Ratal!

Firstly, thanks for the mdfreader module :)
So, I have an issue. I cannot read a mf4 file. The file has been validated with MDFValidator (Vector).
Tried to run it in Linux environment also.

Code:
from mdfreader import mdf4

my_object = mdf4()
my_object.read4(r'c:\temp\jonas\Epic.mf4')

Output:
Traceback (most recent call last):
File "C:/code_projects/mdfreader/main.py", line 4, in
my_object.read4(r'c:\temp\jonas\Epic.mf4')
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 1393, in read4
buf.read(channelSet) # reads raw data from data block with DATA and DATABlock classes
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 350, in read
self[recordID]['data'] = self.load(record, zip=None, nameList=channelSet, sortedFlag=True)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 387, in load
temps.loadHeader(self.fid, self.pointerTodata)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdfinfo4.py", line 93, in loadHeader
fid.seek(pointer)
ValueError: seek of closed file

Thanks in advance!

[ratal]

Hi Meidland,
I lately made modification to allow opening directly from submodule (in master, you might use released version)
However, I would recommend you to open it using main module mdfreader, not mdf4reader:
from mdfreader import mdf my_object=mdf()
Version of file will be detected and correct submodule used.

[meidland]

Hi again!

Now it worked better :) But I'm still having some issues.

Input:

from mdfreader import mdf

my_object = mdf()
my_object.read('my_file.mf4')

Output:

C:\code_projects\mdfreader\mdfenv\Scripts\python.exe C:/code_projects/mdfreader/main.py
Unexpected error: (<class 'ImportError'>, ImportError("No module named 'dataRead'",), <traceback object at 0x03DA9530>)
dataRead crashed, back to python data reading
Unexpected error: (<class 'ImportError'>, ImportError("No module named 'dataRead'",), <traceback object at 0x03DAC5D0>)
dataRead crashed, back to python data reading
Unexpected error: (<class 'ImportError'>, ImportError("No module named 'dataRead'",), <traceback object at 0x03DB07B0>)
dataRead crashed, back to python data reading
Unexpected error: (<class 'ImportError'>, ImportError("No module named 'dataRead'",), <traceback object at 0x03D93080>)
dataRead crashed, back to python data reading
Traceback (most recent call last):
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 1200, in readBitarray
from dataRead import dataRead
ImportError: No module named 'dataRead'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/code_projects/mdfreader/main.py", line 4, in
my_object.read(r'c:\temp\jonas\Epic.mf4')
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdfreader.py", line 362, in read
self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 1393, in read4
buf.read(channelSet) # reads raw data from data block with DATA and DATABlock classes
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 357, in read
temp = temp.load(record, zip=None, nameList=channelSet, sortedFlag=True)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 443, in load
temps['data'] = self.load(record, zip=temps['hl_zip_type'], nameList=nameList, sortedFlag=sortedFlag)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 432, in load
temps['data'] = DATABlock(record, parent_block=data_block, channelSet=nameList, sortedFlag=sortedFlag)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 136, in DATABlock
return record.readBitarray(parent_block['data'], channelSet)
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 1259, in readBitarray
for i in range(self.numberOfRecords)]
File "C:\code_projects\mdfreader\mdfenv\lib\site-packages\mdfreader\mdf4reader.py", line 1259, in
for i in range(self.numberOfRecords)]
struct.error: unpack requires a bytes object of length 8

[ratal]

Hi Meidland,
Your last lines are duplicated and I do not have final error line. But I guess this is this code : self[chan].CFormat.unpack(temp[i].tobytes())[0]
I supposed the temp data type does not match the CFormat. Maybe you can print it to confirm ?
Regarding dataRead cython extension, it is apparently not compiled properly.

[ratal]

I guess you compiled dataRead and solved your issue (the python backup is not as robust)
As no feed back since a while closed but do not hesitate to reopen

[Nimi42]

I'm having a similar issue. Don't know the exact problem.
The same code works for another file.

Traceback (most recent call last):
File "C:\Users\mousavin\workspace\Clustering\run.py", line 19, in
reader.read(src)
File "C:\AppData\Anaconda3\lib\site-packages\mdfreader\mdfreader.py", line 398, in read
convertAfterRead, filterChannelNames, compression)
File "C:\AppData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py", line 874, in read3
info.readCGBlock(info.fid, dataGroup, minimal=minimal)
File "C:\AppData\Anaconda3\lib\site-packages\mdfreader\mdfinfo3.py", line 238, in readCGBlock
read_tx_block(fid, self['CNBlock'][dg][cg][channel]['pointerToSignalIdentifierBlock'])
File "C:\AppData\Anaconda3\lib\site-packages\mdfreader\mdfinfo3.py", line 590, in read_tx_block
block_size) = tx_struct.unpack(fid.read(4))
struct.error: unpack requires a buffer of 4 bytes

[Nimi42]

I fixed it myself, but I'm not sure if my solution is correct.

For some reason mdfinfo3.py", line 590, in read_tx_block
fid.read(4) returns an empty byte.

An if condition fixes the problem.

def read_tx_block(fid, pointer):
    """ reads text block """
    if pointer != 0 and pointer is not None:
        fid.seek(pointer)
        read_bytes = fid.read(4)
        if read_bytes:  # <--- I added this line
            (block_type,block_size) = tx_struct.unpack(read_bytes)
            text = unpack('{}s'.format(block_size - 4), fid.read(block_size - 4))[0]

            return text.rstrip(b'\x00').decode('latin1', 'replace')  # .encode('utf-8')

Dunno, if my file is corrupt or not. Could you please clarify.

Edit:
fid.read(4) should not be called twice

[danielhrisca]

@Nimi42
Hi, can you send the file (you can find my email on my profile page).

[ratal]

Looking in internet, empty return from read would mean either file is empty or pointer is at end of file.
Can you print pointer and check it is within file ? You can confirm by reading it with MDFValidator from Vector. Or you can send the file as proposed Daniel.
Be carefull with your modification, calling 2 times fid.read(4) consecutively will create lot of issues as it will not return the same portion of file.

[Nimi42]

First of all,

thank you for your quick reply.

@ratal
Yes. I just realised. It does not fix the problem though. The file can be read with this work around.
The labels are correct, but the values are not.

I might add that it is possible to open up the file with MDA.

Since I could not find your mail address I did not send you the file though.

@danielhrisca
I send you the file via mail.

Edit:
Seems like the mdf version is very old. Something like 214.
Is the mdfreader module able to handle these old version?.

[danielhrisca]

@ratal
I think mdfreader reads 228 bytes for CNBLOCK. In .dat version 2.14 the CNBLOCK only has 222 bytes (there is no display name and additional byte offset).

[danielhrisca]

2.14 file

3.10 file

[Nimi42]

Well unfortunately I can't change the format in the recording tool, because I am not the one recording
the files.

Also asammdf could not handle the 2.14 version either.

It is possible to use MDA to convert the files, and use asammdf or mdfreader to read
them with python, but that is rather tedious.

I was hoping the libs would be able to handle it.

[danielhrisca]

@Nimi42
I was able to parse the file that you have send over email with the code in the development branch. Do you have problems with other files?

[ratal]

My email address is [email protected]
I will also have a look at it.

[Nimi42]

@ratal
Thank you very much

@danielhrisca
I tried the module

from asammdf import MDF
            
reader = MDF(src)
reader.select(['VehV_v'])

throws an exception with asammdf (2.7.1):

  File "C:\Users\mousavin\AppData\Roaming\Python\Python36\site-packages\asammdf\mdf.py", line 59, in __init__
    raise MdfException(message.format(name, version))
asammdf.utils.MdfException: "C:\AppData\MF_RDE-Bewertung\Daten\20171020_HN-LI8220_Stadt_00001.dat" is not a supported MDF file; "    " file version was found

Edit:
I might add that I'm a little bit confused by the API, because in the code it seems like
filter and select do the same thing.

[danielhrisca]

@Nimi42
I've added initial support for MDF version 2.14 to the development branch https://github.com/danielhrisca/asammdf/tree/development/asammdf . I think issues not related to mdfreader should continue on the asammdf issue tracker.

[Nimi42]

@danielhrisca

Oh yes I see. But the values are still not correct. The same as with my quick fix above.
It does not seem to be that easy to add support for the old version.

I'm sorry for the trouble.

[Nimi42]

np.set_printoptions(threshold=np.nan)

reader = MDF(os.path.join(root, f))
signal = reader.select(['Zeit'])
print(signal[0].samples)

Prints:

[  1.08697600e+03   4.26298573e+04   3.08700774e+04   0.00000000e+00
   0.00000000e+00   0.00000000e+00 ... ]

But it should be:

[   4.246      5.238      6.238      7.238      8.23799    9.23799 ... ]

Similarly with mdfreader

[ratal]

issue reopened

[danielhrisca]

But it should be:

[ 4.246 5.238 6.238 7.238 8.23799 9.23799 ... ]

Similarly with mdfreader

What application did you use to confirm those values?

[danielhrisca]

@Nimi42
You're right. CANape shows correct values. It must be something strange with the data layout in version 2 (compared to versions 3 and 4)

[ratal]

Currently mdfreader is not able to read version 2.x of MDF. As it seems relatively simple from 3.x, it could be implemented but it will require a bit of time.

[raffsa]

Hello,

I have a similar Problem that i get an error code when i read my mf4 file. Is there any solution to this?

---

StopIteration Traceback (most recent call last)
in ()
13
14
---> 15 mdf = MDF('/media/data2/jupyter_notebooks/sacherr/Files_test/213-2047_20170306205838_70A.mf4')

/usr/local/lib/python3.6/site-packages/asammdf/mdf.py in init(self, name, memory, version, callback)
96 self._mdf = MDF3(name, memory, callback=callback)
97 elif version in MDF4_VERSIONS:
---> 98 self._mdf = MDF4(name, memory, callback=callback)
99 elif version in MDF2_VERSIONS:
100 self._mdf = MDF2(name, memory, callback=callback)

/usr/local/lib/python3.6/site-packages/asammdf/mdf_v4.py in init(self, name, memory, version, callback)
340 if name:
341 self._file = open(self.name, 'rb')
--> 342 self._read()
343
344 else:

/usr/local/lib/python3.6/site-packages/asammdf/mdf_v4.py in _read(self)
596 size=size,
597 )
--> 598 data = next(data)
599
600 if record_id_nr == 0:

StopIteration:

[danielhrisca]

@raffsa
this is obviously an error in the asammdf package, not in mdfreader. Please open an issue there.