tests? #8

r00t- · 2015-01-08T14:31:22Z

No description provided.

r00t- · 2015-01-09T12:31:03Z

there currently are some basic unit tests using the unity test framework

we probably want to get some real-world example data and make tests from that.

mbehr1 · 2015-03-19T18:54:17Z

I'd prefer this as well. But to get some real-world examples we need a logging facility for the real data coming from the serial port.

devZer0 · 2017-09-15T10:08:34Z

https://github.com/devZer0/libsml-testing

andig · 2017-09-15T11:48:43Z

https://github.com/devZer0/libsml-testing

Interesting. @devZer0 did you by chance open a bug for the libsml crash?

devZer0 · 2017-09-15T11:52:37Z

it's a while ago that i worked with all that stuff. need to find my sources for commandline parser tool i tried to develop. if i find it and compile it against latest libsml and it still crashes, i will file a bug report. not sure why i didn't at that time....

devZer0 · 2018-01-20T21:22:50Z

what about adding my libsml-testing sml binary dumps to the volkszaehler-git repo or even add those test files to libsml repo ? think that would raise awareness....

furthermore, i found a way to somewhat simulate a serial port, even with setting a virtual baudrate - this way parser tools can be tested which do not support reading from a file or from stdin or developers can test an application with smartmeters they don't have physical access to.

maybe there are even better settings for socat, didn't test into depth, but basically it works and i think this is quite interesting/useful:

To simulate a eHz/Smart-Meter for development or troubleshooting, you can to the following:

get socat and execute
# socat -d -d pty,raw,echo=0 pty,raw,echo=0

socat will connect two pty's and look like this:

# socat -d -d pty,raw,echo=0 pty,raw,echo=0
2018/01/20 22:08:48 socat[8029] N PTY is /dev/pts/6
2018/01/20 22:08:48 socat[8029] N PTY is /dev/pts/9
2018/01/20 22:08:48 socat[8029] N starting data transfer loop with FDs [5,5] and [7,7]
connect your favourite SML parser/tool to one of the pty's socat did attach to:

# ./sml_server /dev/pts/6
now feed a binary sml dump to the other pty's:

# cat EMH_eHZ-GW8E2A500AK2.bin >/dev/pts/9

you can throttle the data rate with "pv" (pipe viewer):

# cat EMH_eHZ-GW8E2A500AK2.bin | pv -L 9600 >/dev/pts/9
watch output from your favourite SML parser/tool or troubleshoot/debug...

andig · 2018-01-21T07:16:42Z

Warum nicht einfach den. Dump einlesen? Woher die dumps- aus Deinem Repo?

devZer0 · 2018-01-21T13:55:53Z

weil es tools geben könnte die nicht von file oder stdin lesen können, weil sie nur auf serielle schnittstelle ausgelegt sind? so wie sml_server bis vor kurzem?

ja, ich meine die dumps aus meinem repo. wenn "testing" bei libsml ne rolle spielt wäre ansonsten ein verweis hilfreich, es wär ja schön wenn die dump-sammlung noch etwas wachsen würde...

andig · 2018-01-22T07:50:49Z

weil es tools geben könnte die nicht von file oder stdin lesen können, weil sie nur auf serielle schnittstelle ausgelegt sind? so wie sml_server bis vor kurzem?

Dann ist das ein Designfehler :)

es wär ja schön wenn die dump-sammlung noch etwas wachsen würde...

Gerne! Magst Du einen PR machen?

andig · 2018-01-22T08:09:30Z

@devZer0 I'd be happy to include your collected dumps in this repo as "real life" test data. Is that ok for you?

hmueller01 · 2018-01-22T08:16:31Z

I do not think that dumps belong in a lib repository. They should be kept separately with a link in the Readme.md.

Reason: Not all users need this just to compile the lib ...

andig · 2018-01-22T08:19:25Z

I do not think that dumps belong in a lib repository. They should be kept separately with a link in the Readme.md.

I still think its a good proposal for testing. How else?

hmueller01 · 2018-01-22T08:23:38Z

Yes, they are. And it is good that they exist. But not needed by everyone, as I added on my last post.

devZer0 · 2018-01-22T10:10:37Z

weil es tools geben könnte die nicht von file oder stdin lesen können, weil sie nur auf serielle
schnittstelle ausgelegt sind? so wie sml_server bis vor kurzem?

Dann ist das ein Designfehler :)

ja, aber da kann ja der endanwender nichts für oder der, der das problem eines endanwender anaylsieren will - umdesignen kann es nur der, der in seinem tool libsml implementiert hat und stdio/file-io vergessen hat...

bzgl. der binary dumps würde ich mir wünschen daß sie teil des volkszaehler-repos würden oder in libsml ein verweis auf mein repo aufgenommen wird.

hmueller01 · 2018-01-22T10:21:44Z

Also wäre ja z. B. volkszaehler/libsml-meter-dumps ein guter Ort, oder?

devZer0 · 2018-01-22T10:36:47Z

ja. schaut doch bitte neben den dumps mal über die übrigen files drüber und bewertet gerne auch das gegenwärtige verfahren so einen dump zu ziehen, wir können das repo ja auch neu aufziehen.

libsml würde ich aus dem repo-namen glaube ich rauslassen weil es ja mit libsml nicht in direktem zusammenhang steht. das sind binärwürste von echten eHz die sich zufällig zum testen von libsml eignen...

Vielleicht reicht "SMLBinaryDumps" als Name?

andig · 2018-01-22T11:02:14Z

sml-samples? 2018-01-22 11:36 GMT+01:00 devZer0 <[email protected]>:

…

ja. schaut doch bitte neben den dumps mal über die übrigen files drüber und bewertet gerne auch das gegenwärtige verfahren so einen dump zu ziehen, wir können das repo ja auch neu aufziehen. libsml würde ich aus dem repo-namen glaube ich rauslassen weil es ja mit libsml nicht in direktem zusammenhang steht. das sind binärwürste von echten eHz die sich zufällig zum testen von libsml eignen... Vielleicht reicht "SMLBinaryDumps" als Name? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALR74FyJI-gV2yERsVVWbioujwh3GmAks5tNGS_gaJpZM4DP4jF> .

andig · 2019-01-21T16:20:28Z

See proposal in #67

devZer0 · 2019-04-11T19:38:28Z

am i the first to stream and decode sml from a git repo on the internet ? :D :D :D

curl -s https://github.com/devZer0/libsml-testing | grep "js-navigation-open" | grep ".bin" | awk -F'title="|" id=' '{print $2}' | while read file;do curl -s https://raw.githubusercontent.com/devZer0/libsml-testing/master/$file | ./examples/sml_server - ;done

devZer0 · 2019-04-11T20:16:16Z

i have spend some thoughts on how to test libsml for stability/reliability to make sure it would not hang , get stuck and burn cpu or even crash.

Maybe someone wants to take a look to decide if this one is a good approach for "data obfuscation/scrambling testing" ?

what i'm doing is inserting bit-flips at random position in sml binary dump and let sml_server eat that "poisoned data" to see if it hangs or crashes

$ cat flipbit.pl
#!/usr/bin/perl
# taken from https://unix.stackexchange.com/questions/196251/change-only-one-bit-in-a-file
# Arguments:   byte (starting from 0),  bit (0-7),  filename (otherwise stdin)
$byte = shift(@ARGV);
$bit = shift(@ARGV);
undef $/;
$file=<>;
substr($file,$byte,1) = substr($file,$byte,1) ^ chr(1<<$bit);
print $file;

$ cat test.sh
#!/bin/bash
while true
 do
  ls -1 libsml-testing/*.bin  |
    while read filename
      do
       size=$(stat -c %s $filename)
       randpos=$((1 + RANDOM % $size ))
       randbit=$((1 + RANDOM %8))
       echo $filename $size $randpos $randbit
       cat $filename | ./flipbit.pl $randsize $randbit | ./libsml/examples/sml_server - >/dev/null
      done
 done

$ ./test.sh |head -n 50
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 378 6
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 1474 4
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 2105 4
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 2425 1
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 2078 5
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 2494 8
libsml: warning: could not read the whole file
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 336 8
libsml: warning: could not read the whole file
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 96 7
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 1933 5
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 529 6
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 3571 3
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 1779 7
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 3980 3
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 4096 2
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 1103 1
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 1001 3
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 2814 3
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 3708 1
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 775 5
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 1108 5
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 2948 2
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 1505 7
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 393 2
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 4085 6
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 2386 7
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 921 6
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 457 1
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 3010 5
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 368 7
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 771 1
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 4033 6
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 145 3
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 2476 3
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 2102 5
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 1388 1
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 4003 4
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 1512 8
libsml: warning: could not read the whole file
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 3461 5
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 4016 8
libsml: warning: could not read the whole file
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 2902 1
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 3518 8
libsml: warning: could not read the whole file
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 3162 1
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 660 6
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 2863 1
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 3945 3
libsml-testing/DrNeuhaus_SMARTY_ix-130.bin 4096 1942 3
libsml-testing/EMH_eHZ-GW8E2A500AK2.bin 4096 1947 6
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P_1.bin 4096 122 6
libsml-testing/EMH_eHZ-HW8E2A5L0EK2P.bin 4096 3334 7
libsml-testing/EMH_eHZ-HW8E2AWL0EK2P.bin 4096 849 7

andig · 2019-04-12T05:57:00Z

am i the first to stream and decode sml from a git repo on the internet ? :D :D :D

Would be easier to just integrate the files here and use them on travis-ci.

Maybe someone wants to take a look to decide if this one is a good approach for "data obfuscation/scrambling testing" ?

I imagine you would need a higher fuzzing rate or keep running this a large number of times before you would see any effect. I‘d rather add a flip percentage.

devZer0 · 2020-01-31T15:50:08Z

as i have crashes of sml_server binary when my optical reader was not adjusted correctly, i assume libsml is not sufficiently crash safe wen it's being sent random data.

i guess many of vzlogger crashes are a result of this.

could someone recommend some tools to do "in stream data obfuscation", like swapping positions, removing/truncating/zeroing parts of the stream?

update:
found this one
https://github.com/nisrine/Fault-Injection-Tool

andig · 2020-01-31T18:28:09Z

Should be simple to build one. Anything that can pipe stdin to stdout and allows to configure a bit flip probability (or any other type of error situations). Could build one if you need it.

andig · 2020-01-31T18:29:00Z

Actually, re-reading the thread- you already have the perl script, what else is needed?

devZer0 · 2020-01-31T21:00:48Z

i think more complex "scrambling" would be useful:

duplication of datastream contents
deletion of datastream contents
cutting out parts of the datastream
more intense/random overwrite/fuzzing
simulating serial communication problems, i.e. "virtually" receiving data with the wrong bitrate for example, as this would probably flip more bits at once

in other words: feed all sorts of "damaged" data to libsml to see how it behaves.

does it hang? (->watchdog to kill it and restart?)
does it crash? (->restart)
does it output wrong/problematic information which could make some other tool go crazy?

andig · 2020-02-01T09:39:42Z

Repeating myself here- why not simply add to the perl script above?

devZer0 · 2020-02-01T11:33:44Z

if it would be simple for me, i would have done it....

r00t- added enhancement question labels Jan 8, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests? #8

tests? #8

r00t- commented Jan 8, 2015

r00t- commented Jan 9, 2015

mbehr1 commented Mar 19, 2015

devZer0 commented Sep 15, 2017

andig commented Sep 15, 2017

devZer0 commented Sep 15, 2017 •

edited

Loading

devZer0 commented Jan 20, 2018 •

edited

Loading

andig commented Jan 21, 2018

devZer0 commented Jan 21, 2018

andig commented Jan 22, 2018 •

edited

Loading

andig commented Jan 22, 2018

hmueller01 commented Jan 22, 2018 •

edited

Loading

andig commented Jan 22, 2018

hmueller01 commented Jan 22, 2018

devZer0 commented Jan 22, 2018

hmueller01 commented Jan 22, 2018

devZer0 commented Jan 22, 2018

andig commented Jan 22, 2018 via email •

edited

Loading

andig commented Jan 21, 2019

devZer0 commented Apr 11, 2019

devZer0 commented Apr 11, 2019 •

edited

Loading

andig commented Apr 12, 2019

devZer0 commented Jan 31, 2020 •

edited

Loading

andig commented Jan 31, 2020

andig commented Jan 31, 2020

devZer0 commented Jan 31, 2020

andig commented Feb 1, 2020

devZer0 commented Feb 1, 2020

tests? #8

tests? #8

Comments

r00t- commented Jan 8, 2015

r00t- commented Jan 9, 2015

mbehr1 commented Mar 19, 2015

devZer0 commented Sep 15, 2017

andig commented Sep 15, 2017

devZer0 commented Sep 15, 2017 • edited Loading

devZer0 commented Jan 20, 2018 • edited Loading

andig commented Jan 21, 2018

devZer0 commented Jan 21, 2018

andig commented Jan 22, 2018 • edited Loading

andig commented Jan 22, 2018

hmueller01 commented Jan 22, 2018 • edited Loading

andig commented Jan 22, 2018

hmueller01 commented Jan 22, 2018

devZer0 commented Jan 22, 2018

hmueller01 commented Jan 22, 2018

devZer0 commented Jan 22, 2018

andig commented Jan 22, 2018 via email • edited Loading

andig commented Jan 21, 2019

devZer0 commented Apr 11, 2019

devZer0 commented Apr 11, 2019 • edited Loading

andig commented Apr 12, 2019

devZer0 commented Jan 31, 2020 • edited Loading

andig commented Jan 31, 2020

andig commented Jan 31, 2020

devZer0 commented Jan 31, 2020

andig commented Feb 1, 2020

devZer0 commented Feb 1, 2020

devZer0 commented Sep 15, 2017 •

edited

Loading

devZer0 commented Jan 20, 2018 •

edited

Loading

andig commented Jan 22, 2018 •

edited

Loading

hmueller01 commented Jan 22, 2018 •

edited

Loading

andig commented Jan 22, 2018 via email •

edited

Loading

devZer0 commented Apr 11, 2019 •

edited

Loading

devZer0 commented Jan 31, 2020 •

edited

Loading