Skip to content

Commit

Permalink
Merge pull request #13 from dist-sys/refactor_v0.6.3
Browse files Browse the repository at this point in the history 
Refactor v0.6.3
  • Loading branch information
@nabon
nabon authored Sep 9, 2020
2 parents ff5f767 + 33ad4f3 commit b1c08bc
Show file tree
Hide file tree
Showing 20 changed files with 903 additions and 991 deletions.
132 changes: 59 additions & 73 deletions doc/usage_en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ Download the archive file (zip or tar) from: https://github.com/dist-sys/mqttloa
By extracting it, you can get the following files.

```
mqttloader
+-- bin
mqttloader/
+-- bin/
+-- mqttloader
+-- mqttloader.bat
+-- lib
+-- lib/
+-- logging.properties
```

Scripts for executing MQTTLoader is in *bin* directory.
Expand All @@ -26,10 +27,11 @@ You can display the help by:
`$ ./mqttloader -h`

```
MQTTLoader version 0.7.0
usage: mqttloader.Loader -b <arg> [-v <arg>] [-p <arg>] [-s <arg>] [-pq
<arg>] [-sq <arg>] [-ss] [-r] [-t <arg>] [-d <arg>] [-m <arg>] [-ru
<arg>] [-rd <arg>] [-i <arg>] [-st <arg>] [-et <arg>] [-l <arg>]
[-n <arg>] [-tf <arg>] [-lf <arg>] [-h]
[-n <arg>] [-im] [-h]
-b,--broker <arg> Broker URL. E.g., tcp://127.0.0.1:1883
-v,--version <arg> MQTT version ("3" for 3.1.1 or "5" for 5.0).
:
Expand All @@ -48,6 +50,7 @@ For example, the following command uses a public MQTT broker provided by HiveMQ.

### Run on multiple machines
You can run MQTTLoader on multiple machines.

Running both publishers and subscribers on a single machine may cause mutual influence, e.g., the subscribers' receiving load lowers the publishers' throughput.
By running publishers and subscribers separately on different machines, you can avoid such mutual influence.

Expand Down Expand Up @@ -81,28 +84,27 @@ Please refer to **3. Parameteres of MQTTLoader** for more details of each parame
| -ss | | Enable shared subscription. By default, it is disabled. Valid for only MQTT v5.0. <br>If it is enabled, a message is delivered to one of the subscribers. |
| -r | | Enable retain for the messages sent by publishers. By default, it is disabled. |
| -t \<arg\> | mqttloader-test-topic | Topic name to be used. |
| -d \<arg\> | 20 | The size of data (payload of messages to be published) in bytes. |
| -d \<arg\> | 20 | The size of data (payload of messages to be published) in bytes. It must be equal to or larger than 8. |
| -m \<arg\> | 100 | Number of messages sent by **each** publisher. |
| -ru \<arg\> | 0 | Ramp-up time in seconds. <br>See **4. How to read the results** for details. |
| -rd \<arg\> | 0 | Ramp-down time in seconds. <br>See **4. How to read the results** for details. |
| -i \<arg\> | 0 | Publish interval in milliseconds. |
| -st \<arg\> | 5 | Timeout for receiving messages by subscribers in seconds. |
| -et \<arg\> | 60 | Maximum execution time for measurement in seconds. |
| -l \<arg\> | WARNING | Log level. <br>Valid values are `SEVERE`/`WARNING`/`INFO`/`ALL`. |
| -l \<arg\> | INFO | Log level. <br>Valid values are `SEVERE`/`WARNING`/`INFO`/`ALL`. |
| -n \<arg\> | (none) | URL of the NTP server. By setting this, time synchronization is enabled. <br>Ex. `ntp.nict.jp` |
| -tf \<arg\> | (none) | File name to write out the throughput data. By default, file output is disabled. |
| -lf \<arg\> | (none) | File name to write out the latency data. By default, file output is disabled. |
| -im \<arg\> | (none) | Run MQTTLoader by in-memory mode. By default, MQTTLoader writes out measurement records to a file. |
| -h | | Display help. |

MQTTLoader starts to terminate when all of the following conditions are met.
- All publishers complete to send out messages.
- The time specified by the parameter `-st` elapses from the last time subscribers receive a message.

MQTTLoader also starts to terminate when the time specified by the parameter `-et` elapses, even if there are in-flight messages.
Thus, `-et` should be long sufficiently.
Thus, if you want to test fixed number of messages, `-et` should be long sufficiently.

If you want to do measurement with fixed time period, you can set the measurement time by the parameter `-et`.
Note that you need to set sufficiently large value to the parameter `-m`.
In this case, you need to set sufficiently large value to the parameter `-m`.

By setting the parameter `-n`, MQTTLoader obtains the offset time from the specified NTP server and reflects it to calculate throughput and latency.
It might be useful for running multiple MQTTLoader on different machines.
Expand All @@ -116,86 +118,71 @@ MQTTLoader displays results like the following on standard output.
Maximum throughput[msg/s]: 18622
Average throughput[msg/s]: 16666.666666666668
Number of published messages: 100000
Throughput[msg/s]: 11955, 16427, 18430, 18030, 18622, 16536
Per second throughput[msg/s]: 11955, 16427, 18430, 18030, 18622, 16536
-----Subscriber-----
Maximum throughput[msg/s]: 18620
Average throughput[msg/s]: 16666.666666666668
Number of received messages: 100000
Throughput[msg/s]: 11218, 16414, 18426, 18026, 18620, 17296
Per second throughput[msg/s]: 11218, 16414, 18426, 18026, 18620, 17296
Maximum latency[ms]: 81
Average latency[ms]: 42.23691
```
For each publisher, MQTTLoader counts the number of messages sent for each second.
MQTTLoader counts the number of messages sent by publishers.
If QoS level is set to 1 or 2, counting is done when receiving PUBACK or PUBCOMP respectively.

After completion, MQTTLoader collects the counted numbers from all publishers and calculates the maximum throughput, the average throughput, and the number of published messages.
`Throughput[msg/s]` is the list of throughputs, which are the sum of each second for all publishers.
Note that these calculation exclude the beginning and trailing seconds that have 0 messages.
Below is an example of calculating throughputs in the case that two publishers, A and B, send messages.

| Elapsed seconds from starting measurement | # of meessages from A | # of messages from B | Throughputs |
|:-----------|:------------|:------------|:------------|
| 0 | 0 | 0 | Excluded |
| 1 | 3 | 0 | 3 |
| 2 | 4 | 3 | 7 |
| 3 | 5 | 5 | 10 |
| 4 | 0 | 0 | 0 |
| 5 | 3 | 4 | 7 |
| 6 | 2 | 2 | 4 |
| 7 | 0 | 0 | Excluded |
| 8 | 0 | 0 | Excluded |

By using the parameterse `-ru` and `-rd`, you can further exclude the beginning and trailing data.
If you set `-ru 1 -rd 1` in the above example, the following data is used.

| Elapsed seconds from starting measurement | # of meessages from A | # of messages from B | Throughputs |
|:-----------|:------------|:------------|:------------|
| 2 | 4 | 3 | 7 |
| 3 | 5 | 5 | 10 |
| 4 | 0 | 0 | 0 |
| 5 | 3 | 4 | 7 |
After completion, MQTTLoader calculates the maximum throughput, the average throughput, and the number of published messages.
`Per second throughput[msg/s]` is the time series of throughputs per second.

By using the parameterse `-ru` and `-rd`, you can exclude the beginning and trailing data.
If you set `-ru 1 -rd 1` for example, the beginning one second and the trailing one second are excluded.

For subscribers, throughputs are calculated as same as the above for the received messages.
In addition, the maximum latency and the average latency are calculated.
Latency is the required time from sending out by a publisher to receiving by a subscriber.
Each message has a timestamp of sending out in its payload and the subscriber receives it calculates the latency.

To calculate the latency accurately, the clocks of pubilshers and subscribers should be the same or synchronized.
Thus, when running multiple MQTTLoader on different machines (e.g., publishers on a machine and subscriber on another), enabling `-n` parameter can improve the calculation of latency.
When running multiple MQTTLoader on different machines (e.g., publishers on a machine and subscriber on another), it is better to use `-n` parameter.
By using `-n` parameter, MQTTLoader acquires time information from the specified NTP server and uses it for timestamps and calculation.

### Data to file
By specifying the file name with `-tf` parameter, you can obtain throughput data like the following.
### Send/Receive record file
By default, MQTTLoader writes out the record of sending/receiving MQTT messages to a file.
As shown below, a file `mqttloader_xxxxxxxx-xxxxxx.csv` is created in `mqttloader` directory.
The file name is generated from the measurement start time.
Note that in the case of running MQTTLoader by Gradle or IDE, the file is created in the current working directory.

```
Measurement start time: 2020-09-01 18:33:38.122 JST
Measurement end time: 2020-09-01 18:33:54.104 JST
SLOT, mqttloaderclient-pub000000, mqttloaderclient-sub000000
0, 11955, 11218
1, 16427, 16414
2, 18430, 18426
3, 18030, 18026
4, 18622, 18620
5, 16536, 17296
mqttloader/
+-- bin/
+-- mqttloader
+-- mqttloader.bat
+-- lib/
+-- logging.properties
+-- mqttloader_xxxxxxxx-xxxxxx.csv
```
This indicates the throughput for each second for each publisher.
The data that used to calculate the summary data in the standard output is written out.

By specifying the file name with `-lf` parameter, you can obtain latency data like the following.
The file `mqttloader_xxxxxxxx-xxxxxx.csv` has records like the following:

```
Measurement start time: 2020-09-01 18:33:38.122 JST
Measurement end time: 2020-09-01 18:33:54.104 JST
mqttloaderclient-sub000000, mqttloaderclient-sub000001
7, 7
4, 4
3, 3
4, 4
3, 4
3, 3
4, 4
3, 4
1599643916416,ml-EeiE-p-00001,S,
1599643916416,ml-EeiE-p-00000,S,
1599643916419,ml-EeiE-s-00000,R,3
1599643916422,ml-EeiE-p-00001,S,
:
:
```
This indicates the latency for each message for each subscriber.

Each line, consists of comma-separeted values, indicates the following data.
In the case that the event type is `R`, latency data follows.

```
timestamp (Unix time), client ID, event type (S: send, R: receive), latency
```

Although MQTTLoader outputs the measurement result to the console, you can use the above .csv file for further analysis.
If you want to avoid the influence of file I/O on the measurement, you can run MQTTLoader with in-memory mode by using the `-im` parameter.
In this case, the above .csv file is not created.

---
---
Expand All @@ -216,11 +203,10 @@ Clone the MQTTLoader repository from GitHub: `$ git clone [email protected]:dist-sy
The structure of the directories/files is as follows:

```
mqttloader
+-- docs
+-- src
mqttloader/
+-- doc/
+-- src/
+-- build.gradle
+-- logging.properties
:
```

Expand All @@ -237,9 +223,9 @@ If successful, *build* directory is created under *\<ROOT_DIR\>*.
You can find *distributions* directory under the *build* directory.

```
<ROOT_DIR>
+-- build
+-- distributions
<ROOT_DIR>/
+-- build/
+-- distributions/
+-- mqttloader.tar
+-- mqttloader.zip
```
Expand Down
Loading

0 comments on commit b1c08bc

Please sign in to comment.