Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect Read (Avg) calculation in Local mining stats for coordinated mining #598

Open
kslazarev opened this issue Aug 1, 2024 · 2 comments

Comments

@kslazarev
Copy link

I've noticed an issue with the Read (Avg) calculation in the Local mining stats for coordinated mining. The Read (Avg) values seem to be incorrectly calculated across all metrics, while the Hash (Avg) appears to be correct (proportional Data Size).

Here are the stats from three nodes in our coordinated mining setup:

Node1 (exit_node):

  • Data Size: 57.3 TiB
  • Partitions: 20
  • Read (Avg): 3829.9 MiB/s (expected 3482 MiB/s)
  • Hash (Avg): 5046 h/s
Local mining stats:
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
| Partition | Data Size | % of Max |   Read (Cur)  |   Read (Avg)  |  Read (Ideal) | Hash (Cur) | Hash (Avg) | Hash (Ideal) |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
|     Total |  57.3 TiB |     30 % |  4457.2 MiB/s |  3829.9 MiB/s |  2056.2 MiB/s |   5616 h/s |   5046 h/s |     1989 h/s |
|         1 |   1.7 TiB |     52 % |    87.4 MiB/s |   109.6 MiB/s |    62.2 MiB/s |    182 h/s |    146 h/s |       60 h/s |
|         4 |   2.6 TiB |     78 % |   299.3 MiB/s |   176.1 MiB/s |    92.1 MiB/s |    335 h/s |    225 h/s |       89 h/s |
|         8 |   2.3 TiB |     70 % |   351.6 MiB/s |   163.4 MiB/s |    83.2 MiB/s |    574 h/s |    205 h/s |       80 h/s |
|         9 |   2.6 TiB |     79 % |   210.6 MiB/s |   182.3 MiB/s |    93.0 MiB/s |    196 h/s |    230 h/s |       90 h/s |
|        13 |   3.3 TiB |     99 % |   219.8 MiB/s |   226.9 MiB/s |   117.0 MiB/s |    249 h/s |    285 h/s |      113 h/s |
|        14 |   3.2 TiB |     99 % |   139.9 MiB/s |   186.8 MiB/s |   116.4 MiB/s |    124 h/s |    286 h/s |      112 h/s |
|        17 |   3.2 TiB |     99 % |   328.4 MiB/s |   229.2 MiB/s |   116.4 MiB/s |    472 h/s |    295 h/s |      112 h/s |
|        19 |   2.8 TiB |     86 % |   270.8 MiB/s |   195.4 MiB/s |   102.0 MiB/s |    336 h/s |    249 h/s |       98 h/s |
|        24 |   3.2 TiB |     98 % |   195.2 MiB/s |   221.9 MiB/s |   115.3 MiB/s |    322 h/s |    281 h/s |      111 h/s |
|        26 |   3.2 TiB |     98 % |   229.4 MiB/s |   216.0 MiB/s |   115.9 MiB/s |    285 h/s |    279 h/s |      112 h/s |
|        27 |   3.2 TiB |     98 % |   189.8 MiB/s |   218.7 MiB/s |   115.8 MiB/s |    276 h/s |    288 h/s |      112 h/s |
|        30 |   1.9 TiB |     58 % |   154.2 MiB/s |   130.5 MiB/s |    68.6 MiB/s |    196 h/s |    168 h/s |       66 h/s |
|        32 |   2.1 TiB |     65 % |   330.5 MiB/s |   146.3 MiB/s |    76.4 MiB/s |    228 h/s |    186 h/s |       73 h/s |
|        33 |   2.8 TiB |     85 % |   162.8 MiB/s |   192.6 MiB/s |    99.9 MiB/s |    123 h/s |    243 h/s |       96 h/s |
|        34 |   3.2 TiB |     96 % |   209.7 MiB/s |   216.3 MiB/s |   113.4 MiB/s |    225 h/s |    277 h/s |      109 h/s |
|        38 |   3.1 TiB |     94 % |   279.9 MiB/s |   217.1 MiB/s |   111.3 MiB/s |    269 h/s |    276 h/s |      107 h/s |
|        45 |   3.3 TiB |     99 % |   169.7 MiB/s |   218.9 MiB/s |   116.8 MiB/s |    247 h/s |    287 h/s |      113 h/s |
|        46 |   3.1 TiB |     94 % |   148.4 MiB/s |   167.5 MiB/s |   111.2 MiB/s |    339 h/s |    261 h/s |      107 h/s |
|        50 |   3.2 TiB |     98 % |   159.9 MiB/s |   201.1 MiB/s |   116.0 MiB/s |    164 h/s |    286 h/s |      112 h/s |
|        51 |   3.2 TiB |     96 % |   319.9 MiB/s |   213.2 MiB/s |   113.3 MiB/s |    464 h/s |    283 h/s |      109 h/s |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+

Coordinated mining cluster stats:
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                 Peer | H1 Out (Cur) | H1 Out (Avg) | H1 In (Cur) | H1 In (Avg) | H2 Out |  H2 In |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                  All |     3861 h/s |     3983 h/s |    4016 h/s |    3392 h/s |      0 |      0 |
|     95.54.82.50:1986 |     2272 h/s |     2218 h/s |    2035 h/s |    1560 h/s |      0 |      0 |
|     95.54.82.50:1985 |     1589 h/s |     1765 h/s |    1981 h/s |    1831 h/s |      0 |      0 |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+

Node2:

  • Data Size: 59.1 TiB
  • Partitions: 20
  • Read (Avg): 3115.4 MiB/s (expected 3594 MiB/s)
  • Hash (Avg): 5394 h/s
Local mining stats:
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
| Partition | Data Size | % of Max |   Read (Cur)  |   Read (Avg)  |  Read (Ideal) | Hash (Cur) | Hash (Avg) | Hash (Ideal) |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
|     Total |  59.1 TiB |     31 % |  3182.4 MiB/s |  3115.4 MiB/s |  2356.4 MiB/s |   5391 h/s |   5394 h/s |     2332 h/s |
|         0 |   1.9 TiB |     57 % |    88.3 MiB/s |   109.1 MiB/s |    75.4 MiB/s |    148 h/s |    172 h/s |       74 h/s |
|         2 |   2.0 TiB |     61 % |   108.3 MiB/s |   119.7 MiB/s |    80.3 MiB/s |    131 h/s |    181 h/s |       79 h/s |
|         3 |   2.7 TiB |     81 % |   112.5 MiB/s |   128.7 MiB/s |   106.1 MiB/s |    202 h/s |    242 h/s |      105 h/s |
|         7 |   2.4 TiB |     73 % |   151.2 MiB/s |   141.2 MiB/s |    95.6 MiB/s |    275 h/s |    224 h/s |       94 h/s |
|        10 |   3.0 TiB |     91 % |   217.9 MiB/s |   176.1 MiB/s |   120.0 MiB/s |    240 h/s |    280 h/s |      118 h/s |
|        11 |   3.2 TiB |     97 % |   138.4 MiB/s |   140.2 MiB/s |   126.7 MiB/s |    246 h/s |    291 h/s |      125 h/s |
|        18 |   3.2 TiB |     99 % |   156.5 MiB/s |   175.8 MiB/s |   129.4 MiB/s |    364 h/s |    296 h/s |      128 h/s |
|        20 |   3.2 TiB |     97 % |   168.5 MiB/s |   169.0 MiB/s |   127.4 MiB/s |    280 h/s |    290 h/s |      126 h/s |
|        22 |   3.2 TiB |     98 % |   195.3 MiB/s |   186.9 MiB/s |   128.6 MiB/s |    263 h/s |    299 h/s |      127 h/s |
|        25 |   3.2 TiB |     97 % |   148.6 MiB/s |   140.3 MiB/s |   127.7 MiB/s |    393 h/s |    285 h/s |      126 h/s |
|        28 |   3.2 TiB |     99 % |   228.1 MiB/s |   189.4 MiB/s |   129.2 MiB/s |    210 h/s |    304 h/s |      127 h/s |
|        29 |   3.2 TiB |     98 % |   185.9 MiB/s |   171.9 MiB/s |   128.1 MiB/s |    315 h/s |    290 h/s |      126 h/s |
|        31 |   2.0 TiB |     61 % |   111.3 MiB/s |   106.3 MiB/s |    80.5 MiB/s |    226 h/s |    182 h/s |       79 h/s |
|        36 |   3.2 TiB |     98 % |   138.5 MiB/s |   167.3 MiB/s |   128.7 MiB/s |    201 h/s |    284 h/s |      127 h/s |
|        40 |   3.3 TiB |     99 % |   158.7 MiB/s |   169.4 MiB/s |   129.7 MiB/s |    336 h/s |    303 h/s |      128 h/s |
|        41 |   3.2 TiB |     99 % |   158.6 MiB/s |   168.9 MiB/s |   129.3 MiB/s |    353 h/s |    295 h/s |      127 h/s |
|        42 |   3.2 TiB |     99 % |   222.1 MiB/s |   188.7 MiB/s |   129.3 MiB/s |    416 h/s |    295 h/s |      128 h/s |
|        49 |   3.2 TiB |     98 % |   168.6 MiB/s |   168.7 MiB/s |   128.4 MiB/s |    173 h/s |    296 h/s |      127 h/s |
|        52 |   3.2 TiB |     98 % |   157.6 MiB/s |   153.1 MiB/s |   128.9 MiB/s |    283 h/s |    288 h/s |      127 h/s |
|        53 |   3.2 TiB |     97 % |   167.6 MiB/s |   144.9 MiB/s |   127.3 MiB/s |    328 h/s |    288 h/s |      126 h/s |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+

Coordinated mining cluster stats:
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                 Peer | H1 Out (Cur) | H1 Out (Avg) | H1 In (Cur) | H1 In (Avg) | H2 Out |  H2 In |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                  All |     3288 h/s |     3420 h/s |    4015 h/s |    3885 h/s |      0 |      0 |
|     95.54.82.50:1985 |     1428 h/s |     1519 h/s |    1779 h/s |    1788 h/s |      0 |      0 |
|     95.54.82.50:1984 |     1859 h/s |     1900 h/s |    2235 h/s |    2096 h/s |      0 |      0 |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+

Node3:

  • Data Size: 50.7 TiB
  • Partitions: 16
  • Read (Avg): 3333.9 MiB/s (expected 3078 MiB/s)
  • Hash (Avg): 4472 h/s
Local mining stats:
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
| Partition | Data Size | % of Max |   Read (Cur)  |   Read (Avg)  |  Read (Ideal) | Hash (Cur) | Hash (Avg) | Hash (Ideal) |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
|     Total |  50.7 TiB |     27 % |  3478.8 MiB/s |  3333.9 MiB/s |  1965.0 MiB/s |   4609 h/s |   4472 h/s |     1734 h/s |
|         5 |   2.8 TiB |     83 % |   230.0 MiB/s |   178.4 MiB/s |   106.6 MiB/s |    276 h/s |    240 h/s |       94 h/s |
|         6 |   2.8 TiB |     85 % |   185.7 MiB/s |   198.0 MiB/s |   109.0 MiB/s |    235 h/s |    249 h/s |       96 h/s |
|        12 |   3.2 TiB |     98 % |   179.2 MiB/s |   208.8 MiB/s |   125.1 MiB/s |    250 h/s |    283 h/s |      110 h/s |
|        15 |   3.3 TiB |     99 % |   169.6 MiB/s |   195.8 MiB/s |   126.3 MiB/s |    395 h/s |    286 h/s |      111 h/s |
|        16 |   3.2 TiB |     98 % |   208.8 MiB/s |   227.4 MiB/s |   125.5 MiB/s |    278 h/s |    288 h/s |      110 h/s |
|        21 |   3.3 TiB |     99 % |   219.8 MiB/s |   225.4 MiB/s |   126.2 MiB/s |    256 h/s |    291 h/s |      111 h/s |
|        23 |   3.2 TiB |     97 % |   228.5 MiB/s |   216.1 MiB/s |   123.4 MiB/s |    386 h/s |    277 h/s |      108 h/s |
|        35 |   3.2 TiB |     96 % |   235.7 MiB/s |   214.0 MiB/s |   122.5 MiB/s |    324 h/s |    277 h/s |      108 h/s |
|        37 |   3.2 TiB |     98 % |   249.7 MiB/s |   199.3 MiB/s |   125.0 MiB/s |    376 h/s |    287 h/s |      110 h/s |
|        39 |   3.3 TiB |     99 % |   189.6 MiB/s |   227.0 MiB/s |   126.6 MiB/s |    176 h/s |    287 h/s |      111 h/s |
|        43 |   3.3 TiB |     99 % |   139.9 MiB/s |   227.8 MiB/s |   126.8 MiB/s |    123 h/s |    292 h/s |      111 h/s |
|        44 |   3.2 TiB |     98 % |   339.5 MiB/s |   217.3 MiB/s |   125.6 MiB/s |    287 h/s |    285 h/s |      110 h/s |
|        47 |   3.1 TiB |     95 % |   263.6 MiB/s |   213.8 MiB/s |   121.7 MiB/s |    166 h/s |    275 h/s |      107 h/s |
|        48 |   3.3 TiB |     99 % |   189.8 MiB/s |   189.3 MiB/s |   126.1 MiB/s |    496 h/s |    287 h/s |      111 h/s |
|        54 |   3.3 TiB |     99 % |   239.8 MiB/s |   196.3 MiB/s |   126.4 MiB/s |    395 h/s |    290 h/s |      111 h/s |
|        55 |   3.1 TiB |     96 % |   209.8 MiB/s |   199.2 MiB/s |   122.1 MiB/s |    182 h/s |    271 h/s |      107 h/s |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+

Coordinated mining cluster stats:
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                 Peer | H1 Out (Cur) | H1 Out (Avg) | H1 In (Cur) | H1 In (Avg) | H2 Out |  H2 In |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                  All |     3670 h/s |     3775 h/s |    3224 h/s |    3042 h/s |      0 |      0 |
|     95.54.82.50:1986 |     1673 h/s |     1889 h/s |    1695 h/s |    1283 h/s |      0 |      0 |
|     95.54.82.50:1984 |     1997 h/s |     1885 h/s |    1528 h/s |    1759 h/s |      0 |      0 |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+

As you can see, the Read (Avg) values don't match the expected values, while the Hash (Avg) seems to be correct. This discrepancy is consistent across all nodes in our setup.

Steps to reproduce:

  1. Set up a coordinated mining environment with multiple nodes
  2. Run the mining process for a sufficient period to gather stable metrics
  3. Compare the reported Read (Avg) values with the expected values based on the node's specifications

Expected behavior:
The Read (Avg) values should closely match the expected values for each node, similar to how the Hash (Avg) values appear to be correct.

Actual behavior:
The Read (Avg) values are significantly different from the expected values, while Hash (Avg) values seem to be correct.

Additional information:

  • Arweave version: master 55dbf97
  • Operating system: ubuntu 22.04.3 lts
@JamesPiechota
Copy link
Collaborator

Yeah. I believe we're incorrectly reporting cached reads as reads. Peers batch H1 jobs and sometimes the same recall range will be represented in multiple batches. In those cases we cache the read to avoid going to disk repeatedly. The cached reads probably should not be shown in the report.

@bgmastermind
Copy link

Or they should as a separate stat named cache hit %

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants