Use index hint for idx_contracts_fcid_timestamp, MySQL does not select it otherwise

[peterjan]

I was still seeing SLOW SQL warnings for SELECT * FROM contracts WHERE contracts.timestamp >= 1709984700000 AND contracts.timestamp < 1710071100000 AND contracts.fcid = '<binary>' LIMIT 1, after EXPLAINing I found that MySQL doesn't use the index we added.

[ChrisSchinnerl]

2 questions before I approve this

1. Did you manually verify the speedup on gompa to get some numbers for with vs without index hint?
2. Is SQLite using the index? If not we might want hints for SQLite as well in which case numbers from Arequipa would be great as well.

[peterjan]

2 questions before I approve this

1. Did you manually verify the speedup on gompa to get some numbers for with vs without index hint?
2. Is SQLite using the index? If not we might want hints for SQLite as well in which case numbers from Arequipa would be great as well.

SELECT * FROM contracts WHERE contracts.timestamp >= 1709984700000 AND contracts.timestamp < 1710071100000 AND contracts.fcid = '<binary>' LIMIT 1,

1. Not really. I deployed this to a node where I was seeing SLOW SQL logs and they disappeared after deploying this. That's not terribly scientific though but since this index was added for exactly this purpose I figured we might as well hint it. The query plan goes from Using index condition; Using where to Using index condition so it can only ever be a speedup I think. Generally I never deploy something to gompa to test because I'm scared of Nate.

2. Yes. The test I added asserts that exact index is used by SQLite by default.

[ChrisSchinnerl]

1. Not really. I deployed this to a node where I was seeing SLOW SQL logs and they disappeared after deploying this. That's not terribly scientific though but since this index was added for exactly this purpose I figured we might as well hint it. The query plan goes from Using index condition; Using where to Using index condition so it can only ever be a speedup I think. Generally I never deploy something to gompa to test because I'm scared of Nate.

The hint is great but it would be great to measure the result rather than relying on the logging. You definitely shouldn't deploy to gompa but you can still pick one of the queries that were logged as slow and run it directly against gompa's database with and without the hint.

[peterjan]

The hint is great but it would be great to measure the result rather than relying on the logging. You definitely shouldn't deploy to gompa but you can still pick one of the queries that were logged as slow and run it directly against gompa's database with and without the hint.

I measured it on gompa and didn't really see a measurable difference with or without the hint. It hovers around 100ms both with and without the hint. I don't think looking at the execution time is a good way of evaluating query performance though, especially not if it's not an obvious boost. It's somewhat decent if you disable query caching (which I did) but ultimately you have to rely on the query plan I think.

We can close this out if you want, we should maybe remove the index though in that case. We added it here, maybe it was a case of two optimisations where one did the bulk of the work and the other didn't really matter all that much?

[ChrisSchinnerl]

The hint is great but it would be great to measure the result rather than relying on the logging. You definitely shouldn't deploy to gompa but you can still pick one of the queries that were logged as slow and run it directly against gompa's database with and without the hint.

I measured it on gompa and didn't really see a measurable difference with or without the hint. It hovers around 100ms both with and without the hint. I don't think looking at the execution time is a good way of evaluating query performance though, especially not if it's not an obvious boost. It's somewhat decent if you disable query caching (which I did) but ultimately you have to rely on the query plan I think.

We can close this out if you want, we should maybe remove the index though in that case. We added it here, maybe it was a case of two optimisations where one did the bulk of the work and the other didn't really matter all that much?

How did you run it on gompa? 100ms seems way too slow. I searched for the query that I ran when I added the index and this is the result I'm getting on gompa.

mysql> SELECT hex(fcid) FROM contracts WHERE contracts.timestamp >= 1706106000000 AND contracts.timestamp < 1708698000000 - 86400000 AND contracts.fcid = x'00195BA795481F5941ADB10463ADC89D12BED28EE61B4A684451DF8E3C5F9402' LIMIT 1;
+------------------------------------------------------------------+
| hex(fcid)                                                        |
+------------------------------------------------------------------+
| 00195BA795481F5941ADB10463ADC89D12BED28EE61B4A684451DF8E3C5F9402 |
+------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> EXPLAIN SELECT hex(fcid) FROM contracts WHERE contracts.timestamp >= 1706106000000 AND contracts.timestamp < 1708698000000 - 86400000 AND contracts.fcid = x'00195BA795481F5941ADB10463ADC89D12BED28EE61B4A684451DF8E3C5F9402' LIMIT 1;
+----+-------------+-----------+------------+-------+--------------------------------------------------------------------------+------------------------------+---------+------+------+----------+--------------------------+
| id | select_type | table     | partitions | type  | possible_keys                                                            | key                          | key_len | ref  | rows | filtered | Extra                    |
+----+-------------+-----------+------------+-------+--------------------------------------------------------------------------+------------------------------+---------+------+------+----------+--------------------------+
|  1 | SIMPLE      | contracts | NULL       | range | idx_contracts_timestamp,idx_contracts_fc_id,idx_contracts_fcid_timestamp | idx_contracts_fcid_timestamp | 42      | NULL | 2437 |   100.00 | Using where; Using index |
+----+-------------+-----------+------------+-------+--------------------------------------------------------------------------+------------------------------+---------+------+------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)

Even without hint it appears to be using the index. So maybe that's why you aren't seeing a difference.

[peterjan]

Hm interesting on my personal node it is not using that index. I guess because I'm on SQL 8.0.32 maybe...

mysql> EXPLAIN SELECT hex(fcid) FROM contracts WHERE contracts.timestamp >= 1706106000000 AND contracts.timestamp < 1708698000000 - 86400000 AND contracts.fcid = x'00195BA795481F5941ADB10463ADC89D12BED28EE61B4A684451DF8E3C5F9402' LIMIT 1
    -> ;
+----+-------------+-----------+------------+------+--------------------------------------------------------------------------+---------------------+---------+-------+------+----------+------------------------------------+
| id | select_type | table     | partitions | type | possible_keys                                                            | key                 | key_len | ref   | rows | filtered | Extra                              |
+----+-------------+-----------+------------+------+--------------------------------------------------------------------------+---------------------+---------+-------+------+----------+------------------------------------+
|  1 | SIMPLE      | contracts | NULL       | ref  | idx_contracts_timestamp,idx_contracts_fc_id,idx_contracts_fcid_timestamp | idx_contracts_fc_id | 34      | const |    1 |     5.00 | Using index condition; Using where |
+----+-------------+-----------+------------+------+--------------------------------------------------------------------------+---------------------+---------+-------+------+----------+------------------------------------+
1 row in set, 1 warning (0.00 sec)

You're also right by the way, the reason I wasn't seeing a speedup on gompa is because I was testing with and without HINT but never EXPLAINed it on gompa... so I was just running the query using the index all the time. We can close it out but I guess many people might be on an older MySQL version. It's not even that old it's jan 2023...