Query sorting is sometimes slower than not sorting

[aprokop]

First observed in #445, but spinning in this issue. So far, only have Cuda observations, and the fact that ExaWind problem is slower for sorted in Serial.

This is with master branch (11d91b1).

Benchmark results (Summit V100)

Darn. Did not expect this.

sorted_vs_unsorted_master.pdf

DBSCAN with HACC data (Summit V100)

Uh-oh.

Halo finder (minpts = 2).

Timer	Sorted	Unsorted	Diff
total time	0.828	0.660	-20%
-- construction	0.097	0.087
-- query+cluster	0.711	0.552	-23%
-- postprocess	0.021	0.021

DBSCAN (minpts = 5)

Timer	Sorted	Unsorted	Diff
total time	1.140	0.885	-23%
-- construction	0.087	0.087
-- query+cluster	1.033	0.778	-25%
---- neigh	0.076	0.063	-18%
---- query	0.952	0.709	-26%
-- postprocess	0.020	0.020

🤷🏻‍♂️

One thing that is unclear to me at this point is whether this is simply explained by the cost of the sort compared to the query runtime, or if there's something else going on. No, cannot be explained by sort. In DBSCAN problem, queries are primitives with radius. Thus, sorting takes a fraction of the construction. The difference in query times exceeds that by far.