assertColumnEquality reuses assertLargeDatasetEquality

[snithish]

@MrPowers to address #25 what can be done is reuse reuse all the logic we have in DatasetComparer as a POC I have converted assertColumnEquality to resue assertLargeDatasetEquality. This make for a clean codebase with optimal code reuse. I understand the ArrayPrettyPrint is the USP, but I believe we can achieve this in other ways.

[MrPowers]

@snithish - The assertLargeDatasetEquality method is a lot slower than assertSmallDataFrameEquality and assertColumnEquality. See this blog post where I talk about how replacing assertLargeDataFrameEquality with assertSmallDataFrameEquality reduced the test suite runtime by 31%.

I am all for code reuse, but the main purpose of this library is to create helper methods for tests that run as fast as possible. Any thoughts on how we can have the best of both worlds? 😎

[snithish]

@MrPowers agreed, did you find any performance difference between assertSmallDataFrameEquality and assertColumnEquality? I believe the performance kick comes from the collect we do in both of them, so they should be equally performant, in which case we will have two broad categories of equality assertions, one which performs a collect and one without.

[MrPowers]

@snithish - I think assertSmallDataFrameEquality and assertColumnEquality are pretty much the same performance wise (don't have any benchmarking to back that up, just my guess).

I agree that we should have two broad categories of equality assertions: the ones that use collect and the ones don't. assertSmallDataFrameEquality and assertColumnEquality should share the same underlying code. assertLargeDataFrameEquality and assertLargeColumnEquality (doesn't exist yet) should share the same underlying code.

Or maybe we should just get rid of assertLargeDataFrameEquality all together... I never use it. I don't know, maybe other folks use it.

Please feel free to tear up the codebase as you see fit 😉 I am always down for improvements. All I care about are tests that run really, really fast and error messages that are beautiful. Anything that gets those results will make me happy. I know you'll be able to work your magic and write some beautifully designed code that gets all the results and uses nice Scala code.

[snithish]

@MrPowers I will try to simplify it down to asserting with and without collect. We can benchmark and decide after that. Should have something up by the end of this week.

[snithish]

@MrPowers finally got to refactoring it to use small dataset equality for column equality, have a look at let me know your thoughts.