Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow relative precision in equality comparison #108

Open
nightscape opened this issue May 18, 2022 · 0 comments
Open

Allow relative precision in equality comparison #108

nightscape opened this issue May 18, 2022 · 0 comments

Comments

@nightscape
Copy link
Contributor

Hi @MrPowers,

I'd like to switch over from spark-testing-base to spark-fast-tests in spark-excel.
I'm using property-based testing to cover a wide range of edge cases when writing and reading Excel files.
The problem with this is that I don't (want to) have control over the range of values for e.g. a double column.
When using assertApproximateDataFrameEquality I can only specify an absolute precision which would be too big for some random numbers and too small for others.
Would you be open to a PR that adds this as additional capability?
I might be able to do this in a source- but not binary-compatible way sth. like this:

trait NumericComparison {
  def compare[T : Numeric](t1: T, t2: T): Boolean
}

case class AbsoluteNumericComparison(precision: Double) extends NumericComparison {
  def compare[T : Numeric](t1: T, t2: T): Boolean = ???
}

case class RelativeNumericComparison(precision: Double) extends NumericComparison {
  def compare[T : Numeric](t1: T, t2: T): Boolean = ???
}

object NumericComparison {
  implicit def doubleToAbsoluteNumericComparison(precision: Double) = AbsoluteNumericComparison(precision)
}

def assertApproximateDataFrameEquality(df1: DataFrame, df2: DataFrame, precision: NumericComparison) = ???

This would give room for further extensions to comparisons.

A simpler alternative might be

def assertApproximateDataFrameEquality(df1: DataFrame, df2: DataFrame, precision: Double, precisionIsRelative: Boolean = false) = ???

Do you think this could work, or do you see another way to add relative precision?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant