You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug, including details regarding any error messages, version, and platform.
Converting from decimal.Decimal to PyAarrows Decimal has given me the pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss error.
I understand that to fix this, I have to ensure that the DecimalType (I am using PySpark, and their schema) I have to max the precision and scaling to the value I want to store.
However, to not allocate more memory than needed, I want to right-size the data; however, receiving the above error in production, without any indication of which value is outside this range, is frustrating and not very helpful.
There are similar lacks in many other error messages regarding invalid data
Component(s)
Python
The text was updated successfully, but these errors were encountered:
Describe the bug, including details regarding any error messages, version, and platform.
Converting from decimal.Decimal to PyAarrows Decimal has given me the
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss
error.I understand that to fix this, I have to ensure that the DecimalType (I am using PySpark, and their schema) I have to max the precision and scaling to the value I want to store.
However, to not allocate more memory than needed, I want to right-size the data; however, receiving the above error in production, without any indication of which value is outside this range, is frustrating and not very helpful.
There are similar lacks in many other error messages regarding invalid data
Component(s)
Python
The text was updated successfully, but these errors were encountered: