Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error message regarding invalid values (i.e Rescaling Decimal128 value would cause data loss) does not show the offending values #44061

Open
C0DK opened this issue Sep 11, 2024 · 0 comments

Comments

@C0DK
Copy link

C0DK commented Sep 11, 2024

Describe the bug, including details regarding any error messages, version, and platform.

Converting from decimal.Decimal to PyAarrows Decimal has given me the
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss error.

I understand that to fix this, I have to ensure that the DecimalType (I am using PySpark, and their schema) I have to max the precision and scaling to the value I want to store.

However, to not allocate more memory than needed, I want to right-size the data; however, receiving the above error in production, without any indication of which value is outside this range, is frustrating and not very helpful.

There are similar lacks in many other error messages regarding invalid data

Component(s)

Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant