Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up returned tuple where step wrapper passes "check_size" back to phase with results #133

Open
lisad opened this issue May 29, 2024 · 0 comments

Comments

@lisad
Copy link
Owner

lisad commented May 29, 2024

I added this return tuple, although I don't like it, because the phase is the place that has the context and the previous number of rows of the main dataset. However, the step wrapper is the place where the step's extra settings are known, e.g.:

@batch_step(check_size=False)
def my_batch_step(batch, context)
return batch

When the phase gets this back it gets 'result, check_size' in order to know whether to check the size or if the step is supposed to resize.

Could we remove this return tuple? The phase passes a context into the step function call, which translates to a call to _step_wrapper. In that function, if you pass context into self.postprocess, then we ALMOST have all the information in the postprocess function. The only thing missing is the previous row size. Also we would need to duplicate the size checks in both batch_step and dataframe_step but that can be factored out as a function so not a big deal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant