Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add requirements list for Pipeline #720

Merged
merged 29 commits into from
Jun 27, 2024
Merged

Conversation

plaguss
Copy link
Contributor

@plaguss plaguss commented Jun 11, 2024

Description

This PR adds a new attribute to requirements to BasePipeline to keep track of the dependencies needed to run a Pipeline. The pipeline.dump() method now contains a new key with the requirements, if any.

We can include requirements at the Pipeline level, and ideally we would add requirements for custom steps via
@requirements decorator, to avoid making the step definition more verbose.

It will throw a ValueError before running and show the dependencies that aren't already installed in your environment.

@requirements(["distilabel>=0.0.1"])
class CustomStep(Step):
    @property
    def inputs(self) -> List[str]:
        return ["instruction"]

    @property
    def outputs(self) -> List[str]:
        return ["response"]

    def process(self, inputs: StepInput) -> StepOutput:  # type: ignore
        for input in inputs:
            input["response"] = "unit test"
        yield inputs

with BasePipeline(
    name="unit-test-pipeline", requirements=["random_requirement"]
) as pipeline:
    gen_step = DummyGeneratorStep()
    step1_0 = DummyStep()
    step2 = CustomStep()

    gen_step >> step1_0 >> step2
pipeline.run()

@plaguss plaguss self-assigned this Jun 11, 2024
@plaguss plaguss added this to the 1.3.0 milestone Jun 11, 2024
@plaguss plaguss added the enhancement New feature or request label Jun 11, 2024
Copy link

codspeed-hq bot commented Jun 11, 2024

CodSpeed Performance Report

Merging #720 will not alter performance

Comparing pipeline-requirements (77d8211) with develop (63ee8c5)

Summary

✅ 1 untouched benchmarks

@plaguss plaguss linked an issue Jun 11, 2024 that may be closed by this pull request
@plaguss plaguss marked this pull request as ready for review June 13, 2024 13:13
@plaguss plaguss requested review from gabrielmbmb and alvarobartt and removed request for gabrielmbmb June 13, 2024 13:13
Base automatically changed from develop to main June 18, 2024 12:36
@gabrielmbmb gabrielmbmb changed the base branch from main to develop June 19, 2024 11:21
Copy link
Member

@gabrielmbmb gabrielmbmb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

src/distilabel/mixins/requirements.py Outdated Show resolved Hide resolved
src/distilabel/mixins/requirements.py Outdated Show resolved Hide resolved
src/distilabel/mixins/requirements.py Outdated Show resolved Hide resolved
src/distilabel/pipeline/base.py Outdated Show resolved Hide resolved
src/distilabel/utils/requirements.py Outdated Show resolved Hide resolved
@plaguss plaguss requested a review from gabrielmbmb June 27, 2024 08:18
Copy link
Member

@gabrielmbmb gabrielmbmb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

mkdocs.yml Outdated Show resolved Hide resolved
src/distilabel/pipeline/base.py Outdated Show resolved Hide resolved
@gabrielmbmb gabrielmbmb changed the title Add requirements list for a pipeline Add requirements list for Pipeline Jun 27, 2024
@gabrielmbmb gabrielmbmb merged commit c357090 into develop Jun 27, 2024
7 checks passed
@gabrielmbmb gabrielmbmb deleted the pipeline-requirements branch June 27, 2024 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Add requirements in Pipeline
2 participants