Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tracker: Airflow 2.x support #3165

Open
shalberd opened this issue Jun 21, 2023 · 4 comments
Open

tracker: Airflow 2.x support #3165

shalberd opened this issue Jun 21, 2023 · 4 comments
Assignees
Labels
kind:enhancement New feature or request platform: pipeline-Airflow Related to usage of Apache Airflow as pipeline runtime

Comments

@shalberd
Copy link
Contributor

shalberd commented Jun 21, 2023

Is your feature request related to a problem? Please describe.
Several issues with Airflow 2.x. The intent of this ticket is to track efforts related to solving the issues one by one

#3109 fix provider package catalog connector code to be Airflow 2 compatible

#2392 generic issue Airflow 2 support request

#3166 generic pipeline editor needs airflow 2 related fixes

#2124 package catalog connector import fixes needed for airflow 2

Describe the solution you'd like
Gradually, enable airflow 2.x support from Elyra components
Specifically, Airflow 2.6.0, which came out in May 2023, or higher.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Starting with generic pipeline editor (python files, scripts) and rendering to DAG.

#3166

@shalberd shalberd added the kind:enhancement New feature or request label Jun 21, 2023
@shalberd shalberd changed the title Airflow 2.x suport Airflow 2.x support Jun 21, 2023
@shalberd shalberd changed the title Airflow 2.x support tracker: Airflow 2.x support Jun 21, 2023
@ianonavy
Copy link

@shalberd For future contributors to be aware, is backwards compatibility with 1.x a goal given that Airflow 1.x is no longer supported upstream?

@lresende
Copy link
Member

@ianonavy I believe we were mentioning that this would be a new runtime, which then supports new operators, etc which I believe is incompatible in nature from Airflow 1.x otherwise we could have just used 1.x.

@shalberd
Copy link
Contributor Author

shalberd commented Jun 21, 2023

I believe we were mentioning that this would be a new runtime

@lresende do you mean you still want to support Airflow 1.x in parallel? I mean, who really is still using Airflow 1.x or the last intermediate 1.x release with backport functionality? It's been ages ago.

Just to confirm, do you propose having parallel logic / processors and classes for airflow 1.x as well as greater than 2.x?

@shalberd
Copy link
Contributor Author

@ianonavy backward compatibility with Airflow 1.x is no longer the goal for the next major Elyra release 4.x that the solutions to the tracker tickets will be a part of.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:enhancement New feature or request platform: pipeline-Airflow Related to usage of Apache Airflow as pipeline runtime
Projects
None yet
Development

No branches or pull requests

3 participants