PR: Migrate the Spyder update mechanism to the conda-based standalone application

[mrclary]

Description of Changes

• All conda installations (Anaconda, conda-forge, our conda-based installers) will use conda-forge to check for available updates. Any other installations will use PyPi to check for updates.
  The reasons for this are as follows. Users of our conda-based installers, conda-forge installs, and PyPi installs will only be alerted to updates when its actually available to be installed. This may not be the case if checking against GitHub. Anaconda main channel installs will almost certainly be notified before the version is actually available on this channel, however, our install instructions should allow them safely install from the conda-forge channel without breaking their environment. Since anaconda main updates have extremely long lead times, users likely will not be alerted to our new conda-based installer until 6.1.0 is released. I think there is a large advantage to alerting these users sooner and makes this is an acceptable tradeoff.
• For current versions that are not conda-base installs, or there is a major version update for a conda-based install, the user is prompted to download and install the update.
• For all release types, upon commencing install Spyder will quit and the installation/update process will proceed.
• If the update downloaded a conda-based installer, an interactive GUI uninstaller will launch (Windows) or an interactive uninstaller is executed at a shell terminal (macOS, Linux). For all platforms, the interactive GUI installer is launched upon successful completion of the uninstall process.
• If the update did not require downloading a conda-based installer, a shell terminal will inform the user that Spyder is being updated via conda, and conda's standard output is displayed.
• In all scenarios, Spyder will start/restart upon successful install/update.
• For current versions that are not conda-based installs, only downloading the conda-based installer will result in installing an update. If the user rejects this option, the user is only instructed how to perform the update, but no automatica update will be peformed.
• A new preference is added to Application->Advanced settings that permits the user to consider unstable releases when checking for updates. The default is to consider only stable releases.

Questions

* Is it possible to use Spyder's restart mechanism for installations? * I suspect that it is not possible. * For the downloaded installer, an entirely new environment will be created and the restart subprocess will attempt to execute code that no longer exists (if uninstalled), or not execute the code from the new environment. * For the downloaded installer, an attempt to uninstall the existing environment may fail because the restart subprocess may have a lock on files in that environment. * For conda updates, an attempt to overwrite code in the existing environment may fail because the restart subprocess may have a lock on files in the environment. * Should Spyder quit before any new update proceeds (and possibly restart afterward)? * I think this is the safest thing to do for the same reasons mentioned in the previous question. * Do we want interactive installers to automatically open Spyder upon successful completion? Or provide the option to open Spyder upon successful completion? Or not open Spyder at all? * Do we want Spyder's update dialog to provide the user with the option to restart Spyder after updating? * close now and update * close now and update and restart * close later and update * close later and update and restart * Any other possibilities? * When the user proceeds to update from Spyder's dialog, do we want to run the downloaded installers in batch mode (non-interactive)?

Proposed path forward

* When the user proceeds with the install, the user will have the option to install on close (same as currently implemented). * If proceeding with the install immediately, emit sig_install_on_close_requested and automatically close Spyder. I.e. Spyder will always close before updating; the user just determines when this will occur. * The install procedure (interactive installer or conda update) will occur in a subshell and so should not block any file operations in the environment. * If the install procedure runs the downloaded installer, then the install script will first uninstall an existing conda-based installation before launching the interactive downloaded installer. * The downloaded interactive installer will automatically start Spyder upon successful installation. This is accomplished via a post-install script and is _not_ presented as an option, e.g. "Do you want to start Spyder?". * If the install procedure is a conda update, then part of the subshell command will start Spyder after the conda update successfully completes.

Note that the downloaded installer will always be run interactively. The primary reason for this is that the user will be able to see the progress of the install. If the macOS installer is run in batch mode, then the only way to view the progress (several minutes!) will be to have the installer batch script launched in a terminal and tail stream the install.log file (similar to our GitHub workflow). I don't know if this is possible on Windows. In either case, the interactive installer will be more aesthetically pleasing to the user rather than a mysterious terminal window popping up and spitting out a bunch of seemingly random garbage. Unfortunately, this is unavoidable for the case where we update through conda.

Interactive Testing

1. Install latest 6.0.0a2: Uninstall or rename your existing Spyder installation. If you already have a conda-based installation, you won't be able to install over it; it must be uninstalled first. For installations other than conda-based, the conda-based installer may not over-write an existing application shortcut, so the shortcut will need to be renamed or deleted. Then download and install 6.0.0a2.
2. Update conda-based installation.
   1. Checkout the mrclary:issue-20831-cbi-updater branch in your local Spyder repository.
   2. Modify the following files in your Spyder repository:
      • spyder/__init__.py. Change the version on line 32 to (5, 4, 4, "dev0").
      • spyder/plugins/application/container.py. Change the delay interval on line 339 from 60000 ms to 10000 ms. This is not strictly necessary, but it is easier to test when only waiting 10s instead of 60s to trigger the updater at Spyder startup.
      • spyder/plugins/application/scripts/install.[sh|bat]. Remove -y from the conda install command on line 16 (.sh) and line 66 (.bat) to allow you to cancel a conda update without clobbering your existing install.
      • spyder/plugins/application/scripts/install.sh. Add || true to the end of line 36 to allow you to cancel an uninstall without aborting the call to the installer.
      • spyder/workers/updates.py. Change line 92 to read

        if False:  # is_conda_env(pyexec=sys.executable):
        

        This will ensure that PyPi is used to check for updates. This is only needed for testing in order to see both stable and unstable releases (conda-forge doesn't list 6.0.0a2).
   3. Run the following command in a terminal/cmd window to update the conda-based Spyder.

      conda run --live-stream -p path/to/spyder-6/envs/spyder-runtime python path/to/spyder/install_dev_repos.py --not-editable
      

      Be sure to substitute the correct paths to your conda-based installation and your local Spyder git repo, respectively.
3. Start or restart Spyder.
4. The updater should run at Spyder launch.
   1. Testing for a minor update, Spyder will be updated in-place using conda mechanisms.
      • You should be alerted that Spyder 5.4.5 is available and whether you want to proceed to install it. Click "Yes".
      • Spyder will quit.
      • A terminal/cmd window appears and displays the progress of the conda update. After the environment solve, do not confirm the changes: type "n" and hit return. You do not want to proceed with the conda update.
      • After the terminal/cmd window closes, Spyder will launch.
   2. Testing for a major update, a conda-based installer will be downloaded and launched.
      • You should be alerted to Spyder 5.4.5 being available and whether you want to proceed to install it. Click "No".
      • Open Preferences->Application->Advanced settings and uncheck "Check for stable releases only".
      • Click "Check for updates..." in the Help menu.
      • You should be alerted that version Spyder 6.0.0a2 is available and whether you want to download and install it. Click "Yes". This will initiate downloading the conda-based installer.
      • After the download is complete, a message box will appear asking if you want to proceed with the installation. Click "Yes".
      • Spyder will quit.
      • Spyder's uninstall script will launch interactively in a terminal asking if you wish to uninstall Spyder (macOS or Linux), or a Windows Uninstaller will launch (Windows). Cancel the uninstall operation.
      • After the uninstall terminal/window is closed, the installer should launch. Cancel the installer.

GIFs

For the following demonstrations, I modify Spyder's `__init__.py` to spoof the version in order to trigger updates.

• Setting the version to non-stable and prior to 6.0.0a1 will trigger a major version upgrade (download installer). For the purposes of the demonstration, I disable exiting on error in order to cancel the uninstaller and still show the installer. I also cancel the installer. Upon successful install Spyder will start.
• Setting the version to stable and prior to 5.4.3 will trigger a minor version upgrade (conda install). For the purposes of the demonstration, I removed the --yes flag in order to be able to cancel the conda install (this is destructive). Spyder starts up after the install is complete.

Windows GIF

Linux GIF

macOS GIF

[mrclary]

I've added questions and proposals to the PR description. Pinging @spyder-ide/core-developers for feedback.

[mrclary]

@jaimergp, is it possible to add custom options to the package installer (constructor)? Specifically an option to open the application shortcut after completion.

I suppose this would be simple enough to add in a post-install script for Linux, since the installer is already a shell script, and the post-install script could request user input.
Perhaps this option can be included in an nsis template for Windows?
Can this option be added to the macOS package installer?
In all cases, can this option be selected/deselected when run in batch mode? i.e. providing commandline arguments/flags when run from the installer is run from the commandline?

[jaimergp]

You can use post-install scripts in all three platforms, and they can be disabled with the corresponding flags. There will be checkboxes (with a custom description if you want) to disable them too. So I think you can hack something together.

On Windows, you can also customize the "conclusion" page with your own NSIS code. See this example.

[mrclary]

You can use post-install scripts in all three platforms, and they can be disabled with the corresponding flags. There will be checkboxes (with a custom description if you want) to disable them too. So I think you can hack something together.

Is it possible to have multiple post-install scripts, each with their own enable/disable flags? Both macOS and Linux must do other things in post-install, independent of a "start application" option. So the option to start the application must be independent of other post-install actions.

Perhaps our post-install actions can be done in the pre-install script instead, leaving the post-install script dedicated to starting the application.

[mrclary]

@spyder-ide/core-developers, I've updated the "Proposed path forward" in the PR description. I invite your comments.

[jaimergp]

Is it possible to have multiple post-install scripts, each with their own enable/disable flags? Both macOS and Linux must do other things in post-install, independent of a "start application" option. So the option to start the application must be independent of other post-install actions.

Unfortunately not. A "launch after installation" is a common feature request for an installer though, so maybe we could add it in constructor directly. However I don't have the time to research how to do this with PKGs right now (I think there's a global postinstall script we can use for that, maybe you can hack your PKG and the Distribution.xml file after generating it with constructor?). If you figure that out, we can probably add a flag somewhere at some point!

[mrclary]

@spyder-ide/core-developers, I updated the proposed path forward based on our meeting discussion today.

[jitseniesen]

I updated the proposed path forward based on our meeting discussion today.

Looks good to me, but I never used the stand-alone installer so it is all theoretical for me.

[ccordoba12]

@mrclary, could you post some gifs in the OP of how the new update process looks in different OSes? Thanks!

[mrclary]

@ccordoba12 @CAM-Gerlach,
For minor and micro releases, our updater currently looks for updates only in conda's main channel, https://repo.anaconda.com/pkgs/main. Are we interested in expanding to or replacing with conda-forge? If so, is there an efficient way to read the index for the spyder channel (and spyder_dev label)? I can't find an analog to https://repo.anaconda.com/pkgs/main/<arch>/repodata.json.

[mrclary]

FYI, channeldata.json is very fast and seems to be the most convenient route for extracting the latest Spyder version.

[mrclary]

@spyder-ide/core-developers, I've un-drafted this PR. Please refer to the PR description for instructions on how to test the changes made here.
If you have any questions, please contact me.

[ccordoba12]

Closing as superseded by #21483.