Upgrade to Yarn Modern v4 #6446
Conversation
| - run: yarn install --production=false --frozen-lockfile --ignore-scripts | ||
| - run: yarn install --immutable |
There was a problem hiding this comment.
--frozen-lockfile is replaced by --immutable. This means the install will abort with an error exit code if the lockfile was to be modified by installing third party libraries.
There was a problem hiding this comment.
--ignore-scripts is replaced by enableScripts: false in .yarnrc.yml. This disables post-installation scripts from third party libraries.
There was a problem hiding this comment.
--production=false was originally used to ensure devDeps were not installed during ci-builds. However, it's not a flag that applies to Yarn Modern, and Yarn will not install any package listed in devDependencies if the NODE_ENV environment variable is set to production, which is the case in many of our build scripts anyway.
| # Based on https://github.com/github/gitignore/blob/master/Node.gitignore | ||
| # Yarn https://yarnpkg.com/getting-started/qa#which-files-should-be-gitignored |
There was a problem hiding this comment.
This root .gitignore file was generated years ago and contained a lot of "extras" that don't apply to FEM. I've cleaned it up and included the recommended Yarn files.
| @@ -1,9 +1,27 @@ | |||
| # Based on https://github.com/github/gitignore/blob/master/Node.gitignore | |||
|
|
|||
| # https://yarnpkg.com/getting-started/qa#which-files-should-be-gitignored | |||
There was a problem hiding this comment.
Same as comment.
| @@ -0,0 +1,3 @@ | |||
| nodeLinker: node-modules | |||
| yarnPath: .yarn/releases/yarn-4.5.1.cjs | |||
| enableScripts: false | |||
There was a problem hiding this comment.
Docs for this are here: https://yarnpkg.com/configuration/yarnrc#enableScripts. We don't want to run post-install scripts from third party libraries, and this config replaces --ignore-scripts from Yarn Classic.
| [](https://travis-ci.com/zooniverse/front-end-monorepo) | ||
| [](https://coveralls.io/github/zooniverse/front-end-monorepo?branch=master) | ||
| [](https://pullreminders.com?ref=badge) | ||
|
|
||
| [](https://lernajs.io/) | ||
| [](https://github.com/zooniverse/front-end-monorepo/blob/master/LICENSE.md) | ||
|  | ||
|
|
||
| ️Take a look at [our roadmap](https://trello.com/b/yg0r4dG5/front-end-rebuild-roadmap)! 🛣️ | ||
|
|
There was a problem hiding this comment.
The deleted links no longer exist.
| @@ -39,7 +34,7 @@ Node, git, and yarn can be installed through [homebrew](https://brew.sh/) on Mac | |||
|
|
|||
| This monorepo is managed with [Yarn Workspaces](https://yarnpkg.com/lang/en/docs/workspaces/). | |||
|
|
|||
| Yarn Workspaces allow us to maintain package modularity for javascript projects that have interdependency. Organizationally, they allows us to track issues, pull requests, and progress for all related packages in one place. | |||
| Yarn Workspaces allow us to maintain package modularity for javascript projects that have interdependency. Organizationally, they allows us to track issues, pull requests, and progress for all related packages in one place. See documentation for [Corepack](https://yarnpkg.com/corepack), too. | |||
There was a problem hiding this comment.
You might need to use Corepack with Yarn Modern.
| printf 'Installing dependencies...\n' | ||
| yarn install --immutable | ||
| printf '\n' |
There was a problem hiding this comment.
The same concept as #6393. Install once, and then build the rest of the packages. This brings the bootstrap file up to date with the CI build scripts.
| "node": ">=16.18" | ||
| "node": ">=20.5" |
There was a problem hiding this comment.
Updated to match package.json in the individual workspaces. I don't think this one matters much because the ci builds reference the files in .github, and each package has its own package.json.
| "devDependencies": { | ||
| "@babel/cli": "^7.25.9", |
There was a problem hiding this comment.
Had to install @babel/cli in a couple of our libraries because they use the babel command in the build scripts.
|
Review (attempt) currently in progress. PSA: you may need to run "yarn install" first.When I switch to this branch and try running (Full version)❗ It appears that I first need to run Additional notes:
Additional debugging notes
|
PR Review (ongoing)Manual testing update
|
There was a problem hiding this comment.
PR Review
Packages: (root) / (everything)
This PR updates the monorepo to use Yarn v4.
- This is purely a dev-side/build-side update. No features are added/removed/changed, and the development & build processes themselves should look basically the same.
Code changes look fine, as far as I can tell, and basic testing indicates that most major packages build & run fine locally. There's a new caveat about running yarn install, but see the 'Status' section at the end for full details.
Testing (CLI)
Basic pass/fail criteria for this review is "can I run some of FEM's major packages on the local dev server?"
Base setup:
- yarn installed on local zsh CLI: 4.5.1
- yarn corepack enabled?: yes
Testing (app-project):
- Switch to branch:
git checkout yarn-modern - Initialise: on root, run 🆕
yarn install, andyarn bootstrap - Run app-project locally:
cd packages/app-project,yarn dev - Open test project: https://local.zooniverse.org:3000/projects/darkeshard/test-project-2021
- Choose a random workflow, classify, and submit.
✅ Tests look good. app-project appears to work on local.
Testing (Docker)
Basic pass/fail criteria for this review is the same as above.
- Initialise: on root, run
docker-compose build - Run app-project locally:
cd packages/app-project,docker-compose up - Open test project: https://local.zooniverse.org:3000/projects/darkeshard/test-project-2021
❓ Tests look odd - the server runs, but on HTTP (https is blocked), and all I'm getting are 404 pages. Which means the local dev server is running and the app-project code is there, but there's probably something wonky in a config, or my test methods need revision. (I rarely, if ever, use Docker for local dev. I'm only testing this to be thorough.)
🤷 I'm marking this as a "OK, this is fine, I guess" since I'm not too worried about Docker, but we may need to revisit this if another developer uses docker as their main development method.
Testing (CLI, Tabula Rasa Edition)
Basic pass/fail criteria for this review is "can I run some of FEM's major packages on the local dev server, if I install the Front End Monorepo from scratch ?"
Tabula Rasa Policy:
- delete existing
/front-end-monorepofolder, or move it elsewhere. - Install a fresh copy of FEM:
git clone [email protected]:zooniverse/front-end-monorepo.git
Testing (app-project):
- Proceed with the same steps as Testing (CLI) above.
✅ Tests still look good. This tabula rasa test confirms the necessity of the 🆕 yarn install step during initialisation.
Status
Changes LGTM, but I'd like to highlight a new "required step" on the dev side that seems to have appeared as a result of the Yarn 4 update.
❗ 🆕 yarn install is now required before running yarn scripts
- Information for Zooniverse devs: when switching to this branch for the first time (and after this PR is merged), you may need to run
yarn install, at least once, before runningyarn bootstrap.- Once installed, you can run yarn bootstrap with no issue, as many times as you like.
- You also need to run
yarn installonce after runningyarn panic. - You know we need to run yarn install when: you attempt to run yarn panic or yarn bootstrap, but instead see the error message: "Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)"
- Actions to be taken: The README needs to be updated with this information. (@goplayoutside3 : this can be part of this PR, or I can make the doc update a separate PR - whatever seems cleaner.)
Documentation Update
### Debugging the production release
Make sure you have pulled the latest production version.
"""sh <== change these to tildes
git pull --tags -f
""" <== change these to tildes
Check out the latest production release.
"""sh <== change these to tildes
git checkout production-release
""" <== change these to tildes
// NEW
You'll need to run yarn install _once_ for the next yarn script to work. Otherwise, you may encounter the error message, _"Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)"_
// NEW
"""sh <== change these to tildes
yarn install
""" <== change these to tildes
Run the bootstrap script to build all the libraries and apps. You can use `bootstrap:es6` here for a faster build if you don't want to run the tests.
"""sh <== change these to tildes
yarn bootstrap
""" <== change these to tildes
### With Node and yarn
Alternatively, you can install Node 20 and yarn and build the monorepo packages.
"""sh <== change these to tildes
git clone [email protected]:zooniverse/front-end-monorepo.git
cd front-end-monorepo
yarn install <== NEW
yarn bootstrap
""" <== change these to tildes
The `bootstrap` script will install the dependencies and build any local packages used as dependencies.
// NEW
`yarn install` is required to be run _once_ for any yarn script to work. If you see an error message like _"Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)",_ it means you haven't run `yarn install` yet.
The Docker build is a production build, so staging projects will 404, if that helps. front-end-monorepo/docker-compose.yml Lines 17 to 21 in b4008d2 |
Done! Thanks for the suggestions, and I added them to the root Readme.
I can confirm this is the case for app-project. However, app-root runs via The culprit for app-project's behavior is getServerSideAPIHost.js. The host I have no idea how to fix this error, nor do I ever use Docker Desktop for development. I tried running The docker build actions do work for deploying, for instance this recent action. I'm likely not going to spend much more time trying to get Docker to work locally because we collectively don't use it for development. |
This means you’ve run a local build with front-end-monorepo/docker-compose.yml Lines 17 to 21 in b4008d2 |
This is the cause of the problem. You should run See the Docker instructions here: The development environment in |
|
I've opened #6465 with fixes for the local Docker builds, and updated documentation. The |
It just occurred to me that I should probably explain this. If you look at the Next.js apps running on Node in production, they use HTTP. HTTPS is handled upstream, so an incoming request for https://www.zooniverse.org is passed through to the Next.js app as a HTTP request, then the response is proxied back to the browser over HTTPS. This means that when you run the production apps/images locally, they don't support HTTPS. |
|
Thanks! That's helpful to know. I want to continue a discussion here that was started in #6465 (that PR is about to be merged) @eatyourgreens you mentioned the size of the FEM image.
In this PR,
😞 Might need to look into yarn workspaces focus. |
|
I think Yarn 4 will automatically ignore dev dependencies, as long as The build is a multi-stage build, with two stages:
|
|
I could also be totally wrong about Yarn 4 production installs and |
I think you're right about this because I remember reading about it when first creating this PR, but now I can't find it documented 🤔 Will continue to investigate... |
|
I'm also happy to answer any other questions that come up about the Docker builds, (if I know the answers.) |
|
Ugh using Docker Desktop to build an image of this branch is even larger 🫠 This screenshot is from And strangely enough, with |
|
This looks more like what I'd expect from
jimodonnell@Jims-MacBook-Pro front-end-monorepo % docker compose run --rm prod-shell
[+] Creating 1/0
✔ Network front-end-monorepo_default Created 0.0s
/usr/src # du -sh *
790.3M node_modules
4.0K package.json
641.7M packages
824.0K yarn.lock
/usr/src # |
|
I'm not sure why Yarn 4 is creating 2GB builds. I don't think those would break deploys, but they'd definitely slow down uploads/downloads to GitHub Container Repository. I don't know if GHCR has any usage limits either. I vaguely remember that, about a year ago, the monorepo ran into errors in GitHub Actions because the production images got really large. |
|
@goplayoutside3 I've opened #6469 to try and make it easier to debug the production build. One thing I noticed from the old docker compose setup: front-end-monorepo/docker-compose.yml Lines 4 to 5 in 958e806 |
|
Here's a recent tutorial for multi-stage Docker builds, including examples for Next.js. https://labs.iximiuz.com/tutorials/docker-multi-stage-builds |
Linked Issue and/or Talk Post
Yarn Migration docs: https://yarnpkg.com/migration/guide
Toward: #5970
Closes: #1470
Closes: #6391
Describe your changes
--frozen-lockfilecommand to--immutableper the migration guide.enableScripts: falsein the yarn config. This replaces the legacy--ignore-scriptsflag..gitignoreand remove some unrelated rules from it.How to Review
There's a lot of changes required to use Yarn Modern, so please see my comments below on individual file changes.
I deployed this branch to the
fe-project-branchand verified a successful deploy and I could see a project at that url. I might overwritefe-project-branchwith a different PR, but the successful deploy GH Action is linked here.To review:
yarn panicandyarn bootstrap.Checklist
PR Creator - Please cater the checklist to fit the review needed for your code changes.
PR Reviewer - Use the checklist during your review. Each point should be checkmarked or discussed before PR approval.
General
yarn panic && yarn bootstrapordocker-compose up --buildand FEM works as expectedGeneral UX
Example Staging Project: i-fancy-cats
Maintenance