Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and publishing process improvements #1118

Open
6 of 8 tasks
fvoska opened this issue Nov 4, 2022 · 2 comments
Open
6 of 8 tasks

Build and publishing process improvements #1118

fvoska opened this issue Nov 4, 2022 · 2 comments
Assignees

Comments

@fvoska
Copy link
Contributor

fvoska commented Nov 4, 2022

Relevant version

  • 0.x
  • 1.x
  • 2.x

Relevant libraries

  • utils
  • core
  • network
  • jsonapi
  • jsonapi-angular

Breaking change

No

Description

Currently, we use lerna for managing version updates and making new releases. For this to work, all libraries have to be in packages directory and it is expected that all libraries have package.json in packages/<LIB_NAME>/package.json. When a build is triggered, packages generate directory packages/<LIB_NAME>/dist with build files. For release to NPM, the whole packages/<LIB_NAME> dir gets packed and published. There are a couple of issues with this:

  1. package.json that ends up on npm includes things that are relevant only for development - things like scripts, devDependencies, etc.

  2. package.json has to be edited manually to add build-related things like:

    "main": "dist/index.cjs.js",
    "module": "dist/index.esm.js",
    "typings": "dist/index.d.ts",
    

    This is not really something that should be commited at all. If the build process changes (for example, if we switch to a different bundler) and starts generating different output files in dist, we don't want to update package.json manually and commit to the repository.

  3. packages/<LIB_NAME> dir can contain many things which are not relevant for NPM release and we have to list all of those things in .npmignore. In this process we of course miss some things, as can be seen on the example of @datx/core release that includes things like tsconfig.mobx.json:

    image

    tsconfig files are in no way relevant for package that gets pushed to NPM and should not be here. It even references some files which did get properly .npmignored ("extends": "./tsconfig.build.json",) and which are not in the released tarball.

If we move to publishing from dist, we can basically remove all .npmignore files that are currently being maintained tediously for each individual package.

Proposed solution

A solution which would work better is to pack packages/<LIB_NAME>/dist folder for NPM release instead of packing packages/<LIB_NAME>. Build process for all packages should be updated in a way that makes all the required files for publishing to NPM available in packages/<LIB_NAME>/dist. To publish all packages from dist, we would run lerna publish --contents dist

Build process should copy README.md and LICENSE files to dist as well.

Proposed solution should include two package.json files for each package. One that will be used for development and one that will be used as a basis for the build process and packaging (I will call this "release package.json" later in the text). Release package.json should include all the info that is relevant for publishing, but it should not hard-code links to build output files. Instead, the release package.json should be copied over to dist directory and the build process should update it with relevant links to build files.

Example of how this is done for angular packages:

This is the development package.json that should not be the one that is packaged into NPM release - packages/datx-jsonapi-angular/package.json:

{
  "name": "@datx/jsonapi-angular",
  "version": "2.4.6",
  "scripts": {
    "ng": "ng",
    "build": "ng build datx-jsonapi-angular && npm run copy-license && npm run copy-readme",
    "copy-license": "cp ./LICENSE ./dist/LICENSE",
    "copy-readme": "cp ./README.md ./dist/README.md",
    "prepublish": "npm run build",
    "test": "jest"
  },
  "peerDependencies": {
    "@angular/common": ">=12",
    "@angular/core": ">=12",
    "rxjs": "6 || 7"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~14.1.3",
    "@angular/animations": "~14.1.2",
    "@angular/cli": "~14.1.2",
    "@angular/common": "~14.2.0",
    "@angular/compiler": "~14.0.1",
    "@angular/compiler-cli": "~14.0.1",
    "@angular/core": "~14.2.0",
    "@angular/forms": "~14.1.2",
    "@angular/platform-browser": "~14.2.0",
    "@angular/platform-browser-dynamic": "~14.1.2",
    "@angular/router": "~14.2.0",
    "@datx/core": "2.4.6",
    "@datx/jsonapi": "2.4.6",
    "@datx/network": "2.4.6",
    "@datx/utils": "2.4.5",
    "@types/jest": "^28.1.1",
    "@types/node": "^18.7.6",
    "jest": "^28.1.1",
    "jest-preset-angular": "^12.1.0",
    "ng-packagr": "^14.0.1",
    "rxjs": "~7.5.5",
    "tslib": "^2.4.0",
    "typescript": "~4.7.3",
    "zone.js": "~0.11.5"
  }
}

This is the package.json that is copied over for release and is "augmented" with links to build bundle files - packages/datx-jsonapi-angular/projects/datx-jsonapi-angular/package.json:

{
  "name": "@datx/jsonapi-angular",
  "version": "2.4.3-beta.1",
  "description": "DatX mixin for Angular JSON API support",
  "sideEffects": false,
  "repository": {
    "type": "git",
    "url": "git+https://github.com/infinum/datx.git"
  },
  "bugs": {
    "url": "https://github.com/infinum/datx/issues"
  },
  "homepage": "https://github.com/infinum/datx#readme",
  "author": "Infinum JavaScript Team <[email protected]>",
  "license": "MIT",
  "keywords": [
    "datx",
    "mobx",
    "jsonapi",
    "angular"
  ],
  "dependencies": {
    "tslib": "^2.3.0"
  },
  "peerDependencies": {
    "@angular/common": ">=12",
    "@angular/core": ">=12",
    "@datx/jsonapi": "2.4.0",
    "rxjs": "6 || 7"
  }
}

As you can see, this is the file which has all the required metadata for NPM release. It also defines peerDependencies and version, but it has no mention of things like build scripts.

During the build process, this file gets copied over to packages/datx-jsonapi-angular/dist/package.json and gets augmented with links to build files (see what gets appended at the bottom of the file):

{
  "name": "@datx/jsonapi-angular",
  "version": "2.4.7",
  "description": "DatX mixin for Angular JSON API support",
  "sideEffects": false,
  "repository": {
    "type": "git",
    "url": "git+https://github.com/infinum/datx.git"
  },
  "bugs": {
    "url": "https://github.com/infinum/datx/issues"
  },
  "homepage": "https://github.com/infinum/datx#readme",
  "author": "Infinum JavaScript Team <[email protected]>",
  "license": "MIT",
  "keywords": [
    "datx",
    "mobx",
    "jsonapi",
    "angular"
  ],
  "dependencies": {
    "tslib": "^2.3.0"
  },
  "peerDependencies": {
    "@angular/common": ">=12",
    "@angular/core": ">=12",
    "@datx/jsonapi": "2.4.6",
    "rxjs": "6 || 7"
  },
  "module": "fesm2015/datx-jsonapi-angular.mjs",
  "es2020": "fesm2020/datx-jsonapi-angular.mjs",
  "esm2020": "esm2020/datx-jsonapi-angular.mjs",
  "fesm2020": "fesm2020/datx-jsonapi-angular.mjs",
  "fesm2015": "fesm2015/datx-jsonapi-angular.mjs",
  "typings": "index.d.ts",
  "exports": {
    "./package.json": {
      "default": "./package.json"
    },
    ".": {
      "types": "./index.d.ts",
      "esm2020": "./esm2020/datx-jsonapi-angular.mjs",
      "es2020": "./fesm2020/datx-jsonapi-angular.mjs",
      "es2015": "./fesm2015/datx-jsonapi-angular.mjs",
      "node": "./fesm2015/datx-jsonapi-angular.mjs",
      "default": "./fesm2020/datx-jsonapi-angular.mjs"
    }
  }
}

To release to NPM< we have to package packages/datx-jsonapi-angular/dist directory that looks like this:
image

That way we don't have to care about .npmignoring things, all that ends up in dist is what needs to be published.

Unclear things

It is not clear to me how lerna version and publish command will work with two package.json files per project. Maybe we can keep version in development package.json file for each lib to be 0.0.0 and update versions only in release package.json. I do not know how to configure lerna to target specific package.json file (release instead of development file).

@isBatak
Copy link
Contributor

isBatak commented Nov 4, 2022

@fvoska
Copy link
Contributor Author

fvoska commented Nov 4, 2022

@isBatak clean-packages solves only part of the issue - it strips package.json that gets published to NPM. However, you still have to take care of some things manually yourself:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants