Skip to content
This repository has been archived by the owner on Aug 11, 2022. It is now read-only.

release: npm@5.0.0 #16244

Closed
wants to merge 222 commits into from
Closed

release: npm@5.0.0 #16244

wants to merge 222 commits into from

Conversation

zkat
Copy link
Contributor

@zkat zkat commented Apr 4, 2017

NOTE: This PR is a WIP. npm@5 beta is currently underway. See #16510 for deets.

Wowowowowow npm@5!

This release marks months of hard work for the young, scrappy, and hungry CLI team, and includes some changes we've been hoping to do for literally years. npm@5 takes npm a pretty big step forward, significantly improving its performance in almost all common situations, fixing a bunch of old errors due to the architecture, and just generally making it more robust and fault-tolerant. It comes with changes to make life easier for people doing monorepos, for users who want consistency/security guarantees, and brings semver support to git dependencies. See below for all the deets!

Breaking Changes

Feature Summary

There's a bunch of exciting stuff going in here! You can track all the changes planned for npm@5 on its project tracker.

Installer changes

$ npm install
npm added 125, removed 32, updated 148 and moved 5 packages in 5.032s.
$
  • --parseable and --json now work more consistently across various commands, particularly install and ls.

  • Indentation is now detected and preserved for package.json, package-lock.json, and npm-shrinkwrap.json. If the package lock is missing, it will default to package.json's current indentation.

Publishing

Cache Rewrite!

We've been talking about rewriting the cache for a loooong time. So here it is. Lots of exciting stuff ahead. The rewrite will also enable some exciting future features, but we'll talk about those when they're actually in the works. #15666 is the main PR for all these changes. Additional PRs/commits are linked inline.

  • package metadata, package download, and caching infrastructure replaced

  • It's a bit faster. Hopefully it will be noticeable. 🤔

  • With the shrinkwrap and package-lock changes, tarballs will be looked up in the cache by content address (and verified with it)

  • Corrupted cache entries will automatically be removed and re-fetched on integrity check failure.

  • npm CLI now supports tarball hashes with any hash function supported by Node.js. That is, it will use sha512 for tarballs from registries that send a sha512 checksum as the tarball hash. Publishing with sha512 is added by feat(integrity): add integrity field to publish npm-registry-client#157 and may be backfilled by the registry for older entries.

  • remote tarball requests are now properly cached. This means that even if you're missing the integrity field in your npm-shrinkwrap.json, npm will be able to install from the cache.

  • Downloads for large packages are streamed in and out of disk. npm is now able to install packages of """any""" size without running out of memory. Support for publishing them is pending (due to registry limitations).

  • Automatic fallback-to-offline mode. npm will seamlessly use your cache if you are offline, or if you lose access to a particular registry (for example, if you can no longer access a private npm repo, or if your git host is unavailable).

  • A new --prefer-offline option will make npm skip any conditional requests (304 checks) for stale cache data, and only hit the network if something is missing from the cache.

  • A new --prefer-online option that will force npm to revalidate cached data (with 304 checks), ignoring any staleness checks, and refreshing the cache with revalidated, fresh data.

  • A new --offline option will force npm to use the cache or exit. It will error with an ENOTCACHED code if anything it tries to install isn't already in the cache.

  • A new npm cache verify command that will garbage collect your cache, reducing disk usage for things you don't need (-handwave-), and will do full integrity verification on both the index and the content. This will also be hooked into npm doctor as part of its larger suite of checking tools.

  • The new cache is very fault tolerant and supports concurrent access.

    • Multiple npm processes will not corrupt a shared cache.
    • Corrupted data will not be installed. Data is checked on both insertion and extraction, and treated as if it were missing if found to be corrupted. I will literally bake you a cookie if you manage to corrupt the cache in such a way that you end up with the wrong data in your installation (installer bugs notwithstanding).
    • npm cache clear is no longer useful for anything except clearing up disk space.
  • Package metadata is cached separately per registry and package type: you can't have package name conflicts between locally-installed packages, private repo packages, and public repo packages. Identical tarball data will still be shared/deduplicated as long as their hashes match.

  • HTTP cache-related headers and features are "fully" (lol) supported for both metadata and tarball requests -- if you have your own registry, you can define your own cache settings the CLI will obey!

  • prepublishOnly now runs before the tarball to publish is created, after prepare has run.

  • ...and more!

For release task list

  • First downstreaming to Node.js 8 (for 5/9)
  • Update docs for new/changes features
    • prepare on git dependency installation
    • package-lock.json vs shrinkwrap
    • Auto-save
    • sha512 on new publishes
  • Land file specifier branch (including idempotent package-lock and many other fixes)
    • Last work on file specifier branch (complete spec tests)
    • details! links to links,
    • details! removing links
  • Improve behavior for shrinkwrap/package-lock
    • warn if both files are there, preferring shrinkwrap
    • Add new fields from spec: created-with, shrinkwrap-version, package-integrity,
    • full new install semantics around dependency fields, based on shrinkwrap version
    • warn if the shrinkwrap is an incompatible version
      • for npm@5
    • update link spec doc to match shrinkwrap
    • Stop generating shrinkwrap/lockfiles off the filesystem -- calculate the tree w/ the installer
  • Make npm update save files as right type: devDeps end up getting saved as deps
  • Leave uninstallable optionals in the tree in such a way that they don't get installed by diff-trees but they do end up in the lock.
  • Ensure npm install --production doesn't remove dev deps from the lock.
  • Add notice and timing log levels directly to npmlog
  • Land patch about running prepare when installing git dependencies.
  • Fix cross-protocol redirects: can't fetch url which will redirect to another url with different protocol zkat/make-fetch-happen#26

@siddharthkp
Copy link

siddharthkp commented Apr 12, 2017

@zkat Will any of these flags become default?

Some folks bump into addition flags but it doesn't reach everyone (example: --cache-min)
--prefer-offline is something that would make npm installs faster for everyone and probably should be a default.

I understand that it breaks expectations on how people think npm works right now (which is --prefer-offline right now?) But, do you think major version is probably the time to change this expectation?

@zkat
Copy link
Contributor Author

zkat commented Apr 12, 2017

@siddharthkp There's no significant speed difference between --prefer-offline and --prefer-online when I actually test it (I'm surprised too). And I mean that -- the two installs on the repo I do my usual benchmarks are within 1 second of each other, for a minute-long install. Those 304 check are very fast, and pacote/make-fetch-happen are both very very fast at what they do when it comes to networking.

I see you mentioned that you think users would expect --prefer-offline to be the default, but I don't think that's true in practice: Many/most users install newer versions of packages, for example, by doing npm i -S pkg. In this case, using --prefer-offline will not upgrade anything: latest is already locally cached. pacote.manifest() will be satisfied with that, and the user will receive a very stale version of the package manifest. The intention behind --prefer-offline is to give users an option to reduce network usage in-general, if they're aware of the consequences of doing do.

Now, if you're looking for a speedup, the biggest one you can get with npm@5 is to use npm-shrinkwrap.json -- as of this patch, shrinkwrap will include content addresses for all package data, which means that as long as you have a warm cache, the network will not be used at all, and the entire package manifest phase will be completely skipped (because we don't need manifests if we have a shrinkwrap). The speedup from that is often on the order of 2x a non-shrinkwrap cached --prefer-offline install.

So, no, I probably won't use npm@5 to make --prefer-offline the default, since I'm not sure there's a reason to, in practice. Making npm-shrinkwrap be a default, though... that's in the works. Maybe not for npm@5 but I'm pushing for it for sure.

@siddharthkp
Copy link

siddharthkp commented Apr 12, 2017

There's no significant speed difference between --prefer-offline and --prefer-online when I actually test it (I'm surprised too).

😮
What were the network conditions like in your benchmarks?

I had a conversation with @iarna a while back (npm 3) where she educated me about the network requests to the registry even when using cache + the effect of latency on it. (Significant for emerging nations - prominently 2G and latency heavy nations - Australia and such)

Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯

I see you mentioned that you think users would expect --prefer-offline to be the default

Ouch! My bad, I meant --prefer-online. Honest mistake.

The speedup from that is often on the order of 2x a non-shrinkwrap cached --prefer-offline install.

This is another interesting part 🤔
Is this because npm still needs to resolve the dependency tree with --prefer-offline which means additional network requests? Not the same with shrinkwrap.

Making npm-shrinkwrap be a default, though... that's in the works.

💯 Is there a github issue that I can upvote for this?

@zkat
Copy link
Contributor Author

zkat commented Apr 12, 2017

What were the network conditions like in your benchmarks?

I'm on a Bay Area network, which is admittedly absurdly good. If the difference is so small now, though, I imagine it wouldn't be so big: even at 1/10th the speed, that would still be barely 10 seconds added (to a 60-second installation).

Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯

I really can't say that for sure right now. tbh, the difference is so surprisingly small that I think I'm gonna have to spend some time crawling through tests and making sure I don't have some bug. That code definitely has tests attached, though...

Ouch! My bad, I meant --prefer-online. Honest mistake.

Ah. As far as this one goes, pacote obeys cache age settings from a given registry. This is generally happy news for all involved (if we hit the registry less without degrading user experience, that's a win!), and those settings tend to be fairly low: the main npm registry sets Cache-Control: max-age=300, which is in seconds (5 minutes). After those 5 minutes, we'll do the 304 checks again. This seems reasonable to me, specially with the --prefer-offline escape hatch. Anyway, no worries. They're kinda easy to get confused 😁

I would care more if it weren't for shrinkwrap being the main target of optimization, which brings me to...

Is this because npm still needs to resolve the dependency tree

Pretty much this. If we don't have the exact hashes for the tarballs we want to end up installing, npm needs to repeat the process of lookup by name --> resolve version --> get shasum + dependency names from version --> repeat for all recursive dependencies. That's a pretty heavy process.

💯 Is there a github issue that I can upvote for this?

Not quite yet. There's a github project for npm@5 and one of my TODOs is to write a proposal for how shrinkwrap-by-default would work. From what I've talked with @iarna, though, we need to fix other stuff (like npm update) before we even consider this. So, it'll probably be specced out after npm@5 is out and we're ready to start thinking about npm@6.

@siddharthkp
Copy link

If shrinkwrap by default is likely to come in npm6, does it make sense to make --prefer-offline a default in npm5?
Or is that going in the wrong direction?

@viktor-ku
Copy link

viktor-ku commented Apr 12, 2017

Any reasons why yarn is so fast? I really expect npm@5 to be as fast as yarn. How can I help?

@siddharthkp
Copy link

@Kuroljov With npm-shrinkwrap, npm will be in serious competition with yarn.

@zkat Can we start playing around with that somewhere? (Can contribute to your benchmarks?)

@zkat
Copy link
Contributor Author

zkat commented Apr 12, 2017

@Kuroljov This release addresses a lot of performance issues. Right now, npm@5 installs are running anywhere between 2x-5x faster than npm@4. There's a few optimizations left that I don't know if we'll have ready by npm@5, but that number is bound to get significantly bigger (as in, 30%+ from one of those changes)

Some sample benchmarks:

react boilerplate install (including lifecycle scripts):
npm@4 i (no cache): 147.23s
npm@4 i (warm cache): 145.70s
npm@5 i (no cache): 74.30s
npm@5 i (warm cache): 70.85s
npm@5 i (no cache, no shrinkwrap): 99.83s
~/code/npm-website $ npm i --ignore-scripts
warm cache:
npm@4.4.0  - 150s
npm@5.0.0 - 52s
npm@5.0.0 w/ shrinkwrap   - 38s

@zkat
Copy link
Contributor Author

zkat commented Apr 12, 2017

@siddharthkp expect a beta release of npm@5 in the coming weeks. I can't give you the exact date, but I'm currently in the process of reviving npm's massive test suite after such a big change and fixing all my little messups along the way.

What I can tell you is that if you make https://npm.im/make-fetch-happen and https://npm.im/cacache faster and better, it'll be directly reflected on the performance of npm@5. Once https://npm.im/pacote gets its @2 release (probably shortly before npm@beta), the same will apply to that. Those first two projects are relatively stable and usable, though, and I think people can definitely already start contributing significantly to them. There's also https://npm.im/ssri but I don't think that's much of a bottleneck at all rn.

I'll make an announcement somewhere dev-oriented once npm@beta is out, cause I definitely want early adopters to help me find all the things that broke during the rewrite -- because it's a big one.

@zkat
Copy link
Contributor Author

zkat commented Apr 20, 2017

@siddharthkp coming back to this: what are the network conditions where you're at? (since you asked). I'd like to get a better, more realistic perspective on how npm5 handles slow and/or unstable networks and special configurations like proxies, which is all fairly hard to simulate unless you've been deep in that situation, imo. There's now a (semi-closed) beta of npm5 that you can install with npm i -g 'npm/npm#release-beta-5 that will give you an npm5 binary that lives alongside your standard installation. It would be super awesome if folks who are willing to look into weirdness tried it out and gave feedback. I very much want it to be a smooth, seamless experience even in bad network conditions: the worst thing that should happen on unreliable networks is that your download will be slightly slower. Everything else should Just Work™.

zkat and others added 21 commits May 25, 2017 18:55
Previously we were doing this so we could load relationships, but now we can
simply request no fake children.  This forces a fetch-metadata phase for the
lockfile, but does not actually install all the modules.
This can happen when, for example, removing a module from your project that
isn't actually in your node_modules at the moment.
It used to be that we did this check in `is-extraneous` but it turns out to
be much cleaner to keep that pure.  It let's us use `is-extraneous` for
things like pruning.
@zkat zkat added the npm5 label May 26, 2017
@zkat
Copy link
Contributor Author

zkat commented May 27, 2017

This was tagged and released yesterday as npm@5 🎉 thanks everyone!

@zkat zkat closed this May 27, 2017
@zkat zkat deleted the release-next-5 branch May 27, 2017 05:59
@zkat zkat removed the in-progress label May 27, 2017
@olegkorol
Copy link

Hi,

what is the new format of npm-shrinkwrap.json?
After updating npm to v. 5 it won't npm install anymore...

Thanks in advance!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet