Earlier today, July 6, 2016, the npm registry experienced a read outage for 0.5% of all package tarballs for all network regions. Not all packages and versions were affected, but the ones that were affected were completely unavailable during the outage for any region of our CDN.
The unavailable tarballs were offline for about 16 hours, from mid-afternoon PDT on July 5 to early morning July 6. All tarballs should now be available for read.
Here’s the outage timeline:
Over the next hour 502 rates fell to the normal 0.
We’re adding an alert on all 500-class status codes, not just 503s. This alert will catch the category of errors, not simply this specific problem.
We’re also revising our operational playbook to encourage examination of our CDN logs more frequently; we could have caught the problem very soon after introducing it if we had carefully verified that our guess about the source of 502s had resulted in making them vanish from our CDN logging. We can also do better with tools for examining the patterns of errors across POPs, which would have made it clearer to us immediately that the error was not specific to the US East coast and was therefore unlikely to have been caused by an outage in our CDN.
Read on if you would like the details of the bug.
The root cause for this outage was an interesting interaction of file modification time, nginx’s method of generating etags, and cache headers.
We recently examined our CDN caching strategies and learned that we were not caching as effectively as we might, because of a property of nginx. Nginx’s etags are generated using the file modification time as well as its size, roughly as mtime + '-' + the file size in bytes
. This meant that if mtimes for package tarballs varied across our nginx instances, our CDN would treat the files from each server as distinct, and cache them separately. Getting the most from our CDN’s caches and from our users’ local tarball caches is key to good performance on npm installs, so we took steps to make the etags match across all our services.
Our chosen scheme was to set the file modification time to the first 32-bit BE integer from their md5 hash. This was entirely arbitrary but looked sufficient after testing in our staging environment. We produced consistent etags. Unfortunately, the script that applied this change to our production environment failed to clamp the resulting integer, resulting in negative numbers for timestamps. Ordinarily, this would result in a the infamous Dec 31, 1969 date one sees for timestamps before the Unix epoch.
Unfortunately, negative mtimes triggered an nginx bug. Nginx will serve the first request for a file in this state and deliver the negative etag. However, if there is a negative etag in the if-none-match
header nginx attempts to serve a 304 but never completes the request. This resulted in the the bad gateway message returned by our CDN to users attempting to fetch a tarball with the bad mtime.
You can observe this behavior yourself with nginx and curl:
The final request never completes even though nginx has correctly given it a 304 status.
Because this only affected a small subset of tarballs, not including the tarball fetched by our smoketest alert, all servers remained in the pool. We have an alert on above-normal 503 error rates served by our CDN, but this error state produced 502s and was not caught.
All the tarballs that were producing a 502 gateway timeout error turned out to have negative timestamps in their file mtimes. The fix was to touch them all so their times were inconsistent across our servers but valid, thus both busting our CDN’s cache and dodging the nginx behavior.
The logs from our CDN are invaluable, because they tell us what quality of service our users are truly experiencing. Sometimes everything looks green on our own monitoring, but it’s not green from our users’ perspective. The logs are how we know.
Today, npm Enterprise has just grown hugely more extensible and powerful with the release of npm Enterprise add-ons.
It’s now possible to integrate third-parties’ developer tools directly into npm Enterprise. This has the power to combine what were discrete parts of your development workflow into a single user experience, and knock out the barriers that stand in the way of bringing open source development’s many-small-reusable-parts methodology into larger organizations.
npm Enterprise now exposes an API that allows third-party developers to build on top of our npm Enterprise product:
With this deceptively simple functionality, developers can offer a huge amount of value to enrich the process of using npm within the enterprise.
Enterprise developers already want to take advantage of the same code discovery, re-use, and collaboration enjoyed by millions of open source developers, billions of times every month. But this requires accommodating their companies’ op-sec, licensing, and code quality processes, which often predate the modern era.
For example…
In the past, it was possible to manually research the security implications of external code. But with the average npm package relying on over 100 dependencies and subdependencies, this process just doesn’t scale.
Without a way to ensure the security of each package, a company can’t take advantage of open source code.
Software that is missing a license, or that’s governed by a license unblessed by a company’s legal department, simply can’t be used at larger companies. Much like security screening, many companies have relied upon manually reviewing the license requirements of each piece of external code. And just like security research, trying to manually confirm the licensing of every dependency (and their dependencies, and their dependencies…) is impossible to scale.
Enterprise developers need a way to understand the license implications of packages they’re considering using, and companies need a way to certify that all of their projects are legally kosher.
Will bug reports be patched quickly? Is the code written well? Do packages rely on stale or abandoned dependencies? These questions demand answers before an enterprise can consider relying on open source code.
Without a way to quantitatively analyze the quality of every code package in a project, many enterprise teams simply don’t adopt open source code or workflows for mission-critical projects.
Our three launch partners, Node Security Platform, FOSSA, and bitHound, address these concerns, respectively.
You can learn about the specifics of each of them here:
By integrating them directly into the tool that enterprise developers use to browse and manage packages, we make it as easy as possible to scratch enterprise development’s specific itches. As more incredible add-ons join the platform, the barriers to open source-style development at big companies get knocked down, one by one.
The Node Security Platform, FOSSA, and bitHound add-ons are available to existing npm Enterprise customers today. Simply contact us at support@npmjs.com to get set up.
If you’re looking to bring npm Enterprise and add-ons into your enterprise, let us show you how easy it is with a free 30-day trial.
Interested in building your own add-on? Awesome. Stay tuned: API documentation is on its way.
The movement to bring open source code, workflows, and tools into the enterprise is called InnerSource, and it’s the beginning of a revolution.
When companies develop proprietary code the same way communities build open source projects, then the open source community’s methods and tooling become the default way to build software.
Everyone stands to benefit from InnerSource because everyone stands to benefit from building software the right way: open source packages see more adoption and community participation, companies build projects faster and cheaper without re-inventing wheels, and developers are empowered to build amazing things.
Add-ons are an exciting step forward for us. We’re thrilled you’re joining us.
When using npm Enterprise, we sometimes encounter public packages in our private registry that need to fetch resources from the public internet when being installed by a client via npm install
.
Unfortunately, this poses a problem for developers who work in an environment with limited or no access to the public internet.
Let’s take a look at some of the more common types of problems in this area and talk about ways we can work around them.
Note that these problems are not specific to npm Enterprise — but to using certain public packages in any limited-access environment. That being said, there are some things that npm (as an organization and software vendor) can do to better prevent or handle some of these problems. We’re still working to make these improvements.
Typically, developers will discover the problem when installing packages from their private registry. When this happens, we need to determine the type of problem it is and where in the dependency graph the problematic dependency resides.
Here are some common problem types:
Git repo dependency
This is when a package dependency is listed in a package.json
file with a reference to a Git repository instead of with a semver version range. Typically these point to a particular branch or revision in a public GitHub or Bitbucket repository. They are mainly used when the package contents have not been published to the public npm registry.
When the npm client encounters these, it attempts to fetch the package from the Git repository directly, which is a problem for folks who do not have network access to the repository.
Shrinkwrapped package
This is when the internal contents of a package contain an npm-shrinkwrap.json
file that lists a specific version and URL to use for each mentioned package from the dependency tree.
During a normal npm install
, the npm client attempts to fetch the dependencies listed in npm-shrinkwrap.json
directly from the URLs contained in the file. This poses a problem when the client installing the shrinkwrapped package does not have access to the URLs that the shrinkwrap author has access to.
Package with install script or node-gyp dependency
This is when a package attempts to defer some setup process until the package is installed, using a script defined in package.json
, which typically involves building platform-specific binaries or Node add-ons on the client’s machine.
On a typical install, the npm client will find and run these scripts in order to automatically fetch and build the required resources, targeting the platform that the client is running on. But when limited internet access means the necessary resources cannot be fetched, the install will fail. Most likely the package will be unusable until the end result of running the install script on the client’s machine is achieved.
To determine the location of the problematic dependency, we can boil it down to two categories:
Direct dependency
A direct dependency is one that is explicitly listed in your own package.json
file — a dependency that your project/package uses directly in code or in an npm run script.
Transitive dependency
A transitive dependency is one that is not explicitly listed in your own package.json
file — a dependency that comes from anywhere in the tree of your direct dependencies’ dependencies.
The same way publishing a package to the public registry requires access to the public internet, most of these solutions require Internet access, at least on a temporary basis. Once the solution is in place, then access to public resources can be restricted.
For starters, remember that it’s generally a good idea to use the latest version of the npm
client. To install or upgrade to the latest version, regardless of what version of Node you have installed, run npm i -g npm@latest
(and make sure npm -v
prints the version that was installed).
Let’s go over the problem types in more detail.
Unfortunately, a dependency that references a Git repository (instead of a semver range for a published package) must be replaced with a published package. To do this, you’ll need to first publish the Git repository as a package to your npm Enterprise registry and then fork the project with the Git dependency and replace the dependency with the package you published. Then, publish the forked project, and use that package as a dependency (instead of the original).
It’s usually a good idea to open an issue on the project with the Git dependency, politely asking the maintainers to replace the Git dependency, if possible. Generally, we discourage using Git dependencies in package.json
, and it’s typically only used temporarily while a maintainer waits for an upstream fix to be applied and published.
Example: let’s replace the "grunt-mocha-istanbul": "christian-bromann/grunt-mocha-istanbul"
Git dependency defined in version 4.0.4 of the webdriverio
package, assuming that webdriverio
is a direct dependency and grunt-mocha-istanbul
is a transitive dependency.
We’ll tackle this in two main steps: forking and publishing the transitive dependency, and forking and publishing the direct dependency.
Clone the project that is referenced by the Git dependency
Optionally, you can create a remote fork first (e.g., in GitHub or Bitbucket) and then clone your fork locally. Otherwise, you can just clone/download the project directly from the remote repository. It’s a good idea to use source control so you can keep a history of your changes, but you could also probably get away with downloading and extracting the project contents.
Example:
git clone https://github.com/christian-bromann/grunt-mocha-istanbul.git
Create a new branch to hold your customizations
Again, this is so you can keep a history of your changes. It’s probably a good idea to include the current version of the package in the branch name, in case you need to repeat these steps when a later version is available.
Example:
cd grunt-mocha-istanbul
git checkout -b myco-custom-3.0.1
Add your scope to the package name in package.json
In our example, change "grunt-mocha-istanbul"
to "@myco/grunt-mocha-istanbul"
.
Commit your changes to your branch and publish the scoped package to your npm Enterprise registry
Assuming you have already configured npm
to associate your scope to your private registry, publishing should be as simple as npm publish
.
Example:
git add package.json
git commit -m 'add @myco scope to package name'
npm publish
Clone the project’s source code locally
Either create a remote fork first (e.g., in GitHub or Bitbucket) and clone your fork locally, or just clone/download the project directly from the original remote repository. It’s a good idea to use source control so you can keep a history of your changes.
Example:
git clone https://github.com/webdriverio/webdriverio.git
Create a new branch to hold your customizations
This is so you can keep a history of your changes. It’s probably a good idea to include the current version of the package in the branch name, in case you need to repeat these steps when a later version is available.
Example:
cd webdriverio
git checkout -b myco-custom-4.0.4
Add your scope to the package name in package.json
In our example, change "webdriverio"
to "@myco/webdriverio"
.
Replace the Git dependency with the scoped package
This means updating the reference in package.json
, and it may mean updating require()
or import
statements too. You should basically do a find-and-replace, finding the unscoped package name and judiciously replacing it with the scoped package name.
In our example, we only need to update the reference in package.json
from "grunt-mocha-istanbul": "christian-bromann/grunt-mocha-istanbul"
to "@myco/grunt-mocha-istanbul": "^3.0.1"
.
Commit your changes to your branch and publish the scoped package to your npm Enterprise registry
Assuming you’ve already configured npm
to associate your scope to your private registry, publishing should be as simple as npm publish
.
In our example of webdriverio
, we next need to deal with the shrinkwrap URLs before we can publish (handled below). In other scenarios, it may be possible to publish now.
Example:
git add .
git commit -m 'replace git dep with scoped fork'
npm publish
Update your downstream project(s) to use the scoped package as a direct dependency (in package.json
and in any require()
statements)
In our example, this basically means doing a find-and-replace to find references to webdriverio
and judiciously replace them with @myco/webdriverio
. However, webdriverio
also contains an npm-shrinkwrap.json
file. We’ll cover that in the next section.
It just so happens that our sample direct dependency above (webdriverio
) also uses an npm-shrinkwrap.json
file to pin certain dependencies to specific versions. Unfortunately the shrinkwrap file contains hardcoded URLs to the public registry. We need a way to either ignore or fix the URLs.
A quick workaround is to install packages using the --no-shrinkwrap
flag. This will tell the npm
client to ignore any shrinkwrap files it finds in the package dependency tree and, instead, install the dependencies from package.json
in the normal fashion.
This is considered a workaround rather than a long-term solution: it’s possible that installing from package.json
will install versions of dependencies that don’t exactly match the ones listed in npm-shrinkwrap.json
, even though the versions of the package’s direct dependencies are guaranteed to be within the declared semver range.
Example:
npm install webdriverio --no-shrinkwrap
(As noted above, webdriverio@4.0.4
also has a Git dependency, so just ignoring the shrinkwrap isn’t quite enough for this package.)
If you want to use the exact versions from the shrinkwrap file without using the URLs in it, you’ll have to use your own custom fork of the project that contains a modified shrinkwrap file.
Here’s the general idea:
(Note that steps 1-3 are identical to the fork-publish instructions for a direct dependency above. If you’ve already completed them, skip to step 4.)
Clone the project’s source code locally
Either create a remote fork first (e.g., in GitHub or Bitbucket) and clone your fork locally, or just clone/download the project directly from the original remote repository. It’s a good idea to use source control so you can keep a history of your changes.
Example:
git clone https://github.com/webdriverio/webdriverio.git
Create a new branch to hold your customizations
This is so you can keep a history of your changes. It’s probably a good idea to include the current version of the package in the branch name, in case you need to repeat these steps when a later version is available.
Example:
cd webdriverio
git checkout -b myco-custom-4.0.4
Add your scope to the package name in package.json
In our example, change "webdriverio"
to "@myco/webdriverio"
.
Use rewrite-shrinkwrap-urls
to modify npm-shrinkwrap.json
, pointing the URLs to your npm Enterprise registry
Unfortunately this is slightly more complicated than a find-and-replace, since the tarball URL structure of the public registry is different than the one used for an npm Enterprise private registry.
In the example below, replace {your-registry}
with the base URL of your private registry, e.g., https://npm-registry.myco.com
or http://localhost:8080
. The value you use should come from the Full URL of npm Enterprise registry setting in your Enterprise admin UI Settings page.
Example:
npm install -g rewrite-shrinkwrap-urls
rewrite-shrinkwrap-urls -r {your-registry}
git diff npm-shrinkwrap.json
Commit your changes to your branch and publish the scoped package to your npm Enterprise registry
Assuming you’ve already configured npm
to associate your scope to your private registry, publishing should be as simple as npm publish
.
Be mindful of any prepublish
or publish
scripts that may be defined in package.json
. You can try skipping those scripts when publishing via npm publish --ignore-scripts
, but running the scripts may be necessary to put the package into a usable state, e.g., if source transpilation is required.
Example:
git add npm-shrinkwrap.json package.json
git commit -m 'add @myco scope to package name' package.json
git commit -m 'rewrite shrinkwrap urls' npm-shrinkwrap.json
npm publish
Note that a prepublish
script will probably need to install the package’s dependencies in order to run. In this case, npm install
will be executed first. If this happens, it should pull all dependencies in the shrinkwrap file from your registry. If any of those packages don’t yet exist in your registry, you’ll need either to enable the Read Through Cache setting in your Enterprise instance or to manually add the packages to the white-list by running npme add-package webdriverio
from your server’s shell and answering Y
at the prompt to add dependencies.
Update your downstream project(s) to use the scoped package as a direct dependency (in package.json
and in any require()
statements)
In our example, this basically means doing a find-and-replace to find references to webdriverio
and judiciously replace them with @myco/webdriverio
.
This is less than ideal, obviously. We’re currently considering ways to improve handling of shrinkwrapped packages on the server side, but a better solution is not yet available.
Some packages want or need to run some script(s) on installation in order to build platform-specific dependencies or otherwise put the package into a usable state. This approach means that a package can be distributed as platform-independent source without having to prebundle binaries or provide multiple installation options.
Unfortunately this also means that these packages typically need access to the public internet in order to fetch required resources. In these cases, we can’t really do much to work around this approach, other than attempt to isolate the steps of fetching the package from the registry and set up the platform-specific resources it needs.
As a quick first attempt, you can ignore lifecycle scripts when installing packages via npm install {pkg-name} --ignore-scripts
.
Unfortunately, install scripts typically do some sort of platform-specific setup to make the package usable. Thus, you should review the install
or postinstall
scripts from the package’s package.json
file and determine if you need to attempt to run them separately or somehow achieve the same result manually.
When node-gyp is involved in the setup process, the package requires platform-specific binaries to be built and plugged into the Node runtime on the client’s system. In order to build the binaries, the package will typically need to fetch source header files for the Node API.
The best we can do is attempt to setup the node-gyp build toolchain manually. This requires Python and a C/C++ compiler. You can read more about this at the following locations:
General installation: https://github.com/nodejs/node-gyp#installation
Windows issues: https://github.com/nodejs/node-gyp/issues/629
A good example of a package with a node-gyp dependency is node-sass
.
Once the build toolchain is in place, the package’s install script may not need to fetch any external resources.
If you’ve made it all the way to the end, surely you’ll agree that npm could be handling things better to minimize challenges faced by folks with restricted internet access. We feel it’s in the community’s best interest to at least raise awareness of these problems and their potential workarounds until we can get a more robust solution in place.
If you have feedback or questions, as always, please don’t hesitate to let us know.
Today, we’re excited to announce a simple, powerful new way to track changes to the npm registry — and build your own amazing new developer tools: hooks.
Hooks are notifications of npm registry events that you’ve subscribed to. Using hooks, you can build integrations that do something useful (or silly) in response to package changes on the registry.
Each time a package is changed, we’ll send an HTTP POST payload to the URI you’ve configured for your hook. You can add hooks to follow specific packages, to follow all the activity of given npm users, or to follow all the packages in an organization or user scope.
For example, you could watch all packages published in the @npm
scope by setting up a hook for @npm
. If you wanted to watch just lodash
, you could set up a hook for lodash
.
If you have a paid individual or organizational npm account, you can start using hooks right now.
Each user may configure a total of 100 hooks, and how you use them is up to you: you can put all 100 on a single package, or scatter them across 100 different packages. If you use a hook to watch a scope, this counts as a single hook, regardless of how many packages are in the scope. You can watch any open source package on the npm registry, and any private package that you control (you’ll only receive hooks for packages you have permission to see).
Create your first hook right now using the wombat cli tool.
First, install wombat
the usual way: npm install -g wombat
. Then, set up some hooks:
Watch the npm
package:wombat hook add npm https://example.com/webhooks shared-secret-text
Watch the @slack
organization for updates to their API clients:wombat hook add @slack https://example.com/webhooks but-sadly-not-very-secret
Watch the ever-prolific substack:wombat hook add --type=owner substack https://example.com/webhooks this-secret-is-very-shared
Look at all your hooks and when they were last triggered:wombat hook ls
Protip: Wombat has several other interesting commands. wombat --help
will tell you all about them.
We’re also making public an API for working with hooks. Read the docs for details on how you can use the API to manage your hooks without using wombat
.
You can use hooks to trigger integration testing, trigger a deploy, make an announcement in a chat channel, or trigger an update of your own packages.
To get you started, here are some of the things we’ve built while developing hooks for you:
npm-hook-receiver: an example receiver that creates a restify server to listen for hook HTTP posts. Source code.
npm-hook-slack: the world’s simplest Slackbot for reporting package events to Slack; built on npm-hook-receiver
.
captain-hook: a much more interesting Slackbot that lets you manage your webhooks as well as receive the posts.
wombat: a CLI tool for inspecting and editing your hooks. This client exercises the full hooks API. Source code.
ifttt-hook-translator: Code to receive a webhook and translate it to an IFTTT event, which you can then use to trigger anything else you can do on IFTTT.
citgm-harness: This is a proof-of-concept of how node.js’s Canary in the Gold Mine suite might use hooks to drive its package tests. Specific package publications trigger continuous integration testing of a different project, which is one way to test that you haven’t broken your downstream dependents.
We’re releasing hooks as a beta, and where we take it from here is up to you. What do you think about it? Are there other events you’d like to watch? Is 100 hooks just right, too many, or not enough?
We’re really (really) (really) interested to see what you come up with. If you build something useful (or silly) using hooks, don’t be shy to drop us a line or poke us on Twitter.
This is the tip of an excitement iceberg — exciteberg, if you will — of cool new ways to use npm. Watch this space!
npm ♥ you!
The 283,000 (!) packages in the npm Registry are only useful if it’s easy for developers to integrate them into their projects and deploy their code, so we’re excited by any chance to streamline your workflow.
If you work with Bitbucket, starting today, it’s easier than ever to install and publish npm private packages — to either your npm account or your self-hosted npm Enterprise installation.
Bitbucket Pipelines is a new continuous integration service built into Bitbucket Cloud for end-to-end visibility, from coding to deployment. We’re excited that they’re launching with npm support.
Why?
How?…
bitbucket-pipelines.yml
supplied in this repository.NPM_TOKEN
environment variable.
~/.npmrc
, after you log in to the registry.bitbucket-pipelines.yml
supplied in this repository.NPM_TOKEN
environment variable:
~/.npmrc
, after you log in to the registry.NPM_REGISTRY_URL
to the full URL of your private registry (with scheme).Alongside the new pipelines integration, the npm for Bitbucket add-on has been updated to support private modules.
This helps complete an elegant CI/CD workflow:
Get started by installing the add-on now.
We have more exciting integrations and improvements in the … pipeline (sorry), but it helps to know what matters to you. Don’t be shy to share feedback in the comments or hit us up on Twitter.
Last month, we released a “one-click” installer for npm Enterprise on AWS. Fresh on its heels, we’re excited to announce support for Google Compute Engine
Getting npm Enterprise up and running on GCE is easy:
gcloud auth login
.gcloud config set project my-project
gcloud config set compute/zone us-east1-d
Fetch npm’s configuration template:
curl -XGET https://raw.githubusercontent.com/npm/npme-installer/master/npme-gce.jinja > /tmp/npme-gce.jinja
Run the npm Enterprise deploy template:
gcloud deployment-manager deployments create npme-deployment --config /tmp/npme-gce.jinja --properties="zone=us-east1-d"
Note: You can replace us-east1-d
with whatever zone you’d like to deploy to.
When you get back, if you visit your cloud console you’ll see a running server called npm-enterprise
. That’s all there is to it!
You can configure your instance by visiting
port 8800
. Our npm Enterprise will point you in the right direction with information on the various settings that you can configure.
It’s our continued goal to make npm Enterprise painless to run, regardless of your infrastructure. Are we missing a platform that you’d love for us to support? Let us know in the comments.
How many npm users are there? It’s a surprisingly tricky question.
There are a little over 211,000 registered npm users, of whom about 73,000 have published packages. But far more people than that use npm: most things npm does do not require you to login or register. So how many are there, and what are they up to? Our first and most obvious clue is the number of packages they are downloading:
That’s over a billion downloads per week, and that only counts package installs that weren’t already in cache – about 66% of npm package installs don’t require downloading any packages, because they fetch from the cache. Still, the growth is truly, without exaggeration, exponential.
So what’s driving all the downloads? Are the same people building ever-more-complicated applications, downloading more packages to do it? Are the same people running build servers over and over? Or are there actually more people? Our next clue comes from the number of unique IPs hitting the registry:
Here’s a ton of growth again, close to 100% growth year-on-year, but much more linear than the downloads: 3.1 million IPs hit the registry in March. Of course, IP addresses are not people. Some of these IPs are build servers and other robots. Other IP addresses are companies or educational institions that serve thousands or even tens of thousands of people. So while it doesn’t correlate perfectly, generally speaking, more IPs means more people are using npm.
Every time npm runs it generates a unique ID that it sends as a header for every request it makes during that run. This ID is random and not tied to you in any way, and once npm finishes running it is deleted. We use this ID for debugging the registry: it lets us see that these 5, 10 or 50 requests were all part of the same operation, which makes it easier to see what’s going on when somebody has a problem. It also makes it possible to say roughly how many times npm is run – or at least, run in a way that makes a request to the registry. There were 84 million npm sessions in March: this number is growing faster than IPs, but less quickly than downloads.
We can take these last two and combine them:
This number is interesting because it’s not going anywhere. The ratio of packages downloaded to npm sessions is essentially constant (this is not the same as the number of packages downloaded per install, because many npm sessions don’t result in downloads). But this is a clear signal: the number of packages per install isn’t rising. Applications aren’t getting notably more complicated; people are installing packages more often because they are writing more applications.
Here’s a final clue:
The number of packages downloaded by an IP is also rising linearly. So, not only are more people using npm, but the people who are already using npm are using it more and more frequently. And then of course there’s this number:
Another way of counting npm users is counting people who visit npm’s website. This also grew enormously; 400% since we started the company. In the last 90 days, npm saw just over 4 million unique users visit our site. Ordinarily, you take web user numbers with a grain of salt – there’s lots of ways they can be wrong. But combined with the IPs, the sessions, and the download numbers, we think that number is probably accurate, maybe even a little conservative.
There are so many sources of error! There are robots who crawl the registry. There are lots of companies who host their own internal registry caches, or run npm Enterprise, and so have their own npm website and registry and never hit ours. There’s the entire nation of China, which finds it difficult to access npm through the great firewall, and is served by our hard-working friends at cnpmjs. There’s errors that inflate our numbers, and errors that deflate them. If you think we’re way off, let us know. But we think we have enough for a good guess.
We think there are four million npm users, and we think that number is doubling every year. Over at the node.js foundation, they see similar growth numbers. Not only are there more of them, but they’re more engaged than ever before. This is awesome! The 25 very hard working people of npm Inc. thank you for your participation and your contributions to the community, and we hope to see even more of you.
In a previous blog post we showed you how easy it is to run npm Enterprise on Amazon Web Services. Today, we’re happy to announce the public availability of the npm Enterprise Amazon Machine Image (AMI). Now, it’s even easier to run your own private npm registry and website on AWS!
Using our AMI, there is nothing to install. Just launch an instance, configure it using the npm Enterprise admin web UI, and you’re done: it’s a true point-and-click solution for sharing and managing private JavaScript packages within your company.
Let’s take a quick look at the details.
We have AMIs for several AWS regions. When you launch a new instance in the AWS EC2 Console, find the right one by searching for the relevant AMI ID under the Community AMIs tab. Note that new AMI versions are published about every month and include the date of publication in the AMI name.
Here’s a list of the AMI IDs by region:
us-east-1
(N. Virginia): ami-edd65bfa
us-west-1
(N. California): ami-61db9d01
us-west-2
(Oregon): ami-dc34f6bc
eu-central-1
(Frankfurt): ami-5ec13431
ap-southeast-2
(Sydney): ami-7d32181e
Ensure the AMI comes from owner 666882590071
.
If you don’t see your preferred region in the list above, contact our support team, and we’ll get one created for you!
When you launch an instance of the AMI, you’ll need to:
m3.large
or better22
(ssh), 8080
(registry), 8081
(website), and 8800
(npm Enterprise admin UI).pem
key pair: this allows you to ssh
into your server instanceIt’s not necessary, but if you’d prefer to attach an EBS volume for registry data that is separate from the root volume, you can. However, the root EBS volume cannot be smaller than 16 GB.
For more information (or screenshots) on any of the above, see our docs for Running npm Enterprise in AWS.
You don’t have to, but you can ssh
into your EC2 instance to make sure it’s up and running. If you do, you should see a welcome message like the following:
Open your favorite web browser, access your server on port 8800
, and follow the prompts to configure and start your appliance.
You’ll need a license key. If you haven’t already purchased one, you can get a free trial key here.
For more information on configuring npm Enterprise, visit our docs.
That’s it! Once you’ve configured and started the appliance, your private npm registry and website are ready for use. See this document for configuring your npm CLI to use your new private registry.
We’re continually striving to provide you the best solutions for distributing, discovering, and reusing your JavaScript code and packages. We hope this AMI makes it just that much easier to leverage the same tools within your organization that work so well in open source communities around the world - a concept we refer to as InnerSource.
As always, if you have questions or feedback, please reach out.
Effective yesterday morning, all requests to the npm registry are made via HTTPS
.
Practically this means:
http://registry.npmjs.org/pkgname
you get a JSON responsehttp://registry.npmjs.org/pkgname/-/pkgname-1.2.3.tgz
https://registry.npmjs.org/pkgname/-/pkgname-1.2.3.tgz
.http://registry.npmjs.org/pkgname
will 301 (redirect) over to https://registry.npmjs.org/pkgname
No! The CLI client checks a shashum to verify the package and that check always has been over HTTPS.
We’ve developed an ecosystem of tools that you can use to replicate the registry in a way that is resilient to these changes:
_changes
feed: https://skimdb.npmjs.com/registry/_changes?descending=true&limit=10
For every change in a package in the registry, the whole package object (with changes) gets emitted as data on the _changes
feed of CouchDB.
follower: https://github.com/npm/concurrent-couch-follower
Users wishing to follow the changes feed can use our CouchDB follower wrapper, which will ensure you don’t miss any documents even if you process them asynchronously.
normalizer: https://github.com/npm/normalize-registry-metadata
Finally, we also provide a normalizer, so that you can clean up the data you receive, and implement the changes from the changes feed.
We will never stop making replicating public packages utterly trivial. If anything, we’ll keep making it easier.
We believe these tools should minimize any disruption from our transition to HTTPS — but of course there are edge cases! If you experience difficulty, we want to hear about it and help you out. As always, don’t be shy to reach out: support@npmjs.com.
Happy replicating!
Last week, npm@2.15.1
(npm LTS) and npm@3.8.3
were released to latest
. Among other improvements, these fix a vulnerability that could cause the unintentional leakage of bearer tokens.
Here are details on this vulnerability and how it affects you.
An up to date npm is the most secure npm. Update npm to get this patch, as well as other patches:
npm install npm@latest -g
If you believe that your bearer token may have been leaked, invalidate your current npm bearer tokens and rerun npm login
to generate new tokens. Keep in mind that this may cause continuous integration builds in services like Travis to break, in which case you’ll need to update the tokens in your CI server’s configuration.
Since 2014, npm’s registry has used HTTP bearer tokens to authenticate requests from the npm’s command-line interface. A design flaw meant that the CLI was sending these bearer tokens with every request made by logged-in users, regardless of the destination of their request. (The bearers only should have been included for requests made against a registry or registries used for the current install.)
An attacker could exploit this flaw by setting up an HTTP server that could collect authentication information, then use this authentication information to impersonate the users whose tokens they collected. This impersonation would allow them to do anything the compromised users could do, including publishing new versions of packages.
With the fixes we’ve released, the CLI will only send bearer tokens with requests made against a registry.
Maybe.
npm’s CLI team believes that the fix won’t break any existing registry setups. Due to the large number of registry software suites out in the wild, though, it’s possible our change will be breaking in some cases.
If so, please file an issue describing the software you’re using and how it broke. Our team will work with you to mitigate the breakage.
Thanks to Mitar, Will White & the team at Mapbox, Max Motovilov, and James Taylor for reporting this vulnerability to npm. You can learn more about npm’s security policy on our security page.
-/-
Node.js has also posted about this disclosure. You can read that here.