Thank you all so much for the excellent questions!
Hi Silverio, Thanks for hosting this AMA. :)
What's your opinion on Yarn and do you think npm 5 is better than it?
First off thx! HUGE fan of the work produced by the humans of npm Inc. I use your product(s) every day and couldn't operate our business without npm and private modules. I was super lucky to get some time with Kat and they shared a bit about
cipm with me. (It's awesome.) So I have two questions!
0.) Where does cipm fit in the npm story in teh coming year? 1.) What is being planned for the esmodules …situation… that tc39 opted our ecosystem into?
Thx again! 🙏💖
0) aaaaah I want to ship cipm so badly! This scratches an itch we have internally that is so very itchy.
A brief explainer:
npm ci is an installation tool design for CI and deployment environments. If you have a
package-lock.json file in your project,
cipm will install what is described in that file. It will ignore what's in package.json and make no attempt to respect what's in node_modules. It merely blasts the desired installation tree to disk as fast as it can. With a warm cache, this results in installation times on the order of milliseconds per package. (The only speed improvements we are aware of on this, right now, come from un-gzipping in advance. Or writing it in C or something to memory-map the file. The next wave of perf improvements will come in major algorithmic changes in the API that get the cache hot in the first place, I think.)
Kat is working right now on integrating cipm inside of the main npm tool. I think we also need to get auth working for it, so there's some work yet before we get it out. But I'm hoping to ship it this year. This is sort of the culmination of plan that team has had for a while now of upending the model of how the npm cli does its work-- calculate a tree, record the calculation, trust the calculation, sync.
After we ship that, we have The Problem of ES Modules. We already have some internal experimentation on what the hell to do about this mess. I have John-David Dalton's esm on my exploration list. I think it might be a bridge forward.
Some handwavey statements: The server side of JS is largely irrelevant; the browser community is much larger & more actively creative. (Note that I say this even though I personally am a server-side js user, not a browser developer.) In 2 years, the majority of npm's registry will be ES6 modules. Nobody will remember this transitional pain.
What's the best part about working at npm? What do you look for before hiring an engineer?
Absolute #1 with a bullet is how much npm's users love it. THIS IS THE BEST THING EVER.
It is so much fun to work for a company that makes something its users love. Indifference is so painful. Hostility is at least somebody caring! But when you can get this experience in your career, cherish it.
What do I look for when hiring an engineer? Oh, wow, that's a deep question.
The ability to work with other human beings is #1. Are they empathetic to their users and to the people around them? Do they enjoy learning? How are they at communicating with their teammates? How do they handle disagreeing with somebody?
Can they work on a team? would be the summary question, I think. This is what distinguishes good programmers from great ones.
I take it as given that engineers I'm interviewing are smart people and can write software. This is the advantage of hiring at npm: I can safely assume that! This is such a surprising and wonderful experience that I've never had before at a company. (Downside: every time we hire I have to turn down amazing candidates who absolutely could be wonderful with us, but I can only pick one. This is painful.)
Cheese on top of meat vs below the meat!
What do you and npm support? 🤨
I put this one to a vote inside npm: 100% of npm prefers Apple's burger to Google's. HOWEVER this does not end the controversy. Nobody was entirely happy with Apple's burger and the entire company then spent the next half hour arguing about how to fix Apple's burger. Everybody felt the lettuce belonged on top, guarding the bread from the tomato. The question was then where the tomato goes. Does it go right next to the meat? That's the best way to season the meat with tomato-ness. But the cheese needs to be on the meat to melt properly, so OMG the argument continued.
Also I will note that this one question brought everything inside the company to a halt. This is not the first time we've had a food-related internal schism, however. Don't ask any of us what you call the pastry that made with flaky croissant dough and chocolate.
How does npm make the build vs buy (or use existing oss tech) decision?
Oh hey this is a fun question that I am never sure we have the right answer to.
The first thing to observe is that npm is absolutely stuffed to the gills with people who, over and over, decided they wanted to write their own whatever instead of going with the one that existed before. We were founded by somebody who led the node project during its early, obscure hipster days, who decided that the world needed another package manager. Early on we hired people who had experience with node when it was still edgy and unknown, so we have a lot of early-adopter questing minds among us. Then there's Ben Coe, who adopts projects like yargs and istanbul because they were mournful puppies who needed an owner.
So we're biased toward invention.
I like inventing things too. But because my team would invent all day every day, I have to put on the "let's use something not invented here" hat more often than I personally want to wear it.
No inventing our own databases! (Okay, that one's easy because we do not have the team to do that.)
No inventing our own testing framework! Well, um, we have node tap's inventor. And nyc's inventor. And istanbul's maintainer. Um.
No inventing our own http framework! We're using restify! Well, um, we invented one we'll be open-sourcing soon.
No inventing our own sql library! We're using knex! Well, um, we invented one we've already open-sourced.
We haven't written a promises library! We just use bluebird like everybody else.
Is there any plan to provide a similar feature like yarn workspaces or improve npm link for better monorepo develop experience?
The short answer here is yes. The longer answer is that we're still trying to figure out what to do, so I don't think we'll have anything to release soon. We're actively discussing what to do about workspaces inside monorepos. Some of us who work on very large OSS projects (like Ben Coe working on Istanbul) really like the lerna workflow for pulling together a lot of small modules.
You can expect improvements to
npm link soon, however. I know that Rebecca Turner, our cli project lead, has had "make npm link work better" high on her to-do list for a couple of releases now, because npm 5 broke some common workflows there.
I SAVED THE BEST QUESTION FOR LAST.
We had an internal text file that listed a handful of npm expansions. It was early npm employee Zeke's idea to turn it into a package we could take PRs on and integrate with our website.
I asked the company for their favorite expansions, and here are their answers:
- no problem, meatbag (many of us love this one best: me, @isntitvacant, @maybekatz)
- now paging Maciej (Maciej is our late-night operations expert)
- no prescribed meaning (@maciejmalecki is very meta)
- Never Punch Manticores (@soldair thinks this is good advice)
- nonchalantly performs magic (@aeflash is our magician)
- nuǝW pǝuoᴉʇᴉsoԀ ʎlǝʌᴉʇɐƃǝN (@seldo thinks this is awesome)
For a while all of the posts we made to npm's blog were legit npm expansions :)
Also we all participated in the pumpkin expansions easter egg this Halloween. Did you notice it?
What are you favorite tech or non-tech practices on your team?
We are a test-mad company and this is kinda great. Though I have seen the seamy underbelly of bad tests (tests that freeze your implementation not your interface make for far more work than they ever save). But wow, the testing culture here is strong and it makes for a lot of confident deploys.
How do you approach planning new projects? In particular product work vs internal business requests. And the how do you sell these projects to your team?
I wrote a bit about this in another answer, but really it's a mess and I am never sure if I'm doing it right. We can't do everything we want to do. I can list a dozen projects off the top of my head that I know we'll need to work on eventually that we can't imagine tackling today. The paradox of hiring is this is always true, no matter how many people you have on your team. The more you hire, the more your imagination expands. The scope of work you can conceive expands.
So it becomes a matter of balancing the urgent against the important.
I rarely try to sell projects. If you're doing the right project, and the information that led you to make that decision is shared, usually people can understand the decision.
When something's urgent, my team is usually the people who tell me about it, and I don't need to sell it to them. Or we've messed up by allowing a known tire fire to flare up into a user-visible problem. Sometimes I propose work and I get a huge wave of resistance-- this is a sign that I shouldn't even try to sell it because I'm just wrong. When the evidence is in favor of infrastructure work, the case is easy to make. (I'd bet that with infra work I'm more often telling my team to put it off than trying to sell them on doing it. Ugh, what a fate for a responsible engineer.)
How does npm decide which features to build next?
Isaac, our CEO, is in charge of product design and direction at npm, so a lot of big projects start with him saying "this is what we need to build next!" It's then my job, as CTO, to figure out how to execute on that-- how to schedule it, how long it'll take (roughly), what the engineering dependencies are, who will need to work on it, and so on.
All of our scaling and maintenance projects come from me, because keeping the npm registry up & expanding under the load is a huge part of my job. I instigate a lot of forward-looking projects, aimed at handling next year's load, based on what I see happening with our current systems.
I also do a lot of listening to what my team tells me they think needs to be worked on next. This ranges from "hey our metrics workers are making too many connections and not scaling up as a result; we should fix it" to "this code base is really gnarly to work on and we know the problem space better now than we did two years ago; let's rewrite".
And still other projects happen because somebody says "darn it we should do this". Two-factor auth as a project happened because I did an implementation spike on my own, realized it was easy, and said it was time. We hid email addresses on our web site recently because somebody else said, hey! I know how we could do this without putting a huge burden on our support team, and we did it without a lot of planning or ceremony.
I do a lot of juggling of inputs like that to figure out what we can do with the humans we have, what needs to happen now, and what can wait. We are pretty flexible about what we do and we don't have a lot of rigid planning. I think we surf on the right side of flexible vs chaotic, anyway.
What is the best part of being part of NPM?
I answered a very similar question above! The short answer is that working on a product that its users love is one of the best experiences you can have in a career, and I have it at npm. The other great thing is a little self-centered, but I'll admit to it: Sometimes I look at the state of modern web development, about the explosion of frameworks and transpilers and complex things happening in the browser, and I think I helped make this happen. By making npm's registry stable and fast, I made it into a tool that everybody pushing web development forward today just takes for granted. I made a platform that they use without thinking to build things that I cannot possibly have imagined.
This is the thrill you get when you work on a platform. The people who invented node.js also deserve to enjoy this thrill. A decade from now, somebody else will get to smile to themselves knowing they made the web of 2027 awesome.
This is cool.
How npm Inc. is different from other company's ?
For me, the #1 difference is that I get to help run the company, as one of the executive team. This isn't a shallow answer, because this means I experience the difference as one of responsibility and one of design.
I am responsible for npm's success in a way I've never been responsible for another company's success! This means that I think about it differently than I've thought about any previous employer, where I cared but wasn't in control. But where this turns into a difference other people might experience is that I have used that control to try to build an engineering organization that is the best one I can imagine. I want to have a company that I'd love working for, an engineering team where I'd do my best work, taking ideas from every great team I've been part of in the past. (And in some cases, avoiding some things that teams I didn't enjoy being on have done.) You can hear some of my thinking about this in some of my public speaking, and a whole lot in my twitter feed when I get started on How To Engineer.
Here are some specifics:
We really are serious about work-life balance. We expect 35-40 hours a week of quality attention from everybody, and no more. We track vacation time, unlike a lot of start ups, because we think of that time as a mandatory minimum! Everybody is expected to use up all their vacation, every year. We do this because we've learned over our careers that people do their best work when they're rested and happy, and when they have lives that include things other than work.
I also like having the space to be wrong. Not in the sense of failing or breaking things, but in the sense of learning. We try to talk problems through before we act on solving them, and the bigger the problem is the more discussion we give it. We can ask questions, not know how to do something, have to go do research, or even be outright wrong about how to fix a problem, and this is okay. The goal is to be right in the end, not to be right from the start. My favorite thing to do is to propose a bad solution to a problem just to get people jump-started at finding the right solution.
This is one of the cores of the "emotional safety" I talk about in my pinned tweet: https://twitter.com/ceejbot/status/761569569802551300
If you have to constantly prove yourself to a hostile set of colleagues, how can you ever learn anything? You can't. This is no good! I want a team that feels safe with each other and can disagree constructively.
I'm also really proud of our engineering staff meeting, which is not about status or projects or anything like that. Instead we take a discussion topic, new every week, that has something to do with the craft of software engineering, broadly interpreted. We talk about it and share what we know. Topics have been things like:
- how & why do you write tests?
- how do you cope with burnout?
- why and how would you ever tune a kernel?
- what would you lose if you wrote an html 3.2 server-side only web app today in 2017?
- Conway's Law
- "worse is better"
- What the heck is functional reactive programming anyway?
The goal here is to share wisdom & experience & level each other up. It is my favorite meeting of the week.
Isaac Schlueter once called our engineering team the most thoughtful team he's seen, and I think that is an amazing compliment. That's exactly what I was hoping to build.
Now these are aspirations, and sometimes I know we fall short of them. But we try, and we fix things when we realize we've gotten them wrong, and this puts us ahead of so many other companies I've worked for.
In light of your answer on work/life balance, can you share how npm deals with being on call and minor and major emergencies, both in the past and how you may be improving things in the future? E.g. if the person who was responsible for a feature goes on holiday and it breaks, what would happen?
I feel very strongly that being on call should be boring. We have very little tolerance for repeated alerts, or for alerts that aren't meaningful. If we see something happening a lot, we fix it or make plans to fix it.
Emergencies like, say, a database write primary going away on AWS, are things we can't schedule, but we can plan for them so coping with them is as unstressful as we can make it. We can also rehearse by doing replacements at unstressful moments. But the reality is that if everything explodes at 4am on a Tuesday morning when Europe is trying to get through its workday, I will be paged, and I will get out of bed, and I will join my team in fixing things. Not the whole team, because we'll keep as many people rested as we can. And anybody who's up with me fixing things will take the rest of the day off.
I once witnessed first-hand a SAN wipe disaster that was made an order of magnitude worse by sleep-deprived management making well-meaning but stupid decisions because they were sleep-deprived. I have tried to set up processes that will help us avoid this trap at npm.
I can't imagine a situation where we let somebody design a feature and nobody else knew how it worked. Code review ftw! Also design review up front. The reality is that some people are always more expert about some things than others, but we try to keep the "bus factor" for any single thing as high as possible. If you spread the knowledge, you spread the burden.
No Heros, Please.
How does npm compare with yarn, I heard that yarn flattens out the files and npm doesn't?
So.... yarn can optionally flatten files, and this is sometimes what you want, particularly if you are targeting the browser. The default npm cli cannot do that, however, because it is designed to work with node module loader and solve node's specific problems, which are quite different. The intent of npm is to avoid dependency hell by dodging it, through the specifics of that loader-- you can have more than one copy of lodash! Your code gets the one that you specifically asked for!
In other words, I'm a-okay with a world where there are lots of tools that lay out packages on disk in lots of different ways, because the problems they're solving are different.
I made those numbers up just now. Do not quote me. They're all good language, Bront.
Is the AMA over?
It already does. We've got fonts, C libraries, and Go code in the registry already.
Also, how would we know? If it has a
package.json, you can publish it. There are no barriers to publishing whatever you can usefully bundle into a small-ish tarball with a semver tag on it.
We don't mind you publishing C libraries, but we probably won't go out of our way to make the experience as good as what we want it to be for web things.
CJ. Thanks, I have been using npm for years and it's an incredible tool.
I have been following npm for a while on Twitter and the culture seems amazing. Do you have any details to share on npm's philosophy that have led you to find and keep great people?
Start from the beginning with a solid foundation, in my opinion. Your initial team sets the tone for how you mean to go on from there. The personalities of your first ten people will set the personality of your company, whether you realize it or not, whether you mean it to or not. I think this is just how groups of people work, kind of a rule of thumb for how organizations grow.
npm hired me early, and I have been in the Valley for a long time and seen both successes and failures. I've been lucky to be at the sidelines of some fantastic engineering teams. I was nobody and I did nothing much at General Magic, but the people I watched at work there went on to do gigantic things. But I've also seen stupid teams do stupid things, and I managed to learn from them too. So my contribution to npm has been to help steer the engineering side of the company in the direction I think is best-- which has upsides and downsides. I think there are perfectly wonderful people who would have a bad time at npm, and whom I would not judge for having a bad time. (For instance, if you don't thrive on the written word as a way to communicate with your colleagues, you're not going to be happy with the writing-mad culture at npm.)
First and foremost, I hold that the humans who work on the software are more important than the software! We work to live; we do not live to work. The best work is done by well-rounded people who have interesting lives, who come to the office refreshed, who treat the humans around the well, who are in relaxed and happy places.
I try to tell my colleagues how much I enjoy working with them every so often. (Dear colleagues, if I haven't said it to you recently, please hear it now. I enjoy working with you yes YOU.) Give this a shot with a colleague you enjoy working with! Tell them! Tell them why!
I love writing software. I remain amazed and delighted that I can earn a living doing this thing that is so much fun. I want it to stay fun for me and for everybody around me. This simple desire drives a lot of what I do. I hope it rubs off.
What do you think the common mistakes that developers should avoid?
Premature optimization is the root of all evil, attributed to Tony Hoare (the person who invented quicksort) and popularized by Donald Knuth.
Premature optimization is a trap we're prone to fall into just because of who we are as programmers. I'd characterize it as doing anything before you know you need to do it.
- pulling things apart into modules before you need to
- generalizing code before you need to
- making something really fast and hard to understand before you need to
- refactoring before you have a reason to
- acting before you have data to justify the action
Be tidy, yes. Don't use this as an excuse not to lift up your loop invariants, for instance, because that's just not being sloppy. But definitely don't generalize something until you have a reason to generalize. (The rule of three is great here.)
Hi Silverio! Your product is amazing, I have to say it. And it becomes even better with NPM 5.
A second is an easy one: how is the remote job policy there? :)
Most of what npm's engineering team does is work on the registry that manages all the packages and the API that lets the whole world get at that data! Most of what I do as CTO is think about scaling that API! So I am delighted to have more than one client for it. In particular, I think yarn is a solution to a very specific problem-- how Facebook development needs to work-- that is not appropriate for most node users, who are the audience for npm's default cli. Yarn has freedom to do some things differently that we could never do in the default client, which has an awful lot of installed base it must not break.
What I'd like to see is many clients, each of which is free to solve problems unrelated to the core use case. What if you want to do a pessimistic semver resolution algorithm that chooses the lowest version number that matches constraints? I think you should do that if you want to and I'm happy to support it with a high-performance API.
We want to make it really easy for people to write more clients. One of the things our own CLI team has done in the last year is to break out the core pieces of our tool into reusable components that anybody writing an npm client might find handy. In particular, check out these libraries:
- pacote - programmatic npm package and metadata downloader
- cacache - a content-addressable http cache
- make-fetch-happen - fetch implementation with a pluggable cache
- npm-profile - a client for npm's profile API
Check 'em out, and maybe try writing a client!
In response to your second question, about remote job policy: We have an office in Oakland, CA, but about half my engineering team is not in Oakland. We've got people in Poland, the UK, New York, Seattle, Portland, and the desert outside of Los Angeles. So I think we have a welcoming remote policy!