Hello Hashnode folks! We're super excited to be here for this AMA session 😀
The host team is a broad representation of engineering, professional services and management at PayPal, so we can tackle pretty much any question you throw at us.
We're more than happy to discuss our engineering practices, open source work, mistakes from the past and lessons learned, corporate culture at PayPal, "day-in-the-life" questions, our diversity and inclusivity initiatives and more!
Only a couple of topics are off-limits (Legal said so...). We can't give any forward looking statements on company performance, and obviously we can't give away "secret sauce" information.
Looking forward to answering everyone's questions!
-The PayPal Team.
Hi PayPal Devs! It's amazing to see you all going live publicly for the first time (I guess) 😀
How does PayPal write test cases? I mean one single mistake and you can end up having huge loss! 👀
Without going into the details, if you can just explain briefly what goes under the hood, it will be really helpful.
Hello. Thanks for your question. You are right. We do invest considerable time of developers into testing. For front tier we do not have dedicated testing teams. Developers are expected to write and maintain tests for their code. For node.js, we use some awesome open sourced modules on npm like jest, tape, mocha, jasmine(and many many more). Teams have their own freedom to explore on this front. For functional testing in browser, we have our own opensourced framework nemo.js (https://github.com/paypal/nemo). I've seen some teams use Phantom.js.
We do have full fledged Quality Assurance teams for testing Service infrastructures and full end-to-end testing of a product. We have specific environment setups for pre-build developer checks, Build check before release, and we have a Sandbox environment that mimics Production code to test without having to move money from our account and buying stuff online in the name of testing ;) :p
Hello, Paypal devs. It's nice to see you here. I welcome you to the Hashnode DEV community.
Can you please give provide little details about PayPal's tech stack?
Thanks for AMA.
Core Paypal's stack is potentially slightly different than the subsidiaries and acquired companies. At Paypal, for the web-tier we completely moved to node.js in the past 3-4 years. In the client, we have quite some freedom to explore frameworks of our choice (Angular, React, ... even the old timer Backbone in a few cases). For mobile native, the Paypal App uses swift on iOS and Java-Android. For one of our checkout SDKs (which is an effort I am leading currently), we have been exploring using react-native. In the mid-tier we mostly write Java services. A few orchestration services use node.js as well. There are even a few legacy systems using C++. For analytics, we use PIG for mining flat data alongside Hadoop and SQL scripts for db sourced data.
For a big company, we take pride in moving fast to explore newer technologies. At the same time, as a big company, it takes time to deprecate older stacks due to the humongous effort involved :))
What’s the code review process in PayPal like?
We use GitHub enterprise on my team and we have pull requests for every change. Two developers who are familiar with the code are required to review and approve pull requests. Depending on the size and complexity of the pull request, we may ask more folks to look at it and some may even pull it down to their local machine to manually test the experience. For UI changes we'll often include a before/after screenshot or gif to make the review process easier :)
Our team's code review process is mostly similar to Kent Dodds. We take advantage of GitHub's Pull Request Templates to ensure our developers fill in the right details of their change. These vary across projects, but generally include:
- Verification (screenshots, or steps the submitter did to reproduce the issue)
- Relevant links
- Required reviewers
Naturally some changes don't require this much paperwork, and only number 1, 2, are 6 mandatory. For simple or focused projects, we may require only one code reviewer.
Thank you PayPal folks for doing this AMA. I have three questions.
My first question is very fanboyish. I am a huge fan of Peter Theil and would love to know if any of you have had a chance to interact with him. I know he left the company long back, but does he ever visit the PayPal offices still?
What's your favorite thing about working at PayPal?
If I were to choose a payment solution today for my app, should I use PayPal or Braintree? How are the two products differents given that both are from the same team?
Hi Siddarthan, I'm happy to answer your questions here :)
- I joined PayPal 2 years ago right after graduating from college, at the time PayPal and eBay were officially split. I have not met or heard Peter Thiel visited our San Jose campus.
- My favorite is our innovation lab
- It depends on your company's requirement. I am working at our Professional Service organization and my job is helping developers/merchants like you to find the right products. Braintree has a better developer-friendly solution in credit card transactions and it supports PayPal payment. PayPal has a well-known brand and acts like a digital wallet.
Honestly, the most I hear about the PayPal founders is when they get mentioned in financial articles -- e.g., "Today, PayPal founder Elon Musk announced that he's building a hyperloop to Alpha Centauri in the next 5 years...". But honestly, I've never seen them on campus or even heard them talked about that often. (Granted, I work from Omaha most of the time...but I think the same is true of most of our San Jose folks.)
Oh man, so hard to pin it down to one thing. First one: my managers. They're awesome people, and they're always supportive of everything I do. Anytime I run into an issue, I know they've got my back. Second: sabbaticals. Every 5 years, PayPal gives you a 4-week paid sabbatical. (Granted, I stayed home for my first one...but there's a lot of people that come back with some amazing stories about what they did.) Third one: In San Jose, soda is free. (In Omaha, 20oz bottles are $0.50, 12oz cans are $0.25.)
It's hard for me to decide. On the one hand, a lot of effort nowadays is going towards Braintree. On the other hand, I know that the PayPal stack has a lot of awesome solutions as well (hint: we have a lot of solutions that we don't advertise publicly). Bottom line, I'd say this: if you want to be able to do credit cards directly on your site, go with Braintree. They have some awesome JS code that automatically sends your buyers' CC info directly to them, so that you don't have to worry about it passing through your system. If you want the trust that comes with the PayPal checkout (or you don't care about taking credit cards directly on your site), go with PayPal.
I don't have much of an opinion on questions 1 or 3, but I do have a favorite thing about working at PayPal: Autonomy. PayPal trusts its employees to make the right call. We do have policies and processes, but we have a lot of freedom at PayPal and that's important to me :)
Aw, these are great questions!
I've also never interacted with Peter Theil, but Dan, our CEO, is a super great guy and he takes time out of his busy day to talk with recent college grad nobodies like me :)
My favorite things: the culture and the innovation lab. The culture because you can work from home whenever you feel like it, take long lunches if you've had a rough day, or work in the lounge. As long as you get your stuff done on time and your code is up to snuff, you can do whatever you need to do to recharge and keep your mind sharp. The innovation lab is also amazing. It's all about out-of-the-box thinking and daring to dream bigger and crazier ideas... and it's full of nerds who are learning everything from cryptocurrencies to VR. And the lab really expands your mind and they host great hackathons and events.
Is any part of PayPal’s tech outsourced?
How helpful is Node.js for you guys? Heard that Paypal was among the first companies to adapt it.
Hey Shahid, Thanks for your interest in Paypal.
I think the answer to the question below might answer your's as well. https://hashnode.com/ama/with-paypal-developers-cj793fn2q007usrwu2qcazocb#cj7cdgoyr0084nrwta2lpplb1
How often does PayPal refactor the code?
Great question! To refactor is to apply what you've learned today for the problems of both today and tomorrow. Since we write lots of code and are always learning, we constantly take a step back, clean up, write (and rewrite) frameworks, and throw away huge portions of our tech stack for better stuff. Naturally we have to produce functionality too, so as with any software team, it is a balancing act to know when to stop and revisit some code, or produce new features or bug fixes. Since PayPal's codebase is so huge, you could confidently say that a significant portion of it is being refactored at any given time.
What legacy have the likes of Elon Musk and Peter Thiel left that the team still follows?
This is actually quite a common question from friends and relatives alike! Honestly, those two, as well as others from the PayPal Mafia, have been absent from the company for many years. Speaking to the tech stack, vestiges include ancient code, names of frameworks, databases, files, etc. Most of this has since been gutted and replaced with new platforms that fit the changing needs of the business.
What made PayPal engineers switch to Node.js? In other words what problems did you guys face with Java stack which was eventually solved with Node.js?
Hi Dylan, I was part of the team that started the Node @ PayPal transformation. I arrived towards the end of the prototyping stage, so I was heavily involved in the evangelism and educational aspects of the transformation.
Long story short, there's nothing inherently wrong with Java (other than being verbose :) ), however, we had painted ourselves into a corner by building very cumbersome and inflexible frameworks on top of Java.
The front-end development cycle was very slow because of it, so we saw Node as a way to hit the reset button, and start fresh with a framework that would incorporate all the lessons learned from the Java days. (Mostly, what not to do with a framework)
This resulted in the rebirth of our open source program, starting with http://krakenjs.com, our nodeJS framework,
Here's a more in-depth look at the history of our change that I presented at jQuery Conf a couple of years ago. https://www.youtube.com/watch?v=qJVk6KiOEBA
All that said, we still do plenty of work in Java, mostly for midtier / backend services. Our newer frameworks use Spring3/Java8, and are much easier to use than those in the past.
Hey Dylan, Thanks for your question. I used to be a part of the core node.js team as well in the past.
To reiterate on what Lenny said
We wanted to move away from building heavy in-house infrastructure around web app development and try out newer technologies. Earlier Java was used heavily in web-tier. Today, Node.js predominately dominates the web-tier. While we have small mid-tier orchestration services in node.js, Java is still used predominantly in the mid-tier.
With node.js we have been able to extend expertise of our webdevs to full stack from just being on the client side.
Node.js also enabled us to experiment to much more savvier front end and native technologies.
NPM ecosystem was a big win to enable tearing down silos within teams and share work across :))
Here is a talk I gave at Intuit as part of Kraken team which focusses on our decision to move to node :) https://www.youtube.com/watch?v=YNtY5ObDJ8c So far it has been a big win...
I always had a question when building a software. How do you bring reliability in projects which are very critical like money transaction or things like spacecrafts? There is no(or very less) room to make mistakes. So how do you at PayPal do fault tolerance and ensure that system is consistent no matter what happens.
Mission critical systems generally have more testing than others, and PayPal is no different. In addition, PayPal has built-in redundancy and fault-tolerance to prevent bad code, a bad machine, network connection, router, data center, etc. from harming the service we provide to our customers. We have many layers of testing and alerting, so that if something goes wrong, we know almost immediately. If something does make it past all those layers, we have a Site Reliability Engineering (SRE) team and bug tracking system to ensure any customer issue gets the highest attention possible.
Hello Paypal Devs . With no disrespect or malice meant at all, a large percentage of the development community would agree that Paypal's documentation just makes working with Paypal API close to impossible. It's such a pain. Stripe ( Forgive me mentioning this ) dominates the market for payment gateway integrations right now, not because they have better programmers, or better software, but the ease of integration of their API service, and how awesome and organized their documentation is .
Yallz are devs and you must have heard developers hate soo much on the paypal api and documentation.
Why has this not changed over the years ? Is there any particular reason why this has not been updated over the years ? or why this complain has not been taken into consideration ?
Thank you so much , hope to get a satisfying response.
Hi Kati, I would love to answer this question!
First off, I want to acknowledge that the pain our dev community faces is real. Even though our product offerings and documentation have gotten much better over the past couple of years, the way that we present it to the world is... not optimal.
Compared to Stripe, we have a very broad spectrum of payment solutions. This is great, because we can solve pretty much any scenario related to moving money and collecting payments, but it can get overwhelming for somebody who is going to integrate with PayPal for the first time.
Now, to the important question: What are we doing about it?
As of two months ago, a new internal organization was formed at PayPal: The Merchant Integration Platform.
We have a single mission: To ensure that merchants/developers are able to find the right product for their needs (with adequate documentation, tutorials and examples), can easily integrate/test against our sandbox environment, and have a smooth move to production.
I've been at PayPal for nearly seven years (Fun fact: My first role was tech support!), and for the first time, a team has complete end-to-end ownership of the developer experience, as well as a single-purpose charter.
As the leader of this org, my only responsibility is to make sure that your integration experience is the best in the world.
We're taking multiple steps to make sure that everyone's needs are met. We'll be revising our content strategy, to make sure we deliver the right information to the right audience (As opposed to throwing the kitchen sink at you), we'll be updating our developer portal, and we'll be making our sandbox more robust. Also, I'm very excited to be restarting our developer outreach program!
(Shameless plug: If you're reading this, and you think it's an interesting problem to solve, I would love to hire you! https://jobsearch.paypal-corp.com/en-US/job/developer-relations-lead/J3J1NV69F3NFBQB62DL )
For anyone who would like to learn more, please reach out to me directly @LennyMarkus
@kent: How is Glamorous different to Styled-Components?
Hi Walter 👋
It's a pretty common question and the answer is a bit nuanced, but at a high level glamorous uses objects for authoring the CSS and styled-components uses tagged template literals (strings) for authoring the CSS.
You can read about why I created glamorous in the intro blog post. Basically it came down to performance, size, preference, and the ability to apply RTL conversion on the CSS. Also, with styled-components you have to give a name to every element you want to style (by creating a component) whereas with glamorous there are the built-in components which allow you to add the styles directly to the elements and not have to give them a name (like
Container). I also think that the glamorous APIs compose a little better than styled-components.
Good luck to you!
How do you handle burnouts? Would love to know what you guys have to say about this.
PayPal is a company that cares deeply about Work-Life balance, and personal wellness.
Occasionally, there will be a project or a site issue that requires you to burn the midnight oil, but for the majority of us, those are few and far between.
From what I've observed (and have caught myself doing on occasion), most cases of burnout are self-inflicted. When people bite off more than they can chew, they need to be able to recognize and correct it. Failing that, it falls to the managers to point it out, and ensure that work is spread evenly.
Moreover, we're on the laid-back side, and enjoy benefits such as untracked vacation days, and the ability to work from home on occasion. It's very important to point out that our culture does not penalize you for asking for help, nor for enjoying these benefits.
Sometimes burnout simply means boredom with your current role. Our culture also respects people moving laterally to new roles. We have internal tooling for job searches, and managers tend to be supportive of these moves.
We'd rather have a good employee remain here, and be motivated, even if that implies moving to a new team.
My managers and co-workers(@tituswoo is one of my favorites) are my great resources that helped me go through my burn out most of the times. I talked through my difficulties with them, and they gave me different ways to perceive things. I like workout a lot and it definitely helps to release the stress. Our San Jose office has a great indoor gym, soccer field, and tennis court. I try to use as many times as I can :D Also, it's very important to give yourself some unplugged times. We have a great vacation policy, so as long as it's the reasonable/no conflict with works, my manager always give me enough vacation times
Just from architecture perspective, if we have multiple microservices up and running. Is it better to have inter service communication via Queues or via direct API-endpoints.
As digital currency like BitCoin is taking over the world; What is PayPal doing next?
Which primary DB do you use with Node.js and how do you handle DB transactions? Also, do you use any NoSQL databases as a part of your stack?
Pretty much everything we have runs on Oracle. We've been using them since the beginning of PayPal -- in fact, we only recently phased out one of the databases that traced its roots all the way back to Confinity. We figured out pretty early on that Oracle connections are expensive to open, so we actually wrote a thing called the Oracle Connection Cache (or OCC for short) -- it keeps a cache of open connections to our Oracle servers and speeds up DB queries quite a bit.
As far as NoSQL databases...I'm not aware of any that we use on our production systems. I think there might be some that we use for doing analytics on our back-end.
How big is the security team in PayPal and what does their day-to-day work look like?
What security software comes pre-installed on MacBooks issued by PayPal? I am running a startup and have been thinking how to protect the data on my employees' machines.
What tools/IDEs/machines do PayPal engineers use to write code?
Everyone has a fair amount of autonomy at PayPal. So you'll find our answers vary quite a bit on this subject.
When I started at PayPal, I was issued a MacBook Pro and there wasn't really any software pre-installed on it (other than security-related software). From there it's pretty much up to you.
So I use Atom, Terminal.app, and Chrome for most of my work :)
Emacs. Forever Emacs. :-)
In all seriousness, it depends on what I'm doing. Like Kent said, PayPal does allow you a fair amount of autonomy in what software you use to do your job. Lately my main tools have been Atom, Chrome, and iTerm2 -- but I also keep a copy of things like SoapUI, Eclipse, Android Developer Studio, Xcode, and XMLSpy for doing other, more specialized tasks.
First, I use a Mac :) Then I use Visual Studio Code (with a bunch of extensions) as my IDE for all development, and exclusively use the Google Chrome developer tools for all client-side stuff. For one-off huge files/logs, I use Sublime. For design I use Sketch. I also use iTerm as my terminal replacement, and can't live without the zsh shell. For general computer productivity, I use Alfred for searching through files/apps, and Better Snap Tool to give me window snapping capabilities. Oh, and Bartender is really great for hiding all the status bar icons.
Hey paypal devs.. I was a massive advocate of you guys up until stripe came along with a super easy payment api, since then I haven't really used anything else. Just want to know since they came on the scene what kind of board stroke improvements you have made to win back developers :) Sorry if its sensitive but will really help us busy devs read into more details later on and make the switch again :)
Hi Charlie, See my answer to Kati Franz
How do you test your Node.js APIs and code?
Hi there, thanks for the question! This is mostly a duplicate of the tech stack question above, which Poornima provided a pretty comprehensive answer to the unit test frameworks in use here at PayPal. However, I will also clarify that APIs are mostly written in Java, while the the front-end stack is written in Node.js. Outside of standard unit testing, there are functional tests, sandbox tests, and live tests, using a combination of off the shelf Java or Python frameworks and our own scripts.
Thanks for the AMA. :)
What challenges did you face while migrating the infra from Java to Node.js?
What are some of the problems you're facing with Node currently?
Do you guys think that blockchain technology is gonna change, how we do remittance in near future? Your personal views :D
Personally speaking, I don't think it's going to happen in the near future. Until something is really mainstream, it's a pain to get merchants to start accepting something -- and even then, some seem to hold out longer than others. I commend the handful of sites that have started accepting it, but I think that there needs to be some huge push to get the rest of the world to use it -- and I don't know that there's really any driving force that's pushing people to accept Bitcoin right now.
What are considerations and security concerns that are typically involved in building APIs for developers?
We're generally going to be looking at a lot of different factors (depending on what the API is going to do exactly), but -- speaking from experience -- I can tell you that some of the things we look at are:
- What does the API do?
- Is it going to have access to any customer data or PCI-sensitive data?
- What data is it going to expose to the outside world? (We have to pay close attention to applicable privacy laws...especially in places like the EU, where privacy is more strict than in the US)
- Are there going to be legal or compliance impacts?
- Is the application serving this API properly isolated? (e.g., it should only have as much access as it needs to function properly)
- Is the application using prepared statements for any DB queries it's doing?
Depending on the answers to those questions, we tend to do a deeper dive into what the API is doing and how it's doing it to make sure that not only is PayPal going to be safe, but our customers as well.
This is a broad question that can be answered by lots of industry standard information on API development. PayPal is a leader in this space, and is an active member in the OpenAPI Initiative. We have all the same concerns as many other API developers: building developer-friendly APIs with consistent standards, onboarding new users, authentication, app management, SDKs, documentation, sandbox, dashboards, up-time and other *ilities, partners, community engagement, and sunsetting old APIs. All of these require attention to ensure the developer experience is pleasant and meets the needs of your business.
Specifically regarding security concerns: as a financial company, we have to ensure extra effort is placed on ensuring APIs and data behind them are safe and secure behind a solid authentication mechanism. You can read more about it on our REST APIs Getting Started documentation.
What’s the best thing you like about your competitor — Stripe?
I like that they're very developer focused. I mean, heck -- they have the code right on their home page showing you how to run a credit card transaction, and it's not very long. I wish all of our products were that easy. But...there's a couple of growing pains to being a big payments company: (a) regulators look at you with more scrutiny, and (b) you really have to do everything you can to prevent fraud losses. We're handling that pretty well today. I'm not sure Stripe is there just yet.
What does it take to land a job at PayPal? I really like the contributions done by Kent C. Dodds to the OSS.
Is OSS an important factor when PayPal hires?
Hi Duane! 👋 I'm glad you like my OSS contributions :) I like doing it!
So my participation in OSS did help me a bit in getting my job (read the whole story here. But lack of activity on GitHub is not a bad mark on a candidate at all. Being able to participate in open source is a privilege that not everyone has and if we limited ourselves to only those with that privilege (and inclination) we would miss out on some really amazing developers.
When a developer has contributed to open source though that is definitely a plus because it allows interviewers to look at the work of the developer ahead of time and get an idea of skill (both in writing software and interacting with other developers). So if you can manage it, I definitely recommend getting into open source!
I hope that helps! Good luck!
are you guys practicing much of the reactive principles when architecting system, are the feelings all positive? I have worked on a few microservice based systems and i don't think there really is a way to totally isolate any direct inter-service dependancy or am i missing a trick :)
When PayPal decided to move the Web Tier to Node.js, were there any internal conflicts? Difference of opinions? If yes, how was it resolved?
Hi James, See my answer to Dylan https://hashnode.com/ama/with-paypal-developers-cj793fn2q007usrwu2qcazocb#cj7cdgoyr0084nrwta2lpplb1
I would say the biggest internal "conflict" was that eBay (Who owned PayPal at the time of the transformation), was also developing it's own Node framework. We spent quite a bit of time doing pros/cons analysis of both frameworks, trying to decide which features would be more useful.
Some of the early meetings felt a bit like the Thunderdome: Two Frameworks enter. One Framework Leaves
What does a PayPal Engineer's daily life look like (highlights of a working day)?
I think you'll get a different answer from everyone. Mine is a bit unique because I'm a full-time remote developer (not super common at PayPal).
For me, I wake up when my kids wont let me sleep anymore (7:30), I take care of home and family stuff and start work at around 9:00. I work on at platform team in my org (peer-to-peer payments mostly), so I have a pretty flexible schedule and just work on the most important thing I need to work on (my last team has 2 week sprints, standup, etc.). I have a few meetings a week. I help other folks be successful with the technology they're using, and I'm building tools for folks. I normally take about an hour or so for lunch to send my daughter off to kindergarten and put my son down for his nap, then it's back to work until around 5.
I think you will hear a slightly different story from each one of us about how the typical working day looks :) I lead a small team. We all have staggered working hours since we all have different kinds of personal commitments. But we do have core hours at office that overlaps during when we have a stand-up to sync up on our day-to-day tasks as well as any dependencies that blocks us. As a manager I try to keep my team as much out of meetings as possible, so they can have some peace at their desk to work on exciting stuff. Our product and design teams sit right by us. SO we do have regular brainstorming sessions which covers a variety of things: from what goes into an immediate release to a long term vision for the team :))
Hi there, wow, is great to have you guys here!
Below in the questions I read about your current tech stack, how do you think your future tech stack will look like in a couple of years (or maybe more)? Which technologies you are looking forward to try/implement?
Hi Devs, Just wanted to know your opinions on GraphQL
I'm a frontend dev primarily, so I've not looked at implementing GraphQL, and I've only used GraphQL a little bit on the side. I LOVE it and I'd love to be able to use it more! So eh... Those are my opinions 😅
We have a team pioneering GraphQL for one of our Paypal Checkout products. GraphQL is being used on node.js as an orchestrator (mid-tier service) for one of our mobile SDK clients. It is yet to be released, but so far consuming it has been great!! The developer on this project is not online today. But I will definitely bring to his attention, to answer your question in more detail. Cheers!!!
PayPal being a Node.js stack company with Douglas along side. What are your views towards the challenge GO? :D
I think everyone's going to have different opinions here.
I agree with Kent. Even within Paypal, you may hear differing opinions from developers. Today node.js reigns the front tier. But I have heard murmurs, GO is being experimented for some midtier services.
Thanks, no wonder why Ryan Dahl and TJ both switched to GO. But again "Change is the only Constant". Rest we all love JS, but also should embrace other technologies... :D
Do you think NoSQL is a fad and that SQL databases with relaxed schema are the way to go?
Personally, I don't think that NoSQL is a fad. It's simply a tool that solves a specific set of problems.
The real trick to them is recognizing when you're actually facing a problem that they solve. Sometimes, as developers, we get caught up in the excitement of a new technology, and as the saying goes... When you have a hammer, everything looks like a nail :)
Hi, thanks Kent for JS Air podcast. My question is do you guys use noSQL databases if yes then for what purposes.
Hi Nivesh! I don't know the answer to your question, but I just wanted to say you're welcome 😊
Hey there! Thanks for doing AMA. My question is Why is Paypal better than using an online service like WorldPay or Visa? And also, how is PayPal managing confidential data of the users?
PayPal is a digital wallet which means that we have an ecosystem built with consumers and merchants. When a merchant adds PayPal as a payment option to their site, we bring our 203 million (roughly the number at the moment) consumers to the merchant who can use the site with confidence having the trusted PayPal logos on the checkout page. We have seen an increase in overall ecommerce site conversion when we had merchants who added PayPal as a payment option with growing consumer confidence.
This consumer confidence comes from the highest level of security we provide to our customers which is also the second part of your question. PayPal has achieved PCI DSS (Payment Card Industry Data Security Standard) compliance certification under various programs and standards. We are committed to security as well as the protection and integrity of customer data.