PayPal’s Matthew Mengerink: Open Source Essential to Success

PayPal transacts more than US$1,500 every second of every day, with millions of people around the world relying on the robustness of its system.

It comes as a surprise to many people that PayPal runs such a large financial services company on an open source platform, but that’s precisely how we’re able to deal with the two competing demands our business model places on us: security and innovation.

The economic, operational, development and security advantages of open source and Linux put us in the perfect position to both grow and innovate in a safe and secure manner. Here are four reasons why we love our open source system — and four tips for you if you’re thinking of making the switch.

Economics That Can’t Be Beat

Our system environment, built entirely in-house by PayPal developers, consists of thousands of Linux-based, single-rack Unity servers, which host our Web-presentation layer, middleware and user interface.

If we were dealing with hefty mainframe operations, our upgrade costs could be rather eye-watering — $2 million or $3 million at a time. To be able to scale effectively, we’d have to purchase bigger and bigger chunks. On the contrary, our upgrade path is far more affordable, in bite-sized chunks, which makes it far easier for us to manage our business. For any given upgrade, we might be looking at 10 one-thousand-dollar servers that we pop into the mid-tier of the platform. Imagine if your tiny experiment creates a need for the next $2 million capacity increase. Companies must have infrastructure that makes unproven innovation affordable. Cost effective experimentation is key!

Tip: Choose the right platform so that you can afford innovation.

A Developer’s Playground

The benefits our developers get from being able to work in the same platform as the production environment cannot be overstated. Open source environments are not as expensive to recreate as proprietary ones, and being able to work in a live environment enhances efficacy by an order of magnitude. When developers are testing things in the same environment that we actually have in production, we’re far more likely to get a consistent and expected result.

When site problems occur, our developers know exactly where to look to help find a problem. It’s also cost-effective, as we can duplicate a live site in the lab and not incur high licensing charges. This consistency lets PayPal coders write new versions of PayPal production applications, which can be switched on with minimal disruptions.

More importantly than all of this is our developers love Linux. People are far more innovative and engaged when they work on what they love.

Tip: The best problems are those avoided. Avoid unforeseeable errors with uniform environments between development and production.

Increased Flexibility

The generic Linux Lego-block servers that make up our Web tier can be easily shifted around for a variety of tasks, so we can make big shifts in resource allocation whenever we need it. For example, when we need to run our daily batch processing for reconciling payments — a task typically done by mainframes — it’s spread across the middle-tier Linux servers in the datacenter. It’s simply a matter of allocating a higher number of the servers to running batch processes, and we can crunch through it all in a couple of hours.

In closed systems with one large impenetrable box, breakages rarely occur, but when they do they completely disable the system. In our model, the multiple nodes render individual breakages almost irrelevant. This distributed, highly redundant system only works with the cost model offered by Linux.

Tip: When redundancy is inexpensive, you can invest in lots of it.

Safety First

All the benefits we receive from our Linux platform would be irrelevant if they couldn’t coexist with the highest levels of security. Luckily, our open source model allows for a highly repetitive and secure system. Tailored security policies mean there is a built-in layer of mistrust among machines, with all of them operating as if they were in an un-trusted network.

The way we mold the Linux kernel and other open source code we use also helps to make the overall system more secure. Linux servers in our datacenter run Red Hat kernels with custom tweaks to add even more security.

Tip: Open source is not “open” to intruders. It is easy to make extremely secure.

Daily Imperatives

So far, the mix of distributed Linux and open source software and rapid application development of open source code have been an enormous success at PayPal. In a world in which both innovation and security are daily imperatives, Linux provides the technology backbone that has allowed us to enter so many new markets and continue to innovate in existing ones with unparalleled flexibility and responsiveness.

Also, at the end of the day, our technologists are proud that we use it.