This issue highlights the role of open source in modern software development, reflects on the history of open source, and explores some of the side effects and challenges of building software on top of an open source ecosystem. As you’ll see, open source software has been around — even pervasive — for more than 20 years. What’s different now is not how important major open source projects are to us, but how we’ve shifted from building on top of open source projects to building with open source projects. We’re moving into the long tail of open source, depending on hundreds of smaller open source projects in even the smallest application.
The Evolution of Open Source Software
Open source software isn’t new. Great projects like Linux, FreeBSD, Apache and PHP have been around for 20-plus years. But how and when we use open source has changed greatly in that time. In the past, it was common to use a small handful of open source projects in your application, usually linked to lots of custom code and an expensive commercial database. Today, the same project might leverage dozens of open source projects, have far less custom code and use multiple free databases. Those database and software vendors aren’t panicking, though; they’re rapidly converting themselves from software companies to platform companies.
No company demonstrates this better than Microsoft. In the late ’90s, former CEO Steve Ballmer went on a crusade against open source software, calling it a cancer that threatens intellectual property and short-changes companies that need real support, accountability and scale for the software on which they depend. Fast forward to today and Microsoft is the largest contributor of open source code to GitHub, the largest central repository of open source projects. This summer, they decided to buy GitHub for $7.5 billion. Azure, their cloud services platform, is driving this change. In order to fuel its growth, Microsoft has created and contributed to a wealth of open source projects that allow developers to more effectively build on its systems.
IBM is making a similar move to transform into a platform company, announcing their intent to acquire RedHat for $33 billion in October. Google, AWS and others have been playing the game for a long time as well, strategically leveraging and contributing to open source projects to drive the growth of their platforms. In fact, most AWS services begin life as a fairly “out-of-the-box” open source offering that AWS has wrapped their command and control infrastructure around.
Even as companies position themselves as platforms, one of the most significant open source projects ever has been Kubernetes, a 4-year-old project that helps manage infrastructure in a platform-neutral way. The heart of Kubernetes is Docker, a container system that allows developers to easily split complicated systems into hundreds of smaller, more manageable pieces. While that makes it easier to build software, it makes it much harder to run. Kubernetes was a collection of tools that Google used internally for its own systems, and it rapidly went from a collection of tools to a complete platform, portable between cloud providers like AWS, Google and Microsoft. In spite of the platform-neutral nature of Kubernetes, all three companies are major supporters and offer their own Kubernetes solutions. In spite of its roots in Google, Microsoft is currently the largest and most active contributor to Kubernetes — and the project is hosted on GitHub.
Open source software greatly enhances the value of companies that base their products and services on top of the ecosystem. You no longer need to differentiate what happens under the hood; instead, you can focus on features relevant to your customers. This makes products better, particularly as software companies navigate the transition and begin to solve the problem they should’ve solved all along: providing safe, feature-filled platforms on which to run your business.
But while open source is clearly not a cancer, it isn’t exactly a utopia either. Find a bug and need it fixed fast? You’ll often need to fix it yourself and submit a patch. And while any vendor can go out of business, hundreds of open source projects are abandoned every day, potentially leaving you holding the bag. One of the stickier problems with open source is the complex interdependencies between projects. An application that relies on a handful of open source projects directly often brings in hundreds of others via dependency. And some of those dependencies may not meet your standards for integrity and quality. Tracking the “supply chain” of open source projects is becoming a big business.
In spite of the challenges, open source is essential to modern software development. By sharing in the solutions to common, cross-cutting problems, we free ourselves to pursue what is unique to each of our businesses.
In the News
From the Blog
AWS October Updates
There is an endless flood of new announcements on changes to the AWS platform and surrounding ecosystem, and it is easy to miss important announcements. This monthly post will inform you of major announcements, product updates and behind the scenes changes. Continue Reading…
Policy as Code Bot (PacBot) is a platform for continuous compliance monitoring that allows you to monitor your infrastructure based on asset groups and starts monitoring new additions upon discovery. You are able to view reports of your entire cloud footprint and quickly asses your compliance posture, giving you the necessary information to prioritize your most critical violations and remediate them immediately. PacBot’s architecture pulls data from Qualys, Bitbucket, TrendMicro Deep security, and Tripwire just to name a-few. This will make it a highly customizable solution that helps meet your organization’s compliance needs. – Roberto
Kubespy makes it easy to see what your Kubernetes cluster is doing under the hood, so that you can track whether your changes are being handled successfully. Check out Shaun’s blog post to learn more.
Cloud Infrastructure Manager (CIM) (GitHub)
CloudFormation is a powerful and exciting tool for creating AWS infrastructure stacks from templates. However, even with all of the capabilities CloudFormation templates afford us, there is still an amount of user intervention required in deploying CloudFormation templates, and managing existing CloudFormation stacks is a largely manual task. Enter Cloud Infrastructure Manager, an open source project that takes on the task of managing CloudFormation stacks at a meta level. Cloud Infrastructure Manager lets us create templates for deploying and managing CloudFormation stacks that document and codify what were previously manual configuration steps. – David
Even small applications that only directly incorporate a handful of open source projects can indirectly pull in hundreds of others due to dependencies. Even though you might be keeping your direct dependencies up to date, those projects may not be keeping their own dependencies up to date. These types of vulnerabilities fall under the category of “supply chain attacks” and can be difficult to track. GitHub has released tools to help their customers identify these dependency vulnerabilities, but you don’t need to be their customer to protect yourself. Open source projects like the OWASP Dependency Check and commercial services like Snyk can help you catalog your dependencies and identify problems. – Cris