The Long Tail of Open Source (And You)

It is a common story: a company has some internal software that is draining resources, and they think “Aha! We’ll open source it, and then the community will maintain it for us.” Which would be true, if there was a community. But you do not get a community for free by typing git push for the first time. You have to build one. And building one takes effort. A lot of effort.

But before you start that journey, you need to have a realistic expectation of how much work is going to be involved growing your community to the size it needs to be for you to be able to take a step back. Almost certainly, if you do not have the time to maintain it internally, you do not have the time to grow a community around it.

But first, let’s talk about movies.

Meet The Long Tail

If you plotted the distribution of films by gross, you’d end up with something that looks a bit like this. We all know what a blockbuster movie is. It’s one of those movies on the left-hand side. We call this the head, and only a few movies end up here. But these movies also make a lot of money for the film studios, so it’s usually worth a gamble.

But what about all the movies on the right-hand side? We call that the long tail, and that’s where most movies end up. All the many thousands of movies you never hear about each year. All the box office flops, and the direct-to-DVD titles.

This sort of thing is a common pattern of course. For instance, there are a few very famous people, and lots of not-at-all-famous people. And similarly, there are a few very popular open source projects, and many practically-unheard-of open source projects.

The Pareto principle tells us that you can expect 80% of your sales to come from 20% of your products. So you focus your efforts on the head. And this has been a rule of thumb in business since 1906. But one of the most startling facts here is that, when the internet is involved, the area under the long tail is frequently more than the area under the head.

For something that you’re selling, that area equals money. And for a company like Amazon, this means they can make more money selling lots of really obscure products than they can from selling a few popular ones. This is something bricks-and-mortar shops cannot compete with, because they have limited shelf space. They are forced to stock only the most popular items.

The Digitisation Bonus

How is this related to the internet? Well, as things moved online, they were digitised. And once something is digitised, we are no longer constrained by physical space. The marginal cost of an additional record in Amazon’s product database is practically zero.

And on top of that, things that used to require money and equipment and professional skills can now be done on a home computer. Desktop publishing is taken for granted, but when it was first introduced in the ’80s, it was a big deal. Now, anyone with an internet connection and a bit of spare time can get published. A book, or a blog, or an open source project does not have to be commercially viable for you to start working on it. And when lots of people come together and are empowered to contribute a just little bit, huge things can happen.

On GitHub, for example, there are a staggering number of open source projects covering a range of needs, requirements, and use-cases from common to obscure. Usually you can find somebody, somewhere, who has tried to solve the same problem as you. Before the internet, and even to some degree before GitHub, this wouldn’t have been possible.

This is one of the best things about open source. The open source community solves problems that would never have made commercial sense. And this is about more than just operating below the coasean floor. The switch from extrinsic to intrinsic motivation allows us to redefine what “value” means on an individual basis. Is Wikipedia profitable? If you measure it with money, no. But there are as many ways of measuring value as there are contributors.

The personal computer and the internet have fundamentally changed the way we work together. And as things become cheaper, easier, and more accessible, that long tail stretches out.

Which is a bit of a problem if you’re hoping to climb it…

One Weird Trick

When you make your first commit to GitHub, you do so amongst hundreds of thousands of other projects. This is important to remember at the outset. You start at the thinnest end of the tail, along with everyone else. All the projects with a single maintainer and no stars or forks.

Why do I mention this? Because though it should be obvious, many people seem to miss the ramifications. For every Wikipedia, there are a thousand other wikis that never managed to get a second contributor. For every blockbuster Kickstarter project that raises $100,000, there are hundreds that don’t make it.

In fact, I was speaking to the person behind one of those $100,000 Kickstarter projects. And I asked him what the secret was. I thought he was going to tell me that he published the project, and it “went viral”, and the money just started pouring in. This idea that things sometimes just “go viral” is a popular narrative, like some sort of rags to riches story for the internet age. People obsess over it, but few really understand how it works.

You wanna know how it really happened? You want to learn this one weird trick for success that social media experts hate me for?

Hard work and good fortune.

He told me that no, it wasn’t pure luck. In fact, what he’d done was take a bank loan that was about a third of the size of the funding goal. And with that money, he hired a PR firm. The PR firm generated interest, got him interviews with the New York Times, and so on. Turning a Kickstarter project into a blockbuster had been, in itself, a project—one which took a lot of money, determination, and effort. And the project might still have been a flop.

The narrative that projects can go viral at random is phony. If it happens, it happens in such a statistically insignificant number of cases as to not be worth accounting for. But we find the narrative compelling, and so the trope persists. After all, who doesn’t want to win the lottery? Yet as Seth Godin points out, though success may be a lottery, the tickets (paid for with hard work) are very expensive. So this is about investment and risk.

There was a lot of risk for the person behind that $100,000 Kickstarter project. But it paid off. And the same applies to open source. There’s no “natural process” that is going to bring the contributors to your project, just because you happened to publish it on GitHub. Your project is not going to “rise to the top” if you leave it. Instead, it will languish. And it will become one of those projects that people don’t use, because it looks like nobody maintains it. And if you can’t even find people to use your software, how are you going to find people to contribute?

Climbing the Tail

Perhaps your project solves some monumental problem that everyone has, and the contributions start to come at you from all directions. But it is exceptionally unlikely. It is much more likely that if you want to grow a successful, healthy community, you’re going to have to put in a lot of work. And then cross your fingers.

There are lots of ways to climb the long tail. Lots of ways to make your chances in the lottery of success more likely. In a business setting, you might talk about marketing, product management, and community building. And in a sense, that’s that sort of thing you’re going to have to do for an open source project too. And sure, those words have some negative associations, especially in the world of open source. But you can’t grow communities with wishful thinking. These are the sorts of things you’re going to need to do. Even if it’s just you and a Twitter account, tweeting at people who mention your project.

Make your project easy to contribute to, document it, tell people about it, and help them when they get stuck. But whatever you do: don’t expect miracles!

Have you ever been stuck at the thin end of the long tail, struggling to climb to the top? Or do you have a success story about reaching the head with an extremely popular project? Either way, share your wisdom with us! Throw us a comment below.