2021-12-17
Open letter regarding log4j

Hello Richard,

I've just read your article Log4j hack raises serious questions about open source software (FT, December 16, 2021) and I have a a confession to make: Although being an open source software developer made me a multi-millionaire, I still feel frustrated when I look at how open source is evolving.

Allow me to introduce myself. My name is Bruno Lowagie and I wrote a book Entreprenerd: Building a Multi-Million-Dollar Business with Open Source Software. This book was written entirely based on personal experience and it received several 5-star reviews from the people who read it:

Unfortunately, less than 350 people bought the book and I was quite disappointed when it wasn't selected as a nominee for the FT & McKinsey Business Book of the Year. I sent an email about the book to FT-journalists such as John Thornhill and Tim Bradshaw, but they completely ignored my message, just like you are probably going to ignore this email. That shouldn't surprise me as much as it does: open source developers tend to be ignored until something goes wrong.

I apologize in advance for the fact that my email is colored by that frustration. Please understand that the cold shoulder I got from the press regarding my book doesn't make sense to me: Journalists keep "raising questions about open source" but at the same time remain deaf for the answers to those questions.

You probably don't know me, but I don't think I exaggerate when I say that I have more mileage in FOSS than the average developer. I was already writing free software when large corporations considered Linux being a cancer (Steve Ballmer, 2001) and open source being socialism (Shai Agassi, 2005). I have actively contributed to the evolution of open source as a developer (I'm the original developer of "iText" in 2000), as an author (of two iText in Action books published by Manning Publications in 2006 and 2009), and as a start-up founder (of iText Group in 2008). I grew the business for iText from zero revenue (2008) to an eight-figure exit (2015). In case you've never heard about iText: it's an open source PDF library that everyone uses, but no one notices because it lives on the server, where it creates invoices, bank statements, boarding passes, and many other documents in the PDF format. Today, the company I founded is fully owned by three private equity companies backed by Peter Thiel—I hope that at least that name rings a bell.

In 2014, I already wrote an article entitled Heartbleed, an Apache License Business Model Failure? My heart bleeds once more when I read about log4j. In your article, you wrote: "If more money isn’t the answer, then other forms of support are needed, along with social incentives that encourage developers to focus more on secure coding techniques."
In my opinion, that diagnosis isn't entirely correct. You are right when you write that the problem isn't that more money is needed. But you fail to tell that the main problem is that the money doesn't end up in the right pockets and isn't spent the way it should be spent. That's the actual problem and I solved it when I created a sustainable business for iText, making sure I could hire a team of dedicated iText developers, including QA and test engineers.

Let me start by saying that open source software foundations such as the Apache Foundation aren't the answer. The boards of such foundations are supposed to be composed based on merits, but the criteria that are used to define those merits are highly flawed. When dealing with them as a former FOSS developer and project owner, I can never shake the impression that foundations are more about politics than about code. Some foundations are sitting on a huge pile of cash, but none of that cash flows back to the original developers of the projects that were brought under the umbrella of the foundation.

You quote a CTO at Red Hat: "The most critical open-source projects are already maintained by full-time developers working inside companies who rely on using the code in their own products."
In Chapter 16 of my book Entreprenerd, I wrote that "the creation of FOSS shifted from individual developers to IT companies. Yahoo, Facebook, Google, and Netflix are examples of companies that open-sourced part of their code. [...] I have the impression that most open source code is written in the context of employee contracts nowadays. Potential star developers risk becoming nameless employees now that large corporations have appropriated the FOSS movement. One can argue whether FOSS created by independent developers is more desirable over FOSS created by corporations. Granted, my preference for the former is colored by my personal experience, but I sincerely hope that my book will inspire developers to create their own FOSS project, find the right business model for the technology, start a company to develop a business, and by doing so, prove that commercial open source software is the best guarantee for the production of software that is useful, innovative, and of high quality, and at the same time is sustainable, accessible, and affordable."

In my opinion, large corporations such as MS, Google, Amazon... aren't the answer to the problem either. In the same Chapter 16, I also wrote: "Today, these companies are perceived as FOSS heroes because they distribute plenty of FOSS products using a permissive license. They can afford to do so because they have a business model in place that generates sufficient money—selling products that are very different from the FOSS product, but they’re not sharing out of charity. The more developers adopt their FOSS products and frameworks, the more these corporations strengthen their position in the market. The generosity of such companies usually isn’t perpetual. It ends the moment the investments needed to maintain and continue the free offering outweigh the advantages. Giving away value for free is only justified as long as the business model allows it. Users building a business on top of a free framework may wake up one day to discover that the framework they use is no longer supported. [...] I advise FOSS users to work with technology from a company that ensures further development and long-term support, rather than using free stuff that is merely a side effect of the actual business. FOSS users should always be aware of the business model behind the technology they use."

The answer to the question you raise about open source software is simple: The world needs more small, independent pure FOSS vendors. Give them a place in the eco-system and reward them for taking care of crucial niche solutions so that they can also assume responsibility for their work. It's the best incentive to avoid problems such as Heartbleed, Log4Shell, and other disasters, because if such a FOSS vendor fails at its task, its business will fail. One company that fails is less damaging for open source as a movement than the damage that is caused by a foundation failing once again to ensure the quality of one of the projects under its umbrella.

I've said it before and I'll say it again: "Good engineers build great technology; great engineers also create a sustainable business model." If developers are smart enough to build great software, they should be smart enough to generate sufficient resources to ensure QA and maintenance for their projects.
You wrote: "Open-source developers value their independence. But their software now has a central place in business and society, and this world-changing experiment needs to be brought up to date."
That's EXACTLY the message I want to bring with my book: It is irresponsible for a software developer to distribute a project that doesn't generate sufficient financial resources to ensure QA and maintenance. However, that's not what developers are told. The moment large corporations started to embrace open source as a "free resource" (free as in free beer, not as in free speech), it was not interesting for those corporations to tell developers that they also should care about the business. It was not interesting to encourage nerds to claim their independence and become entreprenerds, but maybe now is the time to do it.

It is time for open source project owners to prove that they aren't merely good engineers, but that they are also great engineers. Please stop brainwashing open source project owners into believing they should work for free on their projects, claiming they are already paid a salary as an employee. Turn open source project owners into entrepreneurs and teach them how they can turn their project into a product and monetize it; explain how to invest that money in quality assurance, support and maintenance. Read my article Open Source Survival: A Story from the Trenches for inspiration if you don't know where to start.

Based on previous experiences with journalists, I am 98% sure that you won't even read this mail, but if you do, I would really appreciate it if you read my book. You raised questions; I can't think of a reason why you wouldn't want to read the answers. Please give my book a chance. You won't regret it.

best regards,
Bruno

Update 1

I received the following answer from Richard Waters:

Thanks for your response and sharing your experience.

I certainly agree the question here is what incentives there are for developers not just to write good code but maintain it after. Also agree that financial incentives are important - I think I might have been a little too dismissive of that, didn't mean to dismiss it completely out of hand. But clearly a complex topic - I hope someone does a thorough case study of the Log4j community, would be interesting to see how it functioned and what made it tick. 

In my original mail, I shared a coupon that could be used to download the book for free, but when I check that coupon, I can see that it hasn't been used (yet).

Update 2

on LinkedIn, Pieter Colpaert wrote:

Interesting take Bruno! Agree with your arguments.

You often discuss the issue of FLOSS sustainability and then jump to the conclusion you need a for-profit company to guarantee sustainability in open source. What would be your take on initiatives like Open Collective? This way, multiple companies and individuals can co-fund the time of company developers and freelancers to maintain the upstream code.

This is my answer to his question:

With freelancers, you lack continuity. When freelancers get bored of a technology, they move on and their knowhow gets lost. You can hire new freelancers, but they'll have to start from square one with no access to the original developers. That sounds like a recipe for disaster, doesn't it? If it doesn't, you're a lucky bastard because it means you've never inherited a legacy project. A company can't afford to get bored of it's product. It must ensure continuity. If it doesn't, it will die, unlike freelancers.

Furthermore, who takes responsibility for the code written by freelancers? How do you fire freelancers who screwed up and left the project? Will you sue freelancers who caused problems such as log4shell individually? In an ecosystem consisting of for-profit FOSS vendors, the answers to those questions are easy. Companies will compete and the bad apples will disappear. Accidents will still happen, but at least it will be clear which entity is accountable for what went wrong.

The stakes for a company screwing up are much higher than the stakes of an individual developer writing bad code (unless you are in favor of a system that severely punishes such developers, which is, by the way, an existing risk).
The incentive to avoid problems such as Heartbleed, Log4shell... Is much higher for a for-profit company than for every other setup, hence I put more trust in an ecosystem consisting of small companies than in a world dominated by Big Tech funding foundations bread crumbs to evangelize licenses that jeopardize project owners' chances to build a business.

I talk from personal experience: at iText, several people knew (and know) the ISO standard for PDF inside-out. This standard is a document of about a thousand pages, not including the several thousands of pages with technical notes that accompany the standard. It is very hard to find a freelancer with that much knowhow. During my career, I have seen incredibly bad code that revealed a deep ignorance of the PDF standard. That didn't surprise me: It's too much of an effort for a freelancer to study the PDF specification when hired for PDF development.

I also have bad memories about companies such as Amazon and Google who were avid users of iText, but who never contributed anything back to the project. Google was willing to send me "some goodies" as an appreciation for my work, but you don't maintain an open source project with goodies, do you?

Entreprenerd

Book cover Entreprenerd
Buy Bruno's book

Wintercircus

Ghent Wintercircus Campus

Blog

More...

Topics