Today, Tycho makes an interesting point in Penny Arcade, which is to say, he agrees with me.

A few months ago, when I was first pimping Divmod's Vertex generic peer-to-peer system, the first application I proposed was a BitTorrent-like protocol, but with explicit support for giving users credit. There is a proof-of-concept implementation of this called "sigma" in vertex (but don't try to use it yet, it's full of horrible known bugs - it's just there if you want to help implement it).

When I first mentioned this to a few potentially interested parties, they assumed that the advantage would be increased bandwidth savings on the part of the "content provider". It doesn't provide that, and I'm not sure that one can improve on BT's savings; as Steve Holden said, the only platform Avalanche runs on is PowerPoint. When I revealed what I thought were substantial advantages, this prospective audience sighed and said that BitTorrent was great and that nobody cared about any of my features.

The advantages of a Vertex-implemented BitTorrent would be as follows:

• Vertex provides extremely aggressive NAT-traversal code. If it can get through your firewall, it will. This is important because huge numbers of people using BitTorrent don't care that they're not transmitting data; the seeds are "fast enough" and they don't have the technical knowledge required to configure their firewalls. For some BitTorrent-based applications (such as the Blizzard downloader) such configuration is intentionally impossible; the downloader uses a fixed port number and would therefore require smarts on your gateway to forward properly. I cannot fathom how they could make such a stupid decision - that's certainly not a flaw in BitTorrent itself but it seems to be a common implementation mistake.
• Vertex totally separates the application (file downloading) from the transport mechanism. This means a vastly reduced implementation complexity for implementors (if, like Blizzard, they think they need a totally customized "user experience", they can still use common vertex libraries, or at least implement the protocol separately).
• Most relevant to Tycho's post, Vertex provides strong identity verification for peers. This means that you can provide non-bandwidth-related incentives for people to upload. Imagine that your character got one gold piece for every megabyte of the patch that you uploaded!

This last point I have had an especially difficult time communicating, because I thought that Tycho's point was obvious and I didn't bother to explain it.

So here's the thing: if you're a "content provider" and you use BitTorrent, it's great that it decreases your bandwidth costs, but the message you're sending to your users is, "I don't care about the experience you get using my site. I care about saving money on bandwidth." Especially given that the percentage of full-duplex capable seeders is so poor (see my point about NAT traversal) it is often significantly slower to download a file over BitTorrent than over a straight-up high-upstream link, and of course it is vastly better to simply offer it via a service like Akamai's media delivery.

If you want your users to shoulder part of the burden for you, that's great, and it can improve the experience for them provided you're using the money you've saved for something worthwhile, but at least say thank you for using their upstream. To do that, you have to know who is actually helping you more or less. Since BitTorrent has started using bits of Twisted now, hopefully by the time this project gets off the ground, it won't be some custom file-download protocol that offers this feature, it will simply be the next version of BT itself.

Thankfully Divmod has a client right now who is motivating a bit more maintenance on Vertex. I hope I can present it at the next CodeCon.

Divmod and Twisted have instituted a fairly rigorous development process over the last few months. I'm very happy with the results, but there have been several objections to the strict enforcement of apparently useless or trivial parts of the policy, especially as related to small or mechanical changes.

When I was a wee lad, I took violin lessons according to the Suzuki Method. In the Suzuki Method, the first thing you do in your violin lessons (if you are a 4-year-old, anyway) is make a "violin" out of a cracker jack box and a ruler. For months, I did nothing but sit, then stand, hold the cracker jack box, extend it outward, and place it under my chin, hold it there, remove it, tuck it under my arm, and sit back down. I had to draw a diagram, on a poster, which showed where my feet were supposed to go while performing these steps. Then I stood on that diagram, placing my feet in the appropriate positions.

You are not allowed to use a real violin until you can get every step of this exactly correct. As a kid, that was pretty cool. When you have progressed to the point where you are allowed a real violin, you get to open the box and eat the cracker jacks. Parents, however, were often unhappy with this particular trajectory of violin training. Their 4-year-olds were eating cracker jacks six months after starting violin lessons, and students in other methods (read: their hypercompetitive co-workers' children) were playing Twinkle Twinkle Little Star on their shiny new violins at Christmas and Thanksgiving, much to the approval of the assembled family.

Of course it looks ridiculous to results-obsessed American parents to be messing around with crude drawings of your feet and tiny cardboard boxes when what you're ostensibly doing is learning to play an instrument. Why not move on to the instrument directly and focus on the important parts of playing, like fingering and tone and harmonics and sight-reading and so-forth?

The Suzuki Method was invented by a japanese fellow, and it follows the japanese sensibility that requires even trivial tasks to be performed according to extremely rigid rules. Have you ever watched a sushi chef prepare a meal? The next time you do, observe what they do when they wash their hands. They will always position a bowl of water in exactly the same position. They will always dip their hands in the water the same number of times. The knife will always be placed in the same position, picked up and put down at the same exact moments before and after slicing the fish. I don't know how to prepare sushi myself, but I've observed the pattern at every sushi bar I've sat in, from San Jose to Austin to Florida.

There is a reason why we focus on the small details of the development process. If there is an obviously correct way to do something, it should be done that way. Why expend effort on every method trying to decide whether it "should" be documented? Document everything. Why create elaborate rules for what "should" be tested? Test everything.

Those kids who were playing "twinkle twinkle" six months before I was? Well, I got to play with them later, in school and community orchestras. Some of them even became quite good violinists. However, there were certain differences you can easily pick up on. You can spot a Suzuki student because they sit straight; they hold the bow properly and lift the violin with their chin, not with their arm. A less methodical violinist will slouch, hold the violin wrong, hold the bow at an angle, and generally require more effort to play even simple songs. In fact, they will especially require more effort on simple songs, because the Suzuki student has every automatic aspect of playing completely ingrained in their muscle memory. No conscious thought is expended on the things that don't require it.

Not all the first-chair violinists I've played with learned on the Suzuki method, but they all know how to hold the instrument correctly without thinking about it.

I am pretty unhappy with the way my "paying if you should" essay came out. I am trying to pare it down, and maybe split it up into 3 or 4 more coherent pieces where I'm saying one thing that makes sense.

If you couldn't really figure out what I was trying to get across, it's because I was doing a terrible job of expressing myself. Please read the next version, and try to forget about the previous one.

I hear some big company recently developed a spreadsheet, or something. If you enjoy spreadsheeting online though, check out Numbler.  It's been around for a bit longer, and it's based off of some interesting technology I have heard about.

(Disclaimer: Numbler is not a product of Divmod, Inc.  I have no affiliation with Numbler other than I think Carl is kind of cool, athough I wish he'd put some dang links on that page to the Athena site.)

Update: Carl added that link I mentioned to the "about" page. Thanks, Carl!

A few weeks ago, r0ml posted another thought-provoking piece on the relationship between open source and business. Explained, of course, by using Dickens as a metaphor. Loyal readers of this publication may catch a glimmer there of the origins of my enduring affection for the absurdly extended metaphor.

 

I kept meaning to respond to this, but then Stephe Walli gave me the perfect excuse. Apparently USENIX was in Boston last week (why do I never hear about these things until after the fact? Heck, my last blog entry was about a paper I submitted to a previous USENIX and I hadn't even heard about it) and he chaired a panel on open source businesses.

(Apologies to the participants if the O'Reilly summary of the panel was incorrect in some detail - I haven't had an opportunity to listen to the whole thing yet, so I'm mainly responding to that.)

The panel's participants were talking about start-ups, and the focus seemed to have been strongly on acquisition opportunities, valuation by VCs, and other such meta-meta abstractions of a businesses' success. Granted, those are the ones which make a founder rich, so I'm sure they were interesting to the audience, but I think there is a much deeper question to be addressed in discussions of "open source businesses" which I rarely hear mentioned.

Apparently all the participants also dismissed the value of the community's technical contributions. Community relations is hard, but at least in Divmod's case, we have gotten some really excellent code from the community. It's hard to set up a process to recognize this and facilitate its acceptance - and I know because we really screwed it up with our first round of technology (we got effectively zero contributions to that).

As Mr. Twisted, I can talk about slinging open source code with the best of them, but I'm not a billionaire yet, and I'm also not an executive at a major multinational corporation, so you may disregard the rest of this article as the ramblings of a hopeless optimist whose delusions are so strong that even frequent dealings with the harsh realities of life in a startup have done nothing to disabuse him of his romantic notions.

I am technically an executive at a startup company though, so I have a thing or two to say about these ideas. The lovely thing about being in this position is that if I'm right, I can refer to this article as proof of my awesome prescient genius; if I'm wrong, I will happily (well, mostly happily) continue to languish in obscurity.

Here is my radical proposition: the key to open source in business is not "open source", nor is it a licensing model that models the key intellectual property asset valuation that you wish to reflect to your investors. Forget "open source" for a minute. You want to start a business.

Are you going to provide value to your customers?

This is an important question - one might say the important question of starting a business. There are a number of subsidiary questions: are your customers going to believe that you have provided them with something valuable? Who are your customers? What is the value? How do you decide how much to charge them?

Ultimately though, if you don't have an idea of some value you're going to provide, and some customers you're going to provide it to, you don't have a business, you have a hobby.

One of the key differences between working in a startup and working at an established company is that if you're in a startup, the market sometimes surprises you by answering the question in a way you really didn't anticipate. This rarely happens in a big company: GM sells cars. Nobody is going to walk up to an executive at GM and say, "we would really like to buy model airplanes from you instead, we think model airplanes would be really neat - maybe you should forget about the whole 'car' thing". You can say something like that to an executive at Divmod, though. What I'm really trying to get across here is a story about how that happened. We're getting there, but first I have to refer to the articles I'm responding to.

First, let me address what r0ml is saying. He says that Victorian sensibilities included the idea that only a scoundrel would allow themselves to be oppressed by the receipt of favors. I agree - but not from an abstract, antiquarian sensibility. I believe that most such sensibilities arise from real reasons. Some are bad reasons, but feeling obliged to pay for services rendered has a good one, I think: it makes the relationship very clear. You've received a service, and you've paid for it. In the case where you haven't... you may have paid for the service in a way you did not intend to, or you may owe a debt. As Don Corleone put it, "Some day, and that day may never come, I will call upon you to do a service for me."

Online services today have little reason to respect you. You are abstract; an eyeball, a user, a consumer. You aren't their customer. It makes it hard to have an impact, even a small one. By yourself, your value is so small that they can barely even measure it. The most obvious oppression in this case comes from advertising; it pollutes almost everything we touch, most of all online. You can use an ad blocker, of course, but then, to the service provider, it's as if you don't even exit. Whenever I visit a new site with a top banner ad, left banner ad, right banner ad, interstitial ad, and bottom banner ad, I always think of this quote from Futurama, as Fry expresses disgust when he learns that in the future, ads are broadcast directly into dreams:

Fry: That's awful, it's like brainwashing.

Leela: Didn't you have ads in the 20th century?

Fry: Well sure, but not in our dreams, only on TV and radio...and in magazines. And movies. And at ballgames, on buses, and milk cartons and t-shirts and bananas and written on the sky. But not in dreams, no siree!

Sometimes the oppression is more sinister than advertisements. Sometimes it's more like literal oppression. If you use a free email site, what's to keep them from reading your email? The terms of service are theirs, there are warnings about changes, and if you want to dispute or contest harmful changes to the terms of service for a free site, you have no legal leg to stand on, because you haven't paid for the service, after all. I don't think it's so great that everything is free.

Of course, some paid services are just as bad, but at least there is a clearer understanding of the ramifications there. If you don't like a site's ToS change, you can argue that you paid for a different ToS and there is a limit to how much they should be allowed to change them. You can always take your business elsewhere.

So, while businesses provide value to customers, there is actually a value to the status of "customer" that is important all by itself, independent of any product. That brings me to my next topic: tips.

Stephe quotes Brian Aker as saying that selling software "is like putting out a tip jar". It seems as though this is said derisively, as if there is little you could do that's worse, as a business, than putting out a tip jar.

One reason to deride tips is that they're not required; you have to pay for proprietary software, the argument goes, and of course if it's optional no one will pay you. However, tips don't provide zero value. A tip gives you the status of being a customer, even if the service you received was inexpensive or hard to value directly. It can be used to send a message to the person being tipped; "I thought your service was great" or "I thought your service was lousy", but in America at least, in many industries some level of tip is expected as a simple "you performed the service that you were supposed to". Even bargain-hunters won't welsh on the tip, where it's required.

In fact, food service sales topped $480 billion last year in the USA. (The only number I can find for software was a bit over $380 billion worldwide, but that doesn't appear to be a complete survey.) Since food service industries vary, and tips are generally unreported income, let's say that the average tip was 10%. That roughly $50 billion dollars in tips this year. It seems like you could build a tidy cottage industry on a few tens of billions of dollars. For reference, that's 50 companies the size of Google, or a few tens of thousands of smaller companies.

Given that many Americans are willing to tip five times a week, they might be willing to put forth a little bit more for the open source software that keeps their companies running, if only the culture supported such an expense, and they felt they were directly receiving some kind of service from the authors of the software.

The fellow who made the comment about tips was from MySQL. Now, since it seemed intended to be derisive about the business, I wasn't really expecting to find a tip-based model, but it is still interesting to look at the service that they offer with tips in mind.

They list a feature matrix: "MySQL Server", "Certified, Optimized Software", "Maintenance, Updates, Upgrades". and "Knowledge Base" are all listed as features of the support subscription. A potential customer might look at this list, and think one of two things. Either they don't really understand open source, in which case "certified" and "optimized" might lead them to believe that MySQL Pro was somehow fundamentally different than the free MySQL, which was uncertified and had lots of efficiency problems. Or, If they know about the way open source works, they would realize that they could compile it themselves with optimizations enabled, "certify" that it was in fact software obtained from MySQL AB using checksums, and do the maintenance, updates, and upgrades themselves. (Or, if their distribution supported automated upgrades, as Ubuntu and RHEL do, get the upgrade and maintenance support from somewhere else, effectively for free.)

The other features listed are all about resolving problems with MySQL. The problem with that is, if you have a lot of experience with MySQL, you know it's pretty unlikely that it will fail, and this is only a concern if you are ramping up from a very small to a very large use quickly and you don't know how it will perform.

Okay, I can almost hear the keyboards of ten thousand irate PostgreSQL fans clacking - yes, I know you all have great stories about how InnoDB caused some huge problem, and it tanked your company / pissed off your girlfriend / killed your dog. Consider this: if you have such a story, you probably don't still use MySQL. You've switched to some other database, and so you're not a potential customer anyway. Almost by definition, the people who like MySQL and use it are the ones who don't have problems with it: and there's no reason for them to sign up.

A senior developer, at a company I know of which uses MySQL as a substantial (and critical) portion of their product, summed up the situation for me like this: "All of our MySQL issues were things we could fix by changing the config file."

This particular company has looked into paying for MySQL's services, and concluded that there really isn't anything that MySQL is providing that they want or need. Their method for integrating with their product is fully compliant with the GPL, which is hardly unusual; it's a pretty esoteric use-case to need to modify your database engine for an application.

The problem is, a major part of the service that MySQL AB is providing is not accounted for anywhere here. MySQL (the company) wrote MySQL (the database), and they maintain the code. Both of these services have value. Both, like the dinner you've already eaten when you decide on a tip, have already been delivered.

You might not anticipate any problems you can't fix by fixing the configuration file, but much like you don't want to piss off the wait staff with a poor tip if you're going to eat there again, if you're going to keep using MySQL you probably want them to think fondly of you if you ever have to interact with them for whatever reason - maybe you will have a problem, maybe you'll need a feature, or maybe you will just want some advice about how best to use their product. You want them to think of you as a customer who has a real concern about your favorite feature, not as some anonymous jerk on the forums who won't shut up about your favorite feature.

(I warned you at the beginning of this article that I liked abstruse and extended metaphors, didn't I?)

For some people, this desire to be acknowledged will extend to paying for weird stuff that they don't really need, just to establish the relationship. However, when you shift from being explicit about what the payment is for to making up additional services, you create customer confusion and people start to think they're not supposed to pay you. If you offer a "manager's special dessert", and say "order this to show your appreciation for the service!" people who are dieting will read that, and think, "oh, but I don't want a dessert" and they won't buy it. Some, who understand that it's about compensating the staff for their efforts, will just buy it and throw it out, but that number will be a lot smaller than the number who would leave behind some change if the magic text "a small gratuity is appreciated" were present somewhere.

The punchline to this whole shaggy-dog story is that, as it so happens, Divmod takes tips for maintaining our software. I wish I could say it is because I had this brilliant flash of insight and devised this new, awesome business model for open source, but it's nothing close to that. Not only wasn't it originally my idea, I have even been nervous about mentioning it in my blog until now: only one post here mentioned it so far, and if you weren't reading carefully you might think that was a post about stacking up cans of coffee.

The genesis of this idea was from our users. Divmod doesn't have nearly the volume of users that MySQL does, but we do produce a lot of "cutting-edge technology" that can be applied to wildly different application domains. Originally our goal was (and still is) to provide a consumer service using the technology and work with the community (and use the magic of "open source") simply to improve that technology. We started having a lot of interactions with the community that went something like this, though:

J. Random Hacker: I've got this bug and I have a deadline coming up, any chance I can get my patch merged?

Divmod: Thanks for your time, but we're really focused on improving our customer experience first, so since that patch won't be reviewed soon...

J. Random Hacker: But I need this for a deadline! What can I do to speed it up!??

Divmod: Would you like to retain our consulting services?

J. Random Hacker: No! I don't have lots of money to spend on this, I just need feedback on one patch!

Divmod: Seriously thanks, but we don't have the time to work on this, if you're not going to pay for it.

The market was saying, over and over, "we are using your technology and we want to pay you for it". So, we tried to come up with a way for people to pay us for that.

The restaurant metaphor falls apart eventually, as ridiculously stretched metaphors are wont to do, because we actually do provide an additional service for the "tip". It's too hard to change culture alone, so we provide some fun activities ("club meetings") to go along with the subscription-based tip/fee, but we tried very hard to align those with what our users actually wanted - a say in future developments, additional features, a feeling of a relationship with the Divmod team and a way to know that their feedback was being heard.

MySQL and other companies make money from the top 1% of users or so; the corporate ones who are willing to go through a purchasing cycle and support contract and think it's all worth it. Millions of small sites run on MySQL though, and I don't see how the company is set up to get value from them. There's clearly a lot of value in the middle and high end of the market, since basically every open source company that is surviving and making headlines is competing there, but the real success story of open source is the "long tail". I am not sure that Divmod's model is the one that's going to win there (and I think that soon after we launch our subscription-service revenue is going to outstrip our technology-subscriber revenue) but the first open source technology company that figures out how to cash in just a little on each of the millions of installations of their software "in the wild" is going to win big.

I suspect other open source startups hear this message from their communities in various forms, but we haven't heard it, partially because the community doesn't really know what it wants, and partially because the startups that the media tends to focus on are geared primarily to providing applications and infrastructure for internal use in big corporations.