Lots of people ask me, probably because I have internet chat programs open a
lot but also possibly because it's my god damned job to know,
"What's going on with Divmod?"
Specifically, an area of concern lately has been that no visible
improvements have been made to our flagship
product in quite some time.
As anybody who reads JP's blog or Amir's ought to
know,
we have been hard at work.
We have launched a new application. We have started a few new projects. It may seem as though
we've randomly gone off in a new direction.
We don't have much of a budget for customer relations, so we have been late
in explaining what's going on, but rest assured that there is a plan at work
here, and that your email is safe. We are trying to find incremental ways to
become profitable without requiring open signup to Quotient (the Divmod intelligent
email system), because it is too hard to achieve that goal while we are
still working on short schedules and trying to figure out how the company is
going to keep running from month to month. With no revenue stream, we don't
have the collective attention span required to finish such an ambitious
project, despite the massive effort invested so far.
However, we do have plans for applications in the following months which are
effectively small, vertical slices of the full Quotient experience, and we
will be turning them on one at a time, offering free or low-cost sign-up to
each individually, and finally (when some of these have started paying for
themselves) we will roll them up into a larger service, and migrate all our
existing users' data.
Truly adventurous users may want to move over sooner than that, the
second we have anything email-related at all. Talk to me, and I'll be sure
to let you know when that moment is.
We have just rewritten most of our infrastructure outside of Twisted.
Although it is going to take time, and some intermediary steps to get our
whole email application running on top of the new stuff, it is
definitely going to be worth the wait. The application is going to
get better by an order of magnitude. (Those of you who have layperson family
members who subscribe - please translate that appropriately.)
Personally, I'm immensely relieved by this change of pace. We have brought
on a new employee, Moe Aboulkheir, who is absolutely brilliant. On a team
with too many infrastructure gurus, he (along with Amir) has been the
persistent voice of the application, getting us from an idea to something
resembling a finished product faster than ever before in our history. It
looks like we're going to be replicating that success a few times in the
near future and I can't wait.
Divmod is OK. It's just a flesh wound, really.
We've had worse. Thanks to all you out there reading this for sticking with
us.
I have been avoiding stealing Amir's thunder, so he can be the one to talk
about it, and now he has: Read about Divmod's
first new app in our great overhaul.
Divmod has an idea for an internal accounting system that I think is really
exciting.
(That's right. I think our planned accounting system is cool. Perhaps I've had a bit too much kool-aid this year...)
For a while we wrestled with the idea of "Divmod Dollars" - how are we going to bill for our services out of a fixed pre-paid pool? Each individual "click" in the system should generally be free, charging you only internally,and only billing you if you overflow some large pre-paid limit. This seems to be the key to a successful micropayments system - paying for a reasonable amount of usage up-front so that each transaction doesn't have to worry you. Phone companies do this by charging for "minutes" (or, less popularly, for bits), ISPs for "bits", and airlines do it to a lesser extent by using their rewards program with "miles". For the wide variety of services we had planned there was no physically analagous unit of stuff.
So instead, we're going to charge in an internal unit, like a subway token. We call it the "divmid". The core part of the plan is that all services that you buy from Divmod will have to be purchased in divmids rather than in dollars. Convenient interfaces will be provided in various places to convert from sources of funds such as a credit card, PayPal, or e-gold.
As soon as we can get our accounting server up and running, we are going to start selling Divmids. Divmids will be available to our existing subscribers at a reduced rate, and to the general public at an initial cost of $0.60USD or so.
For now.
We've painfully discovered that e-mail is a bad first service to launch, especially on brand-new infrastructure. We've found, time and time again, that our platform needs tweaking, and while we haven't had problems losing users' data or having serious interruptions in service, the amount of care we have had to put towards the objective of stability and reliability has been prohibitive. For example, as a service like flickr improves, it "gets a massage" several times a month, and nobody minds because of their cute maintenance page. If those same users' email had disappeared (beta disclaimer or no), they probably wouldn't have been so charitable.
So instead, we are going to be publicly launching a set of simpler, more self-contained services, and gradually working up to our grand vision of an automated communications platform. Our first service should be available as soon as next week - I'll let Amir describe that one in a separate post.
As the value and diversity of services you can purchase with one divmid goes up, we will raise the price commensurately. In other words, our customers are effectively our venture capitalists. If you buy divmids, and the services become worth something, you can sell them. It's not a share in the company - and it is therefore not priced out of the reach of mere mortals - but it is something that ultimately represents our success. On the other hand, there is the risk that Divmids will suddenly become valueless, or decrease in value if we can't provide certain services any longer.
We do not plan on providing a way to directly convert Divmids back to dollars directly. Continuing with the subway token metaphor; when you buy a subway token and the transit authority raises its rates, you can't just take your tokens back and turn them into cash. Instead, we plan to set up an exchange where people can sell Divmids to each other rather than buying direct from us.
(That's right. I think our planned accounting system is cool. Perhaps I've had a bit too much kool-aid this year...)
For a while we wrestled with the idea of "Divmod Dollars" - how are we going to bill for our services out of a fixed pre-paid pool? Each individual "click" in the system should generally be free, charging you only internally,and only billing you if you overflow some large pre-paid limit. This seems to be the key to a successful micropayments system - paying for a reasonable amount of usage up-front so that each transaction doesn't have to worry you. Phone companies do this by charging for "minutes" (or, less popularly, for bits), ISPs for "bits", and airlines do it to a lesser extent by using their rewards program with "miles". For the wide variety of services we had planned there was no physically analagous unit of stuff.
So instead, we're going to charge in an internal unit, like a subway token. We call it the "divmid". The core part of the plan is that all services that you buy from Divmod will have to be purchased in divmids rather than in dollars. Convenient interfaces will be provided in various places to convert from sources of funds such as a credit card, PayPal, or e-gold.
As soon as we can get our accounting server up and running, we are going to start selling Divmids. Divmids will be available to our existing subscribers at a reduced rate, and to the general public at an initial cost of $0.60USD or so.
For now.
We've painfully discovered that e-mail is a bad first service to launch, especially on brand-new infrastructure. We've found, time and time again, that our platform needs tweaking, and while we haven't had problems losing users' data or having serious interruptions in service, the amount of care we have had to put towards the objective of stability and reliability has been prohibitive. For example, as a service like flickr improves, it "gets a massage" several times a month, and nobody minds because of their cute maintenance page. If those same users' email had disappeared (beta disclaimer or no), they probably wouldn't have been so charitable.
So instead, we are going to be publicly launching a set of simpler, more self-contained services, and gradually working up to our grand vision of an automated communications platform. Our first service should be available as soon as next week - I'll let Amir describe that one in a separate post.
As the value and diversity of services you can purchase with one divmid goes up, we will raise the price commensurately. In other words, our customers are effectively our venture capitalists. If you buy divmids, and the services become worth something, you can sell them. It's not a share in the company - and it is therefore not priced out of the reach of mere mortals - but it is something that ultimately represents our success. On the other hand, there is the risk that Divmids will suddenly become valueless, or decrease in value if we can't provide certain services any longer.
We do not plan on providing a way to directly convert Divmids back to dollars directly. Continuing with the subway token metaphor; when you buy a subway token and the transit authority raises its rates, you can't just take your tokens back and turn them into cash. Instead, we plan to set up an exchange where people can sell Divmids to each other rather than buying direct from us.
Recent fanfare over Twisted, including the totally awesome book which you
should go buy right now, has gotten me thinking - is Twisted really all
that great? I believe that while it is still probably the best thing out
there for doing what it does, there are a few things I wish had happened
differently. So here's a laundry list of things that I wish Twisted did
differently, and how I would implement them if I were starting from scratch
today. Maybe eventually this will be a roadmap - right now, it's wishful
thinking, and too vague to be any real kind of spec.
In the innermost guts of the reactor, there is no real normalization of events. The reactor is sort of a fused engine block where all of the "work" of dispatching events happens. I'd rather that were unrolled a bit. Especially in today's world of generator-heavy Python, I'd rather that the reactor core look something like a set of wrapped iterators; a base generator that ran "select()" and yielded file descriptors ready for reading / writing; a generator that wrapped that which did OS- and FD-specific I/O, like recv() and send(); a wrapper above that generating application-level request/response pairs, and so forth. Think of this as a web server (in very, very broad strokes, this is not a precise API):
Such a system would also make it a lot clearer what "one reactor iteration" meant. Rather than some arbitrary constellation of behaviors which happened to be scheduled "at the same time", one reactor iteration could be made to correspond exactly with one tick of a user-provided iterator.
Further up from that (but using that facility), I wish that we had included SEDA's notion of a "stage". This would have made a few things a lot easier. For example, it would be nice to have a well-defined notion of a request/response processing webserver that could generate a "response" object, possibly from a thread, but have that "response" be processed entirely asynchronously in the main thread.
In particular having a notion of a "stage" would make it a lot easier to run full database transactions within threads, isolating them from communications code, by stipulating that transactions must produce notifications or network I/O in the form of output objects placed into a queue. Recently I have surveyed some open-source Twisted code in the wild, and answered a bunch of questions, which have implied to me that many Twisted developers now believe that the correct way to interface with a relational database is to turn *EVERY* SQL statement into a Deferred which is handled individually.
This is a tangent, but allow me to offer a bit of advice. The documentation is really poor, and never says this, but using Twisted, or rather ADBAPI, to convert every single SQL statement into a separate transaction and handle its results separately, has a whole slew of problems. First of all, it's slow: you have to acquire and release thread mutexes on every operation. Second, it is unsafe. Your conceptual transactions might be interrupted at any moment, leaving your database in an inconsistent state. Also in the realm of safety, notifications generated from within a transaction that gets rolled back are sent to the network anyway, so two different database-using proceses talking to each other can trivially become inconsistent. Take a look at the 'runInteraction' API and give some thought to what represents a "whole" transaction in your database. Moving this transaction processing out of the main loop *IS* an appropriate use of threads, and in fact adbapi does it internally. This is doubly true if your application or your SQL layer does any caching of SQL results; to be sure that the cache is consistent with the DB, you have to keep track of whether and when transactions are rolled back.
Back to the main point. I also would have designed the reactor access API a bit differently. 'from twisted.internet import reactor' looks convenient, but is highly misleading. Figuring out what reactor your process is currently using is part of a more general problem of execution context. There are other objects that applications wish to find in the same way: the current database connection, for example, the current log monitor, or the current HTTP request. twisted.python.context deals with this in a general manner, but because it is not used consistently to access important objects, it has not been subjected to the testing and refining that it has needed. The worst side-effect of this has been the "context object" abomination that afflicts Nevow and Web2.
I also would have designed Deferreds as more central to the whole thing, and optimized the hell out of them rather than worrying about their performance. For example, it would be a lot easier for many applications if deferLater were the default behavior of callLater. Similarly to deferToThread vs. callFromThread. The main reason that the reactor does not use Deferreds for these, or for the client connection API, is because of a general feeling of nervousness about how it would be hard to implement Deferreds in C, so the reactor API shouldn't require them. In retrospect this is silly (especially now that James Knight has actually gotten further implementing Deferreds in C than anyone else has on getting the reactor implemented there).
In the innermost guts of the reactor, there is no real normalization of events. The reactor is sort of a fused engine block where all of the "work" of dispatching events happens. I'd rather that were unrolled a bit. Especially in today's world of generator-heavy Python, I'd rather that the reactor core look something like a set of wrapped iterators; a base generator that ran "select()" and yielded file descriptors ready for reading / writing; a generator that wrapped that which did OS- and FD-specific I/O, like recv() and send(); a wrapper above that generating application-level request/response pairs, and so forth. Think of this as a web server (in very, very broad strokes, this is not a precise API):
def webServer(self, connection):
for request in parseRequests(connection.inputStream):
response = self.respondTo(request)
yield response
Such a system would also make it a lot clearer what "one reactor iteration" meant. Rather than some arbitrary constellation of behaviors which happened to be scheduled "at the same time", one reactor iteration could be made to correspond exactly with one tick of a user-provided iterator.
Further up from that (but using that facility), I wish that we had included SEDA's notion of a "stage". This would have made a few things a lot easier. For example, it would be nice to have a well-defined notion of a request/response processing webserver that could generate a "response" object, possibly from a thread, but have that "response" be processed entirely asynchronously in the main thread.
In particular having a notion of a "stage" would make it a lot easier to run full database transactions within threads, isolating them from communications code, by stipulating that transactions must produce notifications or network I/O in the form of output objects placed into a queue. Recently I have surveyed some open-source Twisted code in the wild, and answered a bunch of questions, which have implied to me that many Twisted developers now believe that the correct way to interface with a relational database is to turn *EVERY* SQL statement into a Deferred which is handled individually.
This is a tangent, but allow me to offer a bit of advice. The documentation is really poor, and never says this, but using Twisted, or rather ADBAPI, to convert every single SQL statement into a separate transaction and handle its results separately, has a whole slew of problems. First of all, it's slow: you have to acquire and release thread mutexes on every operation. Second, it is unsafe. Your conceptual transactions might be interrupted at any moment, leaving your database in an inconsistent state. Also in the realm of safety, notifications generated from within a transaction that gets rolled back are sent to the network anyway, so two different database-using proceses talking to each other can trivially become inconsistent. Take a look at the 'runInteraction' API and give some thought to what represents a "whole" transaction in your database. Moving this transaction processing out of the main loop *IS* an appropriate use of threads, and in fact adbapi does it internally. This is doubly true if your application or your SQL layer does any caching of SQL results; to be sure that the cache is consistent with the DB, you have to keep track of whether and when transactions are rolled back.
Back to the main point. I also would have designed the reactor access API a bit differently. 'from twisted.internet import reactor' looks convenient, but is highly misleading. Figuring out what reactor your process is currently using is part of a more general problem of execution context. There are other objects that applications wish to find in the same way: the current database connection, for example, the current log monitor, or the current HTTP request. twisted.python.context deals with this in a general manner, but because it is not used consistently to access important objects, it has not been subjected to the testing and refining that it has needed. The worst side-effect of this has been the "context object" abomination that afflicts Nevow and Web2.
I also would have designed Deferreds as more central to the whole thing, and optimized the hell out of them rather than worrying about their performance. For example, it would be a lot easier for many applications if deferLater were the default behavior of callLater. Similarly to deferToThread vs. callFromThread. The main reason that the reactor does not use Deferreds for these, or for the client connection API, is because of a general feeling of nervousness about how it would be hard to implement Deferreds in C, so the reactor API shouldn't require them. In retrospect this is silly (especially now that James Knight has actually gotten further implementing Deferreds in C than anyone else has on getting the reactor implemented there).