The change might seem minor but what it really means is that every contributor has equal rights over their work, as part of the overall body of work. This is about turning ZeroMQ into a commodity now, rather than later. We want to make it commercially and legally trivial for any business using ZeroMQ to contribute, without worrying that they are in fact acting as free labor for iMatix.

The alternative would have been to sign over the copyrights to the FSF or a specially created Foundation but that would still mean every patch and contribution needed to be CLAd or pushed out as MIT/X11 code, both of which are barriers to contributors. If I write an amazing ZeroMQ device, I want the right to publish that under the same LGPL as all other ZeroMQ code. Doesn't require me to be iMatix to do that.

It's appropriate that a distributed decentralized technology with no single point of failure have a distributed, decentralized ownership with no single point of failure.

by pieterh

This is ØMQ, which has as many definitions as it has spellings. What ØMQ does is let programmers build applications that distribute easily across any number of boxes. Yes, Dorothy, I realize you thought programmers could already do this. But right now they need to glue their programs together using cat guts and puppy dog tails. ØMQ saves the kittens and puppies so they can be sent to China to make fur coats. Just kidding. They all live on a magical island off California, populated only by little old ladies.

ØMQ is kind of programmer pr0n. By this I mean that it fulfills fantasies and does it wearing really practically nothing. Admittedly not every programmer gets excited by "perfect scalability" but this software separates the men from the boys. No, I'm not going to dignify your imagination with a crowbar joke. Real programmers prefer ØMQ.

To appreciate ØMQ you have to be a veteran of some battles involving large software systems and human stupidity. One day, ØMQ will be taught in CompSci classes as the missing layer in the networking stack. It's not so much about solving horribly complex technical issues, as about solving a bunch of weaknesses in the human mind that programmers seem to have been too damn proud to admit to for ages.

• We are thick, especially when we're distracted, tired, or overworked, which is the natural state for most of us, most of the time.
• So, we have trouble holding complex things in our minds. They fall out, until we literally learn them by heart.
• So, when we're asked to remember complex concepts, we get them wrong. Yet we're convinced we're doing it right.
• So most large, complex software systems are literally filled with the consequences of stupidity and the better the developer, the more time they spend trying to fix these consequences.

ØMQ starts with this hypothesis: we're all kind of stupid, so let's take it very easy and do things simply. A simpler API, a simpler protocol, simpler abstractions. There's always going to be some unavoidable complexity. A system with a hundred pieces ain't going to be trivial. But if those hundred pieces only connect in three distinct ways, that is a great help. And if those connections all look and work just the same, no matter what's going on underneath, that's even better.

by pieterh

Coffee was once a small African bean that caused goats to leap, until it was banned by imams in Mecca in 1511, who discovered the Streisand Effect five hundred years early.

Today, of course, coffee is one of the five essential ingredients of the modern economy. And nothing captures the essence of coffee - the bitter kick combined with a restorative caffeine burst - than a well-made espresso. The perfect espresso is creamy, tasty, and just bitter enough to leave a sweet after-taste. One should never add sugar or milk.

As a traveller, my usual struggle is to find, in order: espresso, Internet, and power. My fall-back, in the US is to find a Starbucks and order a double espresso, and a tall glass of water. It's cheap espresso: bitter, burnt, and beastly. Thus the water.

You get the best espressos, of course, in Italian, Spanish, and Portuguese coffee bars. The French also do it pretty well. This is thanks to the historical Muslim influence, since it was the Ottomans that brought coffee to Southern Europe from North Africa and Ethiopia.

Elsewhere… it can be pretty desperate. Yes, my name is Pieter, and I am a lifelong coffee addict.

So, it was with some emotion that I ordered a Handpresso Wild on Ebay from a French firm "EspressoDiscount". Was this going to be the answer to decades of searching for the always-available hit? Would this be a cheap plastic disappointment?

The package arrived about a week later and my credit card got a Euro 85.05 charge. The packaging was nice, but irrelevant. This is not about the box but about the espresso. I opened it up and found a neat, solid device looking like a cross between a bike pump and a drug delivery system from some far planet.

Bootstrap problem: the Handpresso demands pods. Not the larger soft pods but the smaller 7 gram "single dose espresso" (ESE) pods. They can be hard to find. I found an 18-pack from Lavazza for 6 Euro, a 12-pack from Rombouts (Belgian coffee brand) for 2.50 Euro, and an 18-pack from Kimbo for around 3 Euro.

Now, how to use the thing. My first trial was at the beach. To use the Handpresso, you twist the handle to unlock and then pump, twenty or thirty times, until the needle hits green. There is no point in over-pumping. You then fill the cup with hot water (I took a vacuum flask with me). You place the coffee pod over the cup, and screw the cap tight on. Then you turn, and press the pressure release button. A small steady jet of hot coffee emerges, as pressurised air fills the water chamber and forces the hot water out.

Surprisingly, it works. You get a small but very good cup of espresso. You also get a lot of stares from people who wonder what the heck you have there.

The Handpresso is very well made. It is solid, has few moving parts, a steel body, and feels like it could last for twenty years or more.

My second trial was on a trip to Bratislava. The Handpresso fits neatly into my camera/computer backpack and adds 500g. I learned a couple of things. First, take out the Handpresso when you go through airport security. You will be asked to explain anyhow, but that way you avoid a full bag search. Second, take enough pods with you. They can be hard to find, and when you start showing your toy to friends and colleagues, the pods get used up very rapidly.

Incidentally, a tip if your airline places limits on hand baggage and you want to avoid checking in your stuff: check-in online and go straight to the gate. They don't (yet) weigh hand baggage at the gate.

You can make the coffee stronger by overfilling the cup and then allowing the pod to soak up the excess water, and leaving it for ten seconds before pushing the pressure release button. You get the same effect by forgetting to pump before adding the water.

My conclusions: if you like espresso and you travel, buy this. I've wanted a portable espresso machine for ages, but this design is genius, and successful. It is not as good as a Southern European espresso served in a smoky bar, but it's very close, especially if you buy the more expensive pods. You might think that pre-ground coffee is bad, but the pods are sealed and firms like Lavazza have done this for decades. I actually have a Lavazza Espresso Point in my office but the Handpresso is just so much easier that it's what I'm using now.

I expect that as soon as the patents run out on this, or perhaps a lot sooner, there will be cheaper imitations hitting the market and we'll see many variations on this idea of hot water pumped by manual action: larger capacities, working with ground coffee rather than pads, with built in water heater, and so on.

And finally, because there is no large mechanical pump that needs to be pre-heated: it's good for the environment. Boiling a small cup of water is cheap and efficient. Leaving a machine to sit, heating the room, is wasteful. The Handpresso fits perfectly with my slogan for better living: "low carbon, high lifestyle".

by pieterh

As a young man I voted for the Greens because they represented the vision of a clean Earth where people worked harder and consumed less. The Green movement was always a strange mix of hippy idealism and hard-nosed pragmatism. Consuming less, and living smarter: not options in a world with limited resources.

But that's not why I'm voting Green tomorrow. The main reason is that since 1999, the Green parties of Europe have consistently been the strongest voice in the European Parliament fighting for what I consider to be an ultra-capitalist vision of pure and real competition unfettered by cartels, monopolies, and political cronyism. When it comes to software patents, criminalisation of copyright law, retention of communications data, and the sharing of software developed with public funds, the European Green parties have consistently taken a consistent pro-market line, uninfluenced by lobbyists. Prominent leaders such as Eva Lichtengerber have hammered home the message: Europe depends on its diverse economy of small, innovative firms. The patent system, especially, is the corrupt tool of egomaniac politicians, monopolists like IBM and Microsoft, self-interested bureaucrats, and cynical speculators. It preys on the economic majority of creative SMEs, and taxes the consumer with fraudulent fees extorted through artificial monopolies.

While the other main parties seem unable to snap out of their wide-eyed love of industrial giants, the Greens have correctly understood that the role of the European Parliament is not to rubber-stamp texts written by industry and polished by the Commission. Government does not serve just industry, it serves everyone, and big industry is small, stupid, greedy, and ultimately wasteful part of the whole.

Thank you, Eva, for all the fights you started and finished in Brussels, on behalf of people like me, who ask nothing except the freedom to create. Tomorrow you get my vote, again.

by pieterh

This is what I predict will drive IPv6: the desire by the "criminal majority" to create invisible and untraceable file sharing networks. Efforts to fight copyright infringers depend on the IP address of the person sharing (uploading) files. IPv4 addresses are limited, and easy to trace to at least an ISP, if not an end user. This lets the content industry push for "3-strikes" legislation, as they are doing around the world.

As in any arms race, it's not over just because one side scores a hit, and history tells us that the content industry is typically responsible for technological innovation, through their clumsy lobbying efforts to regulate the Internet into behavior that would protect their distribution channels.

So the question is not whether the file sharers will discover ways to continue their illicit fun and games, but how.

And the answer is, IMO, encrypted Tor networks that emulate IPv6 networks, running over a physical IPv4 network. The real world won't go to IPv4 for a long time, the inertia is almost unmoveable. But emulation is an easy way to run a second real world inside the real real one. So IPv6 will be emulated, and will be pushed by brilliant minds who seriously just want to be able to download the latest episode of Lost.

And thus my prediction: IPv6 will struggle to make any inroads into the Internet as we know it today but it will get into software stacks, into Linux, into browsers, and eventually into network fabric, through the file sharing community and through the actions of the content industry.

by pieterh

Introduction

Collaboration and innovation stem from cheap communications but not all communication platforms are equal. As a manager I often see people struggle with inefficient or inappropriate communication platforms, and I direct them to use more effective platform, usually with good results.

The following section is my own analysis of what the differences are between various communications media, or "platforms", and what impact these differences have on groups. As far as I know this is the first attempt to analyse communications in this way, surprising since it is one of the essentials of our technological society, along with fire, pizza, and fermented starches served by wenches.

Communication Platforms

Let's look at the most commonly used platforms:

• Face-to-face: the most ancient platform, the basis for Greek democracy and when followed by the appropriate shared dinner, evening at the theatre or moving picture, and glass of red Bordeaux at a quite place around the corner, the continuation of our species.

• Written paper: though stone and baked clay could survive going through the washing machine, flattened papyrus stems were more portable, cheaper and ultimately the basis of organised religion, the modern state, and TV guides.

• Telephone: invented by Graham Bell in order to call his mistress, the concept of two cans separated by a very long cat rapidly caught the public imagination, especially when combined with a low, low monthly connection fee and easy-to-pay rate schedule. These days, phones are by definition mobile (at least in my study).

• Fax: invented by the Japanese in the 1970's so they could send pictures of Godzilla more efficiently, fax rapidly took over the business world because email had not yet been invented. Today fax is mainly used by producers of ink cartridges to promote the sales of ink cartridges for fax machines.

• Chat: the Internet version of the telephone, chat has morphed from many forms, including IRC, which was a significant improvement on all its successors. Internet chat is successful because on the Net, any dog can look like a thirteen year old girl.

• Web pages: the Internet version of the papyrus leaf, taking about the same amount of effort to hammer into usable material. The original web site was built for nuclear research but within 24 hours had been defaced by h4ck0rz who loaded it with porn and made $150,250 before the nuclear scientists took control again, and invented the password.

• Electronic documents, which range from simple text files to the elegant suit-and-tie called PDF. PDF is the only popular communication platform that was invented, and more importantly, not royaly screwed up in version 2.0, by a corporation. Graham Bell and his mistress do not count as a corporation.

• Email: still, after all these years, the most popular way to send electronic mail. Email has had many incarnations, from evil Unixy command-line mailers that asked, "Delete all email" on quitting, to point-and-drool web mailers that brought email and advance-fee fraudsters to the masses. Hotmail has paid for several nice villas along the beach of Lagos Island.

• SMS: we never knew that a few lines of text stuffed into the tail end of mobile phone data packets could become such a huge business. SMS produces half of mobile phone companies profits and two thirds of all family disputes when daddy sees the phone bill.

• Mobile mail: SMS chat for adults, epitomised by the Blackberry. Mobile mail combines the advantages of email with the portability of SMS plus full tax deductability as an essential business expense.

• Wiki, the hippie mutant offspring of web pages and web-based email. Someone said, "if we let anyone edit this page, can it be any worse than the web currently is?" and the surprising answer was, "no". Wiki sites are rarely beautiful, but prove the dictum that "many eyes make any problem shallow".

When we communicate, we have different, and sometimes conflicting needs:

• The cost of accessing the platform (which I'll call "accessibility").
• The cost of making a single statement ("unit cost").
• The informality of a statement ("informality").
• The lack of skill needed to use the medium ("simplicity").
• The speed of receiving and responding to a statement ("latency").
• The number of people we can reach at once at reasonable cost ("fanout").
• The reliability of the medium against tampering ("security").
• The likelyhood that our words can be used against us ("transience").
• The ease of storing and searching a dialogue ("archivability").
• The time we can take to respond to a statement ("escapability").
• The freedom we have to pretend we lost a message ("deniability").
• The freedom we have to hide our identity ("anonymity").
• The bandwidth with respect to non-verbal communications ("emotiveness").
• The ability to carry the platform with us ("portability").

Let's see how each of our listed communications platforms scores on these criteria, and let's score each platform from 1 to 5, where 1 is the least useful or desirable, and 5 is the most. We assume that the goal is to communicate with as many people as possible, as cheaply as possible:

 . | F2f Paper Phone Fax Chat Web Etext Email Sms Mtext Wiki --------------+-------------------------------------------------------- Accessibility | 1 2 5 4 4 5 3 4 5 3 5 Unit cost | 5 2 4 3 5 2 4 5 3 5 4 Informality | 5 1 5 2 5 2 2 3 5 4 3 Simplicity | 5 3 4 2 3 1 1 3 5 3 3 Latency | 5 1 5 3 5 1 2 4 5 5 3 Fanout | 3 3 1 3 1 5 5 4 1 5 5 Security | 5 4 5 3 5 2 1 3 5 4 1 Transience | 5 1 5 2 4 1 1 2 4 2 2 Archivability | 1 2 1 2 2 5 5 5 1 5 5 Escapability | 1 5 2 5 4 5 5 4 1 3 5 Deniability | 1 5 2 3 3 1 3 2 1 2 3 Anonymity | 1 3 3 2 5 3 4 3 1 2 4 Emotiveness | 5 1 4 1 3 1 1 2 1 2 1 Portability | 5 1 4 1 3 1 1 2 5 5 1 --------------+-------------------------------------------------------- Total | 48 38 51 36 50 35 38 45 43 50 45 

We can see that the highest-ranking platforms, in my completely unscientific study are: phone (51 out of a possible 70), chat (50), and mobile text (50 as well). No single product does everything well, and people need to mix and match platforms to get some sanity into their world.

We can also see how different platforms suit different types of discussion. For example if we are negotiating a contract with a hostile or at least untrusted partner, we need high transience and deniability, so that we can negotiate freely, and touchy-feely so that we can use psychology and empathy, but when it comes to getting a real promise we need low deniability and high escapability. So, we need to meet face to face and get the words down on paper.

Take another instance, problem solving. This needs emotiveness, low unit cost, high latency, informality. Nothing else counts. The ideal platforms are thus chat, phone, and face to face. Don't try to solve problems by exchanging PDFs. It will not work, as the Napoleonic court system of written argumentation demonstrates painfully.

You'd think this was obvious, but people still try to use verbal contracts, or negotiate subtle arrangements by email. There are good reasons these don't work, not because people are stupid or liars (it is a temptingly accurate but still not sufficient explanation), but because the communications platforms are just not the right ones to get around the stupidity and dishonesty that is a vital part of every statement we ever make (especially when we think we are being clever and honest).

Don't take my figures too seriously - for one thing, there are many criteria that I've forgotten or ignored. For example - don't laugh - the ability to choose an item in accessorisable colours is a huge selling point for half the consuming public. For teenagers with no fixed place of work (and in most countries, no job), criteria such as portability totally outweigh any others, so a platform like SMS scores relatively low because we are generalising to an entire population. Ironically, those with the greatest need for mobile products are those with the least money.

Messing it Up

Surprisingly, people are often remarkably slow to use the "right" way of communicating. One of my great frustrations when I help a team to work together is to see people stubbornly trying to conduct meetings or solve problems or design specifications by email. (The right platforms would be face-to-face or chat, chat or face-to-face or phone, and paper or web). Don't get me started on phone conferences…

There are two main reasons people don't use the right platform in every situation:

1. The "hammer" syndrome. When you know and like one platform you try to use it everywhere. Despite what you see in the movies, email does not work as a tool for seduction, unless you have William Sheakspeare's writing skills.
2. The cost of alternatives. It is easy to say, "let's meet" but when the other party is in Los Angeles, and you are at number thirteen Oxford Street, London WC 1, it's not so easy.

Social Economics

Many people do not realise that humans organise themselves into structures that are predictable, highly formalised, negotiated, and ultimately defined by behavioural patterns that are embedded in our genetic code.

Whole political religions have been based on ignoring this fact, and worse, on trying to put people into structures that we can honestly call "unnatural". In the worse cases, the attempt to re-educate people into artificial organisational structures has resulted in famine, genocide, and destruction of entire swathes of societies.

People organise into structures that are driven by some basic economic principles:

1. All wealth is created by specialisation and by trade between groups that have specialised in their particular areas.
2. Trade is the exchange of goods, services, information, or knowledge, and wealth is the accumulation of goods, credit, information, or knowledge.
3. The scope of trade is defined by available transports: sea, rail, road for material goods; communications platforms for information and knowledge.
4. Fair trade requires accounting, rules, and a neutral but strong authority to impose these.
5. Trade creates changes in wealth distribution that act to mix society.

The flux of families, towns, regions, and societies is driven by these factors, in cycles that are entirely chaotic, in the mathematical sense of the word. If we want to understand human history we can add two more principles:

1. Over time, authority always becomes corrupt so that a society must revolt, stagnate and die, or go to war.
2. Human activity eventually exhausts resources (water, trees, fish, minerals) so that a society must adapt, move, or collapse.

To a large extent the human mind has evolved a set of tools that are capable of creating very large organisational machines through the dual processes of specialisation and trade. To a large extent these machines are competitive, collaborative, destructive, aggressive, and in many cases, downright insane. But it's a great party.

Pink Fax Machines

The original and most striking specialisation in the human genome is the ancient split into female and male. Anyone who honestly claims that men and women are the same, bar the pressures and lessons of society, should re-read my sentence on stupidity and dishonesty. Equality is one thing we all should fight for but difference is what makes us successful, and interesting.

Understand the way we think, as men and as women, as young and old, and you understand the reason different people prefer different types of communications.

Let's look at some basic differences. These apply generally, not universally:

• Men prefer to command, to speak to many at once, to be anonymous and talk to strangers. Men generally use emotions less than women but are happier to travel to distant places and learn new technology in order to communicate.
• Women prefer to discuss and exchange information, to speak to other woman individually, and to know exactly who they are speaking to. Women do not like to learn new technology unless it's obviously useful.

Looking at the criteria for communications platforms, we can group criteria into those that are generally more important for women, those that are generally more important for men, and those we can safely assume are valuable to both genders:

• Accessibility, informality, simplicity, security, transience, and emotiveness are more important for women.
• Fanout, achivability, escapability, deniability, and anonymity are more important to men.

Unit cost, latency and portability are important to everyone, so we can take our criteria and remove those that are not important, giving us a gender-biased calculation of "favourite platform".

For men, we get this table:

 . | F2f Paper Phone Fax Chat Web Etext Email Sms Mtext Wiki --------------+-------------------------------------------------------- Accessibility | 0 0 0 0 0 0 0 0 0 0 0 Unit cost | 5 2 4 3 5 2 4 5 3 5 4 Informality | 0 0 0 0 0 0 0 0 0 0 0 Simplicity | 0 0 0 0 0 0 0 0 0 0 0 Latency | 5 1 5 3 5 1 2 4 5 5 3 Fanout | 3 3 1 3 1 5 5 4 1 5 5 Security | 0 0 0 0 0 0 0 0 0 0 0 Transience | 0 0 0 0 0 0 0 0 0 0 0 Archivability | 1 2 1 2 2 5 5 5 1 5 5 Escapability | 1 5 2 5 4 5 5 4 1 3 5 Deniability | 1 5 2 3 3 1 3 2 1 2 3 Anonymity | 1 3 3 2 5 3 4 3 1 2 4 Emotiveness | 0 0 0 0 0 0 0 0 0 0 0 Portability | 5 5 5 1 1 1 1 1 5 5 1 --------------+-------------------------------------------------------- 22 26 23 22 26 23 29 28 18 32 30 

And for women, we get this table:

 . | F2f Paper Phone Fax Chat Web Etext Email Sms Mtext Wiki --------------+-------------------------------------------------------- Accessibility | 1 2 5 4 4 5 3 4 5 3 5 Unit cost | 5 2 4 3 5 2 4 5 3 5 4 Informality | 5 1 5 2 5 2 2 3 5 4 3 Simplicity | 5 3 4 2 3 1 1 3 5 3 3 Latency | 5 1 5 3 5 1 2 4 5 5 3 Fanout | 0 0 0 0 0 0 0 0 0 0 0 Security | 5 4 5 3 5 2 1 3 5 4 1 Transience | 5 1 5 2 4 1 1 2 4 2 2 Archivability | 0 0 0 0 0 0 0 0 0 0 0 Escapability | 0 0 0 0 0 0 0 0 0 0 0 Deniability | 0 0 0 0 0 0 0 0 0 0 0 Anonymity | 0 0 0 0 0 0 0 0 0 0 0 Emotiveness | 5 1 4 1 3 1 1 2 1 2 1 Portability | 5 5 5 1 1 1 1 1 5 5 1 --------------+-------------------------------------------------------- 41 20 42 21 35 16 16 27 38 33 23 

So, I conclude that men will prefer mobile text, wikis, and electronic texts above all, while women will prefer phone, face-to-face, and SMS.

Again, don't take my opinions too seriously (I certainly don't). It's just one way of looking at things, and my real point is to demonstrate that platforms are different, for predictable reasons, and these reasons are useful to try to understand when we choose platforms for our own work or for others' work.

Let's get back to organisation. Men and women communicate differently, as we all know, and organise differently. It's not accidental that women have trouble in business structures since these generally use male communication techniques, except for meetings, which most men hate but many women enjoy.

The size of an organisational structure, and its dynamics, depends entirely on the mix of communications platforms that are available to its members.

It thus follows that the most effective organisation is the one with the most accessible and complete mix of communications platforms, so that its members can collaborate in whatever way produces the most specialisation, the most efficient communication, and thus the most wealth.

by pieterh

Introduction

The more you know about something, the harder it is to build it.

We went live with the first production deployment of AMQP (on OpenAMQ) at the end of 2006, handling about half a billion messages a day. It was a massive project: design a new protocol, build an industrial-strength implementation, and migrate a major application onto this new infrastructure. Two and a half years sounds a lot but in that time we made three complete redesigns of AMQP, and OpenAMQ, before we had designs that were simple enough to work reliably, and still do what was needed. Functional simplicity is the hardest aspect of design.

It's been hard to translate that success into a final AMQP specification. Part of the problem was failure to agree on what "final" meant. Part of the problem is that AMQP addresses a large problem that cannot be solved quickly. Part of the problem is that the people working on AMQP - including myself - were still building the tools and experience needed.

AMQP will not, in my opinion, be solved in one or two steps, nor in one or two years. The work that has been done already over four years is very important. It has already created a healthy market of competing, interoperable, messaging products. The AMQP/0.9.1 specification, which was published last year, raises the bar: it is lean, precise, coherent, and tested.

But AMQP/0.9.1 is not the final destination. As a contract of interoperability and stability, it's excellent, almost perfect. But it has a limited view of the problem, and it is still more complex than it should be.

The thing about infrastructure is that once it is in place, it is horribly expensive to fix. The complexity in AMQP/0.9.1, if not solved, will be multiplied many times when AMQP expands to cover the fully-reliable messaging, low latency, multicast, and other features we expect to see in AMQP/1.0.

Complexity is easy, simplicity is hard. My day job is to design simplicity by identifying and removing complexity. In this article I'll do that to AMQP, and identify ten areas where there is unnecessary, and in the long term dangerous, complexity, and in each case I'll provide recommendations for solving that complexity.

My ten principles for AMQP are:

1. Make small pieces: AMQP should be a fabric of protocols, not one large protocol.
2. A layered architecture: these protocols should form a clearly layered architecture.
3. Delegate the design work: each protocol is a job for a small, competent team.
4. Leverage the community: open the design process to anyone who wants to participate.
5. Leverage competition: allow competing teams to make competing designs, at any level.
6. Leverage the market: use adoption, over time, to identify which designs are best.
7. Use natural syntax: the framing at each level should fit the needs of that level.
8. Use natural semantics: the interactions between peers at each level should be simple.
9. Push blame to the edges: brokers should never compensate for poorly-written applications.
10. Deconstruct the broker: the architecture should support zero, one, or more brokers.

Since theory without practice is useless, the RestMS project acts as a demonstrator of most of these principles.

Make small pieces

AMQP is on paper a single protocol¹. This makes it easier to market. It makes it much harder to improve, and the 0-10 specification shows this: it's 197 pages, compared to about 40 for AMQP/0.9.1. I've recommended since 2006 that we refactor AMQP into smaller pieces, each solvable by a small team of 1-3 people. For example, separate protocols for version negotiation, for control commands, for message transfer, for transactions. These pieces need freedom to evolve independently, so that experimentation can be cleanly (contractually) separated from standardization.

A layered architecture

AMQP has no clearly layered architecture. The refactoring of AMQP into small protocols must be based around a solid, coherent, layered architecture. This should be the core of what AMQP 'is', the fundamental agreement around which detailed designs can be built. The architecture should specify what each layer does, in terms of its interactions or interfaces with higher and lower layers. The goal of the architecture is to compartementalize innovation, to guarantee space for contributions, and to ensure the overall coherence of AMQP. The layered architecture is what defines the major version of the protocol.

Delegate the design work

AMQP is specified by a tiny, exclusve team. When a complex problem is broken into pieces, shaped by a clear architecture, it becomes possible for teams to work independently and asynchronously, on different pieces. This is the only way to solve large problems: break them down and let people specialize in different areas. The AMQP process is today a technical process. It should become an administrative one, defining frameworks for collaboration, and approving specifications when they emerge and have been proved.

Leverage the community

The AMQP process excludes expert user contributors. Given that messaging users are often highly competent engineers who at the least can act as "competent clients" for designers, this is a problem. There are only two justifications for excluding participation. One: it's legally complex. Two: it's technically unsafe. Both these can be solved, by defining suitable legal frameworks and by breaking AMQP into small pieces that anyone can experiment at implementing.

Leverage competition

AMQP aims to get one authoritative design. This assumes that there are no commercial rivalries, but there clearly are. Rather than suppress these, we can use them constructively by allowing more than one answer to any given problem. If one team proposes a way of delivering messages, and another team thinks it can make a better way, it should be allowed to. Perhaps the two teams can agree, but if they don't we should allow two answers to sit on the table at once. This enables competition, so better designs can emerge. It also forces layers to be more orthogonal, and validates the overall architecture.

Leverage the market

AMQP is shielded from market opinion. If users prefer a particular API, for example, that does not feed back into the AMQP process. But it should: market opinion, especially with open source products, is a very valuable indicator of quality. When it comes to protocol designs, quality valuation by the market can be measured by the number of independent open source implementations, and the number of users of such implementations. Rather than try to make quality decisions up-front, it is simpler to use a green-brown model where experimentation allows multiple "green" specifications that are pruned over time to leave a small number, or one, brown specification.

Use the natural syntax

AMQP uses binary framing for control commands ("Queue.Declare") as well as message transfer ("Basic.Deliver"). This is not ideal. Each piece should use the natural syntax. Command protocols should use text, not binary framing. This is how every successful IETF command protocol works. Text framing makes backwards compatibility trivial. There are no performance issues with command protocols, so parsing is not an issue. Text framing is easy to understand, easy to implement. Message transfer protocols should use binary framing, and can be significantly simpler than AMQP's current framing: smaller envelopes, no verbs, no channels, no dynamic addressing.

Use natural semantics

AMQP currently mixes asynchronous and synchronous conversations in one protocol. It makes the combined protocol complex, and it makes error handling tricky. When we split control commands and message transfer into two separate protocols we can make each simple. Control commands work best with synchronous, pessimistic dialogues: each request gets a success/failure response. Message transfer works best with asynchronous, optimistic dialogues: messages are sent with no confirmation, and reliability is layered on top, as higher level protocols based on acknowledgements and retransmissions, transactions, and so on. Messages can be batched, commands should not.

Push blame to the edges

AMQP currently pushes problems upstream. Application private queues are held on the broker. Slow consumers cause these queues to back-up, and probably the biggest issue for production use is that servers run out of memory, and crash. Applying flow-control to publishers is the wrong solution. Messages should be pushed without pity to the edges, and if these are too slow, that should count as a defect in the edge application, and handled locally: drop old or new messages, or raise a fatal error. Problems at the edge should not be allowed to move upstream, period.

Deconstruct the broker

Currently, AMQP considers a broker to be a mainframe: big, important, central. This is certainly one use case but it is a very bad design for high data volumes: one big central bottleneck. It is wiser to treat the broker as a host for arbitrary queues and routers, and to accept that in many cases, such applications optimally reside at the edges, not a central point. For example, holding private queues centrally makes brokers fragile, while pushing private queues to the edges makes brokers more robust (by pushing blame to the edges). Similarly, routing can be done at the publisher edge. Only the Wolfpack pattern (one-to-one-of-many) absolutely remands a central broker.

Conclusions

The AMQP protocol and process are complex in ways that make it difficult to build a Version 1.0 and which in the long run may affect the success of the AMQP project itself. I've outlined ten principles that will in my view solve the complexity. Most of these proposals are years old and have been made many times to the AMQP working group.

by pieterh

We came to the fourth annual AMQP face-to-face, and the first one with a public conference. The organization by Matthew Arrot, Rita Bauer and the rest of his team was impeccable. You'll see from the first photo set, the USCD location was breath taking and we were spoilt with wonderful lunches, views, weather.

AMQP'09 was a unique event in many ways but mostly because it brought together such a diverse group of people, both an expanding AMQP work group, but also the core of a community of expert users. One of the weaknesses in the AMQP vision up to now has been that the "end user" has been represented only by large global businesses. At AMQP'09 the public seemed mainly to represent small teams, and their needs form an important part of the mix. iMatix is, of course a small team, but ironically our clients are all huge firms. Nonetheless, I often find the response of someone who has precious little time and money to spend on learning and using a technology more enlightening that that of someone with budgets.

The main reason for the public session on Day 1 was to present AMQP/1.0. Rafi and Rob did an excellent job presenting their work. Everyone who came to the event left with their own conclusions. Gone are the exchanges, just as everyone was starting to understand these. At one time on Day 2, the closed group was discussing "XML exchanges" for an hour or so, before I raised my hand to remind: "gentlemen, in 1.0 exchanges are gone." Our mental models take time to come to life, and they die hard.

In place of algorithmic exchanges that feed messages into a population of queues, we will have queues of messages that are scanned by a population of algorithmic links. While the design seems to work for mainstream scenarios, it is not yet clear exactly what this fixes from the 0.9.1 design. We lack a clear explanation of what is broken in 0.9.1, what the possible solutions are, and why the design chosen for 1.0 is the best solution. The argumentation, as one would say.

The argumentation is going to be needed, as more and more people invest in 0.9.1, and will continue to do so over the next year or so until AMQP/1.0 is mature enough to be supported by working products.

Some teams have exploited the exchange-binding-queue model elegantly: FastMQ's ZeroMQ does brokerless messaging by moving the exchange and binding to one peer, and the queue to another. It's unclear how this can work using the new queue-link model.

The difference between the old and the new models is, as far as I can see that the old model is easier to visualize as a flow of messages through exchanges into queues. The new model feels more like pointer manipulation. Maybe more efficient, but also more abstract. Given that it took several years to explain the old AMQP model (it is always totally obvious after six months of reflection), the extra effort needed to understand the new model may prove to be a problem.

People who are used to AMQP/0.9.1 should try to recall how they felt when first reading the exchange-binding-queue explanations. It was obvious, then it was not, and then after much re-reading, it became obvious again. Perhaps this was just alien technology. Or maybe these models are really hard work to understand, even (especially) when they are simple.

My in-progress judgement on AMQP/1.0 is that it looks good, but it is unclear why the changes were needed, and why an evolutionary approach was not possible. I'd have liked to have seen experimentation with the new model, using the existing AMQP/0.9.1 transport, for example. After all, it is trivial to define new classes in AMQP to work with a new set of broker-side objects. Or, I'd like to see the new proposed transport layer running with the existing AMQP/0.9.1 model. Bringing out a completely new AMQP model running on a completely new AMQP transport layer forces all implementers and users - brokers, clients and applications - to throw away their old code and move to new code, in one step. This seems designed to fail.

Having said this, if a new AMQP model is needed, or if a new AMQP transport is needed, we should make them. AMQP is still young enough to absorb this kind of change and if there were serious flaws with the exchange-binding-queue model, it's exactly the early adopters who will understand the need for redesign.

However: such change must be managed and its downstream impact must be minimized. It should be introduced in phases so that the new designs can be matured and validated by real use and competing implementations.

On the afternoon of Day 2 the group split into separate tracks, one to discuss technology and the other to discuss process, and legal aspects. Patents. It's a subject that causes distress because at their core patents (due to their cost and the risk they bring) are a game for large firms only. It is one of the few areas where large and small firms play in the same field, and whenever patents enter the picture the field tilts inexorably away from the smaller teams.

And like I said, it's the smaller teams that are key IMO to AMQP's success.

I'd like to report that the working group came to a firm decision to keep patents far away from AMQP but unfortunately this is not what happened. There were no decisions as such. There was no real criticism of Red Hat's act of patenting that famous XML exchange. Someone joked that one benefit of dropping exchanges in 1.0 was that this patent would become irrelevant.

However, standards committees are slow machines and the AMQP working group moves faster than most. We did make excellent progress on the idea that AMQP was more than one document, and consisted of a sort of ecosystem of specifications. We already have this, informally. The ecosystem is inevitable because AMQP is not a 100% solution. It needs supporting specs in all directions. Either vendors make these privately, with a chance of patenting, or they make them publicly, with a unilateral patent license such as the wiki.amqp.org site uses.

One can never stop a firm from patenting some extension to AMQP. But one can encourage firms to publish their specs in a form that precludes this. Not publishing the spec for functionality then becomes a clear statement of intent. Patents are an issue. but transparency around exactly what is safe goes a long way to solving that issue.

For a firm like iMatix, whether or not we invest in AMQP/1.0 depends very much on whether the AMQP process expands to include such an ecology of open, unilateral specifications. A big jump to new designs is one thing. A jump into a space that may have patent ambushes, quite another. If we're vocally sensitive to this issue it's because we've been hit by it before. In 2005, a meagre patent claim by Belgian firm AllisBlue caused us to exit the mobile applications market and abandon about Euro 250,000 and three years of investment into a mobile apps platform called SMS@. Any patenting activity around AMQP, no matter what the stated intentions are, is a threat to implementer and users of AMQP, and will keep smaller companies away from the table.

I'll continue to blog on this issue, and the progress (or lack of it) that the AMQP working group makes, over the next months and years.

On Day 2… well, on Friday I decided to skip class and head for the desert. The 9-hour jet lag and committee discussions on software patents began to turn my brain to mush. The state desert park of Anza-Borrego, the largest park in California, is just two hours' drive north-west from San Diego. A stretch of highway, and then a long windy trail over mountains and valleys, until one comes to this massive area surrounded by mountains.

When one drives in the desert, one uses an all-wheel drive and GPS. Halfway into Anza-Borrego my GPS died. My heart froze. GPS, while giving a godlike wisdom of where to turn, also makes one stupid. A paper map would make a good backup. In any case I had a gallon of water (89c at Walmart leaving San Diego) and stout shoes. Then I saw a small 'reset' hole on the GPS. Found tiny twig, inserted, pressed, and yay! GPS rebooted and I was safe again.

The hot and cold deserts, along with impenetrable rain forests, are those rare parts of the dry surface of the planet that have not been moulded by human activity. Some roads cross Anza-Borrego but mostly it's moulded only by nature: rising crusts, erosion from water and wind. The plain is hot and dry. The plants are all thorns and bristles. Many were in flower. The beauty of the desert flower is accentuated by its rarity in the dry landscape.

As one walks the hills, one is wafted by the scents of flowers and plants. One particularly sweet smell, a cross between sage and lavender, I traced to a small pine shrub which just had particularly fragrant leaves.

The air rustles with the noises of plants in the wind and the buzz of invisible insects. The ground is sandy where it's flat, rocky on the slopes. There are burrows of mice, or snakes. I saw neither. There are palms where the locals pump water up from the water bed. In places it comes up by itself, and springs dot the park. In the hills you see the work of flash floods, which carve out gullies in the rock. The desert comes alive at night but as the sun set I decided to find a place with a real bed.

I filled up and overnighted in a motel in Borrego Springs, a town of retirees with legs muscled from climbing the desert paths. Like the sheep the area is named after. I did not see any Borrego sheep, they hid well.

The empty wilderness contrasted with the utterly refined beaches of La Jolla (which, Rainey explained, is pronounced "la Hoya"). Neither felt anything like home, which is presently a calm street in the old canal zone of Brussels.

But one thing I realized: AMQP, like any city, and unlike the desert plains and mountains of Anza-Borrego, thrive or fail according to the diversity and energy of the people that inhabit it. It's people and openness, not technology, that make standards a success. And bringing together such a diverse and energetic group of people, the AMQP 2009 event smelled of success.

by pieterh

Brian welcomed Microsoft to the AMQP working group:

Just as Red Hat has been adding native AMQP support into the Linux platform and ecosystem at Fedora and through Red Hat Enterprise MRG, Microsoft is bringing AMQP support to Windows and its ecosystem. Between Linux and Windows, AMQP will become a standard messaging facility on the vast majority of operating systems and server platforms.

And he listed the Working group members, omitting the other implementers (Rabbit Technologies and iMatix):

The AMQP working group already has a well-esteemed set of members, ranging from software vendors like Red Hat to hardware vendors like Cisco to end-users like JP Morgan Chase, Goldman Sachs, Credit Suisse, and Deutsche Börse (see the full list of participants at http://amqp.org).

Ironically, given the later exposure of a stealth patent very close to AMQP by Red Hat, he wrote:

So, there is no threat of Microsoft holding the AMQP standard hostage via patent threats.

So much for Red Hat's public statements. But what do they say behind doors, to their clients, to journalists, and to partners? Well, Brian has been helpful in pointing us to a position paper he wrote for an Indiana University workshop on Cyberinfrastructure Software Sustainability on March 26 & 27th 2009.

In that paper, which I've copied here in case it disappears from that site, Brian explains Red Hat's true position:

Red Hat teamed up with one of its customers, JP Morgan Chase (JPMC), to create an open protocol standard around messaging, AMQP.

This is strange news. iMatix was working closely with JPMC in 2004-6 when we (iMatix and JPMC) designed AMQP, and we never saw anyone from Red Hat until they reviewed the specifications and helped create the AMQP workgroup. And to the best of my knowledge JPMC was not a Red Hat customer except for Linux licenses.

Maybe Brian just meant to type "iMatix" and wrote "Red Hat" out of habit instead. This happens. I do it all the time. But sadly there is more…

JPMC, like many other banks, had developed its own messaging software to meet its high-end messaging requirements. However, JPMC had also written down the specification of its work, and this proved to be a good starting point for creating an open messaging protocol standard. Red Hat and JPMC created a legal contract to form the AMQP working group, which would develop this new standard as AMQP in an open and IP-unencumbered manner. Then, they started bringing in many additional companies to collaborate in this working group.

I will now state this for the record. iMatix was hired by JPMC in 2004 specifically to (a) develop AMQP, (b) develop an open source implementation, and (c) migrate JPMC's largest investment bank application onto this new stack. It was a tough triple project done on brutal deadlines with a deep emotional cost. To demonstrate how much iMatix invested in this project: my wife was pregnant with our second baby, which died in utero at eight months¹. While she was in hospital in Brussels, I was at JPMC in London with my team, where we had spent most of the previous months, making things work.

We took this work utterly seriously and in 2006, we were ready with a very solid specification that is essentially unchanged - many fixes, and unused parts removed - from the AMQP/0.9.1 specification that is used by most AMQP users today. Red Hat received this specification and their major contribution to it, infamous as an example of incompetent protocol 'design' pushed through by bluster and force, was the AMQP/0.9-work-in-progress specification. Read it, and weep, if you have the courage.

It is quite trivial to check that iMatix registered the AMQP IANA port (5672), and the amqp.org, amqp.com, and amqp.net domain names.

Brian's text continues to celebrate the "Linux to Windows interoperability" that seems to echo through Red Hat's approach to software patents over the years.

But back to Red Hat's mythical invention of AMQP. Maybe Brian Che is just ignorant. Let's roll with that for a second. All this specification and accuracy stuff is complex, after all. But in 2007 - yes, eighteen months ago - I pointed the AMQP work group to a Red Hat news article which said,

Indeed, Red Hat's APQM[sic] began its life as proprietary messaging software at financial services giant JP Morgan Chase, said Red Hat Chief Technology Officer Brian Stevens in an interview here during Oracle OpenWorld.

Different Brian, same set of lies. The AMQP work group asked Red Hat for an explanation for this article. "The journalist got it wrong", we were told. We were invited to speak to a Red Hat VP if we wanted clarification.

But I think Brian Che has given us all the clarification we need.

by pieterh

The computer is an Asus Eee 1000 portable, with an extended battery. The software is Linux, wrapped up as Eeebuntu. Here is a screenshot showing the power consumption at 6.4W, and battery life of 13 hours.

I bought the Eee 1000 at J&R's in New York this month, list price $375 plus sales tax. The machine is a top-of-the-line Eee model, with all the features that are sometimes dropped off cheaper Eee's like the 1000HD. The main reason I bought this machine was that it did not have Windows, and came with a 40GB SSD instead of a cheaper, more fragile moving hard drive. At this price, the SSD is a must-have for a portable computer: 25% of portable computers die because they are dropped.

The battery is a "13000mAh" thing off Ebay, posted from Hong Kong and costing $75. In fact it only holds about 10500mAh. They offered to replace it, but 10500mAh is still fine. It's a large "hammer-head" battery but it fits the Eee 1000 neatly and acts as a prop. Check Ebay to see what these batteries look like and cost.

Xandros… ah, Xandros. I used to use this when I first switched to Linux for my desktops, mainly because Xandros used to be Corel Linux and they were one of the first firms to really invest in Linux in the late 1990's, so they kind of earned a soft spot in my heart. Xandros also has a file manager that worked really well, especially with Windows file shares. Today, Gnome and KDE have caught up.

On the Eee 1000, Xandros does two things very well. It boots and suspends/restores incredibly quickly. Boot time is about 15 seconds. Second, it does something magic with the network card, picking up dozens of networks where other Linuxes pick up a handful.

But for a serious user, Xandros sucks. Their repositories are old, and since the main reason for using Linux is the vast amount of great free software, this is really a pain. Xandros works for my five-year old daughter. It does not work for me.

So I tried a variety of EEE-friendly Linux distros. It's easy: download an ISO and use unetbootin to burn to a 2GB USB stick, then reboot and press Esc to get the choice of booting from the USB stick. I tried several distros, with unhappy results. [www.mandriva.coom Mandriva] (supposed to work well with the wifi) did not boot properly. Easy-Peasy, which used to be called Ubuntu-eee and which I've used before, installed fine but just acted weird (like running the installer each time it booted), and showing a horrid green pixelated "Easy Peasy" logo on the desktop. Besides, the name really does not work for me. Then I tried Eeebuntu and this is what I'm now using.

There is a bunch of stuff I had to do before the system was what I'd consider properly set-up. None of these are vital but they all make it better:

• use manual disk partitioning so the system goes on the 8GB SSD and /home goes on the 32GB SSD (the 1000 in fact has two solid-state disks).
• tweak the file systems to reduce SSD write accesses as explained in this blog posting by Ter.
• fix the "software sources" to include the normal Ubunbtu repositories
• install Kate (better than Gedit), Gimp, and a bunch of other good tools
• install powertop
• go into gconf-editor and fix the top panel to auto-hide to 0 pixels, with no animation, and with shorter (200msec) delays to appear and hide.
• remove the bottom panel

Now, the Linux works both as a netbook remix, with the easy access desktop, and a proper full Ubuntu, with the panel menu.

Eeebunbtu uses the array.org kernels and has a decent driver for the RaLink 0781 wifi card. Now, here is where I start to dislike Xandros. They have closed drivers for the RaLink card that use undocumented features. Presumably money changed hands. All very well, but when Xandros get 99% of their software for free, it is unethical to try to get a competitive advantage by exploiting secret knowledge. RaLink do provide open source drivers but they are buggy and imperfect. In the end, this is bad for Asus: users have a choice between three imperfect worlds: Windows XP with its viruses and malwares, Xandros, with its childish limitations, and Linux with its weak wifi drivers.

Asus: please use only hardware that is fully documented and fully supported in Linux.

However, the wifi works acceptably and better than some other portables. Now, to get that 13 hours of battery, I switch off the wifi, dim the screen, and exit Firefox. When I'm on the road, I'm mostly writing, so I need my text editor (Kate).

Unbelievably, the 1000 draws only 6.4 Watts in this state. For comparison, my Thinkpad X60 draws 9.8W (and gets 7 hours from its battery) and the Eee 1000HD draws 11.5W (and gets around 6 hours from the hammerhead battery).

Highly recommended.

by pieterh

From the Devil's Wiki:

US "democracy" is not an intellectual exercise. Each team attempts to get the highest score, the winner gets control over the world's most powerful army and secret services for four years, and the right to steal as many dollars as they can transfer to off-shore bank accounts through fraudulent contracts with their friends' companies.

Let's start with the how. How on earth did someone with a clear and major handicap win the top job? Forget Obama's obvious assets, let's ask how a "black" man won. I'm putting that in quotes because like my kids, half European and half African, Obama is no more "black" than he is "white". As if humans were not all mongrels.

But I remember many American friends telling me that no matter how good a candidate Obama was, he could not beat the deep racial barriers that divide their country. "In Ohio, sixty percent of voters say they'll vote Democrat", one of my friends told me in August, "but only forty percent would vote for Obama."

For me, the key sign that Obama was going to win was his fund raising. Raising money needs a machine. This is the machine that propels - or mostly, fails to - individuals into high office, and keeps them there, protecting them from endless attack like an immune system.

Obama clearly stood out of the other Democrat candidates with a machine that was orders of magnitude more powerful. More powerful than the Clintons, who's machine is sixteen years' mature. Unusual and, I believe, significant.

As we saw, Obama's machine then made silly putty of the Republicans' monster, a machine that has had eight years and untold billions to grow into every branch of government: the courts, the intelligence services, the military, every single federal agency down to local school boards was (and still is) part of the Republican party machine. And Obama beat this machine like the Karate Kid carefully putting down a bully.

Charisma? McCain and Palin? A great slogan? Yes, and no. These were all part of the result but not really the point. There's a reason why there was not widespread vote fraud in 2008. Every time there was a "glitch", lawyers paid by the Democrats descended and began to take notes. Compare this to 2000 and 2004, when painfully obvious frauds stole the White House.

Not just a pretty face, Obama represents the most powerful political machine we've ever seen, any place, any time. The Democratic party puts to shame the old Russian and Chinese Communist parties, the European and Japanese fascists, and every ideology ever invented by a tinpot dictator desperate to hold onto power at any cost.

And yet I don't see the Democrats going on genocidal wars, nor in fact doing anything particularly stupid except perhaps arguing a lot over petty details, while agreeing on and executing the fundamentals with unparalleled speed, accuracy, and efficiency.

It is, in my view, the End of Politics.

In fact, Obama was not elected by the traditional Democratic party. That party wanted a Clinton, Obama was elected by the young Democrats, who represent the majority of educated Internet users with any interest in politics. And who were angry with Bush the first time, angry with him the second time, furious when Bush sided with every business and lobbyist who hates the Internet, from the old entertainment industries, old telecoms, to old software.

Note that I don't say "majority of educated US Internet users". Obama has 4M supporters on Facebook. I bet 50% of them are not Americans.

I repeat, it's the End of Politics.

In the Economics of Evil I explained how online communities use a Bad Guy to grow and prosper. Why Wikipedia exists because of, not despite, the trolls and vandals. Thus the machine that selected and then elected Obama exists thanks to eight years of Republican misrule and abuse of power that felt, and perhaps was, a small step away from a slide towards totalitarianism.

I think that Bush actually hastened the End of Politics by ten years. I've been studying the growing political power of the Internet society for some years but I had not expect to see it crystallize so soon. That eight years of anger and frustration turned into something powerful, determined, and well organized, as online communities tend to be.

Now, how does this translate into the End of Politics, and what does this imply?

The Republicans, by being extreme in their hostility to science, truth, and society, promoted the growth of the machine that would crush them. But the Republican machine is not yet dead. It has roots in local and regional politics across America, it has lots of money - stolen from the public purse through endless insider contracts - and it has time. The game is now Obama's to lose.

So Obama needs his machine and he needs to feed it, invest in it, and protect it from attack. It's not enough to give executive orders when at every level, people can subvert, sabotage, and strangle the progress he wants to make. Obama needs what he had in the elections: an army of volunteers who jump onto every problem and "solve it", as the Internet tends to solve problems like Scientology and regressive copyright.

There is, most definitely, a large and powerful section of US society that hates what Obama represents, and is determined to fight it. If it cannot fight from the top, it will fight from the bottom and sides. An insurgency, perhaps, though with different kinds of bombs. We are going to see eight years more of conflict, and this fight will be definitive. We are, I think, fighting for our very soul as a global society. On the one hand, we have the Bush doctrine of "Power exists to bully those I consider my enemies". On the other hand, we have the Obama doctrine of "We". The contrast could not be greater.

This fight is absolutely necessary, and it is obvious that the Democrats will triumph and Obama will serve his eight years in extraordinary style and accomplishing extraordinary things. If the dreaded disaster happens, and an extremist - a true terrorist, then - murders him, another person will take his place and do much the same. Because, as I explained, this is not about one man but about a massive popular movement. A movement that coordinates through the web, brings together the best, brightest, most optimistic of our society, and which has global roots.

Obama is not the first black US president. He is, however, the first Internet president, and in order to survive the next eight years he must convert every political instrument in the US, and then beyond, to a form compatible with the Internet. This means transparent, online, participatory, and honest.

The Internet knows no politics except as a response to external threats. There are no Internet political parties, no Blues and Greens. The most extreme we get is Digg vs. Fark, KDE vs. Gnome, and that ends with everyone stealing everyone else's ideas and competing like mad until it all looks fantastic, and all looks the same. On the Internet, your worst competitor is in fact your best friend.

So let's work through this again:

• Obama has an ambitious program to reform the errors of his predecessors.
• He will face subtle but massive resistance at all levels of government.
• To win over this, he must bring all levels of federal government into his machine.
• This means, bringing them online.
• At that point, the politics will be replaced by simple work.

In fact traditional politics works by arbitraging power, like traditional banks work by arbitraging money. Both systems have been rendered obsolete by ever-cheaper information technology. The banks did not go to high-risk leveraged instruments because they were stupid. They did that because they had no choice: every traditional way of making 5% profits (above inflation) has been ground away until they cannot make more than 1%. At this point, a commercial bank has no reason to exist.

(Banks will, I suggest, become fully nationalized arms of the state, providing an essential service but without profit. This process has already started.)

As Obama begins the conversion of the old, closed, systems of political arbitrage, he will have to liberate the Internet from anything that restricts it, anything that presents a threat to it, anything that stops it growing. Because if he does not address threats to the Internet, his enemies will use them to attack his power base.

What are the threats to the Internet? We know them, of course. They are:

• The telecoms cartels, which are strangling wireless communications.
• The music and movie lobbyists, which are pushing for filtering and Internet policing.
• Many governments, especially in the UK, who still see their own citizens as fundamentally dangerous.
• The patent system, which holds weapons of mass destruction against any knowledge economy.
• Certain failing software businesses, which were granted immunity from competition law under Bush.

And so, my predictions are fairly easy. Obama will do the right thing, because he has no choice:

• He will end the War on Terror and turn US foreign policy to peacekeeping, not war making.
• He will prosecute many in the old Bush regime but not Dubya, who has done a deal with the new chief.
• He will take America to join the International Criminal Court.
• He will rapidly get the Federal Trade Commission to re-open dusty dossiers on abuse of monopoly by Microsoft, Big Pharma, and telecoms companies.
• He will address copyright and propose a reform, which will be adopted. Specifically, he will be outspoken on the obligation of businesses to license their work under "socially acceptable" licenses.
• He will address the patent system, and propose a reform, which will be adopted. Specifically, he will be outspoken on the issue of software patents and the high cost of drugs.
• He will explicitly promote the use of, development of, and investment in, free and open source software by the Federal government.
• He will end the War on Drugs and stop federal prosecution of marijuana use, which many states now tolerate.
• He will push the deployment of wired and wireless Internet to every home, with subsidies and competition.
• He will push for reforms in education that give students more freedom through online studying.
• He will push for a fundamental reform of the United Nations and he will invest in that, and he will turn it into something never seen before.
• He will encourage, bribe, and force, other governments to adopt similar policies.

Some of these will happen rapidly, some slowly, some only later. Much of it will be obscured by the coming traumas - unemployment, depression, deflation - so will only be visible after the fact. The impact of these reforms is impossible to overstate but when the global middle classes are finally liberated, and given full voice, the world will, I think, become a place fit for my children to grow up in.

At the end of the day, we will tip into something the opposite of the world as imagined by those who ran it for the last eight years. Their vision was of a world of monsters and Gods, of brutality and darkness, of blood and treasure for those strong enough to seize it. Obama's vision is that of every Internet community: life is scientific, truths are real, idiots are for making fun of, and good Internet connections, everywhere one is, is more important than a large car.

by pieterh

by pieterh

It's all about tradeoffs: weight vs. stamina. If I'm going for a few hours down to the cafe to work, I don't need an all-day battery. If I'm flying to the States, I want enough charge for eight hours of work, including videos if needed.

Carrying extra batteries is fine in theory but swapping batteries in and out is a pain. Further, you can only charge them by using them. This means you can't leave that extra battery charging somewhere. Before a trip you have to spend time explicitly recharging it… a pain. And since batteries are different for each notebook, they're expensive and sit around wasted when you change notebooks.

So here is a simpler solution. The power supply, that small black slab that transforms 110v/240v into the 12v the notebook needs, it has an interface. Onto that interface you can plug an extra booster battery, also a small black slab. It charges when the PSU is plugged in, and feeds the 12v output when not. You can get boosters of different capacities - 2.5Ah, 5Ah, 10Ah, etc. Boosters conform to a standard interface so there are many vendors, who actually compete on price and quality.

The nice thing is that you can, when you travel, keep your spare batteries in your bag, plugged into the PSU, so the laptop on your knees remains light. And of course the same boosters can work with any PSU, any laptop.

by pieterh

For a random meeting we decided to stay at this hotel in Bratislava:

This is what my room looked like:

The hotel is a kind of live-in surrealist piece of 3D art where every surface, object, and view is shaped and decorated. It's just above the train station, a half hour walk from the center of town. I'd probably go back there just to check if it was still as weird as before.

by pieterh

Engadget reported today of an Apple "notebook" event on October 14th. Here are my predictions of what we'll see in Apple's new line up. Just hunches:

• Solid-state disk offerings across the board, 64GB up to 256GB on the top of the line notebooks.
• LED backlighting on all models.
• Keyboard backlighting on all models except the lower end models.
• Speaking of "lower end", a relatively cheap and small notebook, not a netbook but close. Thin, white, plastic, $800, 11" screen.
• At least one model with 3G built in and sold in conjunction with mobile operators and iPhone contracts. Can you say "Macbook web access included in contract"?
• Aluminum cases for the high end, cheapo plastic for the proles.
• At least one model with a touch screen, starting the move towards an Apple tablet.
• More case color options. Time to accessorize!
• New Macbook Air with a removable battery and 256Gb SSD but still no second USB port. Rats!

by pieterh

Here is what happened: I have Kubuntu running very nicely on a Thinkpad X61, and I wanted to get a second-hand X40 for traveling. If you don't know these machines, they're worth looking at. Light, tough, and they run for up to 8 hours on one charge if you push the right buttons. Last week at a random meeting I noticed that about half the participants seemed to have the same little notebook. The X40 is the older version of this ultralight model.

I got my second hand X40, and of course Kubuntu installs fine. But I wanted to clone my existing, tweaked notebook. Most explanations assume the drive sizes are the same but my X61 has a 160 Gb drive and the new (old) X40 a 40-gig drive.

So here is how I got a perfect copy onto the smaller machine. You will need a portable USB drive, or a large USB stick.

Make your backups

First, create a tar file of all system files:

 sudo bash cd /[root directory on removable media] tar -czf system-backup.tar.gz --one-file-system /bin /etc /lib /sbin /usr /var 

Second, create a tar file of your home directory

 sudo bash cd /[root directory on removable media] tar -czf home-backup.tar.gz --one-file-system /home/myname 

Prepare the target system

I installed Kubuntu from the same CD I'd used originally. This ensures I have the same Linux kernel version.

Restore the backups

I attached my USB disk (and confirmed KDE's proposal to mount and browse the disk) and opened a shell window. Then I restored the two backups:

 sudo bash cd / tar -xzf [path to removable disk]/system-backup.tar.gz tar -xzf [path to removable disk]/home-backup.tar.gz 

Reboot

That was it. My X40 is now a perfect mirror of the X61, including all configurations, stored passwords, ssh keys, etc.

by pieterh

It's kind of a "thank you" to the residents of the city for putting up with the chaos and traffic all year long.

This week it's started to be great weather and the beach is filled with people of all ages, sizes, colors. Uniquely for a city with large immigrant populations, Brussels seems to mix really well. Not everyone likes the crowds, the smell of roasting fish, merguez, and goat and beef barbecues. Not everyone likes the endless line of little bars, serving every possible kind of drink. Not everyone likes the playgrounds, the beach volleyball, the boules, the long stretches of sand and little trees.

But for those with a little time on a warm evening or relaxed weekend, Bruxelles-les-Bains shows what a relaxed, uncomplicated city Brussels has become, as people of all origins and colors mix happily and without taboos.

All that's missing, for me, is wifi so I could write this blog on the beach.

by pieterh

Now, one of my examples for non-activist people is Wikipedia, which is one of the most explosively successful on-line communities. The question is, who is the bad guy in Wikipedia?

At first sight, Wikipedia is a purely positive thing. People contribute knowledge, discuss, edit articles. True, there are some idiots, trolls, and vandals, but overall the results seem to be great. So great that Wikipedia does better than any traditional expert-based encyclopedia. So, does this mean a positive community can work?

The other day some friends and I were discussing a new project to create a kind of wikipedia-style encyclopedia of art and antique objects. The question came up… what about edit rights? Who can edit? And in the answer, I finally understood why Wikipedia works, and why the bad guy principle is so right.

"Anyone can edit," I proposed. "But what about vandals, idiots, and trolls?" came the question. "Let them edit, it's part of the process. The original authors get annoyed, fix the articles, and get emotionally attached to the whole thing. The more edit wars you have, the more people care, and the stronger the community".

So, lacking a clear external bad guy, those people who make your life miserable when you try to do something useful may, in fact, be exactly what you depend on to get out of bed in the morning. Here's a toast to the trolls!

by pieterh

I've now put them onto a new site, http://www.aliopacto.com. Alio pacto is Latin for "a different way", it was the title of a book I started some years ago. One of the stories on that site is from the book, the others are random things, the products of early mornings and days on trains.

by pieterh

I've been taking photos. Nikon D40X with 18-135mm lens, a fast and efficient tool that produces the quality I was used to in the old days of B&W film SLRs.

But is the "shutter" the best way to take a picture?

Let me introduce the concept of "Continuous Exposure" or CE. A CE camera has no shutter, though it does have a button for taking pictures. CE needs decent but not perfect optics, it needs no optical zoom, can achieve film speeds of several million ISO, can achieve resolutions of many orders of magnitude better than the actual sensor. The one thing CE is not good at is capturing fast motion.

CE works as follows. The sensor is always active and captures an image continuously, sending it to an image buffer for processing. The image buffer has a very high resolution, as much as 250 megapixels. The sensor data is added, in real time, to the image buffer, using algorithms to detect movement and shake. The image buffer is constantly re-centred around the sensor data. As the image buffer aquires sensor data, it refines the detail of areas where there is more data, so that even in an image with very light and dark areas, both will aquire fine detail. As the sensor provides more and more data for an image, the image buffer gets more and more detail, achieving its full resolution after a fraction of a second or in some cases, a second or two. When the image buffer has fully resolved the image, a small 'Ready' indicator shows on the camera, and the user can take the image. A picture taken before the sensor is ready will show less detail in some areas, especially those moving more rapidly.

Why use CE? Mainly, it replaces expensive components with software and processing power. A much cheaper sensor, combined with a large image buffer and efficient processor can produce better photos than the best professional equipment. Optical zooming becomes less important, as sufficient detail can be aquired to make very smooth digital zooms. Low light hand-held photography becomes much easier. CE can be combined with video recording; lower-resolution video, and high quality selected images.

CE does not, of course, exist. The way to make it would be to prove the software algorithms capable of producing high quality images from video streams (which can act as a kind of CE sensor simulation). The algorithms, once designed, would need to be turned into silicon, since general purpose computers are not feasible in low power cameras.

by pieterh