IPv6
Written: 1242822093|%e %B %Y, %H:%M (ipv6 piracy progress)
A prediction of how and when IPv6 will become widely used.
This is what I predict will drive IPv6: the desire by the "criminal majority" to create invisible and untraceable file sharing networks. Efforts to fight copyright infringers depend on the IP address of the person sharing (uploading) files. IPv4 addresses are limited, and easy to trace to at least an ISP, if not an end user. This lets the content industry push for "3-strikes" legislation, as they are doing around the world.
As in any arms race, it's not over just because one side scores a hit, and history tells us that the content industry is typically responsible for technological innovation, through their clumsy lobbying efforts to regulate the Internet into behavior that would protect their distribution channels.
So the question is not whether the file sharers will discover ways to continue their illicit fun and games, but how.
And the answer is, IMO, encrypted Tor networks that emulate IPv6 networks, running over a physical IPv4 network. The real world won't go to IPv4 for a long time, the inertia is almost unmoveable. But emulation is an easy way to run a second real world inside the real real one. So IPv6 will be emulated, and will be pushed by brilliant minds who seriously just want to be able to download the latest episode of Lost.
And thus my prediction: IPv6 will struggle to make any inroads into the Internet as we know it today but it will get into software stacks, into Linux, into browsers, and eventually into network fabric, through the file sharing community and through the actions of the content industry.
Comments: 0, Rating: 0
Communications
Written: 1242594042|%e %B %Y, %H:%M (facebook pink)
Cleaning out an old notebook, I found this article that I wrote in 1999. Surprisingly, it still makes sense, so here, saved from the Delete button of history, is an analysis of Communications, and kind of a prediction of Facebook and Twitter.
Introduction
Collaboration and innovation stem from cheap communications but not all communication platforms are equal. As a manager I often see people struggle with inefficient or inappropriate communication platforms, and I direct them to use more effective platform, usually with good results.
The following section is my own analysis of what the differences are between various communications media, or "platforms", and what impact these differences have on groups. As far as I know this is the first attempt to analyse communications in this way, surprising since it is one of the essentials of our technological society, along with fire, pizza, and fermented starches served by wenches.
Communication Platforms
Let's look at the most commonly used platforms:
- Face-to-face: the most ancient platform, the basis for Greek democracy and when followed by the appropriate shared dinner, evening at the theatre or moving picture, and glass of red Bordeaux at a quite place around the corner, the continuation of our species.
- Written paper: though stone and baked clay could survive going through the washing machine, flattened papyrus stems were more portable, cheaper and ultimately the basis of organised religion, the modern state, and TV guides.
- Telephone: invented by Graham Bell in order to call his mistress, the concept of two cans separated by a very long cat rapidly caught the public imagination, especially when combined with a low, low monthly connection fee and easy-to-pay rate schedule. These days, phones are by definition mobile (at least in my study).
- Fax: invented by the Japanese in the 1970's so they could send pictures of Godzilla more efficiently, fax rapidly took over the business world because email had not yet been invented. Today fax is mainly used by producers of ink cartridges to promote the sales of ink cartridges for fax machines.
- Chat: the Internet version of the telephone, chat has morphed from many forms, including IRC, which was a significant improvement on all its successors. Internet chat is successful because on the Net, any dog can look like a thirteen year old girl.
- Web pages: the Internet version of the papyrus leaf, taking about the same amount of effort to hammer into usable material. The original web site was built for nuclear research but within 24 hours had been defaced by h4ck0rz who loaded it with porn and made $150,250 before the nuclear scientists took control again, and invented the password.
- Electronic documents, which range from simple text files to the elegant suit-and-tie called PDF. PDF is the only popular communication platform that was invented, and more importantly, not royaly screwed up in version 2.0, by a corporation. Graham Bell and his mistress do not count as a corporation.
- Email: still, after all these years, the most popular way to send electronic mail. Email has had many incarnations, from evil Unixy command-line mailers that asked, "Delete all email" on quitting, to point-and-drool web mailers that brought email and advance-fee fraudsters to the masses. Hotmail has paid for several nice villas along the beach of Lagos Island.
- SMS: we never knew that a few lines of text stuffed into the tail end of mobile phone data packets could become such a huge business. SMS produces half of mobile phone companies profits and two thirds of all family disputes when daddy sees the phone bill.
- Mobile mail: SMS chat for adults, epitomised by the Blackberry. Mobile mail combines the advantages of email with the portability of SMS plus full tax deductability as an essential business expense.
- Wiki, the hippie mutant offspring of web pages and web-based email. Someone said, "if we let anyone edit this page, can it be any worse than the web currently is?" and the surprising answer was, "no". Wiki sites are rarely beautiful, but prove the dictum that "many eyes make any problem shallow".
When we communicate, we have different, and sometimes conflicting needs:
- The cost of accessing the platform (which I'll call "accessibility").
- The cost of making a single statement ("unit cost").
- The informality of a statement ("informality").
- The lack of skill needed to use the medium ("simplicity").
- The speed of receiving and responding to a statement ("latency").
- The number of people we can reach at once at reasonable cost ("fanout").
- The reliability of the medium against tampering ("security").
- The likelyhood that our words can be used against us ("transience").
- The ease of storing and searching a dialogue ("archivability").
- The time we can take to respond to a statement ("escapability").
- The freedom we have to pretend we lost a message ("deniability").
- The freedom we have to hide our identity ("anonymity").
- The bandwidth with respect to non-verbal communications ("emotiveness").
- The ability to carry the platform with us ("portability").
Let's see how each of our listed communications platforms scores on these criteria, and let's score each platform from 1 to 5, where 1 is the least useful or desirable, and 5 is the most. We assume that the goal is to communicate with as many people as possible, as cheaply as possible:
. | F2f Paper Phone Fax Chat Web Etext Email Sms Mtext Wiki
--------------+--------------------------------------------------------
Accessibility | 1 2 5 4 4 5 3 4 5 3 5
Unit cost | 5 2 4 3 5 2 4 5 3 5 4
Informality | 5 1 5 2 5 2 2 3 5 4 3
Simplicity | 5 3 4 2 3 1 1 3 5 3 3
Latency | 5 1 5 3 5 1 2 4 5 5 3
Fanout | 3 3 1 3 1 5 5 4 1 5 5
Security | 5 4 5 3 5 2 1 3 5 4 1
Transience | 5 1 5 2 4 1 1 2 4 2 2
Archivability | 1 2 1 2 2 5 5 5 1 5 5
Escapability | 1 5 2 5 4 5 5 4 1 3 5
Deniability | 1 5 2 3 3 1 3 2 1 2 3
Anonymity | 1 3 3 2 5 3 4 3 1 2 4
Emotiveness | 5 1 4 1 3 1 1 2 1 2 1
Portability | 5 1 4 1 3 1 1 2 5 5 1
--------------+--------------------------------------------------------
Total | 48 38 51 36 50 35 38 45 43 50 45
We can see that the highest-ranking platforms, in my completely unscientific study are: phone (51 out of a possible 70), chat (50), and mobile text (50 as well). No single product does everything well, and people need to mix and match platforms to get some sanity into their world.
We can also see how different platforms suit different types of discussion. For example if we are negotiating a contract with a hostile or at least untrusted partner, we need high transience and deniability, so that we can negotiate freely, and touchy-feely so that we can use psychology and empathy, but when it comes to getting a real promise we need low deniability and high escapability. So, we need to meet face to face and get the words down on paper.
Take another instance, problem solving. This needs emotiveness, low unit cost, high latency, informality. Nothing else counts. The ideal platforms are thus chat, phone, and face to face. Don't try to solve problems by exchanging PDFs. It will not work, as the Napoleonic court system of written argumentation demonstrates painfully.
You'd think this was obvious, but people still try to use verbal contracts, or negotiate subtle arrangements by email. There are good reasons these don't work, not because people are stupid or liars (it is a temptingly accurate but still not sufficient explanation), but because the communications platforms are just not the right ones to get around the stupidity and dishonesty that is a vital part of every statement we ever make (especially when we think we are being clever and honest).
Don't take my figures too seriously - for one thing, there are many criteria that I've forgotten or ignored. For example - don't laugh - the ability to choose an item in accessorisable colours is a huge selling point for half the consuming public. For teenagers with no fixed place of work (and in most countries, no job), criteria such as portability totally outweigh any others, so a platform like SMS scores relatively low because we are generalising to an entire population. Ironically, those with the greatest need for mobile products are those with the least money.
Messing it Up
Surprisingly, people are often remarkably slow to use the "right" way of communicating. One of my great frustrations when I help a team to work together is to see people stubbornly trying to conduct meetings or solve problems or design specifications by email. (The right platforms would be face-to-face or chat, chat or face-to-face or phone, and paper or web). Don't get me started on phone conferences…
There are two main reasons people don't use the right platform in every situation:
- The "hammer" syndrome. When you know and like one platform you try to use it everywhere. Despite what you see in the movies, email does not work as a tool for seduction, unless you have William Sheakspeare's writing skills.
- The cost of alternatives. It is easy to say, "let's meet" but when the other party is in Los Angeles, and you are at number thirteen Oxford Street, London WC 1, it's not so easy.
Social Economics
Many people do not realise that humans organise themselves into structures that are predictable, highly formalised, negotiated, and ultimately defined by behavioural patterns that are embedded in our genetic code.
Whole political religions have been based on ignoring this fact, and worse, on trying to put people into structures that we can honestly call "unnatural". In the worse cases, the attempt to re-educate people into artificial organisational structures has resulted in famine, genocide, and destruction of entire swathes of societies.
People organise into structures that are driven by some basic economic principles:
- All wealth is created by specialisation and by trade between groups that have specialised in their particular areas.
- Trade is the exchange of goods, services, information, or knowledge, and wealth is the accumulation of goods, credit, information, or knowledge.
- The scope of trade is defined by available transports: sea, rail, road for material goods; communications platforms for information and knowledge.
- Fair trade requires accounting, rules, and a neutral but strong authority to impose these.
- Trade creates changes in wealth distribution that act to mix society.
The flux of families, towns, regions, and societies is driven by these factors, in cycles that are entirely chaotic, in the mathematical sense of the word. If we want to understand human history we can add two more principles:
- Over time, authority always becomes corrupt so that a society must revolt, stagnate and die, or go to war.
- Human activity eventually exhausts resources (water, trees, fish, minerals) so that a society must adapt, move, or collapse.
To a large extent the human mind has evolved a set of tools that are capable of creating very large organisational machines through the dual processes of specialisation and trade. To a large extent these machines are competitive, collaborative, destructive, aggressive, and in many cases, downright insane. But it's a great party.
Pink Fax Machines
The original and most striking specialisation in the human genome is the ancient split into female and male. Anyone who honestly claims that men and women are the same, bar the pressures and lessons of society, should re-read my sentence on stupidity and dishonesty. Equality is one thing we all should fight for but difference is what makes us successful, and interesting.
Understand the way we think, as men and as women, as young and old, and you understand the reason different people prefer different types of communications.
Let's look at some basic differences. These apply generally, not universally:
- Men prefer to command, to speak to many at once, to be anonymous and talk to strangers. Men generally use emotions less than women but are happier to travel to distant places and learn new technology in order to communicate.
- Women prefer to discuss and exchange information, to speak to other woman individually, and to know exactly who they are speaking to. Women do not like to learn new technology unless it's obviously useful.
Looking at the criteria for communications platforms, we can group criteria into those that are generally more important for women, those that are generally more important for men, and those we can safely assume are valuable to both genders:
- Accessibility, informality, simplicity, security, transience, and emotiveness are more important for women.
- Fanout, achivability, escapability, deniability, and anonymity are more important to men.
Unit cost, latency and portability are important to everyone, so we can take our criteria and remove those that are not important, giving us a gender-biased calculation of "favourite platform".
For men, we get this table:
. | F2f Paper Phone Fax Chat Web Etext Email Sms Mtext Wiki
--------------+--------------------------------------------------------
Accessibility | 0 0 0 0 0 0 0 0 0 0 0
Unit cost | 5 2 4 3 5 2 4 5 3 5 4
Informality | 0 0 0 0 0 0 0 0 0 0 0
Simplicity | 0 0 0 0 0 0 0 0 0 0 0
Latency | 5 1 5 3 5 1 2 4 5 5 3
Fanout | 3 3 1 3 1 5 5 4 1 5 5
Security | 0 0 0 0 0 0 0 0 0 0 0
Transience | 0 0 0 0 0 0 0 0 0 0 0
Archivability | 1 2 1 2 2 5 5 5 1 5 5
Escapability | 1 5 2 5 4 5 5 4 1 3 5
Deniability | 1 5 2 3 3 1 3 2 1 2 3
Anonymity | 1 3 3 2 5 3 4 3 1 2 4
Emotiveness | 0 0 0 0 0 0 0 0 0 0 0
Portability | 5 5 5 1 1 1 1 1 5 5 1
--------------+--------------------------------------------------------
22 26 23 22 26 23 29 28 18 32 30
And for women, we get this table:
. | F2f Paper Phone Fax Chat Web Etext Email Sms Mtext Wiki
--------------+--------------------------------------------------------
Accessibility | 1 2 5 4 4 5 3 4 5 3 5
Unit cost | 5 2 4 3 5 2 4 5 3 5 4
Informality | 5 1 5 2 5 2 2 3 5 4 3
Simplicity | 5 3 4 2 3 1 1 3 5 3 3
Latency | 5 1 5 3 5 1 2 4 5 5 3
Fanout | 0 0 0 0 0 0 0 0 0 0 0
Security | 5 4 5 3 5 2 1 3 5 4 1
Transience | 5 1 5 2 4 1 1 2 4 2 2
Archivability | 0 0 0 0 0 0 0 0 0 0 0
Escapability | 0 0 0 0 0 0 0 0 0 0 0
Deniability | 0 0 0 0 0 0 0 0 0 0 0
Anonymity | 0 0 0 0 0 0 0 0 0 0 0
Emotiveness | 5 1 4 1 3 1 1 2 1 2 1
Portability | 5 5 5 1 1 1 1 1 5 5 1
--------------+--------------------------------------------------------
41 20 42 21 35 16 16 27 38 33 23
So, I conclude that men will prefer mobile text, wikis, and electronic texts above all, while women will prefer phone, face-to-face, and SMS.
Again, don't take my opinions too seriously (I certainly don't). It's just one way of looking at things, and my real point is to demonstrate that platforms are different, for predictable reasons, and these reasons are useful to try to understand when we choose platforms for our own work or for others' work.
Let's get back to organisation. Men and women communicate differently, as we all know, and organise differently. It's not accidental that women have trouble in business structures since these generally use male communication techniques, except for meetings, which most men hate but many women enjoy.
The size of an organisational structure, and its dynamics, depends entirely on the mix of communications platforms that are available to its members.
It thus follows that the most effective organisation is the one with the most accessible and complete mix of communications platforms, so that its members can collaborate in whatever way produces the most specialisation, the most efficient communication, and thus the most wealth.
Comments: 2, Rating: 0
10 Principles for AMQP
Written: 1241352409|%e %B %Y, %H:%M (amqp community sermon)
Ten ways that AMQP can be made simpler, more backwards compatible, more interesting, and overall more enjoyable and successful for all who work on it and use it.
Introduction
The more you know about something, the harder it is to build it.
We went live with the first production deployment of AMQP (on OpenAMQ) at the end of 2006, handling about half a billion messages a day. It was a massive project: design a new protocol, build an industrial-strength implementation, and migrate a major application onto this new infrastructure. Two and a half years sounds a lot but in that time we made three complete redesigns of AMQP, and OpenAMQ, before we had designs that were simple enough to work reliably, and still do what was needed. Functional simplicity is the hardest aspect of design.
It's been hard to translate that success into a final AMQP specification. Part of the problem was failure to agree on what "final" meant. Part of the problem is that AMQP addresses a large problem that cannot be solved quickly. Part of the problem is that the people working on AMQP - including myself - were still building the tools and experience needed.
AMQP will not, in my opinion, be solved in one or two steps, nor in one or two years. The work that has been done already over four years is very important. It has already created a healthy market of competing, interoperable, messaging products. The AMQP/0.9.1 specification, which was published last year, raises the bar: it is lean, precise, coherent, and tested.
But AMQP/0.9.1 is not the final destination. As a contract of interoperability and stability, it's excellent, almost perfect. But it has a limited view of the problem, and it is still more complex than it should be.
The thing about infrastructure is that once it is in place, it is horribly expensive to fix. The complexity in AMQP/0.9.1, if not solved, will be multiplied many times when AMQP expands to cover the fully-reliable messaging, low latency, multicast, and other features we expect to see in AMQP/1.0.
Complexity is easy, simplicity is hard. My day job is to design simplicity by identifying and removing complexity. In this article I'll do that to AMQP, and identify ten areas where there is unnecessary, and in the long term dangerous, complexity, and in each case I'll provide recommendations for solving that complexity.
My ten principles for AMQP are:
- Make small pieces: AMQP should be a fabric of protocols, not one large protocol.
- A layered architecture: these protocols should form a clearly layered architecture.
- Delegate the design work: each protocol is a job for a small, competent team.
- Leverage the community: open the design process to anyone who wants to participate.
- Leverage competition: allow competing teams to make competing designs, at any level.
- Leverage the market: use adoption, over time, to identify which designs are best.
- Use natural syntax: the framing at each level should fit the needs of that level.
- Use natural semantics: the interactions between peers at each level should be simple.
- Push blame to the edges: brokers should never compensate for poorly-written applications.
- Deconstruct the broker: the architecture should support zero, one, or more brokers.
Since theory without practice is useless, the RestMS project acts as a demonstrator of most of these principles.
Make small pieces
AMQP is on paper a single protocol1. This makes it easier to market. It makes it much harder to improve, and the 0-10 specification shows this: it's 197 pages, compared to about 40 for AMQP/0.9.1. I've recommended since 2006 that we refactor AMQP into smaller pieces, each solvable by a small team of 1-3 people. For example, separate protocols for version negotiation, for control commands, for message transfer, for transactions. These pieces need freedom to evolve independently, so that experimentation can be cleanly (contractually) separated from standardization.
A layered architecture
AMQP has no clearly layered architecture. The refactoring of AMQP into small protocols must be based around a solid, coherent, layered architecture. This should be the core of what AMQP 'is', the fundamental agreement around which detailed designs can be built. The architecture should specify what each layer does, in terms of its interactions or interfaces with higher and lower layers. The goal of the architecture is to compartementalize innovation, to guarantee space for contributions, and to ensure the overall coherence of AMQP. The layered architecture is what defines the major version of the protocol.
Delegate the design work
AMQP is specified by a tiny, exclusve team. When a complex problem is broken into pieces, shaped by a clear architecture, it becomes possible for teams to work independently and asynchronously, on different pieces. This is the only way to solve large problems: break them down and let people specialize in different areas. The AMQP process is today a technical process. It should become an administrative one, defining frameworks for collaboration, and approving specifications when they emerge and have been proved.
Leverage the community
The AMQP process excludes expert user contributors. Given that messaging users are often highly competent engineers who at the least can act as "competent clients" for designers, this is a problem. There are only two justifications for excluding participation. One: it's legally complex. Two: it's technically unsafe. Both these can be solved, by defining suitable legal frameworks and by breaking AMQP into small pieces that anyone can experiment at implementing.
Leverage competition
AMQP aims to get one authoritative design. This assumes that there are no commercial rivalries, but there clearly are. Rather than suppress these, we can use them constructively by allowing more than one answer to any given problem. If one team proposes a way of delivering messages, and another team thinks it can make a better way, it should be allowed to. Perhaps the two teams can agree, but if they don't we should allow two answers to sit on the table at once. This enables competition, so better designs can emerge. It also forces layers to be more orthogonal, and validates the overall architecture.
Leverage the market
AMQP is shielded from market opinion. If users prefer a particular API, for example, that does not feed back into the AMQP process. But it should: market opinion, especially with open source products, is a very valuable indicator of quality. When it comes to protocol designs, quality valuation by the market can be measured by the number of independent open source implementations, and the number of users of such implementations. Rather than try to make quality decisions up-front, it is simpler to use a green-brown model where experimentation allows multiple "green" specifications that are pruned over time to leave a small number, or one, brown specification.
Use the natural syntax
AMQP uses binary framing for control commands ("Queue.Declare") as well as message transfer ("Basic.Deliver"). This is not ideal. Each piece should use the natural syntax. Command protocols should use text, not binary framing. This is how every successful IETF command protocol works. Text framing makes backwards compatibility trivial. There are no performance issues with command protocols, so parsing is not an issue. Text framing is easy to understand, easy to implement. Message transfer protocols should use binary framing, and can be significantly simpler than AMQP's current framing: smaller envelopes, no verbs, no channels, no dynamic addressing.
Use natural semantics
AMQP currently mixes asynchronous and synchronous conversations in one protocol. It makes the combined protocol complex, and it makes error handling tricky. When we split control commands and message transfer into two separate protocols we can make each simple. Control commands work best with synchronous, pessimistic dialogues: each request gets a success/failure response. Message transfer works best with asynchronous, optimistic dialogues: messages are sent with no confirmation, and reliability is layered on top, as higher level protocols based on acknowledgements and retransmissions, transactions, and so on. Messages can be batched, commands should not.
Push blame to the edges
AMQP currently pushes problems upstream. Application private queues are held on the broker. Slow consumers cause these queues to back-up, and probably the biggest issue for production use is that servers run out of memory, and crash. Applying flow-control to publishers is the wrong solution. Messages should be pushed without pity to the edges, and if these are too slow, that should count as a defect in the edge application, and handled locally: drop old or new messages, or raise a fatal error. Problems at the edge should not be allowed to move upstream, period.
Deconstruct the broker
Currently, AMQP considers a broker to be a mainframe: big, important, central. This is certainly one use case but it is a very bad design for high data volumes: one big central bottleneck. It is wiser to treat the broker as a host for arbitrary queues and routers, and to accept that in many cases, such applications optimally reside at the edges, not a central point. For example, holding private queues centrally makes brokers fragile, while pushing private queues to the edges makes brokers more robust (by pushing blame to the edges). Similarly, routing can be done at the publisher edge. Only the Wolfpack pattern (one-to-one-of-many) absolutely remands a central broker.
Conclusions
The AMQP protocol and process are complex in ways that make it difficult to build a Version 1.0 and which in the long run may affect the success of the AMQP project itself. I've outlined ten principles that will in my view solve the complexity. Most of these proposals are years old and have been made many times to the AMQP working group.
