The previous episodes focused on laws of math (generality of computation) and physics (quantum mechanics makes it easy to build general purpose computers). Now we concentrate on laws of economics in the industrial age. In the beginning of the industrial age, factories lowered the cost of goods dramatically from before — a textile factory is much more efficient than a tailor making suits, and so it is much cheaper to clothe ourselves. However, as factories evolved, it became more and more important to drive the cost even lower — whereas before the competition was with other tailors, now the competition was with other factories. So, economies of scale and efficient production lines developed. Soon after that, supply chain management and moving factories to the most cost-efficient places were developed.

Let’s say that you want to make T-shirts with political slogans. You’ve got your slogan writers, huddled and coming up with good slogans. They’re a sunk cost — you’ll pay them the same no matter how many T-shirts you make, or how much they cost. Now you build a big factory to make T-shirts, and print slogans on them. The company next door specializes in T-shirts with band logos. They build a big factory to make T-shirts and print logos on them. Someone realizes there is money to be made in this scenario — build a factory to manufacture T-shirts, and send them in large crates to places that will print stuff on them. Because their factory is bigger, their cost for making T-shirts are lower, and they pass some of the savings onto you, the political slogan company. In turn, you pass some of the savings onto your customers, and everyone is happier.

Lesson: When economies of scale hit, you want to try and buy standard components for the product you manufacture.

Now suppose you build a car. The anti-lock brake system need to figure out when the car is sliding, and start “pumping” the brake. You can build a system that will contain gyroscopes that attach to some handle that pulls a wire that pumps the brake. Or, you can

• Buy an accelerorometer
• Buy a general purpose chip
• Write software for this chip, based on input from the accelorometer
• Connect this software to a motor that controls the brake

Note that in this scenario, we source some standard parts. Even though the parts are more complicated, it turns out that we save a lot of money by using standard chips and accelerometers. Note that putting the software on the chip is just copying it to some standard storage device — extremely cheap. ABS brakes become cheaper, and people die less. Overall, a good thing.

The same sort of logic causes a lot of things that used to be done with handles, levers and pulleys to be done with a general purpose chip and some software. Your car is full of them. Your microwave has some. Your television has some. Each of those devices is basically a computer connected to some strange peripherals (brakes, microwave-rays or screens). In a world with strong copyright regulations, those devices’ software are locked to us, even though we own the device. Where darkness goes, evil deeds covered by the darkness soon follow. The police in Evil Regime Country wants to make sure nobody can run from them. They mandate all cars sold in ERC must have, in their software, a special switch that when a certain bluetooth signal is sent (many cars nowadays have computers connected to bluetooth receptors), the software controlling the automatic gear system makes sure the car will not go over second gear. If the ERC police are smart about using this signal, nobody will ever know.

In a world with strong copyright protection, big corporations control your life — and they are corruptible, and if darkness holds, corrupt.

Recapping the previous episode:

• It’s easy to define a system that is as powerful as any other computational system (Turing complete)
• It’s impossible to say what any computation does (Rice’s theorem)
• You can use bits (0s and 1s) to encode any sort of data — music, text or anything else.

In this episode, I will have a lot more to say about bits. Before I delve into bits, though, I want to talk about computers. While abstract systems that do computations abound, most modern computers are fairly similar. The theoretical computational system they are built to resemble are so-called “Von Neuman Machines“. A modern computer has lots of peripherals, but strip away the peripherals and you are left with:

1. A “microchip” which implements instructions like “Add Register1 and Register2 and put the Results in Register3″
2. Memory — mapping of “addresses” (index numbers) to “values”
3. Instructions on the microchip of the sort “Treat register 1 as an address, and fetch the value there into register 2″

The exact instruction set that the microchip implements depends on the type, but ultimately, as we saw before, it does not matter too much — all computational systems are equivalent. What is important is “Moore’s Law”, of which many variants exist but ultimately says that:

1. Microchips can perform more calculations per second every year.
2. The amount of memory available for a given price keeps growing.

An important computer peripheral, which almost all computers have a variant of, is a storage device. That ranges from a magnetic hard drive to a micro-SD card. All that is important, for our purposes, about those is:

1. Accessing them is slower than memory
2. They are bigger (have more addresses) than memory for the same price [a lot more -- frequently 10x or 30x]
3. They grow bigger for the same price every year

Another important computer peripheral is the network card. Networks connect computers to each other.

Now, let’s remember something we said earlier — there is no way to reliably detect any sequence of 0s and 1s that encodes a computation that we do not like — say, a Turing machine outputting a fanfiction about Micky Mouse having sex with Pluto. Therefore, if a network card allows you to send two distinct messages [if it only allows you to send one message, it's kind of a sucky network card], you can send some “illegal” Turing machine encoding. Moreover, you will be able to store this illegal Turing machine faster and more cheaply every year.

So as an inescapable conclusion of (1) The math (b) Moore’s law we see that the law cannot be enforced, and it costs less and less every year to evade this law. The laws of the universe (such as how easy it is to use Quantum Mechanics to implement computation on the stuff that is abundant on every ocean beach, or Rice’s theorem) care nothing for Walt Disney or a starving artist — they are simple there, immutable and unforgiving, and humans must learn to deal with them.

One way to deal with them would be to “follow the money”: enforce the law only when someone is breaking it for corporate-level gain. This is not the way the legal system has gone through. Instead, they turned to God — “Crypto. Really good, standards-defined crypto”.

So to understand what transpired, it is important to understand the basics of cryptography. From the beginning of time (as humans count time, I guess), people have communicated with each other — “Look, Ug, I found an antelope and killed it. Help me eat it?” Not long afterwards, eavesdropping begun — “Hey, everyone, Ung has an antelope.” The next level was to use codes — “Look, Ug, I found a You-know-what and you-know-what-ted it. Help me you-know-what it?”, and so the battle begun.

Codes, like in the example above, have to balance two issues. The person to whom they are intended must be able to decipher them (Ung better hope Ug will get the right message). The person to whom they are not intended for must not be able to decipher them (or once again, the whole village will know about the antilope). Fast-forward to Greek times, Caeasar had the eponymous cipher, based on shifting every letter in the alphabet by a certain amount and the Nazis had Enigma. Word to the wise: use neither of those, as they both have been “broken”. “Broken” is a term cryptographers use to say “the eavesdropper can read the messages which are not intended for them”. Cryptographers spend a lot of time trying to figure out how to read those…

But back to the basics: In computer-based cryptography, we have a secret, S. S can be considered, like everything else a computer handles, as a certain sequence of bits. For cryptography to work, two parties have to pre-agree on S. (Side note: I will not be covering public-key cryptography here.) Then, we must be able to compute the “encryption”: a computation that takes M, a message, and calculates a function that depends on M and S. Then we must be able to compute “decryption”: a function that takes the encrypted message, E, and S, and returns to us M. Next, it should be impossible (or at least, very hard), to compute M from E without S.

How do we know that something is “very hard”? Well, if we have a guess for S, we can check that the decryption gives us a plausible message. How hard is it to guess S? Returning to the quotation at the beginning of the section, “standards-based crypto” is usually at least 128 bits of S (also known as the “key size”). That means that there are 2 to the power of 128 options. It is largely agreed that the smallest time-frame relevant for computation is the time it crosses a photon (light particle) to cross a hydrogen atom. A hydrogen atom is about 10**-11 meters long. The speed of light is about 10**8 meters per second, which means it takes more than 10**-20 seconds, which is more than 2**-70 seconds. Thus, if we have 2**128 options, we need more than 2**58 seconds. 2**30 seconds is more than a year, so it will be more than 2**28 years, which is about 32 million years. I used various approximations above, but the conclusion still stands — and more and more modern crypto uses 256-bit sized keys, which is not quite the heat death of the universe, but the point still stands: you cannot guess and hope to win. However, it turns out to be an open problem whether any question where you can easily verify guesses (the formal name for that is NP) is “hard” (the formal name for “easy” problems is P). By open, I mean that many computer scientists have tried tackling this problem over the last 50 years, with no success (and little progress).

This means, in particular, we cannot know that any cryptographic computation is really “good”. However, what we can do, and the second part of why “really good” is followed by “standards-based”, is to ask really smart people to try and solve a cryptographic problem. If they can’t, after trying really hard, we assume that the problem cannot be solved, and we used it as our encryption mechanism.

In modern times, the names “Ug” and “Ung”, perfectly good though they may be, have fallen into disuse. Modern cryptographers usually talk about “Alice” and “Bob” wanting to transmit messages, and “Eve” wanting to listen to them surreptitiously.

So here is an example of a cryptographic system: if you buy a DVD in the store, it is encrypted with a special key. When you buy a DVD player in the store, is contains the key. Your DVD player is a computer, with special software that decrypts the DVD, with a special key, and then plays it. So Alice, which is the Hollywood studio, encrypted the movie contents. Then Bob, the DVD, decrypts it. Bob, to play the DVD, must have the key. You bought the DVD, so there is nothing stopping you from opening up the DVD player, and taking the key, is there?

Well, there are two things. Once is simple: Hollywood, before giving the DVD maker the key, make sure that the DVD player is hard to “tamper” with. The DVD maker must put the key in a special chip, glued to the microchip, and that self-destructs if people tamper with it. As you can imagine, creative people have found ways to defeat that self-destruction. And so, Hollywood convinced the US government to pass a law called “Digital Millenium Copyright Act”. The DMCA says that

1. It is illegal to tamper with the DVD that you bought and get the key out.
2. It is illegal to tell someone the key that you dug out of the DVD.
3. It is illegal to tell someone how to tamper with the DVD to get the key out.
4. It is illegal to tell someone where to find instructions on how to tamper with the DVD, or how to get the key.

You might have noticed that 2-4 are restrictions on speech. There are numbers that are so illegal, that not only are you not allowed to write them on a piece of paper and give them to your friend, but if someone spray-painted them on a building, you are not allowed to tell anyone where that building is. Whether you’re allowed to tell them where they can find a map with the building starred in it is, I believe, still up for debate.

The incredulity continues as you find songs on You Tube and Flag images that encode illegal numbers. This means that certain songs and flags are now illegal. In fact, it very well might be that the Wikipedia page on “Illegal numbers” is already illegal, since it contains data that can allow recovery of these numbers. I wish I were kidding, but I am not.

I, personally, am not a copyright extremist. I am not committed to abolish copyright. However, I think that understanding the math and physics of computation are important, because otherwise we end up making songs illegal. When Hollywood claims that copyright infringement might cause jobs lost, and that this necessitates stronger copyright law, we should first ask “Will this make singing songs illegal?”

This is why programmers are concerned about copyright law — because we understand all of the above. I hope, after reading this, you are also concerned!

Before we even start, I would like to point out an excellent description for why we usually expect explanations to be simpler then they really are. Please do read this. I’ll wait, really. If you think you do not need to read it, that’s actually evidence that you do — you expect my explanation to be simpler than it is, and that you do not need that bit of background knowledge…

Welcome back! One important concept introduced in the article linked above is “Word of Power”. (If you didn’t read it, now is the time to fix that issue…) I will try to introduce the new concepts using Words of Power, saying the word, and then linking it to the power behind it.

The first Word of Power will be “Universal Turing Machine”. You may remember a previous post of mine about Alan Turing, one of the greatest giants on whose shoulders we have the privilege to be standing on. But I want to start by talking about another great giant, John McCarthy. McCarthy wrote a paper with the somewhat unassuming name, Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I. A more appropriate title would have been “paving the way forward for programming for the next 50 years”, you see. What McCarthy did in the paper was to lay down a simple explanation of a system for defining computation. Nowadays, a lot of computation is done, appropriately, on electronic computers. Before we had those, however, “computer” was a job title of a person who applied computation rules to symbols on paper to get results — much like the rules we learned in primary school for doing long multiplication.

In both sense of the word “computer”, the goal is the same: apply rules to manipulate symbols. McCarthy defined a specific set of rules to manipulate symbols that he called “Lisp”. Lisp used S-expressions (short for “symbolic expressions”) to define computations. Read this sentence over again, because it might be the most important sentence in here: even though Lisp was a set of rules to manipulate S-expressions, those S-expressions define other computations. If you only learned to manipulate S-expressions, someone could write an S-expression that did, say, long multiplication. Then, if you manipulated the S-expression for long multiplication followed by S-expressions representing numbers, you would do long multiplication. Now comes the fun part — you could even write the rules for manipulating S-expressions using S-expressions. You might think that for this to work, the rules for manipulating S-expressions would be really complicated. This is not true — you can see the manipulation rules here. Sure, it’s a bit long, but doesn’t look like more than you needed to learn to do arithmetic, right?

Well, what use would rules for manipulating S-expressions when already know how to manipulate S-expressions? Maybe very little, if not for a few other thing. I hope you have seen a demonstration of the game of life, once upon a time. If you haven’t, I recommend that you read the wikipedia article. With 4 simple rules, much much simpler than McCarthy’s S-expression rules, much is possible. How much? Well, the proof is long and difficult, but mathematicians have found a Game of Life configuration that will manipulate S-expressions. So if you only knew to manipulate squares on graph paper according to the rules of the Game of Life, seemingly easier than manipulating S-expressions, we could just give you a (pretty hefty) Game of Life that would cause you to manipulate S-expressions. Wait, but manipulating S-expressions lets you do arithmetic, right? So you could do arithmetic too!

Wait, what about arithmetic? Well it turns out, if you just know how to add 1 to numbers, and do a bunch of other trivial things (like substitute numbers for other numbers), we can give you a instructions that will let you manipulate the Game of Life. Or other instructions, that will manipulate S-expressions. It turns out that if you can do interesting enough computations, it doesn’t matter a whole bunch what you know to do — you can find an initial input (S-expression, Game of Life configuration, or arithmetic instructions) that will let you do anything. Anything? Well, not quite!

Another interesting result (called “Rice’s Theorem”) is that you cannot write an S-expression (or any of the other things) to take any S-expression (or any of the other things) and say anything interesting about what it does to any input. Notice that I didn’t say “respectively”, and didn’t mean to — you can’t make an S-expression that will take any Game of Life configuration, and will say anything interesting about what it does. You can write S-expressions that will take some S-expressions (or …) and say something interesting about them — but not any S-expression (or …).

What has all this to do with our friend, Alan Turing? Well, Turing invented the original thing that stands for “…”, and he called it a Turing machine. A Turing machine has an internal state, and reads from and writes to a tape. The interesting thing about a Turing machine? You can write a Turing machine that will manipulate S-expressions. Or play the Game of Life. Or perform arithmetic. Any of those Turing machines can do anything S-expressions do! All of those are called “Universal Turing Machine”, because any computation possible by any other Turing machine, or S-expression, or Game of Life, or arithmetic, can also be done by them. You only need One Turing machine, or one S-expression, or one Game of Life configuration, or one set of arithmetic rules and you can do anything anyone else can do, if maybe somewhat slowly. Fortunately, or unfortunately, all those computations also have the same basic flaw — you can never write one of those to say anything interesting about any computation that comes their way (this will be important later!)

Let us recap:

1. Computations that manipulate symbols can be done either electronically or by a human, equally well, if not equally fast.
2. You can define a “universal computation”, that if you learn to do, anyone can get you to do any other computation by writing the correct input.
3. It is impossible for you to know anything interesting about any possible computational input.

Those three concepts (“a lot of interesting manipulations are equivalent”, “universal manipulations are possible” and “nothing interesting to say”) are the core of what computers are. A computer chip does very specific manipulations ([1]). However, programmers, by writing carefully thought out inputs, can get it to perform any computation ([2]). Lastly, it is impossible to write a computer chip that will be able to say anything interesting about any specific program ([3]). Of course, crafting these programs is difficult — this is why programmers use “programming languages”, which are even more “universal computation instructions”. A language is called “Turing complete” when you can use it to build a Turing machine (which would also have allowed you to build an S-expression manipulator, or Game of Life player, etc.). In short, all we have said so far applies equally well to most programming languages. There are niche non-Turing-complete languages, used where being able to say something interesting about any possible program is important — but here’s the rub — it’s actually hard to invent something which is not Turing complete. As we saw above, even very simple things will be able to build Turing machines.

Computers, languages and programmers are all becoming better at making computations that were “possible” a decade ago be “fast” now. There is an enormous economic pressure on that — after all, the more you can do “fast”, the more you can do “more and bigger”, and people like “more and bigger”.

Now, we come to a very simple computation — copying. A “copier” can be defined in multiple ways, but let’s suppose, for the sake of argument, that we just want to replicate the input twice (“have two copies”). It is easy to write an S-expression to do it. Therefore, any Turing complete environment can do it. But remember — it is impossible to say anything “interesting” about any possible S-expression, and therefore, it is impossible to write a computation that outputs the correct answer for “this S-expression is not a copier”. Wait, what? Yes, that’s right. If an S-expression has access to some input, it is impossible to know for sure that it will not copy it.

The next Word of Power I wish to introduce is “bits”. A “bit” (short for binary digit) is a place-holder for something that can be either 0 or 1. Anything that can represent at least two states (say, the light switch in your kitchen) is a bit. Now, let’s say that you have a set of symbols — say musical notes. I hope you have seen “Sounds of Music”, and remember the best song ever — “Do, a deer” (etc.). Our symbols are “Do, Re, Mi, Fa, Sol, La, Ti”. As the song makes the case, we can write any piece of music with just those (yes, I know about sharps, flats and octaves…bear with me). Now, let’s assign “bit patterns” to each note:

• Do — 000
• Re — 001
• Mi — 010
• Fa — 011
• Sol — 100
• La — 101
• Ti — 110

Now, if we want to write the notes for the first line of “Twinkle, Twinkle” (Do Do Sol Sol Ti Ti Sol) we can instead write 000000100100110110100 in bits. If we have those bits, we can just take them in threes, and convert them back to the musical notes. We can do the same with any set of symbols — say, the alphabet. This is actually how, more or less, computers store text — they can convert the alphabet into bits, and save those bits. Then they can manipulate those bits using carefully crafted programs to, say, replace the word “Foo” with “Bar”. Or, say, copy them. If we have the bits to a fan fiction of Mickey Mouse having sex with Pluto, we can copy those too. If instead, we have the bits for an S-expression that will generate the bits for a fan fiction of Mickey Mouse having sex with Pluto, we could copy those too. But wait, this is funny — there is no way to know for certain that certain bits are not an S-expression that will produce the bits of Mickey Mouse having sex with Pluto.

A fan fiction of Mickey Mouse having sex with Pluto is a dangerous thing. It is, as the law and case law currently stands, a violation of copyright law. In general, it is not protected under the Fair Use doctrine, and this means that writing this fan fiction is illegal. Copying this fan fiction is illegal. What’s more, Disney has an incentive (or at least, believes it has an incentive) to prevent copying this fan fiction around. When incentive (or perceived incentive) and legal powers combine, the result is expected — Disney would dearly love to have an automatic way to prevent computers from copying this fan fiction. Or, say, from copying the bits that represent the video of “Cars 2″.

Remember what I said above — it’s impossible to have a way to know for certain which bits are actually an S-expression that will create the bits for the “Cars 2″ video. Although Disney has the incentive (this is a matter of economics and psychology, ultimately the results of the forces of evolution) and the legal powers (this is a matter of social convention), the math, hard and unyielding, doesn’t care. Math doesn’t care about evolution. Math doesn’t care about society. Math is math, and the math says that you can’t build a computer that will only copy things if they’re not S-expressions that produce the Cars 2 video, no matter how much you want to.

Join me next episode, when I explain the basics of cryptography, and how they pertain to the issue of copyright law.

Get into a serious relationship

Get engaged

Move to a team at work where I fit in better

Move to a nicer apartment

Get a loan (to build up my credit history)

Go to a Less Wrong meet-up

Get a talk accepted at PyCon

Get more recommendations on my Linked In profile

Write Rationalist Fan Fiction

Contact my Congressperson

Dear Congresswoman Speier,

I am writing to you regarding the so-called “SOPA”, or “Stop Piracy Online Act” which will be up for debate and a vote soon (possibly as soon as December 15th). I am concerned about the implications of the bill for my rights, as well as those of others in the United States.

Let me prefix this by saying that the bill will do nothing to “Stop Online Piracy”. Technologies like Freenet[1], Tor[2] or VPN[3] already exist for dedicated pirates to work around any technical or legal hurdles the US government might throw in their path.

However, what the bill *will* do is to create a US equivalent of the so-called “Great Firewall of China” — a centralized place for the government to stop any communications it does not want. It wouldn’t work against pirates, of course, which would find any small hole in the implementation and smuggle everything through that (just like they do in China — a police state!). But it will mean that new forms of art or technology can be stopped by a unilateral act of the government.

I, like many of your constituents, work in the hi-tech industry. I depend on free communication, and unfettered access to information, for my livelihood. If you put obstacles in front of Silicon Valley’s business, competitors outside the US will out-innovate, and eventually out-compete us.

This bill does not just threaten my rights to free expression, and my access to new forms of communication. This bill threatens my ability to make money, all in the name of protecting out-moded business models.

Last but not least, you will notice that the amounts of television, music and movies available has not decreased with the internet — even if we just limit our search to mass-media. If you factor in over 50 hours a minute of video uploaded to YouTube, you will see that this bill is not necessary in order to “To promote the Progress of Science and useful Arts”[Article I, Section 8, Clause 8, United States Constitution]

I thank you in advance for your consideration, and hope that you will choose to vote according to the interests of your constituents — against the SOPA bill.

Yours,
Moshe Z.

[1]http://freenetproject.org/
[2]https://www.torproject.org/
[3]https://www.ipredator.se/

Different communication mechanisms have different trade-offs: XML-RPC is ubiqutous, JSON-RPC is almost as popular but more space/time efficient, AMQP allows efficient routing and so on. It is advantageous to write application code which is completely transport-agnostic. Deciding how to split components into processes is a decision more efficiently done at the tail-end of a project, when the knowledge of which processes are more reliable, or need to communicate more, is known. Described here is the system we use to allow application code to be unaware of those distinction, pushing those decisions into deployment or installation time.

Introduction

When starting to implement a complicated system in Python, there are many unknowns: amount of communications between processes, typical process bugs and deployment targets. It is highly useful to allow writing the business logic code in a way that is agnostic to the answers to those questions. This also allows the communication layer to be written as a quick prototype, and optimized later. Since infrastructure costs are frequently minimized during the demo phase, this kind of approach allows quick demos during development without any cross-team code rewrite. The trade-off in using this kind of approach is that the feature set assumed from the transport has to be a least-common-denominator subset of all “possible” feature sets. This, in turn, pushes some of the work that could be done by more sophisticated transport to the Python-level infrastructure and sometimes, in the worst cases, into application code. The most egregious example is that any system that supports HTTP-based RPC mechanism (XML-RPC, JSON-RPC or SOAP) must not assume any kind of asynchronous communication mechanism.

Interface Definition Language

Many systems do not have an IDL of their own — or it is so weak (as in the case of XML-RPC Introspection) so as to be worthless. Interface definition languages give us two benefits: it reduces the amount of trust between processes, since processes can only send valid data to each other, and it allows us to declare explicitly which methods are supported by a given component. Our Interface Definition Language takes advantage of Python meta-classes to look like::

 class ThingyData(Data):
     """
     Data which identifies a target
     """
     uuid=lambda:str
     metadata=lambdaict

 class ThingiesOfType(Data):
     """
     Collection of targets of the same type
     """
     type=lambda:str
     thingies=lambda:array(ThingyData)

 class SearchInterface(Interface):
     """
     Interface to search functionality.
     """
     name='com.vmware.example.version1.Search'
     @public
     def RunQuery(query=str, maxResults=int):
         """run a query

         @param query: a query to run

         @param maxResults: maximum number of results to return

         @param results: query results
         """
         return dict(results=array(ThingiesOfType))

Since we cannot depend on the underlying transport to contain objects or references, we assume all communications happen in terms of “simple” (nested) data types: ints, strings, floats, uniformly typed lists, string to string dictionaries and string to vartype dictionaries. In order to do so in a dependable way, without application code having to translate manually, we have the Implementation metaclass::

 class SearchImplementation(object):
     __metaclass__ = Implementation
     interface = SearchInterface
     def RunQuery(self, query):
         d = defer.Deferred()
         myThingies= Structure('ThingiesOfType')
         myThingies.type = 'Something'
         myThingies.thingies = []
         reactor.callLater(5, d.callback, [myThingies])
         return d

It is possible to return either a deferred or a value from a method in an implementation. Note that there is no need for type checking inside application code — bad types would trigger an exception early on in the calling process.

Clients

Communication needs two sides. So far we have described how to write a server. We want client code to be equally transport-agnostic. In order to do that, we have a dynamic client object.

 def _simplifyObject(obj):
     if isinstance(obj, (int, float, str, bool, dict)):
         return obj
     elif isinstance(obj, list):
         return [_simplifyObject(x) for x in obj]
     elif isinstance(obj, Structure):
         return dict([(key, _simplifyObject(value))
                        for key, value in obj])
      else:
         raise ValueError("cannot simplify", obj)

 class MultiClient(object):
    def __init__(self):
        self.ifaces = set()
        self.routing = {}
    def add(self, iface, provider):
        self.ifaces.add(iface)
        for method in iface.methods:
            self.routing[iface.name+"."+method] = provider
    def callRemote(self, name, *args, **kwargs):
        args = map(_simplifyObject, args)
        provider = self.routing[name]
        return provider.callRemote(name, *args)

This means we can drop in any objects that support callRemote, commit to which interfaces are available for them, and the client code does all multiplexing. From inside the application code, we need no idea about which provider gives which interface — all we need is a callRemote.

Transport and Processes

The server code is the non-transport-agnostic part. It takes the fully-qualified name of a function, runs that function and treats the results as a (possibly nested) lists of implementations that should run in the given process. For the XML-RPC transport, it also expects on the command-line to have a list of which ports are running which interfaces. It creats XML-RPC proxies for those ports, and drops them into the singleton client. Running all servers manually means a lot of manual stitching up and repetition. In order to save that repetition, we have a so-called multiserver, using the Twisted Process Monitoring module (which takes care of starting and restarting processes). The multiserver gets a list of processes, specified in a JSON file. It then calculates what should be its client list, and runs all processes with the same client list, using the same server. We plan to add support for JSON-RPC and HTTPS as command-line switches to the multiserver, which would then run all processes, and assumptions, appropriately. Note, however, that a smarter multiserver could decide that some processes should use JSON-RPC and some XML-RPC, and run them all appropriately. Also note that currently the multiserver can only support one component per process, even though its underlying server is capable of more. We also plan to have the multiserver set a special environment variable “SECRET_KEY”. This key would be a randomized string — all clients would send it, and all servers would require it before they process input. In this way, a malicious process running with different privileges on the same machine will not be able to get into the system.

Unified Front

We want to have a single port on which all public parts of the interface are exposed. We can do this in a way that is transport agnostic, even if it is not agnostic to some of the implementation details of Interface or the singleton client.

 def makeForwardingFunction(name, outargs):
     def retFunc(self, *args):
         remoteReturn = singletonClient.callRemote(name, *args)
         def cb(remoteReturnValue):
             return dict([(aname, argumentParse(tp, remoteReturnValue[aname]))
                              for aname, tp in outargs.iteritems()])
         remoteReturn.addCallback(cb)
         return remoteReturn
     return retFunc

 def makeForwarder(iface):
     d = dict(interface=iface)
     for name, func in iface.methods.iteritems():
         if func.isPublic:
             d[name] = makeForwardingFunction(iface.name+'.'+name, func.outargs)
     return Implementation('Forwarder', (), d)()

 def unifiedFront():
     for iface in singletonClient.ifaces:
         yield makeForwarder(iface, client.client)

Note that we could have set our multiserver to serve the unified front using one protocol (e.g., XML-RPC over HTTP) while running all clients using another one (e.g., AMQP). While this requires some complicated code in the multiserver, it does not require *any* kind of application code support.

Conclusion

With the right kind of separation between application code and transport code, it is possible to support complicated interaction topologies while completely hiding them from the application code. Python metaclasses, duck-typing and high level of dynamism makes writing this kind of code fairly easy.

The non-Joss-ity comes at the end, where Superman does not have to actually face the consequences of stopping the Hackensack-rocket first (and letting Lois die), because he just spins the Earth around in reverse, undoing time. Come on! That’s no way to make the hero truly suffer. Oh, no, in the Joss version of the movie, Superman realizes he has to turn back time. But he can only do it by some super-pain-staking method, say using his microscopic vision and amazing memory to build atom-by-atom a computer model of the Earth as it existed just before the bombs. This takes millenia, which luckily superman, not aging, has.

Now superman has to make a choice — does he flip the switch, causing the entire Earth to reassemble into its pre-bomb existence? Sure, in some sense he kills everyone on Earth, but in a different sense, what right to life to they have compared to the people who would have lived had the bomb not killed Lois? Even though Superman is in love with a girl he knows will not exist in the re-written reality, he does it. He hugs Lois, but grieves for what might have been.

– THE END –

For example, “gold is just as rare to them as it is to you.” Really? Is the Earth somehow so extra-ordinarily special in terms of gold, that they would come here, where an intelligent species on the verge of discovering space technology really likes it gold, instead of trying anything else not so involved? For example, mining for gold on any other planet in the solar system? (I’m not going to even mention solutions like surrounding a nuclear reactor with lead, and using a centrifuge to separate out gold from lead).

But I’m not going to just complain. I’m going to show what it should have been — what alternate premise could have made the movie make sense!

On an alien planet, there lived two intelligent species, from completely parallel evolutionary lines. Where we evolved a sense of humor as a signal of intelligence, one of them, species R, developed a sense of taste — for intelligent species. The R males would find a member of the other intelligent species, trap it, and present it to the female as part of their mating ritual, very early on in their evolution. Eventually, they would be herding the other species, P, and would develop complete disgust for “meat that doesn’t talk”. For the Rs, eating a meal that doesn’t talk is a way of subsistence on sub-par food — like eating worms for us, a way to survive in the wild while reaching civilization, not a part of their diet.

In fact, they begun to be connesuirs of intelligent species meat. They fly around the galaxy, find interesting intelligences, and eat them. As soon as they detect signs of intelligence, they swoop in and start munching about. For Earth, that meant playing around with electricity, and early radio waves, was the first sign. Soon after some of the early experiments in 1872, the vast probe system they laid out informed them that Earth had intelligent life. Squishy, soft and tiny, those intelligences were unlikely to prove a threat. But how could they get the amount of food they needed?

Enter the gold. Where gold is found, the humans go, the aliens discover. They started stealing the western gold not because they liked it, but because they knew it would get the humans to them — and into their ships, where they could be packaged and kept for a while. They had to experiment with preparation, and an unfortunate number of people were burned to a crisp from culinary experiments gone wrong.

Ella’s species, of course, was lured in with the same trick — stealing their gold to get the people. She has never, and sadly will never, discover it was all a ploy for the aliens previous planet-wide feast.

Mushroom Alfreddo (there’s lots of variants — but this one is really easy!)

Ingredients:

4 Cups of Mushrooms (as many different ones you can find)
1 Carton of Heavy Whipped Cream
2-3 Tsp. of butter

Preparation

Slice the mushrooms into medium pieces.
Heat a non-stick pan, and put the butter on it evenly. Pour in the mushrooms and mix with the butter. The butter should coat all mushrooms — add more if necessary. Wait until the mushrooms start losing significant volume (1/2-2/3). Add the heavy whipped cream, turn down the stove to low heat, and mix. Add salt, pepper and garlic to taste, and wait until the sauce is thick enough.

This recipe is meant to be done as you’re cooking pasta in another pot. If you have a timer in the kitchen, or are good at keeping time, this is easy. Otherwise, do the pasta first. (When you’re done cooking pasta, pour water out, add a little bit of olive oil. This will give the pasta a nice taste and will stop it from sticking together.)

Israeli Salad

2-3 servings

Ingredients

2 Large Cucumbers
4 Mid-size tomatoes
1/3 Yellow onion
1 Small lemon
1 Tablespoon of olive oil
[Adjust quantities according to size of vegetables]

Preparation

1. Take out the tomatoes’ seeds/liquids, and chop the rest to small pieces
2. Peel the cucumbers, and chop to small pieces
3. Peel the onion, and chop it to small pieces
4. Mix them in a bowl. The best mixing instrument is hands (wash first!)
5. Pour juice from the lemon on the vegetables.
6. Pour olive oil on the vegetables.
7. Salt&pepper (black, ground) to taste
8. Mix again to season evenly.

Variants

• Add cooked/canned corn kernels (mexican style)
• Add chopped hard-boiled egg (breakfast style)
• Add cottage cheese (lunch style)
• Add chopped Israeli pickled cucumber
• Substitute the onion with chives (fancy style)
• Add some extra-fine chopped hot pepper

Any subset of these works — so this is actually 64 different recipes, all easy!

Eventually, even if, as was common, every couple had only two children, the population would increase indefinitely. Rak shrugged. It would get taken care of, of course, it was too big to be allowed to be handled by people. He felt Pom’s hand in his, caressing his fingers. He brought up Pom’s hand to his lips, and kissed it. Pom turned to him, and Rak nodded.

“I’m tired too,” Rak said, knowing Pom was not much of a night person. The couple walked back, hand in hand, to the youth dormitory. Rak could feel the hairs on the back of his neck rising, as he felt Pom’s hand caress his ass, then drop teasingly. “Come on,” said Pom, “let’s go to bed.” Rak nodded.

In a post-coital bliss, Pom and Rak lay next to each other, hand in hand, staring at the ceiling. Pom hoped it would be a time of silent contemplation, but he knew his boyfriend too well. Rak wasn’t good with silence.

“So what do you think about the story?” said Rak. “Not much,” said Pom, “it was just a cliched love story, as far as I could tell.” “No!” said Rak emphatically, “it wasn’t about the plot, it was about the setting. The eld times, when the villages were not so crowded together.” Pom shrugged. Rak was like that, he knew. “It’s just a story,” Pom said, “of the eld times.”

He wasn’t really sure what Rak was getting at, but he knew how to get the story out of Rak. Play it cool, and just a little bit dumb, and Rak would explode in a big rant, and it would be all over — the sooner the better. Pom really did need his beauty sleep. He smiled, as Rak launched into the rant –

“Not only are all stories of eld times consistent about this, it’s also inevitable. If people are born, there will be more people. More people means more villages, since villages cannot have more than fifty people.

Eventually, the earth will be covered with villages,” he stopped for a quick breath and continued, “and it’s *already happening*. What does that mean for the future? Where will people go? Where will our kids go?”

Pom winced. He knew Rak was much more serious about this relationship than he was. He *thought* he was in love with Rak, but he wanted to take things slow. Whenever Rak talked about kids, or a future, Pom felt a little uneasy.

True, he was fifteen, and it was time to think about a committed relationship, but he wanted more time to think, to get in touch with his feelings, to really make sure.

Pom shrugged. “You do realize we do not live in prehistoric times, right?” he said, “it is not up to us.” Pom looked up, a gesture meant to symbolize looking at O, though ve was omnipresent. Pom felt like that should have been the end of the debate. O would take care of them, as ve always has, since the dawn of history.

He could hear Rak drawing in a breath, preparing a clever retort, and decided to forestall it. He turned to Rak, and pressed his lips Rak’s. Argument over.

The next morning Rak woke up early, and knew what he needed to do. He walked outside, across the paved path, and into the woods. He breathed deep breaths, in and out, to get himself into the right state of mind. Communing with O needed a certain inner calm, and Rak knew O would not appear until he was calm enough. Rak reached the tree he thought of as “the O tree”. O did reside everywhere, but humans needed a place to associate with O. Rak stood in front of the tree, and intoned the ritual phrase. “O, I want to speak to you.”

In the tree, a face appeared. O, looking every bit as majestic and powerful as Rak remembered him, was the face in the tree. “Tell me your problem, child.” Rak assumed O already knew, but wanted Rak to phrase it himself.

“The population is growing. Soon, there will be no more room for new villages.”
“Did you think I did not know?”
“I’m sure you know. But I wish to know your plan.”
“Humanity will be going outside.”
“Outside of what?”
“The earth.”

Rak stared at O, comprehension slowly dawning. “You mean,” he said, “we’re going into space?”

“Some of you, at least.”

Rak understood now the reason he was talking to O.

“I want to go.”
“As I knew you would. But you cannot go alone. You are almost of age, Rak, and only couples will go.”

Rak understood. Everyone knew that was best — to live life with another, to share joys and sorrows. He would need to marry Pom, and convince him to go with him. It would not be easy — Pom did not like big changes. But, after all, relationships cannot always be easy.

Pom nodded. He still felt uneasy, but he could see how much Rak wanted this, and he loved Rak. He forced himself into a smile, and said, “Sure, Rak! Let’s get married and go to space!” Pom enjoyed seeing the happiness on Rak’s face. They kissed, and Rak melted into his arms. This, thought Pom, was bliss.

The bliss only lasted a week, only to be replaced by pure horror. Kids? Never seeing anyone they knew, ever again? Pom was having nightmares about the vacuum of space sucking out all his joy, as a throng of kids was clamoring for attention. He evaded Rak’s gaze more and more, and started to find excuses to avoid Rak’s company. In the end, there was only one thing he could do.

“Dear Rak,” read the note that Rak noticed on the kitchen table. He had a feeling he knew what was coming — declarations of love, together with heart-breaking explanations for why they could not be together. He needed neither. He crumpled the note, and through it in out. It disintegrated in mid-air, and Rak was happy to see it go. It didn’t change what happened, of course. He had lost the love of his life, and nothing will ever be OK again.

“You have to come!” insisted San. Rak shrugged at her. “Come on! It’s going to be a beautiful ballet, and you know you love the ballet!” Rak nodded, still not convinced. San added, gently, “If you too were meant to be, you would be together. O willing, you’ll find someone. You’re a great guy.” Rak’s eyes teared, and San hugged him. He took a deep breath. “OK,” he said, let’s go.

The ballet was as beautiful as promised. The village dancing troupe was amazing, as Rak expected. He enjoyed looking at the fluid motions, the seemingly effortless flights through the air. He couldn’t avert his eyes. Pom was not much for the ballet, so they have not gone in a while. He forgot how much fun it was. When the dance was over, Rak was the first one to clap, the first to stand up, the last to stop clapping. San smiled at him, and said “lets’s go congratulate the dancers.” Rak knew some people did it, but he never mustered the courage, not until now.

“Amazing performance,” said Rak, as they went backstage. The head of the troupe, Jom, smiled at them. “Thank you! I think I know your names — Rak and San?” They nodded quickly. “I always host the dancers for a snack at my place,” he said, “would you like to join us?”

“I grow the apples myself,” Jom said, as Rak bit into a juicy one. He knew, of course, where the cookies and milk came from. Tag’s milk, doubtless, and Kop’s excellent chocolate chip cookies. Rak mused of the days long ago, when goods from far away places were exchanged. It was so much simpler to have the village grow all of its needs, freely shared. “Rak?” said Jom, and Rak realized he was day dreaming. “Wonderful apples,” Rak said, and meant every word. The apples were just sweet enough, just hard enough — he almost felt guilty destroying such perfection.

He noticed one of the dancers watching in amusement as he enjoyed the apple. “I’m Nal,” the dancer introduced himself to Rak, “I’ve rarely seen someone enjoy an apple that much.” Rak nodded. “I guess I…haven’t had one in a while. I forgot how much I like them. I’m Rak.”

Rak went outside, and started carving. Wooden utensils, he long knew, was to be his life-job. He was always drawn to the feeling of the wood, the subtleties that lay beneath the simple block. His hands were running the knife over the wood, as his thoughts were running around themselves. “Hi!” a voice interrupted his reverie, almost causing him to cut himself. He looked up. “Mom!” he said.

“I’ve heard about you and Pom,” she said. “I thought you would be interested to know that it’s not until I was 17 that I met your father.” Rak wasn’t sure if he was. It was…his mother, after all. “I was with my previous boyfriend for three years, and we were sure we were going to get married. But your father came along and..well…”

“Mom!” said Rak. “I have a point,” she said, “I’m not just enjoying you squirming there.” Rak fidgeted. “The right person always comes along. O gives us the opportunities. We just have to notice them. I know it feels like the end of the world right now…but things will get better.” She hugged him. “I promise.”

They did.

“We’re ready, O,” said Nal, a few weeks later, holding Rak’s hand. “Take us to space.” They would be, Nal and Rak knew, among the first of the colonists. As they stepped onto the space elevator, hand in hand, they were amazed at the speed O had built it. Pre-singularity humanity, they knew, thought of space elevators as complicated, expensive project. O (or to use the full name, SysOp) had finished it in weeks.

The day when Rak died started out like any other day. The colony was humming along, fifty people alone in space. The trouble started when the alarms blared. Rak looked up. “Pressure warning,” he said, surprised, “and just my luck to be on emergency duty.” Rak grabbed his gear and ran to the location indicated in the map. It was probably a false alarm, he thought to himself, but he was the one who was supposed to confirm it.

His eyes darkened as he saw the problem. Someone has been careless, apparently. The crack had to be there for a while, but someone taped over the crack sensor. There has been an earlier warning, probably, about a crack. Now a whole plate was gone, and patching that will be a pain. The air was whooshing out, past Rak’s ears, but he had plenty of time. The first thing to do was to report it. The only problem with sending a report was that the nearest comm unit was near the missing plate.

Rak pulled himself along the wall, carefully. A few more feet, and he will get to the comm unit, he thought. He would have, if another plate has not gotten loose. As he rolled into space, he remembered safety procedure was to wear a space suit first. He flew into the vacuum of space, hanging on by a thread. He opened his mouth, avoiding explosion. He calmly counted down the seconds until suffocation, and then it was all over.

The next thing Rak knew was looking into Nal’s eyes. O, of course, brought him back. Nal would have to nurse him back to health. That, of course, was as it should be.