Hacker News: Groupthink’R'Us

Do you want to write a post that will appear in Hacker News?

Here are a few blog headers that are sure to win you votes. Just write some filler and publish:

• Any post containing the words “iterate” and “learn”. Bonus points if you can weave in words about how failing is good and failing fast is better. In your post, deride the need for any planning with an explanation that “you cannot assume anything”, and explain how the best way to succeed is to iterate quickly.
• Explain why ad-supported products are a bad idea. Bonus points if you quote pointless statistics, explain about freemium or announce why nano-, pico-, or milli- payment systems will finally enable people to pay for stuff.
• Explain why start-ups are better than big companies. Use one of the common themes of flexibility and ability to take risks.

Seriously, all of these subjects have been done to death. We get them, really we do. The pros and cons have been explained, and making a more extreme post with an even more black-and-white picture is not what we want. And honestly? Debunking those is just as bad (done to death, we know it). Feel free to write such a post if it adds to your ego to have something rise to the top of HN. I’ll be here, waiting for more interesting posts there…

Managing Large Twisted Projects

Introduction

When working on a Twisted-centric project, it can be hard to recruit programmers. While Python, especially in recent years, has achieved main-stream status, Twisted is not quite there yet. It seems to be the case that recruiting Python programmers is, today, a solved problem. In the case where programmers unfamiliar with Python need to be recruited, resources (both free, cheap and expensive) abound for getting them up to par.

In the Twisted world, especially for large projects, programmers who have no previous experience with Twisted will need to be recruited. In all software projects, time is of the essence. An efficient, effective, way to get Python programmers into Twisted is desired. This is the case in VMware Israel (formerly B-hive), where the flagship product AppSpeed has multiple portions written in Twisted.

Teaching Resources

The broadest tutorial for Twisted is the “finger tutorial”. It leads the reader from writing the most basic finger server into a finger server supporting IRC and XML-RPC backends, going through a variety of Twisted programming techniques. Its broadness is also its chief downside, causing many newbies to learn many concepts which are irrelevant to their immediate needs, hiding the forest inside a density of trees.

The Twisted howto collection is another excellent resources. Two primary howtos are the “Writing Servers” which covers, in a focused, no-nonsense manner, the steps needing to write a server for a new protocol. Very little depth and “why” are covered, which is often as much as an upside as a downside. The other important howto is “Deferred Execution”, covering the basics of deferreds. Several other resources for covering deferreds in-depth are available.

Architecture

It is often the case that one, or more, of the people beginning a project have experience in Twisted. It is worthwhile to think how to attempt having a clean interface to those parts which need more such experience so that people writing other parts can hold their own with less such experience. Protocol details, for example, are often uninteresting, have a natural API and require non-trivial coding to deal with all the possibilities of packet portions arriving. The best wrapper for such details, if at all possible, is with an object whose methods return deferreds, and expose as few of the protocol details as possible.
An infrastructure containing such objects will help people learn Twisted in parts, while becoming productive programmers on the project early on.

Another useful part of the architecture is to compensate for beginners’ mistakes. The most popular such mistake is trying to do too much in a single callback, halting the reactor loop. Two solutions, often used in tandem, are useful for this problem. One is to limit the damage by trying to separate loosely-coupled tasks into separate processes, and using infrastructure such as AMP to communicate between those. In case one reactor is paused for too long, the others will still function correctly. The other solution is watchdogging: have a relatively-frequent “LoopingCall” start early in the infrastructure do some measurable action (sending a UDP packet, touching a file) and have a separate watchdog check for this action. If too long passes without the action being performed, the reactor is not functioning up to par — and the process should be rebooted. This, of course, necessitates correct recovery from rebooting: a rudimentary persistence framework, to say the least.

In places where long-running algorithms are required, or are merely easier to write, “deferToThread” is a useful abstraction. It allows the Twisted-beginner to shed all thoughts about events, and to write regular Python code. Those deferred-returning APIs will come in handy as they are used through reactor.blockingCallFromThread. It is useful to make sure blocking API meant to run in a different thread lives in separate modules — it makes code reviews easier when knowing in advance whether “time.sleep” or “reactor.callLater” is a bug.

Deferred-oriented infrastructure, a communication platform and a watchdog are three things most large Twisted projects will sport at one point or another. The sooner they are done, and the better, the better overall quality of the project will be — especially in a project which is introducing a large portion of its programmers to Twisted.

Summary

In most big Twisted-oriented projects, a large portions of the programmers, even if Python experts, will not know Twisted before starting on the project. It is useful to plan for this in advance, allowing an easy Twisted initiation for the beginner. This can be done in the three prongs of focusing on unavoidable Twisted issues first (deferreds and non-blocking rather than low-level APIs), compensating in the architecture for common mistakes and allowing a transfer of non-Twisted Python knowledge by allowing threaded abstractions.

The New “Star Trek”

Warning: extensive spoilers

Nothing is Sustainable

I rememer two distinct episodes in my life: when I first heard the word “sustainable”, and a few days later, when I was hit with the “wow” moment of understanding it. So simple! So elegant! It’s like a budget — if you spend (whatever) less than when you bring in, you can survive indefinitely. The ecological equivalent to Paul Graham’s “Ramen Profitable” — immortality depends on a positive life-sustainer flow.

This idea is indeed elegant, with one small flow. The second law of thermodynamics means that no process is really sustainable. In the case of Paul Graham’s “Ramen Profitable”, it doesn’t matter — it’s close enough to infinity that other issues (not negative cash flow) will cause the company to close if it doesn’t make it big fast enough. But for ecological processes, they’re finite. That’s reality.

So I would like to offer an alternative to discussion in terms of sustainability (which, like I indicated, is impossible). Let’s discuss processes in terms of “runway”. Again, this term derives from start-up thinking: the runway a company has is the amount of cash it has divided by the burn rate. By the time we reach the end of the runway, we have to “evolve or die” — find another process, or die out of insufficient resources. Oil has “runway”: how long we can keep without finding a way to use energy that doesn’t come from old dinosaurs. Emissions have “runway”: how long we can keep pumping emissions into the atmosphere before making areas unlivable. We should calculate the runways, and make sure to finance programs with estimated time of finish before the end of the runway, while understanding that nothing is sustainable, and we need to evolve or die.

In these terms, the “sustainability” advocates say we must reduce the burn-rate to extend the runway. That’s one good idea of course, but it’s irrelevant before we made sure we understand how to get off the runway into the next process, and how long it is going to take. Even the most wild-flying start-up has some kind of plan, and so should we!

Mailer for (Virtual) Appliances

Many virtual appliances want to communicate by e-mail: mostly by sending. The standard solution for that is to use some off-the-shelf mailer, write a process that communicates user-configuration to the mailer configuration file, and be done with it.

There is just one problem: all mailers suck, big time. I thought it was just true for mail user agents (MUAs) but apparently it is true for MTAs too. Let’s go down the list, shall we?

• qmail: more or less impossible to distribute in compiled form, and so useless for virtual appliances
• postfix: IPL has the obnoxious patent clause which is an unacceptable legal risk for companies
• sendmail: Ideally, you’d like whoever is writing the configuration generator to keep his sanity
• smail: too weak, the first page warns about an exploit that existed since 1992
• exim: complicated configuration, random bugs (bad recovery from unresponsive DNS servers, e.g.) Best of a pretty poor bunch

I don’t think I’m missing any big competitor. Now, for appliances, you do not need things like relaying (unless the goal of the appliance is to be a mail server, in which case, man, good luck with that). You do not need things like mailboxes, and usually no way to get mail would be fine. (Anyone who would like to reply to mail from an appliance would usually prefer the mail to go to a specialized mail server, and for the appliance to be able to access its mail via POP/IMAP.)

So, the ideal mailer for a Virtual Appliance would have

• An easy way to use the two most popular configuration: “send all e-mails through a relay” or “use MX”
• Accessible way to report problems with sending: either permanent errors or giving up on retry attempts (the standard solutions, replying by mail, are not useful in the case of the appliance). Ideal would be a parseable logfile (in addition to the usual developer/adminstrator logfile).
• No open port 25 from the outside
• It’s an appliance — let’s make it easy to control the outgoing queue with a simple configuration
• Easy way to configure retry attempts
• Ideally, as few bugs as possible

For that much configuration, we basically need the following configuration variables:

• “Use Smart Host”, boolean
• “Smart host” (IP/Name, irrelevant if “Use Smart Host” is False)
• Log files “prefix” (use suffix of _main and _parsed)
• Queue place
• Queue max size
• retry_attempts = (list of numbers, seconds to wait between retry attempts)
• SMTP listening port

Next, we’ll need the sendmail executable so programs can, indeed, send e-mail. It’s best to make this compatible with the regular sendmail — it will make porting code to the appliance easier. So, as long as we open the listening port for localhost only (the only option), our sendmail executable can parse, connect  to the local port, and deliver the message via SMTP.

Note that most appliances do not need to support a high load of ourgoing mail. Therefore, it is perfectly reasonable to write this in a high level language, such as Python. It would be nice to support the kill -HUP functionality to reread the configuration file.

I think this is a nice design document — and I think there’s a good chance I’ll be working on this in the near future. Anyone with me?

Memory allocation strategies: conservative lifetime estimation

I hope that most readers do not have to deal with these questions anymore — ideally, people should be using infrastructure that does memory management by itself. However, it is still often enough the case that for a specific use-case, one needs to deal with manual memory management. This is done, classically, with what I like to call malloc/free matching dance. You go through the code, analyze it, and make sure that every malloc is matched up with a free.

The problem with this strategy is that bugs are not *local*. When you do malloc, you effectively insert a bug — there is no corresponding free yet. The solution for that bug is that somewhere far away you add a “free” call. This match up, particularily in complicated code, is non-trivial. It is clear why it is popular, though — it can be thought of as an attempt to do a John Henry — simulate the way an automatic memory reclamation code would work, freeing the memory as soon as it knows it is unneeded. Much like in the classic folk tale, this strategy does not bode well for its instigators.

An alternative is “conservative lifteimte estimation”. When doing the allocation, make a conservative estimate about when you will need this memory no longer. This might mean a literal time (”in one hour”) or when an event passes (a function returns, a connection is finished). As soon as the allocation is done, attach the “free code” to the event, and forget about it. Come the event, free all the memory.

I have used this strategy to fix rampant memory allocation bugs in two cases. In one case, I could prove that the memory will be needed in only one re-entrant function. I used a static buffer, wrote a simple allocator (no free, pointer to free space incremented each allocation), and marked it as “clear” before the function. In the other case, I wrote a class called a “Guard”, to which objects of class “Actions” could be attached. When the guard executed its destructor, it would call all the actions. Add to that a template DeleteAction inheriting Action and destroying a pointer to T when the action was called, defining a stack-Guard in some functions, and voila, we had a way to make “function end event” a first class object in C++.

This strategy is older than that, of course. The classical unix trick of spawning a process, killing it, and inside the process never calling free, is just an OS-supported version of the same idea. Apache uses pools, which make a “liftetime estimation” a first class object, and uses at-most-n-requests-per-child as another usage of this strategy. While this strategy frequently overallocates, it allows you to write code which is obviously correct in many cases.

Real-world Software Engineering

Here is an idea for a “real-world software engineering” class, that I would be really happy if it was offered by universities.

Choose a type of software, for example, a wiki. Choose the technology, and the language (let’s say — Python, Django). Write this wiki, before the class begins. The students have to work, in pairs, on fixing a bug in the wiki. This is exercise 1. The succeeding exercise each involve changing the requirements, and asking for a new version — basically, a new version every week or two. The wiki starts out without any unit tests, but every area the students toucuh has to have unit tests added to it to see the change is ok and didn’t hurt directly related functionality.

The class itself, other than the exercises, will cover topics like using unit tests, source control, bug management systems and so forth. It will be expected the students will actually use these techniques during the exercises — but for the first time in their life, they’ll use it on real code, for good reasons, and they’ll need to use them correctly to do the right thing. I expect the exercises to be most of the grade, with a test to go over correct understanding of the concepts.

I feel like this class would be the best preparation for students in a software-engineering/CS degree for the real life software industry, as I know it. Any comments? Anyone know of a university giving such classes?

Assume AJAX

Assume AJAX.

If you are writing a web application, and you hope for it to be popular, assume AJAX. You will add it at some point. And you know what? You’re going to be duplicating a lot of server-side code in JavaScript.

Let’s start with validation. Server-side validation is a must, client-side validation is crucial for a good user experince. Next comes the typical “query from DB” presentation style (and it does not matter if it’s google serch result of facebook friend feed): a list of things, all alike. You’ll want to be able to refresh those in real-time. Another case for code duplication. If AJAX is going to show up anyway, wouldn’t we want to make it easier and with less duplicate bugs?

Well, there is one obvious solution. Have it all in AJAX. Send over static HTML files, and fill up the content with AJAX. Do the validation in AJAX, and send in remote calls (the remote calls will hopefully just have exceptions if the values are not the right type). Have stuff in JSON-RPC or something RESTful, and let AJAX do the heavy lifting. In the future, that might be an option.

But if you care about user experience, you don’t want to do that. It would take way too long for the user to get the initial approximation, compared to a traditional solution. The best of all worlds would be a server that just did the AJAX for you. It would be completely untrusted, and would serve HTML by getting the static file and running the same AJAX code being run from the client. All it would take is just another load event (say, OnServerLoad) which would be for the server. You would want a fairly compatible “browser” as the basis for the server. Perhaps one which knew how to do separation between processes well?

Yep, server-side chromium with a few API tweaks would be the perfect template engine. Put it, and your actual running-code-accessing-privileged-stuff *-RPC/REST server behind a simple squid reverse proxy with appropriate rules, have the AJAX do the calls to the backend, and you have an AJAX-ready application with your JavaScript framework of choice baked right in.

(Addendum: Jaxer seems to be close, but still no cigar)

Web Application Frameworks

So, you want to write a web application, and you’re looking for a framework?

I bet you don’t. I bet what you are really trying to write is a fancy CMS. “Wait!”, I hear you cry. No, you wait…this is my post.

Let me explain what I mean by CMS (Content Management System): it’s a system that is involved in moving some content from some users to others, where “anonymous” is also considered a user. Often, the ability to integrate mail (being able to send mail in and most importantly to get mail notifications) is also considered a part of CMS.

Where does that leave us? Facebook: a CMS. Twitter: a CMS. Livejournal: a CMS. Gmail: a CMS. Google Search? Nope, that’s not a CMS: that’s an actual web application. It gets input, it runs some code, it shows output.

Ever wondered why ORM is so popular in “web applications”? Because when you’re primarily moving content about, describing that content is a really important part of the equation.

CMSs are basically about communication. This is why they are so popular: people enjoy to communicate. A lot. (Hey, people reading my blog!)

That’s fine. If you wanna write a CMS, go ahead. Use Django, or RoR or whatever. I bet some of these are better than others — but because I have no experience with CMSs, I cannot give good advice on that topic.

If, however, you want to write an actual web application, not a fancy CMS, what will you use? Certainly not Django or RoR. That helps you very little. Yes, it’s got “a request object” (that wraps around 3 WSGI objects/methods). It’s got a “dispatcher” (how long would it take you to write something that maps URLs to code? 5 minutes). And it’s got a templating system.

But here is the point — I don’t know if Django’s templating system is any better than any other. I’m not a fan, of course — I would use Nevow’s template, or perhaps ZPT. But templating is really about choosing a language, so arguments tend to be almost purely emotional here.

So what’s an example of a web application? Well, let’s enter my mind for a minute, and see what we find. These days, I’m involved with AppSpeed, an application-level performance monitoring system. It’s an application that helps you monitor your performance. But it’s not out yet, so let’s look at other things in roughly the same category: managing your wireless router. It’s about the same general niche — data center management. Except this is about the tiny data center you have in your house.

The requirements for both are the same — we want to use the web application to *understand* and *control* what’s going on in the infrastructure. There’s no ORM: if you’re keeping data, be it configuration, logs or history, you already have that choice before you started writing the web part.

So, given that dispatch is trivial, and the ORM is irrelevant, you need to choose two things: a templating system, and a JavaScript framework if you will use any AJAX at all. Templating systems are a dime a dozen, and JavaScript frameworks are likewise — and any combination works more-or-less as well as any other.

So, what’s my point? Web application frameworks are really CMS application frameworks. If you want to write a CMS, go ahead — it might get popular and worth billions of dollars.

But if you write a web UI for a non-CMS applications, just choose a random templating system, and enjoy…as long as you remember that any commonly-known wisdom about web application has a good chance of not applying to you.

Birthday Party Quest 1

> look
You see three buildings marked A, B and C
> go building A
You go to building A
There is a list of names.
> read list
None of the names look familiar
> go building B
You go to building B
There is a list of names.
> read list
None of the names look familiar
> go building C
You go to building C
There is a list of names.
> read list
None of the names look familiar
> inventory
cellphone
car keys
pouch
backpack
> look backpack
The backpack contains:
 USB A-to-mini-B wires
 Tums
 earplugs
 Lenovo laptop
> look pouch
The pouch contains:
 eeePC laptop
> look cellphone
The battery appears to be dead
> charge cellphone
Charge cellphone with what?
> charge cellphone with eeePC laptop and USB wire
Cellphone seems to be charging
...
eeePC battery dead.
> look cellphone
Wireless appears to be off. The message reads "Not enough battery power for wireless"
> press "Recent calls" on cellphone
One of the recent calls appears to be from a friend
> charge cellphone with Lenovo laptop and USB wire
Cellphone charging
...
> look cellphone
The message reads "Restore connections? [OK] [Cancel]"
> click ok
Cellphone wireless is now on
> call friend on cellphone
Your friend tells you to go to building C, floor 16, apt. 61
> go to building C, floor 16, apt. 61
You have found the birthday party! Congratulations.
You win