Managing Large Twisted Projects


When working on a Twisted-centric project, it can be hard to recruit programmers. While Python, especially in recent years, has achieved main-stream status, Twisted is not quite there yet. It seems to be the case that recruiting Python programmers is, today, a solved problem. In the case where programmers unfamiliar with Python need to be recruited, resources (both free, cheap and expensive) abound for getting them up to par.

In the Twisted world, especially for large projects, programmers who have no previous experience with Twisted will need to be recruited. In all software projects, time is of the essence. An efficient, effective, way to get Python programmers into Twisted is desired. This is the case in VMware Israel (formerly B-hive), where the flagship product AppSpeed has multiple portions written in Twisted.

Teaching Resources

The broadest tutorial for Twisted is the “finger tutorial”. It leads the reader from writing the most basic finger server into a finger server supporting IRC and XML-RPC backends, going through a variety of Twisted programming techniques. Its broadness is also its chief downside, causing many newbies to learn many concepts which are irrelevant to their immediate needs, hiding the forest inside a density of trees.

The Twisted howto collection is another excellent resources. Two primary howtos are the “Writing Servers” which covers, in a focused, no-nonsense manner, the steps needing to write a server for a new protocol. Very little depth and “why” are covered, which is often as much as an upside as a downside. The other important howto is “Deferred Execution”, covering the basics of deferreds. Several other resources for covering deferreds in-depth are available.


It is often the case that one, or more, of the people beginning a project have experience in Twisted. It is worthwhile to think how to attempt having a clean interface to those parts which need more such experience so that people writing other parts can hold their own with less such experience. Protocol details, for example, are often uninteresting, have a natural API and require non-trivial coding to deal with all the possibilities of packet portions arriving. The best wrapper for such details, if at all possible, is with an object whose methods return deferreds, and expose as few of the protocol details as possible.
An infrastructure containing such objects will help people learn Twisted in parts, while becoming productive programmers on the project early on.

Another useful part of the architecture is to compensate for beginners’ mistakes. The most popular such mistake is trying to do too much in a single callback, halting the reactor loop. Two solutions, often used in tandem, are useful for this problem. One is to limit the damage by trying to separate loosely-coupled tasks into separate processes, and using infrastructure such as AMP to communicate between those. In case one reactor is paused for too long, the others will still function correctly. The other solution is watchdogging: have a relatively-frequent “LoopingCall” start early in the infrastructure do some measurable action (sending a UDP packet, touching a file) and have a separate watchdog check for this action. If too long passes without the action being performed, the reactor is not functioning up to par — and the process should be rebooted. This, of course, necessitates correct recovery from rebooting: a rudimentary persistence framework, to say the least.

In places where long-running algorithms are required, or are merely easier to write, “deferToThread” is a useful abstraction. It allows the Twisted-beginner to shed all thoughts about events, and to write regular Python code. Those deferred-returning APIs will come in handy as they are used through reactor.blockingCallFromThread. It is useful to make sure blocking API meant to run in a different thread lives in separate modules — it makes code reviews easier when knowing in advance whether “time.sleep” or “reactor.callLater” is a bug.

Deferred-oriented infrastructure, a communication platform and a watchdog are three things most large Twisted projects will sport at one point or another. The sooner they are done, and the better, the better overall quality of the project will be — especially in a project which is introducing a large portion of its programmers to Twisted.


In most big Twisted-oriented projects, a large portions of the programmers, even if Python experts, will not know Twisted before starting on the project. It is useful to plan for this in advance, allowing an easy Twisted initiation for the beginner. This can be done in the three prongs of focusing on unavoidable Twisted issues first (deferreds and non-blocking rather than low-level APIs), compensating in the architecture for common mistakes and allowing a transfer of non-Twisted Python knowledge by allowing threaded abstractions.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: