Deploying With Docker

March 23, 2015

Glyph wrote about an interesting way to deploy Python applications to Docker. I wanted to try it out, and exercise ncolony a little. The result is NaNoAuto, which is still very preliminary. It was mostly written as a way to test out deploying to Docker and exercise ncolony. Several notes that I learned the hard way:

  • On Ubuntu, all docker client commands need to run as root to access the docker container’s UNIX domain socket. I think there is a way to run it on a host/port and ask for a username/password, but I have not figured that out.
  • Glyph’s scripts need a recent version of Docker. The built-in version for Ubuntu is too old — make sure to install docker from Docker’s apt repository.
  • The entry point you give Docker must not exit — and so must not daemonize. Glyph’s app used an explicit reactor running it (well, even worse — Klein’s implicit reactor). Civilized people, of course, use ‘twistd’. When building things like a civilized person, please remember to pass the “–nodaemonize” flag (AKA “-n”) to twistd.
  • On my personal TODO list is to add a hash checker to the runner, so that the wheels installed from the internet can have their hashes matched. It is possible that “peep” could play a role here.
  • The scripts are divided into “build-base” and “do-build”. “build-base” is the long-running but fairly stable part — it only changes if you change the basic platform (OS, Python interpreter, needed non-Pythonic parts). The “do-build” part is fairly fast, and needs to be run for every change to a Python file.

Note that this still means a little bit of overhead on top of building a new wheel for each change to a Python file. It should be possible to write something on top of “docker exec” that will copy Python files directly from source to container so as to facilitate even quicker edit/debug cycles.


Announcing: NColony — the infrastructure for your Python

March 14, 2015

For the last few days, I have been working on infrastructure for writing multi-process servers: basically, something that would be a good start-command for your Docker, or something you put in your ‘@reboot’ crontab to bring the entire thing app. It is still not tested, mature or production ready — there are still a lot of unimplemented features. I am putting the announcement up now, though, because at least everything that still needs to be done (as far as I know) to get it feature-ready is documented as an issue.

The code, issues and so on are all on the Github for ncolony. Patches for existing issues are welcome, as well as new issues. Questions, comments and code-critiques are also more than welcome.


sphinx-quickstart: Bad idea, or worst idea?

March 3, 2015

When reading the documentation for sphinx, it suggests getting started by running the sphinx-quickstart “wizard”. Inspired by such beloved pieces of software like PowerPoint, sphinx-quickstart will ask you questions about your project and then manufacture a bunch of files which, when built, produce pretty documentation. This…this is wrong. We don’t start writing a new project in Python by having it ask a bunch of questions and then manufacture setup.y. We document, we have samples and we let people who are, ideally, capable of writing complicated Python module to create those using their favorite editor.

One might think perhaps this is a design limitation of sphinx  but it is not really. It’s just a silly little thing in the documents. For a minimal (and somewhat typical) sphinx set up all you need is:

index.rst

 
PROJECT-NAME-HERE
=================

.. toctree::
   :maxdepth: 2

conf.py

master_doc = 'index'
project = 'PROJECT'
copyright = '2015, YOUR NAME'
author = 'YOUR NAME'
version = release = '0.0.1'

Building the documentation, something that sphinx-quickstart will helpfully create a Makefile (that one calls recursively from their Python Makefile, I guess) and Windows batch file [in sphinx-quickstart’s defense, it asks whether to do those things]. Again, one might have the impression that perhaps sphinx is so complicated one might need those. Here is the complicated command needed to build the documentation:

sphinx-build -b html . _build/html

Sphinx is pretty well documented. Starting from this skeleton, it is easy to add extensions, change configuration options, etc. There is absolutely no excuse for sphinx-quickstart, a blemish on an otherwise pretty nifty documentation system. If you were also turned off by needing to run a quickstart script that fills your source directory with things, fear no more. Build your skeleton yourself, and enjoy beautiful docs!


Clash of Clans: War strategy tips

October 22, 2014

[I wanted a central place to put my CoC Clan War strategy tips so I can refer people to them.]

Defense

  • A town hall outside the walls is great for farming, but your war base should have amazingly good protection of the town hall, otherwise they get, essentially, a free one star (together with the one star for 50% that people can get by just taking down the outermost buildings with an archer rush, this almost always means free two stars).
  • Air defenses should be well protected, or even a simple “take out the air defenses” together with a castle dragon can let even relatively low-level town hall take out pretty impressive bases.
  • Do not bother with walls around your town hall. If giants got to the point where they have nothing to do but to take out your town hall, you have bigger problems and otherwise, the archers will take out your town hall while leaving the wall intact. Instead, use walls to protect the giants’ favorite targets: wizards and mortars.
  • Avoid the “eggshell” design: one strong outer wall with no internal walls. Once the outer wall is breached, your base is effectively wall-free.

Offense

  • Meta tip #1: Remember to question your instincts if you raid for trophies or loot a lot. War has different incentive schemas and different pros and cons.
  • Meta tip #2: [Continuation of meta tip #1] You have more than 30 seconds to figure out your strategy. You can look at the enemy village, and even tap buildings to see ranges. Spend some times thinking about your strategy before sending in your first troop,
  • If you are not the first to attack, watch replays of all previous attacks. Make sure to memorize:
    • Location of traps
    • Location of hidden tesla towers
    • Composition of clan castle
  • Go for the three stars. A one-star attack is almost useless, and a two-star attack is worth only 66% of a three star attack. Only go for the one-star if nothing else seems realistic.
  • The clan castle almost certainly has troops in it. Probably a mix of archers and wizards.
    • Check the range of the clan castle ahead of time, and plan how to draw the troops out.
    • Giants and, to a lesser extent, hog riders, are extremely vulnerable to castle troops, because they do not fight back. Clear castle troops before sending them in, if possible.
    • Make sure to draw out troops and kill them with either a lightening spell or archers/wizards (you’ll probably need the equivalent of 20-30 housing spaces, in either archers or wizards, to demolish a 20-25 sized castle).
    • If you draw out troops, try drawing them out to an area that’s free of other defensive buildings.
    • Make sure you drew out all troops: some will sometimes stay in. Better safe than sorry.
  • If you are going to send in dragons or healers, it is probably a good idea to get rid of the air defenses using giants or hog riders first. Air defenses are extremely effective against flying troops. [This does not apply if you plan your attack taking into the air defenses into account and being OK with the flying troops having a limited life time.]
  • Walls should be dealt with, if possible, by wall breakers and not keeping the giants there hacking at the wall. First, this is an inefficient use of the giants’ hit points, and worse, this wastes a lot of time. Time is of the essence for getting all three stars.
  • Send troops (somewhat) en-masse. Sending the archers or wizards one-by-one just lets the defensive buildings take them out one by one. Even better, send mass amounts of archers or wizards hidden behind giants to let giants soak up the pain while the archers take out buildings.

A Problem With High School Math Education

September 25, 2014

When I studied math in high-school, I absolutely hated geometry. It pretty much made me give up on math entirely, until I renewed my fascination in university. Geometry and I had a hate/disgust relationship since. I am putting it out there so if you want to blame me for being biased, hey, I just made your day easier.

But it made me question: why are we even teaching geometry? Unlike the ancient Egyptians, few of us will need to collect land taxes on rapidly changing land sizes. The usual answer is that geometry is used to teach the art of mathematical proofs. There are lemmas-upon-lemmas building up to interesting results. They show how mathematical rigor is achieved, and what it is useful for.

This is bullshit.

It is bullshit because there is an algorithm one can run to manufacture a proof for every true theorem, or a counter-example for every non-theorem. True, understanding how the algorithm works takes about two years of math undergrad studies. The hand-waving proof is here: first, build up the proof that real number pairs as points and equations of lines as lines obey the Euclidean axioms. Then show that the theory of real numbers is complete. Then show the theory of real numbers is quantifier-free. QED.

When we are teaching kids geometry, we are teaching them tricks to solve something, when an algorithm would do it at well. We are teaching them to be slightly-worse than computers.

The proof above is actually incorrect: Euclidean geometry is missing an axiom for it to work. (The axiom is “there is no line and a triangle such that the line intersects the triangle in exactly one edge”.) Euclid missed it, as have generations after him, because geometry is visually seductive: it is hard to verify proofs formally when the drawings look “correct”. Teaching kids rigor through geometry is teaching them something not even Euclid achieved!

Is there an alternative? I think there is. The theory of finite sets is a beautiful little gem of mathematics. The axioms are the classic ZF axioms, with the infinity axiom inverted (“every one-to-one function from a set to itself is onto”). One can start building the theory the same as building the ZF theory: proving that ordinals exist, building up simple sets and the like — and then introduce the infinity axiom and its inversion, show a tiny taste of the infinity side of the theory, and then introduce the “finiteness axiom”, and show how you can build the natural numbers on top of it.

For the coup-de-grace, finite set theory gives a more natural language to talk about Godel’s incompleteness theorem, showing that we cannot have an algorithm for deciding questions about finite sets — or the natural numbers.

Achieving rigor is much easier: one uses Venn diagrams as intuitions, but a formal derivation as proof. It’s beautiful, it’s nice, and it is the actual basis of all math.


Clash of Clans and the Prisoners’ Dilemma

July 18, 2014

I have started playing, recently, the online game “Clash of Clans”. Clash of Clans has the option of “donating” troops to other clan members. Troops donated stay in a separate area (so they do not take up area for one’s own troops), and also participate in base defense (as opposed to one’s own troops, which are offensive only). In short, being donated-to makes both one’s offense and one’s defense stronger. However, donating troops means paying the cost of training them and reaping no benefit from it.

Does we have the pay-off matrix: assume paying the cost of troops is -1, and the benefit is +2 (it has to be bigger than the cost, since players do build troops for themselves with even less benefits):

Donate/Donate: 1/1
Don’t donate/Don’t donate: 0/0
Donate/Don’t donate: -1/2
Don’t donate/Donate: 2/-1

What I love about this is this is a massive experiment with prisoners’ dilemma on more-or-less average people. As expected, one sees both equilibrium: clans where hardly anyone donates, and clans where everyone donates. In the clans where hardly anyone donates, there is still some reciprocity: (some) people will try to donate “back” to people who donated to them. Since in practice the cost-benefit analysis turns that way, it is sometimes worthwhile to donate even for a .2 chance of a reciprocation, which allows the loop to be fed.

There are reputation effects (the number of troops one donated and one had donated to appears where everyone can see it), which would be assumed to improve the donation rate. However, from my anecdotal experience, being in two clans, the most distinguishing feature is “do you know these people in real life?” Clans made up of people whose reputation effects extend beyond the game are in the good equilibrium. Since I’ve joined the Facebook-employee-only clan, my life has been much better.

I started thinking that Facebook is like that too. In situations where you can help a colleague, where the benefit to the colleague would be disproportional to one’s self, we encourage that kind of thinking. Facebook is in the good equilibrium of the dilemma!

[Plug: We’re hiring engineers and eng. managers. Please contact me on FB with further questions or resumes.]


Thoughts on Warrants

June 30, 2014

Warrants (and similar things, like indictments at preliminary hearings) rely on the concept of “probable cause”. The definition of probable cause varies, but in general requires for something to be more probable than not — or in terms of Bayesian probability (which applies probability to states of mind, not just to repeatable experiments), higher than 0.5 probability. Do our warrants actually obey this ruling? This is a fairly simple test — at least 50% of warrants handed should, eventually, result in evidence leading to conviction[1]. Do they? Hell, in less than 50% of cases does it lead to an indictment[2]!

Since warrant judges hand out a lot of warrants, we can decide to actually measure and automatically disbar judges whose warrant rates fall significantly below 50% rates (e.g., using p values of 0.005 would correspond to having a pretty high Bayesian prior that judges are “good”). The criteria could simply be — “of all warrants handed out where the case has ended, how many convictions have there been” (so any cases still in trial would be eliminated from consideration). After doing this for a while, we can update the Bayesian prior, periodically, to how many judges actually get disbarred by this.

In a perfect world, we would also apply this to convictions — however, since convictions are handled by one-offs (jurors), it is hard to place blame for overturning convictions. However, at least for warrants and non-grand-jury-indictments, this allows a process of automatic continuous improvement of our standards, avoiding the “judges and police co-operate to come up with lots of warrants” problem.

[1] Technically, since conviction is “beyond reasonable doubt”, this is a lighter standard than the true standard of actual guilt. However, based on the “better a hundred guilty people go free…” (Benjamin Franklin) standard, reasonable doubt being >=1% probability, the effect should be minor.

[2] http://www.insurancejournal.com/news/east/2014/06/27/333106.htm


Follow

Get every new post delivered to your Inbox.

Join 307 other followers