 |
A record of some work I do at the Media Lab. |
I flew out to Kansas City to represent Paulina Modlitba's Globetoddler project. It was a whirlwind 3 day trip that included brief tours of the city and Hallmark in addition to a presentation and demo session. Andres took some photos of the trip.
Me, Leonardo and Hallmark's Jeff Manning discuss our plans for world domination.
I've finally posted my writeup of trash talk, a prototype of a mesh-networked public address system. Check it out!
I've fallen in love with markdown. It's really simple and easy to use -- and results in cleaner, more readable, and compliant HTML than I end up with when I code it by hand.
I also love Django, a fantastic python web development framework. And it even has out-of-the-box support for markdown as a template filter. But I don't use django just to do standard database-driven blogging stuff like this blog -- I also use it to run my whole web site, and use it to organize a lot of projects and pages for which the individual look and feel of the page is an integral part of the content. This means that I can't just plug some "data" about a project into a template -- the template is the product. Consequently, many of my pages are their own one-off templates. But that means there are many cases where the content I want to use markdown with isn't coming from some variable from a database, but is part of a template that I'm writing.
Long story short: I want markdown as a template tag, not just as a filter. So I wrote a very simple template tag to do this:
# markdown_tag.py: a markdown template tag for django. from django.contrib.markup.templatetags import markup from django import template register = template.Library() @register.tag(name = "markdown") def markdown_tag(parser, token): args = token.contents[len("markdown"):].strip() nodelist = parser.parse(('endmarkdown',)) parser.delete_first_token() return MarkdownNode(nodelist, args) class MarkdownNode(template.Node): def __init__(self, nodelist, args): self.nodelist = nodelist self.args = args def render(self, context): output = self.nodelist.render(context) return markup.markdown(output, self.args)
To use this, save it as "markdown_tag.py", and put it in a "templatetags" directory in your application, like so:
- project/
- application/
- models.py
- views.py
- urls.py
- templatetags/
- markdown_tag.py
Then, in your template, include the following:
{%load markdown_tag %}
...
{% markdown %}
# This is a markdowned heading!
{% endmarkdown %}
This also supports extensions, if you have the latest version of markdown - as in this example, with the codehilite extension:
{% markdown codehilite %}
:::python
print Hello, world!
{% endmarkdown %}
My first YouTube videos, for two current projects - "Free Press", and "Extensible Media Insurgency Toolkit":
Suppose you want to do secure uploads of data when you don't have a lot of infrastructure like computers and wifi networks and such. Say all you have is a cell phone. You might need to resort to sending SMS messages, saving data to a flash card and trucking it out manually, or if you're lucky, uploading the data over a GPRS connection. You can't rely on the existence of an HTTPS or secure socket connections, so you can't rely on the phone's native TLS implementation to encrypt the data. So you have to write an application that does the encryption for you. Fortunately, there are free java cryptographies available so you don't have to write your own. I chose Bouncy Castle, which has a "lightweight" version which is geared toward mobile phones. Unfortunately, the documentation is also a bit on the lightweight side, so I wrote the following to show how I got asymmetric RSA and symmetric AES encryption working in a J2ME MIDlet.
We want to do secure uploads to a server, but don't want the data to be comprimised should the phone be lost. Consequently, it's best to use asymmetric public key cryptography. That way, the phone only stores the non-secret public key to encrypt the message, and the server which is safely far away from prying eyes stores the private key which decrypts the message. This is in contrast to Jonathan Knudsen's example "whisper" MIDlet which uses a shared key for symmetric encryption between the sender and receiver, and leaves the messages vulnerable should the phone be lost.
Bouncy Castle implements two asymmetric engines in the lightweight API - RSA and ElGamal. I decided to use RSA, primarily because it is far more popular and widely supported, and consequently would be easier to do on the server side. However, the RSA implementation in Bouncy Castle only works for encrypting a single block of data - which is only 127 bytes if you use a fairly standard 1024 bit key. If your message contains a lot of data (such as pictures, videos, or news stories), this is unacceptably small. In addition, RSA is relatively inefficient in comparison to commonly used symmetric block ciphers like AES, and would become very slow with the limited resources of a mobile device. So how do you efficiently encrypt long messages? The recommended solution is to encrypt the whole message using a symmetric algorithm like AES with a randomly generated key, and then to encrypt this key using RSA and send the encrypted key along with the message. The server must then first decrypt the symmetric key using its RSA private key, and then use the symmetric key to decrypt the rest of the message.
One more complication comes our way: the initialization vector. Symmetric block ciphers like AES operate in one of several modes of operation (such as ECB, CBC, CFB, etc.). Basically, these modes describe how the cipher should treat chains of blocks while encrypting a multi-block message. In order for a block cipher to generate a unique ciphertext every time (even if the plaintext is identical), it is necessary to put the block cipher in an initial random state before encrypting. The decrypting receiver must then also have the initialization vector to set its internal state before decryption. The use of an initialization vector in the context of our implementation here may not be necessary (since we randomly generate a new key each time the message is sent, which makes the ciphertext unique regardless of the initial state). However, I am not enough of an expert in cryptography to predict the effects of its omission, so I decided to include it anyway.
Consequently, the message we send now consists of 3 parts:
First step needed is to generate an RSA key pair. The private key will sit safe and secure on the server, the public key will be distributed with the application that uploads encrypted data. Here are excellent instructions for how to generate the key. Briefly, it boils down to the following commands on a linux system:
$ openssl genrsa 1024 > host.key $ chmod 400 host.key
$ openssl req -new -x509 -nodes -sha1 -days 3065 -key host.key > host.cert
$ cat host.cert host.key > host.pem $ chmod 400 host.pem
I spent a long time trying to work with the obscurely named and complex class hierarchy of the BouncyCastle lightweight API in order to import the public RSA key directly into the java app. While I'm sure it's possible, I did not succeed. However, a simpler solution is just to read the only two parts of the public key that matter to the crypto algorithms: the "exponent" and the "modulus", used by the one-way math functions that do the encryption. While this ignores the authentication of the key provided by having a properly signed certificate, my assumption here is that if someone had sufficient access to the phone to replace the certificate stored in the MIDlet's jar with a bogus one, they would've had sufficient access to replace the MIDlet's code anyway, making any authentication moot.
To read the data stored in the public key, run the following command:
$ openssl x509 -noout -fingerprint -text < host.cert
You'll get a whole bunch of information, and the key values we want look like this:
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:ad:7d:3b:17:77:68:c4:18:2e:db:3e:ea:0c:2c:
23:52:0a:34:1a:03:b5:2b:c7:0e:fa:19:68:35:29:
ee:84:35:76:bf:c1:4f:f1:f8:19:4b:1a:ee:9b:f2:
59:ce:ff:da:b2:21:a6:c8:51:4c:9d:31:07:59:e7:
84:90:27:30:4b:e5:3c:a4:e5:ff:1c:6a:b8:c6:13:
65:54:75:1e:c2:17:07:4a:be:41:56:44:ae:b6:8b:
b3:a8:68:66:25:7d:9f:72:e8:09:51:c9:e1:a0:f0:
00:71:b0:ec:13:25:ed:71:ed:9b:b9:cb:af:4c:3d:
06:1b:bb:66:90:97:e2:65:cf
Exponent: 65537 (0x10001)
That's the Modulus (a very long integer in hex), and the Exponent, a much smaller integer. The java BigInteger class doesn't take that gnarly hex format in its constructor, it wants a decimal string. So I wrote the following python script to simply extract the needed integers:
import os os.system("openssl x509 -noout -fingerprint -text < host.cert > host.info") info = open("host.info").read().split('\n') reading_modulus = False modulus_parts = "0x" for line in info: if line.strip().startswith("Modulus"): reading_modulus = True elif line.strip().startswith("Exponent"): exponent = line.strip().split(' ')[1] reading_modulus = False elif reading_modulus: modulus_parts += line.strip().replace(':', '') print exponent print int(modulus_parts, 16)
This will print out the exponent and modulus on two lines. If you save the script as "extract_modulus_and_exponent.py", you can use it like this:
$ python extract_modulus_and_exponent.py > rsa_public_key.res
The file rsa_public_key.res should be stored in the root of your jar to run with the following code.
So we need methods to generate the AES key and initialization vector - for this, we just use Bouncy Castle's implementation of SecureRandom. Next, we encrypt the message using the AESLightEngine in CBC mode. Finally, we encrypt the AES key using the RSA key values read from the resource file generated in the previous step.
import java.io.IOException; import java.io.InputStream; import java.math.BigInteger; import java.security.SecureRandom; import org.bouncycastle.crypto.AsymmetricBlockCipher; import org.bouncycastle.crypto.BlockCipher; import org.bouncycastle.crypto.InvalidCipherTextException; import org.bouncycastle.crypto.encodings.PKCS1Encoding; import org.bouncycastle.crypto.engines.AESLightEngine; import org.bouncycastle.crypto.engines.RSAEngine; import org.bouncycastle.crypto.modes.CBCBlockCipher; import org.bouncycastle.crypto.params.KeyParameter; import org.bouncycastle.crypto.params.ParametersWithIV; import org.bouncycastle.crypto.params.RSAKeyParameters; public class UploadCipher { private static final int AES_KEY_LENGTH = 16; // 16 bytes for AES-128 private static final RSAPublicKey RSA_KEY = new RSAPublicKey("/rsa_public_key.res"); private ParametersWithIV aes_key; private BlockCipher symmetricBlockCipher; private AsymmetricBlockCipher asymmetricBlockCipher; private int symmetricBlockSize; private SecureRandom secureRandom; public UploadCipher() { secureRandom = new SecureRandom(); // Prepare symmetric block cipher for message symmetricBlockCipher = new CBCBlockCipher(new AESLightEngine()); symmetricBlockSize = symmetricBlockCipher.getBlockSize(); createAESKey(); // Prepare asymmetric block cipher for key asymmetricBlockCipher = new PKCS1Encoding(new RSAEngine()); asymmetricBlockCipher.init(true, new RSAKeyParameters(false, RSA_KEY.MODULUS, RSA_KEY.EXPONENT)); } private void createAESKey() { byte[] aes_key_bytes = new byte[AES_KEY_LENGTH]; byte[] iv = new byte[symmetricBlockSize]; secureRandom.nextBytes(aes_key_bytes); secureRandom.nextBytes(iv); aes_key = new ParametersWithIV(new KeyParameter(aes_key_bytes), iv); } public byte[] encrypt(byte[] message) { // initialize block cipher in "encryption" mode symmetricBlockCipher.init(true, aes_key); // pad the message to a multiple of the block size int numBlocks = (message.length / symmetricBlockSize) + 1; byte[] plaintext = new byte[numBlocks * symmetricBlockSize]; System.arraycopy(message, 0, plaintext, 0, message.length); // encrypt! byte[] ciphertext = new byte[numBlocks * symmetricBlockSize]; for (int i = 0; i < ciphertext.length; i += symmetricBlockSize) { symmetricBlockCipher.processBlock(plaintext, i, ciphertext, i); } return ciphertext; } public byte[] getKey() { try { byte[] key = ((KeyParameter) aes_key.getParameters()).getKey(); return asymmetricBlockCipher.processBlock(key, 0, key.length); } catch (InvalidCipherTextException icte) { icte.printStackTrace(); return null; } } public byte[] getIV() { // Encryption here is probably optional; some sources say the IV can be // sent in plaintext. try { byte[] iv = aes_key.getIV(); return asymmetricBlockCipher.processBlock(iv, 0, iv.length); } catch (InvalidCipherTextException icte) { icte.printStackTrace(); return null; } } } class RSAPublicKey { public BigInteger EXPONENT; public BigInteger MODULUS; public RSAPublicKey(String filename) { InputStream in = this.getClass().getResourceAsStream(filename); String contents = new String(); try { int c; while ((c = in.read()) != -1) { contents += (char) c; } } catch (IOException e) { System.err.println("Could not read RSA key resource."); } int linebreak = contents.indexOf("\n"); EXPONENT = new BigInteger(contents.substring(0, linebreak).trim()); MODULUS = new BigInteger(contents.substring(linebreak + 1).trim()); } }
I chose to use the CBC mode for encryption, since it is very fast and very widely supported. For an example using the CFB mode, see Jonathan Knudsen's Whisper application. I've found that despite the need to pad the message to a multiple of the block size for CBC, it is faster than CFB.
For the RSA encryption, I used the PKCS1 encoding scheme - see the Bouncy Castle FAQ for why this is necessary.
In this example, after encryption, I'm uploading the message to a server in a multipart-mime format, where it can be decrypted. The POST data sent includes the encrypted key, the encrypted initialization vector, and the encrypted message. Stay tuned for the full example application (which I'm still working on :). This strategy for uploading data depends on a phone and network that can make GPRS connections, which might not be true in all cases. In a real world application, your method of getting the data to a server might include sending SMS messages, saving data to a flash card, or carrier pigeons.
The following is example PHP code for decrypting the message. For testing purposes, I just wrote code that dumps the contents of a POST request to a file; but if the request is a GET request (such as when you look at it with a web browser), it decrypts the file and displays the results. To do the decryption, PHP depends on the mcrypt and openssl extensions.
WARNING: this is not production code, and is not usable in any real server. Only regard it as an example for how to use PHP's crypto api's to decrypt some message. DO NOT use this code on any public server.
<?php function decrypt_aes($ciphertext, $aes_key, $iv) { return rtrim(mcrypt_decrypt(MCRYPT_RIJNDAEL_128, $aes_key, $ciphertext, MCRYPT_MODE_CBC, $iv), "\0"); } function decrypt_rsa($ciphertext, $private_rsa_key) { $plaintext; openssl_private_decrypt($ciphertext, $plaintext, $private_rsa_key, OPENSSL_PKCS1_PADDING); return $plaintext; } $file = "uploads/test.txt"; if ($_POST) { $fh = fopen($file, 'w'); fwrite($fh, serialize($_POST)); fclose($fh); } else { $post = unserialize(file_get_contents($file)); // Need special permissions for host.pem, which shouldn't be seen by the world ever. $private_rsa_key = openssl_get_privatekey(file_get_contents("host.pem")); // slash stripping necessary for binary data; but beware of security considerations. $aes_key = decrypt_rsa(stripslashes($post['key']), $private_rsa_key); $iv = decrypt_rsa(stripslashes($post['iv']), $private_rsa_key); $message = decrypt_aes(stripslashes($post['message']), $aes_key, $iv); echo $message; } ?>
Stay tuned for the source code for a full application that does the encryption and uploading, as well as a lot of discussion on using techniques like encryption in the service of free press.
I spent a very long time fighting with Eclipse and the EclipseME plugin in order to do a J2ME app using the Bouncy Castle crypto libraries. Here are the steps that finally worked for me in Ubuntu 7.04 using Eclipse 3.1, EclipseME and the latest Bouncy Castle:
Make sure the source folders and output folders are set to "src" and "bin" respectively. This is important: my habit is to use the project folder for both source and class files, but your built packages will have errors if you do this. EclipseME seems to put all the contents of the "source" directory (which, if it's the package directory, includes things like "deployed" and ".settings" and other stuff) in your jar, which causes problems.
Install and set up the BouncyCastle crypto library.
Set up obfuscation. BouncyCastle includes some classes that are reimplementations of system classes (such as java.security.SecureRandom and java.lang.BigInteger). You will receive runtime security errors if your application tries to add these classes to the system. To avoid this, it is necessary to obfuscate the classes (which renames them, and places them in the default package).
Now you are ready to write your crypto code! But note that your development process and debugging are now different. Because bouncycastle depends on obfuscation for the code to run at all, and obfuscation only runs during the "packaging" stage, you can no longer simply run your emulated MIDlet with the WTK emulator to debug. Instead, you must use the following steps to test your program:
You might enjoy a fun array of projects that I completed in How to Make Almost Anything 2007:
