Also available at

Also available at my website http://tosh.me/ and on Twitter @toshafanasiev

Friday 23 December 2011

Ruby List Comprehensions

I'm busy learning Ruby and I'm discovering that I really like it. I've used Python for some time now so naturally it's the differences between these two paradigmatically similar languages that I notice the most.

A feature of Python that I think is really cool is the list comprehension; a way of specifying a list literal using the definition of the list, rather than it's enumeration, as the code below illustrates:

#!/usr/bin/python

# literal as an enumeration
a = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]

# literal as a comprehension
b = [ i for i in range( 10 ) ]

This is a really neat feature that can create a sequence from an arbitrary definition, such as the characters in a string, the odd numbers in a sequence etc. You get the idea.

In Ruby, doing this ...

a = [ 1..100 ]

... doesn't give you an array of one hundred elements, but an array containing a single Range object of 1 to 100.

The fact that list comprehensions seem to be missing from Ruby was initially slightly saddening, until I discovered that Ruby's class definitions, like its strings, are mutable. This is what's really got me excited about this language - you want list comprehensions? Add them.

The duck typing offered by dynamic languages along with Ruby's class mutability makes this sort of thing really easy to do, as demonstrated below.

#!/usr/bin/ruby
#saved as 'list_comp.rb'

class Array
  def from_e!( enum )
    enum.each { |x| self << x }
    return self
  end
  def self.from_e( enum )
    return [].from_e! enum
  end
end

This is cool! It lets you create lists in a really intuitive way using definitions, not enumerations - and the best part is that I've added this to the actual, built-in Array class itself - that's the power of Ruby. I love it.

Here are some lists being created from scratch using the class method:

irb(main):001:0> require './list_comp.rb'
=> true
irb(main):002:0> Array.from_e 1..10
=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
irb(main):003:0> Array.from_e 'How much wood would a woodchuck chuck?'.each_char
=> ["H", "o", "w", " ", "m", "u", "c", "h", " ", "w", "o", "o", "d", " ", "w", "
o", "u", "l", "d", " ", "a", " ", "w", "o", "o", "d", "c", "h", "u", "c", "k", "
 ", "c", "h", "u", "c", "k", "?"]
irb(main):004:0> Array.from_e 0.step 50, 5
=> [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50]
irb(main):005:0> Array.from_e 'one two 3 four 55'.scan /\d+/
=> ["3", "55"]

And here is the instance method in action, modifying an existing array:

irb(main):006:0> a=[0]
=> [0]
irb(main):007:0> a.from_e! 1.step 10,1
=> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Note the ! on the end of the instance version of from_e - I've noticed and really appreciate the convention of marking mutating (non-const, if you like) instance methods using the exclamation mark and I intend to stick to it. Also, I've tried to follow convention by calling the method from_e - to match the to_s, to_i, etc. pattern.

Now it could be that Ruby already supports this sort of thing and I just haven't found it yet but either way I'm really impressed - it seems you can make this language whatever you want it to be. I'm looking forward to learning more - this is only day 3!

Wednesday 21 December 2011

OAuth Request Signing

I've started writing a little app that needs to talk to Twitter - to do this it needs to speak OAuth. It runs on Google App Engine and is written in Python and there are many good OAuth (and even dedicated Twitter) modules written in Python so I could have grabbed one of these and got on with it but that wouldn't have been much fun and wouldn't get me any closer to an understanding of OAuth - so I went ahead and implemented it myself.

Since at this stage I don't need my app to allow users to log in and interact as themselves (actually I'm actively avoiding this sort of requirement) I didn't need to worry about the 'dance' part of OAuth (the token request/authentication/exchange bit) - I only need the request signing part and the keys provided by my Twitter app's dashboard page.

I tweeted from an interactive Python session using my OAuth script for authentication, and have pulled down mentions etc. The implementation is deliberately unoptimised and fragmented - this lets it serve better as an executable reference to the OAuth standard (the various functions link to the specific sections of the standard that they implement) - and also it is good enough for what I want to do right now.

Here it is

'''implementing OAuth ( request signing only ) as set out in http://tools.ietf.org/html/rfc5849

License: This code is free for use by any person, for any purpose but is provided as-is, with no guarantees - use at your own risk. One proviso: drop me a mail and let me know how it worked for you.'''

from urllib import quote
import urllib2
import time
import os
import base64
import hashlib
import hmac

def encode( value ):  
  '''#section-3.6'''
  return quote( value or '', '-._~' )

def enc_params( params ):
  '''#section-3.4.1.3.2'''
  ep = [ '%s=%s' % ( encode( k ), encode( v ) ) for k, v in params.items( ) ]
  return '&'.join( sorted( ep ) )

def b64e( value ):
  '''#section-6.8'''
  return base64.b64encode( value )

def make_base_string( method, url, params ):
  '''#section-3.4.1'''
  return '&'.join( [ encode( method.upper( ) ), encode( url ), encode( enc_params( params ) ) ] )

def get_timestamp( ):
  '''#section-3.3'''
  return str( int( time.time( ) ) )

def get_nonce( ):
  '''#section-3.3'''
  return b64e( os.urandom( 32 ) ).strip( '=' )

def hmac_sha1_sig( base_string, consumer_secret, token_secret='' ):
  '''#section-3.4.2'''
  key = '%s&%s' % ( encode( consumer_secret ), encode( token_secret ) )
  h=hmac.new( key, base_string, hashlib.sha1 )
  return b64e( h.digest( ) )

class OAuthClient( object ):
  def __init__( self, consumer_pair, token_pair ):
    '''takes (consumer_key, consumer_secret), (oauth_token, token_secret)'''
    self.consumer_key, self.consumer_secret = consumer_pair
    self.oauth_token, self.token_secret = token_pair

  def create_oauth_params( self, url, data=None ):
    method = 'POST' if data else 'GET'
    params = {
      'oauth_consumer_key'     : self.consumer_key
    , 'oauth_token'            : self.oauth_token
    , 'oauth_signature_method' : 'HMAC-SHA1'
    , 'oauth_timestamp'        : get_timestamp( )
    , 'oauth_nonce'            : get_nonce( )
    , 'oauth_version'          : '1.0'
    }

    if data: params.update( data )

    base_string = make_base_string( method, url, params )
    params[ 'oauth_signature' ] = hmac_sha1_sig( base_string, self.consumer_secret, self.token_secret )

    return params

  def create_oauth_header( self, url, data=None ):
    params = self.create_oauth_params( url, data )
    header = 'OAuth ' + ', '.join( [ '%s="%s"' % ( k, encode( v ) ) for k, v in params.items( ) ] )

    return header

  def open_url( self, url, data=None ):
    '''returns the resource represented by the url, or a tuple of error code and response text'''
    if data: post_data = enc_params( data )
    else: post_data = None
    r=urllib2.Request( url, post_data )
    r.headers[ 'Authorization' ] = self.create_oauth_header( url, data )

    try:
      return urllib2.urlopen( r ).read( )
    except urllib2.URLError, e:
      return e.code, e.read( )

If you save this as something like oauth.py and insert your own consumer/oauth keys over the placeholders you can try it out at a prompt like so

>>> CONSUMER_KEY = 'CONSUMER-KEY'
>>> CONSUMER_SECRET = 'CONSUMER-SECRET'
>>> OAUTH_TOKEN = 'OAUTH-TOKEN'
>>> TOKEN_SECRET = 'TOKEN-SECRET'
>>> import oauth
>>> c = oauth.OAuthClient( ( CONSUMER_KEY, CONSUMER_SECRET ), ( OAUTH_TOKEN, TOKEN_SECRET ) )
>>> mentions = c.open_url( 'http://api.twitter.com/1/statuses/mentions.json' )
>>> c.open_url( 'http://api.twitter.com/1/statuses/update.json', { 'status' : 'here is my status update, blah, blah. Read this blog: http://blog.tosh.me/. } )


That's it, more to follow on the project it was written for.