Jason Staten

Yesterday, I spent awhile working on a RubyQuiz Challenge involving the parsing of JSON. Although I did create it with minimal viewing of the solution, I cannot take credit for the code or the idea.  I did notice that on the site there was no actual merging of their snippets, so I felt inclined to share.

jsonParser.rb

require "strscan"
class JSONParser
	AST = Struct.new(:value)
	def parse(input)
		@input = StringScanner.new(input)
		parse_value.value
	ensure
		@input.eos? or error("Unexpected data")
	end

	private

	def parse_value
		trim_space
		parse_object or
		parse_array or
		parse_string or
		parse_number or
		parse_keyword or
		error("Invalid Data")
	ensure
		trim_space
	end

	#Parses colon separated object hashes
	def parse_object
		if @input.scan(/\s*\{/)
			obj = Hash.new
			more_parts = false
			while key = parse_string
				@input.scan(/\s*:\s*/) or error("Expected : separator")
				obj[key.value] = parse_value.value
				more_parts = @input.scan(/\s*,\s*/) or break
			end
			error("Missing object pair") if more_parts
			@input.scan(/\s*}/) or error("Unclosed object")
			AST.new(obj)
		else
			false
		end
	end

	def parse_array
		if @input.scan(/\s*\[/) #Arrays start with [
			a = Array.new
			more_items = false
			while current = parse_value
			a << current.value
			more_items = @input.scan(/\s*,\s*/) or break
			end
			error("Missing value") if more_items
			@input.scan(/s*\]/) or error("Unclosed array")
			AST.new(a)
		end
	end

	#Parses string objects
	def parse_string
		if @input.scan(/"/)
			s = String.new
			while current = parse_string_content || parse_string_escape
				s << current.value
			end
			@input.scan(/"/) or error('Unclosed String')
			AST.new(s)
		else
			false
		end
	end

	def parse_string_content
		@input.scan(/[^\\"]+/) and AST.new(@input.matched)
	end

	def parse_string_escape
		if @input.scan(%r{\\["\\/]}) #slashes and quotations
			AST.new(@input.matched[-1])
		elsif @input.scan(/\\[bfnrt]/) #newlines, tabs, etc
			AST.new(eval(%Q{"#{@input.matched}"}))
		elsif @input.scan(/\\u[0-9a-fA-F]{4}/) #Hex integers
			AST.new([Integer("0x#{@input.matched[2..-1]}")].pack("U"))
		else
			false
		end
	end

	def parse_number
		@input.scan(/-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?\b/) and
		AST.new(eval(@input.matched))
	end

	def parse_keyword
		@input.scan(/\b(?:true|false|null)\b/) and
		AST.new(eval(@input.matched.sub("null","nil")))
	end

	def trim_space
		@input.scan(/\s+/)
	end

	def error(message)
			raise "#{message}:  #{@input.peek(@input.string.length)}"
	end
end

For an in depth explanation of the individual portions in this code, refer to the RubyQuiz Site . One way that I’ve tested this code is by accessing the Twitter search api. The following snippet will print the 20 most recent #neumont tweets:

require 'jsonParser'
require 'cgi'
require 'open-uri'
url = "http://search.twitter.com/search.json?q=%23neumont"
parser = JSONParser.new
open(url) {|html| @output = html.read}
obj = parser.parse(@output)
obj["results"].each {|e| puts CGI.unescapeHTML(e["text"])}

For anyone who’s new to the blog, StatenUtil is a personal C# utility library. Consisting mainly of extension methods, its main objective is to simplify frequently used operations. This edition includes adding some Ruby-like looping methods into C#. I understand that Func and Action objects aren’t a real replacement for blocks. However, it’s nice to have some features I miss when not using Ruby.

Int32Extensions.cs

using System;

namespace StatenUtil
{
    public static class Int32Extensions
    {
        public static void Times(this int source, Action<int> action)
        {
            for (int i = 0; i < source; i++)
            {
                action(i);
            }
        }
        public static void UpTo(this int source, int max, Action<int> action)
        {
            for (int i = source; i <= max; i++)
            {
                action(i);
            }
        }
        public static void DownTo(this int source, int min, Action<int> action)
        {
            for (int i = source; i >= min; i--)
            {
                action(i);
            }
        }
        public static int[] Range(this int source, int max)
        {
            int difference = max - source;
            if (difference < 0) return new int[0];
            int[] ret = new int[difference + 1];
            for (int i = 0; i <= difference; i++)
                ret[i] = source+i;
            return ret;
        }

    }
}

This class file revolves around adding functions to the int class. The Times method allows for a specific integer capturing lambda expression to be executed source times. For example, 6.Times(x => Console.WriteLine(x)) would print lines from zero to 6.

UpTo and DownTo are similar to the Times method. However, they loop over a range of integers rather than from zero. Compare these to Ruby’s upto and downto methods.

Range is simply a way to create an array covering a range of integers. The source is the minimum value and max is the largest of the array.

IEnumerableExtensions.cs

using System;
using System.Collections.Generic;

namespace StatenUtil
{
    public static class IEnumerableExtensions
    {
        public static void ForEach<T>(this IEnumerable<T> source, Action<T> action)
        {
            IEnumerator<T> enumerator = source.GetEnumerator();
            while (enumerator.MoveNext())
                action(enumerator.Current);
        }
    }
}

Currently, this class file add inline ForEach capabilities to any generic IEnumerable object, similar to Ruby’s each. Although List has this capability, other IEnumerables like arrays do not.

That’s all for now. However, StatenUtil is a work in progress that’s always being amended. In each significant addition, the details will be shared once again.

I Listen, I Promise

Recently, I recieved a comment on one of my previous posts, Creating a Socket Server and Client in Ruby, that asked about the next step towards taking the simple example to make a working web server. Well, you’ve been heard and I have written up code to do just that. It handles retrieval and sending of headers, handling GET requests, and responding with status.

Request and Response

class RequestInfo
	attr_accessor :headers, :method, :resource
	def initialize
		@headers = {}
	end
end
class ResponseInfo
	attr_accessor :status, :headers, :body
	def initialize
		@headers = {}
		@body = String.new
	end
end

These classes are short and are designed simply to contain the information coming in and going out of the webserver. RequestInfo’s method is used to obtain the method of the request (GET, etc..). Resource is the requested file from the connecting client. Finally, headers is a map of key value pairs corresponding with headers.

ResponseInfo consists of status, which is the first line returned containing 200 OK or 404 Not Found; body,  the content of what’s actually responded with; and headers, the map of headers of the response.

Populate the Request

	request = RequestInfo.new
	response = ResponseInfo.new
	while((current = s.readline).chomp! != "")
		if current[0,3] == "GET"
			methodline = current.split
			request.method = methodline[0]
			request.resource = methodline[1]
		elsif current.match ':'
			headerline = current.split ':',2
			request.headers[headerline[0]] = headerline[1]
		end
	end

This segment handles populating the RequestInfo object, by going through the sent request line by line from the TCPSocket connection s. It retrieves method, resources, and headers in a single while loop.  The headers are retrieved by splitting on the first colon in each line that contains one.

Populate the Respone

	begin
		File.open("public/#{request.resource}","r") do |file|
			while(curline = file.gets)
				response.body << curline
			end
		end
		response.status = "HTTP/1.0 200 OK"
		response.headers["Content-Type"] = "text/html"
		response.headers["Content-Length"] = response.body.length.to_s
	rescue => err
		puts err
		response.status = "HTTP/1.0 404 Not Found"
		response.headers["Content-Type"] = "text/html"
		response.headers["Content-Length"] = "0"
		response.body = ""
	end

Within this segment, we read in a file (looking in a folder called public so the webserver.rb itself cannot be retrieved through a request), appending its contents to the response.body. Afterwards, if all is successful, we respond with a 200 OK along with some other headers like Content-Type and Content-Length. Typically, if there’s an issue it’s because the file cannot be found, so returning a 404 is appropriate.

Note: A suggestion for improving this example is reading the MIME type of the read file so the Content-Type is not hardcoded.

Write Back the Response

	s.write response.status + "\n"
	response.headers.each {|key,value| s.write "#{key}:#{value}\n"}
	s.write "\n" + response.body
	s.close

This writes to the socket connection s, in order, ResponseInfo’s status, headers, and body. Finally, the connection is closed.

All Together Now

class RequestInfo
	attr_accessor :headers, :method, :resource
	def initialize
		@headers = {}
	end
end
class ResponseInfo
	attr_accessor :status, :headers, :body
	def initialize
		@headers = {}
		@body = String.new
	end
end
require "socket"

serv = TCPServer.new('jstaten.com',7881)
loop do
	Thread.start(serv.accept) do |s|
	request = RequestInfo.new
	response = ResponseInfo.new
	while((current = s.readline).chomp! != "")
		if current[0,3] == "GET"
			methodline = current.split
			request.method = methodline[0]
			request.resource = methodline[1]
		elsif current.match ':'
			headerline = current.split ':',2
			request.headers[headerline[0]] = headerline[1]
		end
	end
	if (request.resource == "/")
		request.resource = "index.html"
	end
	begin
		File.open("public/#{request.resource}","r") do |file|
			while(curline = file.gets)
				response.body << curline
			end
		end
		response.status = "HTTP/1.0 200 OK"
		response.headers["Content-Type"] = "text/html"
		response.headers["Content-Length"] = response.body.length.to_s
	rescue => err
		puts err
		response.status = "HTTP/1.0 404 Not Found"
		response.headers["Content-Type"] = "text/html"
		response.headers["Content-Length"] = "0"
		response.body = ""
	end
	s.write response.status + "\n"
	response.headers.each {|key,value| s.write "#{key}:#{value}\n"}
	s.write "\n" + response.body
	s.close
	end
end
end

When the steps are all implemented, you have a working ruby web server. Surely there are improvements to be made upon this creation, but it’s an effective proof-of-concept that can be built off.

I appreciate the feedback from all visitors to my blog, and if you feel that you have any questions or suggestions, feel free to comment away.

Note: Made a few corrections suggested. Thanks!

Remember the Java JAX-WS Web Service discussed what seems like long ago? Currently there is an have an instance of it running on this hosting server (got to love SSH). Experimenting with Ruby has led me to find the simplicity of creating a web service client. The duck typing of the language allows you to not need to generate a bunch of classes to pull in a WSDL (although it is possible using wsdl2r).

require 'soap/wsdlDriver'

URL = 'http://jstaten.com:7070/MathExample/MathService?wsdl'

begin
   driver = SOAP::WSDLDriverFactory.new(URL).create_rpc_driver
   p driver.addInts({:arg0 =>2, :arg1 => 4}).return
rescue => err
   p err.message
end

Steping through this code, you’re simply requiring the soap.wsdlDriver library and setting the URL of the WSDL to be read in. The begin statement is for error handling incase something goes wrong. Inside that, you create a new rpc driver using the WSDLDriverFactory. The “driver” within this code is the instance which contains methods pointing to the web service’s exposed operations. After creating the driver, we call a method on it. Because it requires two parameters (the two integers to be added), we pass it a map. The return field from running that method contains the value of the response. Finally, we rescue by printing the error message. If you’d like to test this against the web service in the code, feel free. Just please don’t abuse it.

I came across a rather interesting discovery today that email is quite simple to send within Ruby. Check it out:

Simple Email

require 'net/smtp'
message = 'Hello Dude'
Net::SMTP.start('127.0.0.1',25) do |smtp|
smtp.send_message(message, 'from@from.com','to@to.com')
end

Granted, this is a very simple example, but the principles are still there. Write a message as a string, start a new SMTP connection, tell it where it’s from and going, and send it on its way.

The problem with this method is that you wind up with some very ugly(nonexistent) headers that spam catchers may not like. One way around this is to include them in the message as follows:

Simple Email with Headers

require 'net/smtp'
message = <<MESSAGE_END
From: test@hello.com
To: to@to.com
Subject: Sample Message

Hello, this is a message
MESSAGE_END
p message
Net::SMTP.start('10.245.27.10',25,'localhost') do |smtp|
smtp.send_message(message, 'test@hello.com','to@to.com')
end

This gives you greater control over the headers. However, I still found this to be a bother for the simple sake of having to type the to field multiple times. That’s the sort of thing that can lead to errors in code (it actually messed me up for a a minute while writing this). So my last bit moves the message into class form, allowing us to keep our sending portion short, clean, and less error prone. This involves two files, emailMessage.rb and email.rb. They are shown below:

emailMessage.rb

class EmailMessage
attr_accessor :to
attr_accessor :from
attr_accessor :message
attr_accessor :subject
def initialize(attributes)
@to = attributes[:to]
@from = attributes[:from]
@message = attributes[:message]
@subject = attributes[:subject]
end

def to_s
<<MESSAGE_END
From: #{@from}
To: #{@to}
Subject: #{@subject}
#{@message}
MESSAGE_END
end
end

email.rb

require 'net/smtp'
require 'emailMessage'
em = EmailMessage.new :to=>;'to@to.com', :from=>;'test@test.com', :subject=>;'mysubject'
em.message = 'This is the message'
Net::SMTP.start('127.0.0.1',25) do |smtp|
smtp.send_message(em.to_s, em.from,em.to)
end

We’re back to a short, clean, email.rb that can be used for sending email with most SMTP.

Using SSL

Some SMTP hosts, including Gmail, requires the usage of SSL to send emails. This is not built into the standard Ruby libraries. However, Stephen Chu has a great post covering how to do it in Ruby.

Back with more Ruby, I bring you some database connectivity because that’s always an important thing to know about a language.

Prerequisites

This tutorial requires that you have the mysql and dbi gems installed. If you have RubyGems installed, it should be as simple as

#gem install mysql
#gem install dbi

However, if you’re having difficulty, surely Google understands.

Also, this requires a MySql database running on your local machine with a database containing a customer table as follows

CREATE TABLE `customer` (
`id` int(11) NOT NULL auto_increment,
`name` varchar(30) NOT NULL,
`telephone` char(10) default NULL,
PRIMARY KEY  (`id`),
UNIQUE KEY `telephone` (`telephone`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

Accessing the Database

require 'rubygems'
gem 'dbi'
require 'dbi'
begin
con = DBI.connect("DBI:Mysql:DATABASE_NAME:localhost","USER","PASSWORD")
result = con.execute "SELECT * FROM customer"
result.fetch_hash do |row|
printf "ID:%d NAME:%s PHONE:%s\n", row["id"], row["name"], row["telephone"]
end
result.finish
rescue DBI::DatabaseError => e
puts "Error, #{e.err}, #{e.errstr}"
ensure
con.disconnect if con
end

Breaking this down into some blocks, the program starts by requiring rubygems and dbi (gem ‘dbi’ allows the dbi libraries to be accessed). DBI is ruby’s interface to use several different databases. In this case we’ll be using MySQL.

The real work begins when we use DBI.connect to create a connection to a database. This connection is what we’ll use to send queries to the database. Be sure to replace DATABASE_NAME, USER, and PASSWORD with the appropriate information for your own database.

Next up, con.execute will return a result set that can be iterated over using fetch_hash. In this case we’ll simply print each result to the console. Notice that rows are hashes that allow you to access information based on column name. Finally, result.finish means we’re done using the result set given to us.

The rescue section simply makes sure to notify of an error if something goes wrong. Also, ensure does just what it says, ensures that the connection is closed if it exists.

Going Further

Inserting Data

Surely the output right now is rather boring because you don’t have any data in your table. Lets put some in, and lets do it with a prepared statement. That way when you build a Ruby app with user input, you’re less susceptible to SQL injection.

cmd = con.prepare "INSERT INTO customer(name,telephone) VALUES(?,?)"
cmd.execute "Bobby","2229998888"

Receiving Entire Datasets and Single Results

Remember working with result.fetch_hash? What about if you want the entire dataset? What about just the first row? You’re in luck, Ruby makes it easy.

#returns an array of DBI::Row objects
result.select_all "SELECT * FROM customer"

#returns the first row
result.select_one "SELECT * FROM customer"

Database access with Ruby is an important skill to know if you’re dealing with storing any sort of information. With this brief overview, grasping the idea should be straight forward.

Over break, I’ve been experimenting with Ruby a little bit. It’s rather different from the C#/Java syntaxes I’ve leared through schooling. However, I’ve enjoyed using it. And for the reference of myself and others, I’m slapping up a how-to for creating a simple socket server and client.

The Server

require "socket"
serv = TCPServer.new('localhost',7885)
count = 0
loop do
Thread.start(serv.accept) do |s|
count += 1
s.write "You are visitor #{count} to my TCP Ruby Server"
s.close
puts "New visitor: #{count}"
end
end

The server begins by getting the socket library that already comes with Ruby. It then creates an instance of TCPServer, binding it to localhost on port 7885. Count is used to keep track of the number of incoming connections the server receives.

Enter the loop, a new thread will be started upon the blocking method serv.accept and during the life of the thread, it will increment the count, send a message to the visitor, close the connection, and finally write the user count to the console.

The Client

require "socket"
client = TCPSocket.new("localhost",7885)
str = ""
while (add = client.recv(100)) != ""
str += add
end
puts str
client.close

The client simply receives all output sent to it.  It opens a connection with localhost on port 7885. After connecting, it recieves a buffer of 100 bytes until no more are read. The buffer is appended to the final output string, and the connection is closed.

And there you have it, a simple Ruby socket connection. From here you can create such things as a web server, or writing your own protocol. Enjoy!