Aaron Feng

PHP has a string tokenizing function called strtok that has an interesting interface compared to many languages. What makes strtok function interesting is that creating the tokenizer and generating tokens utilize the same function. For example:

$tok = strtok("closure fun", "  ");

while ($tok !== false) {
    echo "$tok\n";
    $tok = strtok("  ");
}

The code above tokenizes the input string by a space, and outputs each token on a newline. What makes this function interesting is the ability to maintain state! I don't know what happens under the hood, but it appears to "remember" the input string after the tokenizer has been created. This behavior can be easily mimicked by using a closure variable. Below is an implementation in lisp:

(let ((rest nil))
  (defun strtok (string &optional (token nil token-supplied-p))
    (check-type string string)
    (when token-supplied-p (check-type token string))

    (when (and string token) (setf rest string))
    (let ((items nil)
          (output nil))
      (loop until (or (null rest)) do
           (setf items (cl-ppcre:split 
                        (funcall (lambda () (if token token string)))
                        rest 
                        :limit 2)) 
           (setf output (first items))
           (setf rest (second items))

           (when (not (string= output "")) 
             (return-from strtok output))))))

The "rest" variable outside of the function declaration is the closure variable. The "rest" is in fact invisible to the outside world, and it allows the function to maintain state!

Last post I talked about creating an abstraction using Lisp macros to assist me with my unit tests. If you look at the code, there is an obvious flaw. get-result-for-test-case hard codes the target PHP function in it. It wouldn't be very nice to rewrite a new version of it every time I need to call a different PHP function.

In the original code from last post, the target PHP function was defined in string format. Since I'm working in Lisp I would prefer to define it in sexp format to gain the most flexibility. For example, calling the explode function in PHP would look like the following:

(explode " " "aaron feng")

Let's start the refactoring:

; helper function
(defun quote-string (input)
  (if (typep input 'string)
    (format nil "\"~a\"" input)
    input))

(defun get-function-signature (function-info)
  (let ((function (first function-info))
        (parameters (rest function-info))
        (signature "") 
        (delimiter ""))
    (setf signature (format nil "~a~a" function "("))

    (loop for parameter in parameters do
         (setf signature (format nil "~a~a~a" 
                                 signature
                                 delimiter 
                                 (quote-string parameter)))
         (setf delimiter ", "))

    (setf signature (format nil "~a~a" signature ")"))))

get-function-signature does all the dirty work by converting input function in sexp format to a string that we can eventually send to PHP. Let's incorporate this function back into get-result-for-test-case, so it can accept other functions.

(defun get-result-for-test-case (func-info)
  (with-output-to-string (str)
    (let ((func-string
           (concatenate 'string "print_r(" (get-function-signature func-info) ");")))
      (run-program *php-bin* (list
                              "-r"
                              func-string)
                   :output str))))

With these changes, it will make get-result-for-test-case much more generic. However, there is another problem, not all PHP functions return a string. If the function returns an array, the output will always be "Array". It would be nice if it could convert PHP array back into Lisp list, so it can be used for comparison in the tests. That is exactly what I'm going to do next. Notice that I replaced the "echo" with "print_r", so it will output human readable values for data structure such as array.

(defun parse-php-array (array)
  (let ((parts (split-sequence #\Newline array))
        (output))
    (dolist (part parts)
      (let ((value (get-array-value part)))
        (format t "~a" part)
        (when value (push value output))))
    (reverse output)))

Now if I wrap get-result-for-test-case with parse-php-array, it should convert the output into a Lisp list.

(parse-php-array (get-result-for-test-case `(explode " " "aaron feng")))

Of course, more work can be done to make the code more intelligent, such as, only try to parse array if the output is an array type. Currently the code can only handle Array and String as return type, but it can be extended further if desired.

It's not hard to get caught up on cranking out code when you are in the zone. All of the unit tests are passing one after another, and the implementation just flows out from your finger tips. There's no need to stop, step back and question the work because everything feels so right. Situations like this can really hinder your ability to unleash the power of your language. Especially if Lisp is your language of choice. After all, Lisp is the official programmable programming language.

This recently happened to me. I have been working on a side project (I'll talk about it in later post) that required me to implement string functions in Common Lisp that are similar, in spirit, to ones found in PHP. Since I'm a huge fan of TDD, I started with a failing test. However, before the failing test can be written, I need to confirm the actual behavior in PHP. During my the implementation cycle, I toggled back and forth between PHP and Lisp. This can get tedious over time, but I just kept chugging along.

This morning Kyle Burton showed me the light. He sent me a few pieces of code that remove the tediousness from my implementation cycle. I adopted the code to fit my environment and the testing framework. Without further adieu:

(defparameter *php-bin* "/opt/local/bin/php")

(when (not (probe-file *php-bin*))
  (error 
   "Error: %s was not found, please update *php-bin* to point to your php." 
   *php-bin*))

(defun get-result-for-test-case (instr 
                 &optional (width 75) 
                 (lbreak #\Newline) 
                 (cut nil))
  (with-output-to-string (str)
    (run-program *php-bin* (list 
                "-r" 
                (format nil "echo wordwrap(\"~a\",~a,\"~a\",~a);"
                    instr
                    width
                    lbreak
                    (if cut "true" "false")))
         :output str)))

(get-result-for-test-case "aaron feng" 3)

I was executing PHP code externally just to get the output, so I can use it in my unit tests. Why not ask PHP from Lisp!

The last line calls get-result-for-test-case function which calls the PHP wordwrap function directly and returns the result. Now this removed the need for me having to go outside my Lisp environment while I'm hacking. Why stop here? Why not have a macro that generates the test case using the result from PHP?

(defmacro create-test-case (test-name string &optional (width 75)
                (lbreak #\Newline) (cut nil))
  `(addtest ,test-name
       (let ((input ,string)
         (expected ,(get-result-for-test-case string width lbreak cut)))
     (ensure-same (wordwrap input ,width ,lbreak ,cut) expected))))

(create-test-case empty-string "   " 4)

Voila, instant test case. the last line shows how the test case can be generated on the fly and run. If you are a real ninja, you need to take one more step further.

(defmacro create-test-cases (&rest cases)
 `(prog2
      (deftestsuite wordwrap-suite () ())
      ,@(mapcar #'(lambda (case) (macroexpand-1 `(create-test-case
,@case))) cases)))

(create-test-cases
                (empty-string1       "   ")
                (empty-string2       "   " 4)
                (no-wrap             "foo")
                (wrap-one-word-at-1  "foo" 1))

create-test-cases creates a testing suite to host all the test cases then it generates test cases based on input. This takes WAY LESS typing than hand writing each test case. I no longer have to switch between two environments (PHP and Lisp) because it's all automatically done for me! Thanks Kyle!

Lisp is often viewed as strange in syntax and parentheses centric language that only thing it cares about is processing lists. After all, the name is derived from LISt Processing (LISP). But it couldn't be more wrong!

Pascal Bourguignon has a great post on the lisp mailing explain why Lisp has no syntax. Lisp has no parentheses. Lisp as no lists. For the lazy readers, who don't want to click on the link, I'll give a simple break down of the Bourguignon's post.

Lisp has no syntax

Anyone with knowledge of how a compiler works would know that the job of a parser is to create a parse tree. The parser has knowledge of the syntax of the language so it can transform tokens from the source code into a tree structure. The reason Lisp doesn't have a concrete syntax is because the source code is in parse tree syntax. A parse tree can be represented in many formats.

Lisp has no parentheses

In lisp, a list is constructed by placing elements in the list between parentheses and leading the first parenthesis with a single quote. For example '(a b c). Internally lisp doesn't store or know anything about parentheses.

Lisp has no list

Lisp has no list type. List is an abstraction on top of the CONS cells. A CONS cell is equivalent to a node in a link list. It has two pointers; one points to data and other points to the next node or nil.

I've only covered each point superficially, but Bourguignon's post contains a lot more details.

Many that are close to me have noticed my new obsession with a language that was invented in 1958: Lisp. I don't think there has been a day that goes by that I don't make some kind of reference to Lisp. The more I read about Lisp, the more mind blowing it is to me that most people don't use it. I understand Lisp may be an acquired taste, but there are many interesting concepts/features that can impact one's thinking even when they are not using Lisp.

Everyone complains about the parentheses. Please complain about something original. To tell you the truth, after a couple of weeks, you won't even see the parens anymore. You don't have to be a genius to do it, trust me, I'm definitely not one.

With that said, I would like to recommend three books that have helped me gain an understanding of Lisp. Best of all, they are all free! The only cost is your time, and desire to learn.

• Practical Common Lisp By Peter Siebel
• Common Lisp : A Gentle Introduction to Symbolic Computation By David S. Touretzky
• On Lisp By Paul Graham

If you don't want to pull your hair out, I would suggest you read "On Lisp" last as it contains many advanced topics. It doesn't matter which order you read the other two books. They require no previous Lisp experience. The book by Touretzky is actually targeted for people with no programming experience at all, but I found it useful just to skim through real fast.

What are you still doing here? Click on the links and start reading!

Yesterday was the grand opening of the second Apple store in my area. I was out and about, so I decided to stop by the store around the opening time: 10 am. I have never been to a grand opening of an Apple store before, and to my surprise, there were tons of people waiting in line just to get into the store.

Even though Apple was giving out free t-shirts to the first 1000 customers, there was still an impressive amount of people there.

If you are not barfing and slurping with paredit while you are editing your Lisp code in Emacs, you are missing out. You need to read on.

One of the biggest complaints for beginner Lispers (like myself) is the overwhelming parentheses. After working with Lisp code for a couple of weeks, the parens do disappear. However, you still have to match up the parens even if you don't see them any more. This is where paredit comes in. Paredit will manage all the parens for you, so you never have deal with dangling parens again. It feels annoying at first because it will not allow you to delete a closing paren without removing the opening one. Just like anything else, if you can get over that, it can be quiet useful.

Two of my favorite features are barf and slurp. They are best explained with examples.

; starting with the following sexp

(x (y) (z))

; barf forward...

(x (y)) (z)

; barf forward...

(x) (y) (z)

; slurp forward...

(x (y)) (z)

; slurp forward...

(x (y) (z))

; barf backward...

x ((y) (z))

; barf backward...

x (y) ((z))

; slurp backward...

x ((y) (z))

; slurp backward...

(x (y) (z))

You get the point. There are many more features to paredit besides barf and slurp. If you are interested in learning barfing, slurping and other delicious paredit features, check out this link.

My company has hosted the last two Philly Lambda meetings. Toby DiPasquale gave the first presentation on Scala then Kyle Burton gave a presentation on Lisp. Both presentations were very interesting especially for a functional noob like me. There's a wealth of information amongst this group of people, so it's definitely a good experience.

Currently there are around 60 people on the mailing list. However, there's not much discussion happening on it right now. Hopefully it will pick up eventually. I'm definitely looking forward to seeing what's coming up next.

Well, not in a forever way. Ruby is the language I use mostly on my own time. Since my free time is very limited, I need to shelf it for now. Making room for me to hack some lisp.

Like many people, my free time comes in small chunks. I have been using project euler as a way to learn new languages (also math). The code required to solve each problem is usually very short (30 lines or less). I can usually fit each problem into my small chunks of time very easily. It's a great way to get your hands dirty on the language you want to learn.

For those who follow my blog, you probably noticed that I have been posting project euler solutions in ruby. Since I'm trying to learn lisp now, I'll start to solve the problems using lisp. I'll post the rest of my solutions written in ruby in the following post.

Overall, I still enjoy using ruby. The lack of free time has forced me to make this hard decision.

A couple of months ago I wrote a utility that will download a web page with all the dependencies (css, and images) to your hard drive. All the references in the web page will be changed to refer to your local copy.

I wrote it as a prototype, and it took me 30 to 40 minutes to write it, so I'm sure there is room for improvement. I pointed to a few web pages, such as amazon, ebay, google, and my blog it worked pretty well!

The code requires Why the lucky stiff's Hpricot library. With out further adieu, here is the code below:

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'rio'

module Hpricot
  class Elem
    def is_css
      if self.name == "link"
        self["type"] == "text/css"
      else
        false
      end
    end
    def is_full_path
      if self.name == "link"
        self["href"][0..6] == "http://"
      elsif self.name == "img"
        self["src"][0..6] == "http://"
      else
        false
      end
    end
  end
end

if ARGV.size.zero?
  puts "Missing web page you wish to snatch."
  exit
end

url_scheme = "http://"
url = ARGV[0]
doc = Hpricot(open(url_scheme + url))

Dir.mkdir(url) unless File.directory?(url)

doc.search("link") do |item|
  if item.is_css
    if item.is_full_path
      rio(item['href']) > rio(url)
    else
      rio(url_scheme + url + item['href']) > rio(url)
    end

    # nested style sheets in another style sheet
    css_path = File.dirname(item['href'])
    css_file = File.basename(item['href']).scan(/(.*?\.css)/m).flatten.to_s

    file = File.open(url + "/" + css_file,"r")

    inner_css = file.read.scan(/@import '(.*?\.css)';/m).flatten
    inner_css.each do |css|
      css_url = url_scheme + url + css_path + "/" + css
      rio(css_url) > rio(url)
    end
    file.close

    item['href'] = css_file
  end
end

doc.search("img") do |item|
  if item.is_full_path
    rio(item["src"]) > rio(url)
  else
    rio(url_scheme + url + item["src"]) > rio(url)
  end
  item["src"] = item["src"].split("/")[-1]
end

File.open(url + "/" + url + ".html", "w") do |file|
  file << doc.to_s
end