Erjang - No, its not a typo and its Erlang-on-JVM

There's a new Erlang language in town built on-top of the JVM. Its Erjang. I seem to experience a explosion of hosts of languages built on-top of the Java Virtual Machine from Scala (my current favourite), Clojure (Lisp clone), Jython (Python clone), Groovy and JRuby to name a few; for a developer like myself, i do find myself suddenly swamped with a huge list of flavours in which to train myself in. Enough about me.

So Erjang is a runtime which basically loads the Erlang's BEAM file into the JVM and creates Java classes and its instances. Check out the project's homepage here. Like all languages, it'll need time to be adopted and adopted. Personally, i'll stick to Erlang since it fulfills most of my use cases from bit-hacking (which i absolutely adore) to general purpose computing and i'll love to listen to the use cases in which the author built Erjang. There's always something to be learned.

And i found it! Here's the rationale from the Creator which is ossm (check out the video)

Over the last few years, I have been meeting "Erlang people" more and more often, and I was getting this clear impression that "you people" have some kind of magic ability to reason intuitively about concurrent systems in a way that I could not. That bothered me, so I wanted to learn Erlang. Being a language implementor, the most obvious way to do that is to just go ahead and implement an Erlang VM, right?

The result of this "little exercise" is Erjang, an open-source JVM-based Erlang VM, which I think has potential to be useful as a means to push Erlang into new areas of application, help grow the Erlang community, and most importantly make all those Java-heads out there willing to learn how to do concurrency right. In this "brave new world" of networked services and multi-core everywhere, there are a lot of people who are challenged in providing even basic reliability, efficiency and correctness. My hope is that Erjang's existence can influence some of those outside the Erlang community to do better.

In technical terms, Erjang reads .beam files and compiles them to Java's equivalent .class files which are then read into the running JVM. It runs off a plain Erlang/OTP distribution - it only requires the beam files from there; Erjang itself is written in Java. As off this writing, it can run some non-trivial erlang programs, but is not yet capable of booting OTP [will have to update this abstract as we go along - follow updates on my blog]. Comparing the BEAM virtual machine and Erjang, the most obvious differences are that (a) Erjang will not be able to provide [soft] real-time guarantees since it uses Java's garbage collector, and (b) it has limited support for native code and port drivers (other than file and network I/O). The upside is the new ways this allows us to deploy Erlang systems.

In this presentation I will talk about how Erjang works, what behavior to expect from Erlang programs running on top of Erjang, what I learned along the way, and demonstrate [live] that Erjang runs well enough to be obviously useful.

Tsung - Load Testing Tool in Erlang

You've must have heard of Tsung - a load testing tool. Well i've got the opportunity to learn and use it in my current employment and its really versatile in conducting load tests for web-based applications. It has many features that i've come to appreciate after spending a couple of years with Mercury Interactive (its defunct now after being bought over by HP in 2006); admittedly Mercury's solutions were more versatile than what Tsung can offer but considering that web is the common platform where many applications are being hosted on; i think its a safe bet :)

So, IMO i think

1. Tsung is good for conducting load testing scenarios and executing them in a local / distributed manner
2. Load Testing can be relatively light weight (Erlang processes are hitting the target app) and hence the cost of using relatively heavy weight machines is likely to reduce since there is lesser need to use those machines since more concurrent users can be simulated on 1 machine in Tsung. (I'll see whether this assumption is correct)

Good starting points (URLs of interest):

1. http://tsung.erlang-projects.org/
2. http://www.erlang.org

A load testing tool would be useless if it didn't know how to capture server response data (e.g. checking whether an expected string is returned) and reuse it (e.g. session ids returned from servers which you can use subsequently in a HTTP POST/GET in an URL-rewrite type of string) subsequently, generate dynamic data, read data from an external file (commonly used in storing <username, password> pairs)
So my example would illustrate the logging in to a website (e.g. http://projecteuler.net) and logging out from it - simple enough to illustrate my point.
Next what i did was to record the series of events that mimick my user logging and logging out of the website and this is captured in the tsung_recorder_timestamp.xml
and i used that XML file and included some other stuff so that it looks like what i have for you below (this is a basic load test scenario)

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/local/share/tsung/tsung-1.0.dtd">

<tsung loglevel="info" dumptraffic="false" version="1.0">

<clients>
<client host="localhost" use_controller_vm="true"/>
</clients>
<servers>
<server host="78.110.165.8" port="80" type="tcp"></server>
</servers>

<load>
<arrivalphase phase="1" duration="2" unit="minute">
<users interarrival="1" unit="minute"></users>
</arrivalphase>
<user session="rec20091102-01:30" start_time="0" unit="second"></user>
</load>

<options>
<option name="file_server" value="/tmp/userlist.csv"></option>
</options>

<sessions>
<session name='rec20091102-01:30' probability='100'  type='ts_http'>
<request><http url='http://projecteuler.net/' version='1.1' method='GET'></http></request>
<request><http url='/style_main.css' version='1.1' if_modified_since='Sat, 29 Nov 2008 20:04:00 GMT' method='GET'></http></request>
<request><http url='/images/logo.jpg' version='1.1' if_modified_since='Thu, 28 Dec 2006 14:12:43 GMT' method='GET'></http></request>
<request><http url='/images/icon_register.png' version='1.1' if_modified_since='Fri, 31 Dec 2004 12:48:16 GMT' method='GET'></http></request>
<request><http url='/images/icon_about.png' version='1.1' if_modified_since='Fri, 31 Dec 2004 12:48:28 GMT' method='GET'></http></request>
<request><http url='/images/icon_problems.png' version='1.1' if_modified_since='Fri, 31 Dec 2004 12:48:26 GMT' method='GET'></http></request>
<request><http url='/images/icon_login.png' version='1.1' if_modified_since='Fri, 31 Dec 2004 12:48:32 GMT' method='GET'></http></request>
<request><http url='http://projecteuler.net/images/corner_tr.gif' version='1.1' if_modified_since='Thu, 10 Apr 2008 19:35:02 GMT' method='GET'></http></request>
<request><http url='/images/corner_tl.gif' version='1.1' if_modified_since='Thu, 10 Apr 2008 19:34:41 GMT' method='GET'></http></request>
<request><http url='/images/corner_br.gif' version='1.1' if_modified_since='Thu, 10 Apr 2008 19:35:10 GMT' method='GET'></http></request>
<request><http url='/images/corner_bl.gif' version='1.1' if_modified_since='Thu, 10 Apr 2008 19:34:55 GMT' method='GET'></http></request>
<request><http url='/images/euler_main.jpg' version='1.1' if_modified_since='Mon, 21 Jan 2002 19:18:20 GMT' method='GET'></http></request>

<thinktime random='true' value='2'/>

<request><http url='http://projecteuler.net/index.php?section=login' version='1.1' method='GET'></http></request>

<thinktime random='true' value='6'/>

<request subst="true">
<match do="continue" when="match">Logged in as %%readcsv:getUsername%%</match>
<http url='/index.php' version='1.1'  contents='%%readcsv:getUserString%%' content_type='application/x-www-form-urlencoded' method='POST'></http>
</request>

<thinktime random='true' value='4'/>

<request><http url='/images/icon_tick.png' version='1.1' method='GET'></http></request>

<thinktime random='true' value='4'/>

<thinktime random='true' value='3'/>

<request><http url='http://projecteuler.net/index.php?section=logout' version='1.1' method='GET'></http></request>
</session>
</sessions>

</tsung>

Hence, the main thing you should note is the use of dynamic substitution (e.g. %%readcsv:getUsername%%) where i wrote a simple erlang program to read my username and password from a file (see the XML tag option above) and replacing each simulated user with a valid user id and password.
Next, i checked that the server response contains a string Logged in as XXX where XXX would be dynamically generated by the function (Check out the erlang code for the function, simple stuff).
The erlang program is shown below.

1 -module(readcsv).
2 -export([getUserString/1, getUsername/1]).
3
4 getUserString({Pid, DynVar}) ->
5  {ok, Line} = ts_file_server:get_next_line(),
6  [Uid,Pwd] = string:tokens(Line, ","),
7  "username=" ++ Uid ++ "&password=" ++ Pwd ++ "&login=Login".
8
9 getUsername({Pid, DynVar}) ->
10  {ok, Line} = ts_file_server:get_next_line(),
11  [Uid,_] = string:tokens(Line, ","),
12  Uid.

The above erlang code must be compiled via erlc and you place the .beam file into the directory via a command like this

sudo mv readcsv.beam /usr/local/lib/erlang/lib/tsung-1.3.1/ebin/

Now, running the load test should be alright.
Note: In this load test, i defined a duration of 2 minutes with 2 users since load testing using 800 gazillion users is considered a chargeable offense so DON'T DO IT.


Have fun!

wxErlang

There's another graphics library other than "gs" aka Graphical System which is an adaptation of wxWidgets otherwise known as wxErlang. If you have never tried using wxErlang to run your stuff then you might run into some problems.

Sadly, the documentation on this is pitifully little

For me, i got this error when i attempt to run the default wx libraries shipped with Erlang.

=ERROR REPORT==== 7-Jun-2009::12:17:09 ===
WX Failed loading "wxe_driver"@"/usr/local/lib/erlang/lib/wx-0.98.1/
priv/i386-apple-darwin9.6.0"
** exception error: {load_driver,"dlopen(/usr/local/lib/erlang/lib/
wx-0.98.1/priv/i386-apple-darwin9.6.0/wxe_driver.so, 2): Symbol not
found: __ZN10wxGLCanvas20MacVisibilityChangedEv\n  Referenced from: /
usr/local/lib/erlang/lib/wx-0.98.1/priv/i386-apple-darwin9.6.0/
wxe_driver.so\n  Expected in: flat namespace\n"}
 in function  wxe_server:start/0
 in call from wx:new/1

If this is familiar to you, then i suggest you do the following
1. Read all of this
or if you are like me building wxErlang & BEAM emulator 5.7 from source
1. Download the wxErlang libraries from wxErlang website and unzip to dir wxMac-2.8.10
2. Change to that directory
3. Execute the following command:

./configure --with-opengl --enable-unicode --disable-shared --enable-graphics_ctx

4. Do the following in sequence:

make && make install
     cd contrib/src/stc/
     make && make install

Note that you might need to do a "sudo make install" if you use the default installation prefix which is /usr/local/{bin,lib, etc}
5. Assumed that you've download Erlang-OTP 13B source code and unzipped to a directory
6. Export the PATH variable such that the build directory to your wxErlang is right at the front of the PATH. e.g. export PATH=<wxErlang dir>:<rest of your path>
7. Build your erlang source code as per your preferences/needs as defined in the README

After all that is over, you should be able to launch your wxErlang programs.

An example of lazy evaluation for generating lists

One thing i've learnt from Erlang is only that its very expressive despite its annoyances but a good outcome of this expressiveness is possibly the availability of lazy(delayed) evaluation. In the example of passing lists to functions, the lists are evaluated fully before being passed to functions.

The big deal here is that you could potentially craft a function that's computationally expensive but the user of this function might not need all the results all at once. Example would be using these sorts of functions to obtain database records and to display them on a webpage but every good UI designer knows that its ineffective to display all 15,000 records at 1 time and you could design a web page that displays 10 - 20 records at a time and have "next" buttons/links to display subsequent pages.

So, naturally we can craft functions such that we can have lazy evaluation. A simple example is as follows where the function next/1 behaves like a generator function but only delayed; another way to think of this concept is possibly python generators. In my example below, its not hard to imagine that next/1 could be your computationally expensive operation.

1 -module(lazy_eval).
2 -export([next/1, test/2]).
3
4 next(Num) ->
5         fun() -> [Num|next(Num+1)] end.
6
7 test(Seq, Fun) when Seq =:= 100 -> [Seq];
8 test(Seq, Fun) when Seq < 100 ->
9         [Val|Func] = Fun(),
10 %        io:format("~p ~p~n", [Val, Func()]),
11         [Val | test(Seq + 1, Func) ].

A sample output (with debugging statements):

6> lazy_eval:test(1, lazy_eval:next(1)).
1 [2|#Fun<lazy_eval.0.5801025>]
2 [3|#Fun<lazy_eval.0.5801025>]
3 [4|#Fun<lazy_eval.0.5801025>]
4 [5|#Fun<lazy_eval.0.5801025>]
...
99 [100|#Fun<lazy_eval.0.5801025>]
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,
23,24,25,26,27,28,29|...]

Have fun!

Why Erlang Sucks - by Damien Katz

Found another interesting read why Erlang sucks. This post is not to create some sort of flame war and the end result is crucifying Damien but rather his post is a rather insightful look into the annoying part of Erlang that most Erlang developers must have seen (me including) and felt.

Examples of what i thought was good was his
* comparison of "if" and "case" in Erlang
* Erlang's concept of strings (Which is a real irritance personally to me)
* Erlang's concept of records which is super similar to C's struct (Another irritance but you can get the hang of it...after a while)
* Absolutely love Damien's take of exit(...) and his comments on hara-kiri (Really hilarious)
* I share Damien's criticisms on Erlang's documentation of the libraries

Overall, its a very honest look into the weaknesses of Erlang as a whole and there's more though...you can form your own opinions about the language though.

What is Erlang and what is its role going to be ?

I borrowed the following quote from the book Programming Erlang

For years we have relied on processors getting faster and faster. That trend is ending. Instead, we now have multicore processors - 2, 4 or more processors running in parallel. The problem is, unless your program can execute in parallel, it will use only one of these cores at a time.

The Erlang programming language lets you build highly parallel, distributed, fault-tolerant systems - systems that can exploit these new architectures. It has been used commercially for many years to create massive fault tolerant and highly reliable systems.

Erlang programs run seamlessly on multicore computers with no extra code on your part. Erlang combines ideas from the world of functional programming with techniques for building fault-tolerant systems. The result is a powerful language that makes it much easier to build the massively parallel networked applications of the future.

Therefore, in the coming blogs i am going to write stuff about Erlang and programming in it, in general.

For a brief introduction to the language features, follow this article.