Chewing the Cud (Posts about News)

Life Update

James Aimonetti — Thu, 26 Jan 2012 19:38:18 GMT

Updated the blog to run 3.3.1 - lot of cobwebs around these parts. Hopefully I can be more proactive in blogging about things going on at work, and perhaps starting to write about what I'm up to personally (not that I have much of that right now). Maybe my Google stats will jump over the 0.3 hits I average! Dare to dream!

IT Expo

James Aimonetti — Thu, 07 Oct 2010 13:34:06 GMT

Just returned from IT Expo West last night. Three days of learning, hob-nobbing, and talking myself hoarse about the awesomeness that is 2600hz. We got a decent writeup posted on TMC's site, met quite a few people, collected beaucoup business cards, and generally had a fun time hanging with the team. Super tired but ready to keep building the best hosted PBX software platform! Bonus: See Darren's awesome (yet mildly awkward) video interview! Also, VoIP service providers looking to offset calling costs for their business clients can look at PromoCalling as a way to compete with Google and Skype's free calling plans.

Still Kicking

James Aimonetti — Fri, 17 Sep 2010 13:06:14 GMT

I am still alive and well; just busy. I did write a blog entry for my company, 2600hz. More to come…eventually.

Dealing with CouchDB and Couchbeam

James Aimonetti — Wed, 05 May 2010 14:29:47 GMT

I'll put this disclaimer up right away: My experience with CouchDB and Couchbeam is limited so I am definitely missing out on some knowledge and understanding that would probably make things easier. I won't detail CouchDB's offerings here as I have noted above my limited insight into that world. I will try to comment on using Couchbeam to get documents into and out of CouchDB, as well as writing a layer to take Couchbeam's document format and cleaning it up into slightly prettier Erlang data structures (proplists of binaries just aren't fun to type). The first thing to do is to connect to your database and get the PIDs for the connection and the DB using the couchbeam\_server:start\_connection\_link/{0,1} and couchbeam\_db:open\_or\_create/2. Couchbeam allows you to name the DB PID so you don't have to carry it around with you; just pass {DB\_NAME\_AS\_ATOM, "DB\_NAME\_AS\_LIST"} as the second parameter to open\_or\_create/2. Now calls can use DB\_NAME\_AS\_ATOM instead of the DB process's PID to communicate. The biggest stumbling block out of the gate is the datatypes used by Couchbeam. Look at the API docs and get familiar with the types you'll be expected to provide and receive from the calls to the database and views. A document is either a json\_object() or a 2-tuple {DocID, Revision}. A json\_object is a proplist of [{json\_string(), json\_term()}]. json\_strings are simple, atom or binary. A json\_term is an atom, binary, integer, float, a list of json\_terms, or another json\_object(). Practically, for my needs, all keys are binaries and the majority of values are either binaries or numeric. I do have one place where I need to store a nested json\_object(); wrap it in a 1-tuple as the value and you should be good.

%% nesting a json_object() as a value in a json_object()
ToNest = [{< >, < >}],
[{< >, < >},
 {< >, { ToNest } }].

Saving a doc is pretty easy; create the proplist and pass it to couchbeam\_db:save\_doc/2. One big gotcha when saving a document was whether the < > key existed. If the \_rev key was undefined, a big ugly bad\_match error was thrown with all sorts of chaos going on around it.

Erlang and Webmachine Tres

James Aimonetti — Sat, 24 Apr 2010 12:02:41 GMT

In my first post, I talked about the basic technologies I'd be using, and how to get an interactive shell started in emacs that started up your webmachine application. In this third post, I'll talk a little about building a session handling module. This will hopefully familiarize you with using redirects and cookies in your webmachine resources and using Couchbeam to store and retrieve documents. The high-level usage for sessions I have currently is:

User agent accesses a supported resource (probably a browser accessing the home page initially).
- I am defining the is\_authorized/2 function in all of my resource files that correspond to web pages, as well as the finish\_request/2 function.
- The second parameter in the functions is a record, currently defined as
```
-record(context, {session}).
```
In the is\_authorized/2 function for public facing pages, it returns the tuple:
```
is_authorized(RD, Context) ->
    {true, RD, Context#context{session=session:start(RD)}.
```
- Within session:start/1, I extract the session cookie (if it exists) and retrieve the session document from CouchDB. If the session cookie isn't set, I create a new #session{} record with some defaults pre-filled.
- #+BEGIN_EXAMPLE %% session.hrl -record(session, {'_id', '_rev', user_id, expires, created, storage=[]}).
  
  %% session.erl start(RD) -> get_session_rec(RD).
  
  get_session_id(RD) -> wrq:get_cookie_value(?COOKIE_NAME, RD).
  
  %% get_session_doc(request()) -> not_found | json_object() get_session_doc(RD) -> case get_session_id(RD) of undefined -> not_found; SessDocId -> couchbeam_db:open_doc(?DB, SessDocId) end.
  
  %% get_session_rec(request()) -> session_rec() get_session_rec(RD) -> case get_session_doc(RD) of not_found -> new(); Doc -> from_couchbeam(Doc) end.
  
  new() -> Now = calendar:datetime_to_gregorian_seconds(calendar:local_time()), #session{created=Now, expires=?MAX_AGE, '_id'=couchbeam_util:new_uuid()}.
  
  #+END_EXAMPLE
- So first I pull out the session ID from the cookie header in get\_session\_id/1 which will return either undefined or the ID.
- If the session ID is undefined, get\_session\_doc/1 returns the not\_found atom, which is what Couchbeam returns when it can't find a document. If a session ID is found, I call couchbeam:open\_doc/2 to retrieve the document from CouchDB.
- Finally, if get\_session\_doc/1 returns not\_found, I create a new session record calling new/0. If a document is returned, I call a function that transforms the proplist to the session record.
- Note: Couchbeam will auto assign your '\_id' if the document to be saved doesn't have one or it is undefined. While useful, for sessions it causes a problem as the auto-generated UUIDs are fairly sequential (not exactly but usually only the last four or five characters differ). Calling couchbeam\_util:new\_uuid() creates your random uuids.
- I store the created time in seconds, as well as the expires time (30 minutes worth of seconds right now). This allows me to easily clear ended sessions from CouchDB with a view indexing the session documents by (created+expires).
Now, as the resource request is processed, I can always access the session through the Context parameter.

When the request is done processing, it is time to save the session and set the cookie. In wm\_resource:finish\_request/2 I have:

finish_request(RD, Context) ->
    {true, wm_session:finish(Context#context.session, RD), Context}.

And in session.erl, I define the finish/2 process:

%% finish(#session(), request()) -> request().
%%  save session and set cookie headers
finish(S, RD) -> save_session_rec(S, RD).

save_session_rec(S, RD) ->
    DateTime = calendar:local_time(),
    Now = calendar:datetime_to_gregorian_seconds(DateTime),
    case has_expired(S#session.created + S#session.expires) of
        true -> close(S, RD);
        false ->
            D = to_couchbeam(S#session{created=Now,expires=?MAX_AGE}),
            S1 = from_couchbeam(couchbeam_db:save_doc(?DB, D)),
            set_cookie_header(RD, S1, DateTime, ?MAX_AGE)
    end.

close(S, RD) ->
    DateTime = calendar:local_time(),
    SessionDoc = to_couchbeam(S),
    couchbeam_db:delete_doc(?DB, SessionDoc),
    set_cookie_header(RD, S, DateTime, -1).

set_cookie_header(RD, Session, DateTime, MaxAge) ->
    {CookieHeader, CookieValue} = mochiweb_cookies:cookie(?COOKIE_NAME, Session#session.'_id', [{max_age, MaxAge},
                                                                                                {local_time, DateTime}]),
    wrq:set_resp_header(CookieHeader, CookieValue, RD).

So to finish a session, I check if the current session has expired, and if so close it down. Otherwise, I update the created and exipres properties of the session and save it to CouchDB, and then set the cookie headers to reflect the new times.

For a resource needing valid authentication to be accessible, the is\_authorized/2 is slightly different:

%% wm_resource.erl
is_authorized(RD, Context) ->
    S = session:start(RD),
    case session:is_authorized(S) of
    true -> {true, RD, Context#context{session=S}};
        false ->
            RD0 = wrq:do_redirect(true, RD),
            RD1 = wrq:set_resp_header("Location", "/login", RD0),
            {{halt, 307}, RD1, Context#context{session=S}}
    end.

%% session.erl
is_authorized(S) ->
    S#session.user_id =/= undefined andalso
        not has_expired(S#session.created + S#session.expires).

Obviously you can set your own redirect header. The {halt, 307} is a temporary redirect (since hopefully you've only forgotten to login or been away too long and your session expired).

Erlang, Euler, Primes, and gen_server

James Aimonetti — Fri, 26 Mar 2010 09:33:33 GMT

I have been working on Project Euler problems for a while now and many of them have centered around prime numbers. I've referenced my work with the sieve in other posts but found with a particular problem that some of my functions could benefit from some state being saved (namely the sieve being saved and not re-computed each time). The problem called to count prime factors of numbers and find consecutive numbers that had the same count of prime factors. My primes module had a prime\_factors/1 function that would compute the prime factors and the exponents of those factors (so 644 = 2² * 7 * 23, and primes:prime\_factors(644) would return [{2,2},{7,1},{23,1}]. The prime\_factors/1 looked something like this:

prime_factors(N) ->
    CandidatePrimes = prime_factors(N, primes:queue(N div 2)),
    PrimeFactors = [ X || X < - CandidatePrimes, N rem X =:= 0 ],
    find_factors(PrimeFactors, N, 0, []).

The call to find\_factors/4 takes the factors and finds the exponents and can be ignored for now. The time sink, then, is in generating the CandidatePrimes list. I think my primes:queue/1 function is pretty fast at generating the sieve, and dividing N by two eliminates a lot of unnecessary computation, but when you're calling prime\_factors/1 thousands of times, the call to queue/1 begins to add up. This is where I needed to save some state (the sieve) in between calls. Erlang, fortunately enough, has a module behavior called gen\_server that abstracts away a lot of the server internals and lets you focus on the business bits of the server. I won't discuss it much here as I'm not an authority on it, but Joe Armstrong's book and the Erlang docs have been a great help in understanding what's happening behind the scene. You can view the prime\_server module to see what its current state is code-wise. To speed up prime\_factors/1, I split it into two functions prime\_factors/1 and prime\_factors/2. The functions look like this:

prime_factors(N, CandidatePrimes) ->
    PrimeFactors = [ X || X < - CandidatePrimes, N rem X =:= 0 ],
    find_factors(PrimeFactors, N, 0, []).

prime_factors(N) ->
    prime_factors(N, primes:queue(N div 2)).

Now, if we don't need to save the queue between calls you can still call prime\_factors/1 as usual. The prime\_server module utilizes the prime\_factors/2 function because it initializes its state to contain a primes sieve (either all primes under 1,000,000 if no arg is passed to start\_link, or all primes =< N when using start\_link/1) and the current upper bound. Now when the server handles the call for getting the factors, we pass a pared down list of primes to prime\_factors/2 and get a nice speed boost. Well, the heavy lifting is front-loaded in the initialization of the server (generating the sieve) and in calls that increase the sieve's size. One improvement there might be to save the Table generated during the initial sieve creation and start the loop back up from where it left off (when N > UpTo) but that is for another time. If you choose your initial value for start\_link right, regenerating the sieve should be unnecessary. The last speed boost was noticing that calculating the exponents was an unnecessary step so I wrote a count\_factors/1 and count\_factors/2 that skips the call to find\_factors/4 and returns the length of the list comprehension. With these changes complete, problem 47 went from taking well over 5 minutes to just under 20 seconds to solve brute force.

Purely Functional Data Structures and Me

James Aimonetti — Tue, 23 Mar 2010 10:24:20 GMT

Quick post to say that I've put my dabblings into Chris Okasaki's Purely Functional Data Structures book up as a github repo. I am picking and choosing what exercises and examples to code, with the goal being to slowly cover the whole book. Some concessions are made to fit the ideas into Erlang (like recursively calling an anonymous function), but overall I think the ideas fit nicely. There is a streams lib from Richard Carlsson that I found on the Erlang mailing list in the repo as well that I used for reference for a couple things in chapter 4. I stuck with my streams being represented either as an empty list [] or [term() | fun()] with term() being the calculated value and fun() being the suspension function, instead of the tuple that Richard chose. After reading further in the thread, I understand why (don't confuse people that might use proper list functions on streams) but for my small examples, it was enough to use lists.

Connect to remote erlang shell while inside emacs

James Aimonetti — Thu, 07 Jan 2010 13:41:18 GMT

While developing my top secret project, I have been getting into the fun stuff in Erlang and Emacs. Connecting to a running instance of my app from a remote shell wasn't straightforward to me at first, so below is my documented way of connecting, as well as dropping into the Erlang JCL from within an Emacs erlang shell.

Start yaws: yaws –daemon -sname appname –conf /path/to/yaws.conf
Start emacs, and from within emacs start an Erlang shell with C-c C-z (assuming you have distel configured).
From the Emacs erlang shell, get into Erlang's JCL by typing C-q C-g and pressing enter. A \^G will be printed at the prompt, but won't be evaluated until you press enter. You should see the familiar JCL prompt "User switch command –>".
Type 'j' to see current jobs you have running locally, which is probably just the current shell (1 {shell,start,[init]}).
Type 'r appname@compy' to connect to the remote node identified by appname ( from the -sname parameter ) on the computer compy (usually whatever hostname returns)
Type 'j' to see current jobs, which should list your current shell as "1 {shell,start,[init]}", and a second shell "2* {appname@compy,shell,start,[]}".
Type 'c 2' to connect to the remote shell. You can now run commands in the node's shell. You may have to press enter again to bring up a shell prompt.

james@compy 14:33:34 ~/dev/erlang/app
> yaws --daemon -sname app --conf config/yaws.conf

james@compy 14:34:00 ~/dev/erlang/app
> emacs

Eshell V5.7.4  (abort with ^G)
1> ^G

User switch command
 --> j
   1* {shell,start,[init]}
 --> r app@compy
 --> j
   1  {shell,start,[init]}
   2* {app@compy,shell,start,[]}
 --> c 2

1>

Posting from Emacs

James Aimonetti — Thu, 24 Dec 2009 19:37:11 GMT

I am posting this short message from emacs using the weblogger.el package. Cool!

Erlang, Primes, and the Sieve of Eratosthenes

James Aimonetti — Wed, 16 Dec 2009 12:37:54 GMT

Working through the Project Euler again and using Erlang to do so, there are quite a few problems that deal with primes. It is important, then, to have a library of functions that make generating and validating primes easy. A typical method for generating primes is using the Sieve of Eratosthenes. As discussed in the wiki article, Melissa O'Neill has shown the given Haskell implementation is not a true implementation, and shows a couple versions that are more true to the algorithm and more performant. I took the implementation she described using a priority queue on page 7 of the pdf. I used a skew heap implementation I found as the priority queue, modified it slightly to handle a Key and a Value parameter, and away I went. Here's the implementation I came up with:

%% primes.erl
-module(primes).

-export([queue/1]).

queue(N) ->
    sieve_queue(lists:seq(2, N)).

sieve_queue([]) ->
    [];
sieve_queue([X|XS]) ->
    Table = insert_prime(X, skew_kv:empty()),
    [X | sieve_queue(XS, Table)].

insert_prime(P, Table) ->
    skew_kv:insert(P*P, from(P*P, P), Table).

sieve_queue([], _Table) ->
    [];
sieve_queue([X|XS], Table) ->
    {NextComposite, _Value} = skew_kv:min(Table),
    case  NextComposite =< X of
        true -> sieve_queue(XS, adjust(Table, X));
        _Else -> [X | sieve_queue(XS, insert_prime(X, Table))]
    end.

adjust(Table, X) ->
    {N, [Nprime | NS]} = skew_kv:min(Table),
    case N =< X of
        true ->
            T = skew_kv:delete_min(Table),
            T2 = skew_kv:insert(Nprime, NS(), T),
            adjust(T2, X);
        _Else -> Table
    end.

%% from http://www.erlang.org/cgi-bin/ezmlm-cgi?4:mss:177:khdaceipfabbicmifdhf
%% a lazy list that starts at K and increments by Inc
from(K, Inc) ->
    [K|fun()-> from(K+Inc, Inc) end].

I tried to keep variable names similar to the O'Neill implementation, but I have modified it a bit to use lazy lists, Erlang-style. Not being familiar with Haskell, I think lists there are defined lazily by default, but Erlang needs an explicit construct to do lazy lists. Fortunately, it isn't terribly hard. So, the code! I named the function queue because I was implementing prime number generators using other methods from the paper (the unfaithful sieve and a sieve with a map instead of a queue), hence why you need to call the function with:

1> primes:queue(10).

I don't think I'll expound on the why, as O'Neill does a nice job explaining it in her paper, so I'll focus on the how and leave it to the reader to learn the why. My insert\_prime function differs from the O'Neill insertprime because mine has to account for Erlang-style lazy lists. My assumption is that the map call in O'Neill's creates a lazy list and so all the mappings are not computed, since XS is a potentially huge list of numbers. Instead, I observed that those lists are created by the map call are arithmetic progressions that start at p*p and increase by p. My simple lazy list that accomplishes this is called from/2, which I got from a Joe Armstrong suggestion and adapted to my needs. Let's take a quick look at from/2. We pass in a number K and an increment Inc and return a list with K at the head and a fun as the tail. The fun wraps a call to from/2, which acts as the iterator for the lazy list. So evaluating

1> primes:from(2, 2).
[2|#Fun]

Here's an example showing the iterator in action:

57> [K|F1] = primes:from(2, 2).
[2|#Fun]
58> [K2|F2] = F1().
[4|#Fun]
59> [K3|F3] = F2().
[6|#Fun]
60>

This lazy list is stored in the queue as the value (line 16) for the key P*P. The lazy list is then used in the adjust/2 function when we want to compare the smallest key of the queue to the current number we're evaluating. Line 28 extracts the K into Nprime, and the fun into NS, from the from/2 call earlier. Line 32 shows how to get the next number in the lazy list, NS(), which returns [K+Inc | fun()], and inserts it as the value into the queue. Again, for the why, consult the O'Neill paper. Otherwise, the implementation is roughly the same. There are some improvements and optimizations that could be made, I'm sure, but for a rough draft, it is fairly speedy, minimal memory usage, and for Project Euler, it is more than adequate in producing the primes I need to solve the problems.