Chewing the Cud (Posts about work)

Using Erlang's =maps:groups_from_list/2= function

James Aimonetti — Fri, 28 Mar 2025 22:28:07 GMT

Since OTP 25, Erlang has included a function maps:groups_from_list/2 and I had occasion to use it recently.

In KAZOO, we have a process that monitors other processes, keeping track of their message queue length, heap size, etc (we're on OTP 26 at the moment so need to do this manually).

I wanted to print a table of the tracked processes, ordered first by message queue length, but within the group, sorted by heap size.

-record(proc_info, {pid, message_queue_len, total_heap_size}).

sort_processes(Processes) ->
    %% Groups = #{mql => [#proc_info{}...]
    Grouped = maps:groups_from_list(fun(#proc_info{message_queue_len=MQL}) -> MQL end
				   ,Processes
				   ),
    maps:fold(fun sort_group_by_heap/3, [], Grouped).

sort_group_by_heap(_MQL, ProcInfos, Acc) ->
    lists:keysort(#proc_info.total_heap_size, ProcInfos) ++ Acc.

The first argument is used to extract the each list element a "key" to use when inserting the element into the resulting map. In my case, the grouping was for message_queue_len.

The Grouped map() will contain keys corresponding to the unique set of message queue lengths, and the value will be the list of #proc_info{} records.

There is an arity-3 version where you can map the list element into a new value to be put into the grouping's list of results as well. As I want to sort the records by total_heap_size after grouping, I had no cause to use it.

What I like about Erlang records

Records get a bad rap in Erlang, especially now that maps exist. I find the utility of records to be so great that I reach for them way more often than for maps when I have a fixed amount of fields. Records work really well with the lists:key* functions to, providing the index into the record tuple as above in sort_group_by_heap.

One day I'll write more about how I use records but wanted to mention this specific use as a common one in my work.

Reverse binaries in Erlang

James Aimonetti — Wed, 26 Mar 2025 17:06:38 GMT

Reversing binaries in Erlang

In KAZOO, we have a kz_binary module that performs various functions over binary() data (typically string-ish stuff). Most of this code came before the string module was introduced in OTP 20 (for those that may not be aware, KAZOO started on R15B03 in 2010).

Recently we encountered an issue in a production cluster where a large blob of JSON was being processed and kz_binary:strip_right/2 was called to remove trailing endlines (why is a longer story related to changes in how we are receiving data from an external service that haven't been reflected yet in KAZOO's reception of the data) and getting stuck.

strip_right reverses the binary() and removes the supplied character, then reverses the result back. The existing reverse/1 implementation was cribbed from a Stack Overflow answer:

Existing implementation

reverse(Binary) ->
    Size = byte_size(Binary) * 8,
    <<X:Size/little-integer>> = Binary,
    <<X:Size/big-integer>>.

This works great until Binary gets large. In testing, if the binary() is large (2²² (4194304) bits or 2¹⁹ (524288) bytes), this binary matching will fail. Coincidentally, the JSON blob was 562868 bytes and triggered the crash.

After updating the tests to include a large enough binary to stress the code, I tried a couple different approaches before settling on chunking the binary input up, reversing the chunks into an accumulator, and converting that list of chunks back into a binary.

New implementation - reverse the chunks

%% for large binaries, we need to chunk them up and reverse them
%% REV_CHUNK was chosen because #reasons (through experiments)
-define(REV_CHUNK, 8192).

-spec reverse(binary()) -> binary().
reverse(<<Front:?REV_CHUNK/integer-little, Rest/binary>>) ->
    list_to_binary(reverse(Rest, [<<Front:?REV_CHUNK/integer-big>>]));
reverse(Binary) ->
    %% this fails for large binaries
    Size = bit_size(Binary),
    <<X:Size/little-integer>> = Binary,
    <<X:Size/big-integer>>.

%% side note: it was observed that creating a binary as the
%% accumulator caused execution time to range 30-60ms while using the
%% list as accumulator and converting to binary at the end caused
%% execution time to range 1-6ms typically, hence the list of binaries here
reverse(<<Front:?REV_CHUNK/integer-little, Rest/binary>>, Acc) ->
    reverse(Rest, [<<Front:?REV_CHUNK/integer-big>> | Acc]);
reverse(Rest, Acc) ->
    [reverse(Rest) | Acc].

REV_CHUNK was chosen through some experiments with timer:tc/3 running but there wasn't that much variability. A better perf test might settle on a better power of 2 but at the time, no meaningful reduction in time was observed on the dev machine used.

One other interesting note was using a list for the accumulator vs constructing the resulting binary (eg <<Front/binary, Acc/binary>> in reverse/2) was much faster.

Do note that this approach doesn't take into account Unicode stuff like grapheme clusters (see the Erlang string description and Using Unicode in Erlang for more). Caveat emptor!

Why I enjoy Erlang

I want to highlight two bits of the code above that I appreciate about Erlang that may not be apparent to some.

Clause ordering
First, the order of clauses in the reverse/1 provide some assurances:
- We know that any binary() input that has ?REV_CHUNK bytes (or more) will be handled in the first clause
- Because of that, we know that Binary in the second clause must be smaller than ?REV_CHUNK bytes.
This is a powerful approach I use a lot in Erlang - using clauses to match arguments as specifically as possible. The code is self-documenting on how to handle large vs small binaries (and what "large" is defined as). Recursive functions are easily defined by stating the base case in the first clause (or clauses) and the recursive part as the last clause.

For instance, also from kz_binary:
```
pos(Char, Bin) ->
    pos(Char, Bin, 0).

pos(_Char, <<>>, _Pos) -> -1;
pos(Char, <<Char, _/binary>>, N) -> N;
pos(Char, <<_, Bin/binary>>, N) ->
    pos(Char, Bin, N + 1).
```
1. First clause of pos/3 says we reached the end of Bin and didn't find Char
2. Second clause says we found Char and return the position it was found
3. Third clause does the recursion, taking the first byte off Bin and incrementing the position

List construction via prepend

Prepending elements onto a list is fast and gives Erlang a natural stack data structure. If we look at a trace of how reverse/2 works we can see how elegantly the list construction helps create the solution:

With REV_CHUNK=32 (4 8-byte chunks) when calling kz_binary:reverse(<<"abcdefghijkl">>):

Function	Front	Rest	Acc
rev/1	1684234849	`<<"efghijkl">>`
rev/2	1751606885	`<<"ijkl">>`	`[<<"dcba">>]`
rev/2	1818978921	`<<>>`	`[<<"hgfe">>,<<"dcba">>]`
rev/2		`<<>>`	`[<<"lkji">>,<<"hgfe">>,<<"dcba">>]`

list_to_binary([<<"lkji">>,<<"hgfe">>,<<"dcba">>]) => <<"lkjihgfedcba">>

Of course, this is a manually-written fold. Conceptually the code works the same as:

reverse(Binary) ->
    lists:foldl(fun(<<Nib:?REV_CHUNK/big>>, Acc) -> [Nib | Acc] end
	       ,[]
	       ,[Bin || <<Bin:?REV_CHUNK/little>> <= Binary] % obviously assumes binary has even number of REV_CHUNK chunks
	       ).

Alternative approaches

I looked at some other approaches and found them to be lacking.

Simple reverse using binary accumulator:

%% this one took 9007807 (or 9 seconds) on the big input
reverse(Binary) ->
    reverse(Binary, <<>>).

reverse(<<>>, Acc) -> Acc;
reverse(<<H:1/binary, Rest/binary>>, Acc) ->
    reverse(Rest, <<H/binary, Acc/binary>>).

Using lists:reverse/1:

%% this took a range of 15..64 ms on the big input
reverse(Binary) ->
    list_to_binary(lists:reverse(binary_to_list(Binary))).

2025-03-28: Newer version of `reverse/1`

Of course, while writing this post, I was doing some more reading in the Erlang docs and found that, apparently since R14B, the binary module contains encode_unsigned/2 and decode_unsigned/2 functions!

reverse(Binary) ->
    binary:encode_unsigned(binary:decode_unsigned(Bin, 'big'), 'little').

How lovely! I always like when we can leverage core Erlang code over home-rolled solutions. We have more than enough in KAZOO and I hope we can continue to weed it out where approriate.

Setting up call forwarding in KAZOO

James Aimonetti — Mon, 06 Apr 2020 21:59:59 GMT

With the transition to working from home in full swing, let's look at the ways KAZOO makes it easy to bring your office with you!

From Scratch

Let's say you are a solo business owner who needs to forward your business phone calls to your personal mobile phone. Assuming the business phone number is provisioned for the account already, here's the basic steps to set that up!

Setup the callflow

A callflow in KAZOO is what is executed when a configured phone number, extension, or regexp matches the incoming call. The flow portion will then be a series of actions (like ringing devices, going to voicemail, sending to the carrier, etc) chained together to control how the caller progresses through the call.

Since you want to forward calls from your business number to your mobile, you'll need to use the resources action. The callflow data object would look like:

{
  "flow": {
    "data": {
      "to_did": "{MOBILE_NUMBER}"
    },
    "module": "resources"
  },
  "name": "forward business to mobile",
  "numbers": [
    "{BUSINESS_NUMBER}"
  ]
}

Simply stated, the resource action's to_did will override the normal action to call the to number with to_did instead. The power here is that the same callflow can be used for many numbers (say multiple businesses you might accept calls for).

The basic API call then becomes:

curl -H "X-Auth-Token: {API_AUTH_TOKEN}" \
     -X PUT \
     -d '{"data":{"name":"forward business to mobile","numbers":["{BUSINESS_NUMBER}"],"flow":{"module":"resources","data":{"to_did":"{MOBILE_NUMBER}"}}}}'
     https://api.kazoo.domain/v2/accounts/{ACCOUNT_ID}/callflows

Voicemail considerations

With calls forwarded to the carriers in the manner above, your personal voicemail will answer the call if you are unable to at the time. If this is not desirable, you will need to slightly modify how you make this happen.

What you need is to require the callee (you) to press 1 to accept the call before KAZOO will let connect the caller to your mobile phone. If your personal voicemail tries to pick up the call, KAZOO will not continue and will return to the callflow and try the child of the resources action. Since there is no child, the call will be hung up - not ideal!

To setup the required keypress, you'll need to create a call-forwarded device.

Call-Forwarded Device

KAZOO devices can be configured to represent anything from SIP devices to call-forwarded devices, WebRTC, and other types. The other nice benefit is that you can assign the device, via the owner_id property, to your KAZOO user and get calls to ring any of your devices!

To create the call-forwarded device, you only need to setup the call_forward object:

{
  "call_forward": {
    "enabled": true,
    "number": "{MOBILE_NUMBER}",
    "require_keypress": true
  },
  "name": "WFH device",
  "owner_id": "{USER_ID}"
}

Not strictly required to include owner_id (especially if this is a temporary setup until the office re-opens).

The API command then becomes:

curl -H "X-Auth-Token: {API_AUTH_TOKEN}" \
     -X PUT \
     -d '{"data":{"name":"WFH device","call_forward":{"enabled":true,"number":"{MOBILE_NUMBER}","require_keypress":true},"owner_id":"{USER_ID}"}}'
     https://api.kazoo.domain/v2/accounts/{ACCOUNT_ID}/devices

The callflow action is then changed to:

"flow":{
  "module":"device"
  ,"data":{"id":"{WFH_DEVICE_ID}"}
}

Now when your personal voicemail picks up, KAZOO will not connect the caller and will instead go to the child (which currently is missing!). Let's add your business voicemail box.

Call-forwarding plus Voicemail

If you haven't already created a KAZOO voicemail box its a straight-forward process similar to device and callflow creation via API.

Once you have the voicemail box ID you can modify the callflow's flow to look like:

{
  "flow": {
    "children": {
      "_": {
	"data": {
	  "id": "{VOICEMAIL_BOX_ID}"
	},
	"module": "voicemail"
      }
    },
    "data": {
      "id": "{WFH_DEVICE_ID}"
    },
    "module": "device"
  }
}

Of course now that you have KAZOO voicemail, you can enable voicemail-to-email, transcription (if your cluster supports it), auto-delete from the voicemail box after successfully sending the email, and more.

Configure using the UI

You can also use two of 2600Hz's UI apps, SmartPBX or Advanced Callflows, to achieve the same configurations as the direct API examples above.

Take a walk-through of SmartPBX to see more or jump to the call-forwarding setup.

Customize routing decisions

Sometimes KAZOO callflow actions can't quite encode the flow you would like to achieve, or KAZOO doesn't have access to data sources needed to make routing decisions. Fortunately KAZOO has Pivot to help you take control of call routing when you need more dynamic handling.

The callflow is simple:

{
  "flow": {
    "children": {
      "_": {
	"data": {
	  "id": "{WFH_CALLFLOW_ID}"
	},
	"module": "callflow"
      }
    },
    "data": {
      "method": "POST",
      "req_format": "kazoo",
      "voice_url": "https://your.http.server/pivot/whatever/language/you/like.ext"
    },
    "module": "pivot"
  },
  "name": "dynamic call control",
  "numbers": [
    "{BUSINESS_NUMBER}"
  ]
}

Now any call to {BUSINESS_NUMBER} will issue an HTTP request to your.http.server. If your server fails to respond, the callflow will continue and run the callflow from above (you would need to remove the {BUSINESS_NUMBER} from the numbers array on that callflow so they don't conflict with the Pivot callflow here).

Your server, in your language of choice, will process the request data, consult any data sources, make any business logic decisions, then send a response with the appropriate callflow 'flow' object. For instance, to have callflow mirror the routing to the call-forwarded device, your PHP script might look like:

<?php
header('content-type: application/json');

/* Business logic
 * Database lookups
 * Whatever shenanigans
 */
?>
{
  "module":"device"
  ,"data":{"id":"{WFH_DEVICE_ID}"}
}

Wrap-up

In this post we've walked through what API calls and JSON objects would be needed to setup call-forwarding of a phone number (like the business' phone number) to another phone number (like the owner's personal mobile number). Depending on what voicemail box you'd like to answer if the callee is unavailable will determine which path, device or resource, you use to route the caller.

We also briefly introduced the idea of using Pivot to encode custom logic when routing callers to the call-forwarded device (and provide a child action if the Pivot request fails for any reason).

The building blocks exposed in KAZOO's callflow actions are many (~75 at last count) and run the gamut from low-level - collecting touch tones (DTMF) or playing text (TTS) to the caller, or time of day routing - up to more conceptual actions like leaving and checking voicemails, ringing a user's devices, user directories, and so much more.

Have questions? Join our forum, hop on IRC (Freenode #2600hz) to chat, and start an all-in-one install to test things out.

If you have a business and want to engage on using our hosted platform or other offerings, we at 2600Hz are ready to chat (sales@2600hz.com) and help you build it!

Stateless Property-based testing with Erlang and PropEr

James Aimonetti — Mon, 26 Nov 2018 21:10:27 GMT

In preparation for my talk at CodeBEAM, I thought I'd give an introductory example of some stateless property-based tests as an introduction to property-based testing.

Fine reading and viewing options

First things first, get yourself a copy of the Property-Based Testing book.

Second, there are a number of blog posts and videos online that talk about property-based testing that are far better resources than I would write. In no particular order:

An introduction to property-based testing (using F#) - I like the approach of an adversarial relationship between the code writer and the test writer. How can the test writer ensure the code writer is properly implementing a function without cheating on the implementation just to get tests to pass?
Code checking automation - John Hughes breaks down testing some SMS processing code using Quickcheck and finding an inconsistency in the spec.
Don't write tests! - John Hughes talks about why testing is hard and how property-based testing can make it easier (and maybe more fun!).
Testing web services with QuickCheck - Using QuickCheck (and stateful property testing) to test web services.
The Hitchhiker's Guide to the Unexpected - Fred Hebert talks about using property testing to test his supervisor hierarchy and how it reacts to failure (see here specifically).

Basically any video where John Hughes or Thomas Arts talk QuickCheck or property-based testing is worth a watch!

PropEr

We use the open source, QuickCheck-inspired, testing tool PropEr to develop our tests. The homepage for the project has links to tutorials and other helpful information specifically related to using PropEr.

The Code

Briefly, we want to test the higher-level properties of a function instead of explicitly enumerating the inputs and outputs we expect.

Let us consider the function camel_to_snake/1 which takes a Camel case string (such as ThisIsCamelCase) and converts it to Snake case (such as this_is_snake_case).

camel_to_snake(<<First, Bin/binary>>) ->
    iolist_to_binary([to_lower_char(First)
		     ,[maybe_camel_to_snake(Char) || <<Char>> <= Bin]
		     ]).

maybe_camel_to_snake(Char) ->
    case is_upper_char(Char) of
	'false' -> Char;
	'true' ->
	    [$_, to_lower_char(Char)]
    end.

is_upper_char(Char) ->
    Char >= $A andalso Char =< $Z.

to_lower_char(Char) when is_integer(Char), $A =< Char, Char =< $Z -> Char + 32;
to_lower_char(Char) -> Char.

Pretty simple implementation - if the character is uppercase, convert it to an underscore and the lowercase version of the character.

Now, if we were doing unit tests alone, we might write:

camel_to_snake_test_() ->
    Tests = [{<<"Test">>, <<"test">>}
	    ,{<<"TestKey">>, <<"test_key">>}
	    ,{<<"testKey">>, <<"test_key">>}
	    ,{<<"TestKeySetting">>, <<"test_key_setting">>}
	    ,{<<"testKeySetting">>, <<"test_key_setting">>}
	    ,{<<"TEST">>, <<"t_e_s_t">>}
	    ],
    [?_assertEqual(To, camel_to_snake(From))
     || {From, To} <- Tests
    ].

And for a relatively trivial function like this, we're reasonably confident those tests are adequate:

make eunit
...
  module camel_tests'
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    [done in 0.018 s]
...

Let's see if we can get some more coverage of the input space with property tests!

Properties to test

We have a pretty basic property here - take a string and convert uppercase characters to _{lowercase}.

So how can we generate inputs and outputs that will help test the conversion?

Well, we can generate the CamelCase string character by character and build the snake_case version as we go.

First, the high level property test:

prop_morph() ->
    ?FORALL({Camel, Snake}
	   ,camel_and_snake()
	   ,?WHENFAIL(io:format("~nfailed to morph '~s' to '~s'~n", [Camel, Snake])
		     ,Snake =:= camel_to_snake(Camel)
		     )
	   ).

So `camel_and_snake/0` is a generator that will return a 2-tuple with the built strings, will compare the call to `camel_to_snake/1` against the generated Snake, and if the property fails (evals to false), we output the failing pair for analysis.

camel_and_snake() ->
    camel_and_snake(30). % we don't want overly huge strings so cap them to 30 characters for now

camel_and_snake(Length) ->
    ?LET(CamelAndSnake
	,camel_and_snake(Length, [])
	,begin
	     %% we create a list of [{CamelChar, SnakeChar}]; unzipping results in {CamelChars, SnakeChars} as iolist() data
	     {Camel, Snake} = lists:unzip(lists:reverse(CamelAndSnake)),
	     {list_to_binary(Camel), list_to_binary(Snake)}
	 end
	).

%% Create a list of [{CamelChar, SnakeChar}]
camel_and_snake(0, CamelAndSnake) ->
    CamelAndSnake;
camel_and_snake(Length, CamelAndSnake) ->
    CandS = oneof([upper_char()
		  ,lower_char()
		  ]),

    camel_and_snake(Length-1
		   ,[CandS | CamelAndSnake]
		   ).

%% Lower chars are easy - whatever the camel gets the snake gets too
lower_char() ->
    ?LET(Lower
	 ,choose($a,$z)
	 ,{Lower, Lower}
	).

%% Uppercase just requires a little math
upper_char() ->
    ?LET(Upper
	,choose($A,$Z)
	,{Upper, [$_, Upper+32]}
	).

Running it the first time we see:

(prop_morph)......!
Failed: After 4 test(s).
{<<70,74,101,101,81,90,81,72,112,117,84,74,106,100,100,67,90,98,98,122,80,107,81,77,99,79,113,82,84,99>>,<<95,102,95,106,101,101,95,113,95,122,95,113,95,104,112,117,95,116,95,106,106,100,100,95,99,95,122,98,98,122,95,112,107,95,113,95,109,99,95,111,113,95,114,95,116,99>>}

failed to morph 'FJeeQZQHpuTJjddCZbbzPkQMcOqRTc' to '_f_jee_q_z_q_hpu_t_jjdd_c_zbbz_pk_q_mc_oq_r_tc'

Shrinking ............................................(44 time(s))
{<<65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65>>,<<95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97>>}

failed to morph 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' to '_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a'

Ah, right, if the first character is uppercase, we don't want to introduce the underscore. Let's fix the generator to account for the first character situation:

%% First, we need to choose a different start
camel_and_snake(0, CamelAndSnake) ->
    CamelAndSnake;
camel_and_snake(Length, []) ->
    CandS = oneof([first_upper_char()
		  ,lower_char()
		  ]),
    camel_and_snake(Length-1, [CandS]);
camel_and_snake(Length, CamelAndSnake) ->
    CandS = oneof([upper_char()
		  ,lower_char()
		  ]),

    camel_and_snake(Length-1
		   ,[CandS | CamelAndSnake]
		   ).

%% We define first_upper_char to not include the underscore for the snake version
upper_char() ->
    ?LET(Upper
	,choose($A,$Z)
	,{Upper, [Upper+32]}
	).

Running this through PropEr gives us:

proper_test_ (prop_morph) .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
OK: Passed 500 test(s).

Ship it!

Not quite ready

One of the joys of KAZOO is getting to learn about how telecom works in the wider world. This forces Unicode front and center when dealing with user input (on the plus side, we can use emojis for phone extensions!).

Looking at our code, we see we've naively handled just the Latin-based alphabet. For inspiration, we open up string.erl to see how it handles upper/lowercase conversions:

%% ISO/IEC 8859-1 (latin1) letters are converted, others are ignored
%%

to_lower_char(C) when is_integer(C), $A =< C, C =< $Z ->
    C + 32;
to_lower_char(C) when is_integer(C), 16#C0 =< C, C =< 16#D6 ->
    C + 32;
to_lower_char(C) when is_integer(C), 16#D8 =< C, C =< 16#DE ->
    C + 32;
to_lower_char(C) ->
    C.

to_upper_char(C) when is_integer(C), $a =< C, C =< $z ->
    C - 32;
to_upper_char(C) when is_integer(C), 16#E0 =< C, C =< 16#F6 ->
    C - 32;
to_upper_char(C) when is_integer(C), 16#F8 =< C, C =< 16#FE ->
    C - 32;
to_upper_char(C) ->
    C.

Let's adjust our generators first to see failing cases:

lower_char() ->
    ?LET(Lower
	,union([choose($a,$z)
	       ,choose(16#E0,16#F6)
	       ,choose(16#F8,16#FE)
	       ])
	,{Lower, [Lower]}
	).

first_upper_char() ->
    ?LET(Upper
	,union([choose($A,$Z)
	       ,choose(16#C0,16#D6)
	       ,choose(16#D8,16#DE)
	       ])
	,{Upper, [Upper+32]}
	).

upper_char() ->
    ?LET(Upper
	,union([choose($A,$Z)
	       ,choose(16#C0,16#D6)
	       ,choose(16#D8,16#DE)
	       ])
	,{Upper, [$_, Upper+32]}
	).

Running this, we get some nice failures:

 proper_test_ (prop_morph)...!
Failed: After 1 test(s).
{<<222,240,240,198,253,220,75,212,233,76,248,110,77,83,229,99,195,88,216,250,246,67,227,237,103,240,217,253,220,221>>,<<254,240,240,95,230,253,95,252,95,107,95,244,233,95,108,248,110,95,109,95,115,229,99,95,227,95,120,95,248,250,246,95,99,227,237,103,240,95,249,253,95,252,95,253>>}
failed to morph 'ÞððÆýÜKÔéLønMSåcÃXØúöCãígðÙýÜÝ' to 'þðð_æý_ü_k_ôé_løn_m_såc_ã_x_øúö_cãígð_ùý_ü_ý'

Shrinking ..............................................................(62 time(s))
{<<192,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65>>,<<224,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97>>}
failed to morph 'ÀAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' to 'à_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a'

You can also see that, because we wanted 30-character strings, PropEr generates a shrunk version that is still 30 characters long. Let's inform PropEr that we want to constrain the length of the strings but be a little more flexible so PropEr can generate better failing test cases:

prop_morph() ->
    ?FORALL({Camel, Snake}
	   ,resize(20, camel_and_snake())
	   ,camel_and_snake()
	   ,?WHENFAIL(io:format('user', "~nfailed to morph '~s' to '~s'~n", [Camel, Snake])
		     ,Snake =:= camel_to_snake(Camel)
		     )
	   ).

camel_and_snake() ->
    ?SIZED(Length, camel_and_snake(Length)).

You can read more about resize/2 and the ?SIZED macro but basically they let PropEr know to constrain the size a bit but with more flexibility than a static length.

Running the tests now:

proper_test_ (prop_morph)...!
Failed: After 1 test(s).
{<<220>>,<<252>>}

failed to morph 'Ü' to 'ü'

Shrinking ..(2 time(s))
{<<192>>,<<224>>}

failed to morph 'À' to 'à'

Much easier!

Let's adjust our implementation to account for these upper/lower bounds:

is_upper_char(Char) ->
    (Char >= $A andalso Char =< $Z)
	orelse (16#C0 =< Char andalso Char =< 16#D6)
	orelse (16#D8 =< Char andalso Char =< 16#DE).

to_lower_char(Char) when is_integer(Char), $A =< Char, Char =< $Z -> Char + 32;
to_lower_char(Char) when is_integer(Char), 16#C0 =< Char, Char =< 16#D6 ->
    Char + 32;
to_lower_char(Char) when is_integer(Char), 16#D8 =< Char, Char =< 16#DE ->
    Char + 32;
to_lower_char(Char) -> Char.

And the tests now pass nicely:

 proper_test_ (prop_morph) .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
OK: Passed 500 test(s).

Honing the skill

Thinking in properties is not necessarily intuitive at first. There are many choices for what properties to choose to test, so pick and choose which apply to the particulars of the code you're testing. Another helpful thing is to keep the properties as simple as possible at first. As you build confidence in the generators and property tests, you can layer on more complex properties to ensure the tests are encapsulating the properties of your code.

As you progress down the road of property testing, you will hopefully find that you force yourself to think more deeply about your code and hopefully head off issues before the tests locate them. A great exercise is to pair with a teammate, have one write the implementation and one write the property tests, and compete to see if the tester can find bugs in the implementation.

Running Kazoo VMs with KVM/QEMU

James Aimonetti — Fri, 24 Aug 2018 17:37:59 GMT

I've been running Kazoo in CentOS VMs to replicate production setups to solve support tickets. But getting my laptop talking to the VMs running on my dev server was non-obvious to me. I've finally got it working for an all-in-one Kazoo server and learned a little bit too.

The setup

I have three systems to connect:

Computer	Network IP
My laptop (or any computer on my LAN really)	192.168.1.5
My dev server (beefy blade)	192.168.1.10
My Kazoo VM (using libvirt)	192.168.122.21

So the dev server running libvirt defines a default network of 192.168.122.0/24 and assigns IP addresses to the VMs from that pool.

By default, the dev server (the host) can talk to the VMs on the subnet and the VMs can talk to each other but computers on my LAN cannot talk to the VM subnet.

What to do?

Setup the laptop

The dev server knows how to route to the VMs so we need the laptop to send packets destined for the VM to the dev server. This is actually straight-forward to accomplish by adding a route to the kernel routing table:

First, check out the routing table:

sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100

Now add the new route for the VM subnet IPs to route to the dev server:

sudo ip route add 192.168.122.0/24 via 192.168.1.10 dev eth0

Verify the routes:

sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100
192.168.122.0/24 via 192.168.1.10 dev eth0

Setup the dev server

The main issue on the dev server was the iptables wasn't setup to accept NEW connections to the subnet.

iptables -L FORWARD
...
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED

Just RELATED and ESTABLISHED.

I needed to add NEW to that list:

sudo iptables -I FORWARD -m state -d 192.168.122.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT

You may also need to modify the POSTROUTING to masquerade:

iptables -t nat -A POSTROUTING -j MASQUERADE -o eth0

Setup the VM

On the VM side all I really did was turn of firewalld:

systemctl stop firewalld
systemctl disable firewalld

No plans to open this to the greater Internet so for now this is acceptable to me. :)

Testing

Now that the three servers should be communicating properly, let's take a look. On each server, start a tcpdump:

sudo tcpdump -vv port 8000
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

In this case, look for all traffic on port 8000 (sending or receiving) as that is Kazoo's default API port.

Now, from the laptop, query the base API URL:

laptop$ curl -v http://192.168.122.21:8000

In the various TCP dumps you should see something like:

# Laptop
11:06:24.242471 IP (tos 0x0, ttl 64, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
    laptop.49560 > VM.8000: Flags [S], cksum 0xfcf3 (incorrect -> 0x04e3), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
11:06:24.243341 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    VM.8000 > laptop.49560: Flags [S.], cksum 0x73bf (correct), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
11:06:24.243385 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x12c6), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
11:06:24.243452 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > VM.8000: Flags [P.], cksum 0xfd3e (incorrect -> 0xe0bf), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
11:06:24.244169 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    VM.8000 > laptop.49560: Flags [.], cksum 0x1274 (correct), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
11:06:24.445887 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    VM.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
11:06:24.445911 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f9), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
11:06:24.446377 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [F.], cksum 0xfceb (incorrect -> 0x05f7), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
11:06:24.447064 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    VM.8000 > laptop.49560: Flags [F.], cksum 0x0622 (correct), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
11:06:24.447090 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f4), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

# Dev machine (host)
    laptop.49560 > 192.168.122.21.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.240212 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.122.21.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.240672 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.240702 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > 192.168.122.21.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.241046 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.21.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.442439 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    192.168.122.21.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.443185 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.443660 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.443935 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.21.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.444356 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

# VM
18:06:24.243655 IP (tos 0x0, ttl 63, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
    laptop.49560 > vm.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.243713 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    vm.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.244510 IP (tos 0x0, ttl 63, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.244543 IP (tos 0x0, ttl 63, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > vm.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.244555 IP (tos 0x0, ttl 64, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    vm.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.445989 IP (tos 0x0, ttl 64, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    vm.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.446930 IP (tos 0x0, ttl 63, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.447336 IP (tos 0x0, ttl 63, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.447502 IP (tos 0x0, ttl 64, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    vm.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.448032 IP (tos 0x0, ttl 63, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

Continuing

Instead of having to edit the routing table on every device on the network to forward packets to the dev machine, I setup my router to do it for me. Consult your router's manual for how to add static routes, then add something along the lines of:

Network/HostIP	192.168.122.0	Subnet
Netmask	255.255.255.0
Gateway	192.168.1.10	Dev Server IP
Metric	0
Interface	LAN

Naming may vary but hopefully it is clear enough to get you started.

Next Steps

Setup a cluster of Kazoo VMs with multiple zones and see how it goes
Orchestrate it all with…?

Spellcheck projects using aspell and Makefiles

James Aimonetti — Wed, 25 Jul 2018 15:24:19 GMT

Spellcheck using Aspell and Makefiles

With an international community surrounding Kazoo it is important to provide tools that help everyone stay on the same page with the development cycle.

One of the challenges, as a native speaker of English, is how easily our brains 'fix' misspelt words without us realizing it. Often enough, PRs are accepted where a key into a JSON structure is misspelled and goes unnoticed, time passes until someone realizes the error, and now JSON documents using that key must be supported to prevent previously working data from failing to work now (ask about Kazoo's history with 'wednesday' in the temporal routes callflow action!).

Fortunately there's a nice command line tool, GNU Aspell, that provides spell checking capabilities and custom dictionaries. Aspell does a great job suggesting alternatives when finding a potentially misspelt word, making it easy to find the correct word (or add a word to a personal dictionary).

Personal dictionary

A personal dictionary is helpful to maintain a word list separate and apart from the "default" English dictionary Aspell provides, especially concerning acronyms and jargon specific to the project. When you spellcheck files and find words that are correct but unknown to Aspell, you can add them so Aspell knows not to highlight them again. These words are added to the personal dictionary. Being a plaintext file, the personal dictionary is also easily included in source control so you can distribute it with your project.

Personal replacements

The replacement file can contain automatic replacements; when Aspell encounters a word in the replacement file, it will replace it automatically with the replacement word. Really handy for oft-misspelt words.

Installation

Aspell should be available through most GNU/Linux distros.

Makefile setup

You can easily add a make target to spellcheck a project. Create a splchk.mk file and add:

.PHONY = splchk

DICT = .aspell.en.pws
REPL = .aspell.en.prepl

$(DICT):
	@$(file >$(DICT),personal_ws-1.1 en 0 utf-8)

$(REPL):
	@$(file >$(REPL),personal_repl-1.1 en 0 utf-8)

splchk: $(DICT) $(REPL) $(addsuffix .chk,$(basename $(shell find . -name "*.md" )))

%.chk: %.md
	aspell --home-dir=. --personal=$(DICT) --repl=$(REPL) --lang=en -x check $<

Replace the $(shell find . -name "*.md") with whatever you want to use to find files to spellcheck.

When you run make splchk you will get the opportunity to check the spelling for each file found (if the file doesn't have any issues you won't deal with it). If you find a word that is correct but Aspell highlights it for repair, you can 'a'dd it which will include it in your personal dictionary.