Offering Load

One of the things I like about Perl is it’s enthusiasm for embedding multiple little languages into the main language.  Actually it’s more like they are all woven together.   Erlang shares a bit of that, though it’s more adhoc.  The result is that when you start reading Erlang code you discover all these different things are going on at the same time.

There is the usual code v.s. comments.   There is no multiline comment syntax.  Comments begin with a percent sign and run to the end of the line.  It’s a convention enforced by all the editting modes that one, two, and three percents are implicitly at the level of line, statement, and file respectively.  And file comments tend to be implicitly associated with the following code.

The comments are often written in the stylized documentation language.   And the documentation language has a sublanguage that is wiki like, and some tags that document data types and function signatures.  As far as I know things you say in the documentation about types and function signatures have no effect on the behavior of the program, i.e. like usual your comments can be lies.

In a manner that is reminiscent of C preprocessor’s use of hash, as in #define or #ifdef, the Erlang compiler uses a prefix dash on tags to mark up the code.  Unlike the comments this markup does effect behavior.  In addition to the kinds of things C does with it’s preprocessor (simple macros, conditional compilation, include files), the Erlang compiler uses these to define records, do some limited type declarations, imports and exports, and assert metadata about the file being compiled (i.e. version, author, module, etc).  And finally this is were the behavior mechanism is.

Together this makes simple Erlang programs much more complex to read.  The simplest program will usually have one or two statements in each of 3 notations: doc, Erlang, compiler annotations.  And both the doc and compiler annotations have their own embedded micro languages for special functions.

I don’t know what’s going on in that example with the “%% userdevguid-begin rt_simple:types”?

This post is an attempt to tease out all the state of a Erlang Process.

The overarching design pattern  used to organize Erlang Processes is known as the supervision tree.  The interior nodes in that tree are supervisors while the leaves are workers.  Supervisors concern themselves with reliability and scaling, while the workers consume most of the cycles.  A nice feature of this design is that workers can often be written so they just commit suicide when things go wrong.  But, supervision trees deserves their own discussion, what’s described here includes the raw materials from which that scheme is built out.

Each process includes:

• a process id
• the node the process, along with others, resides within
• a status which is one of runable, running, waiting, suspended
• a bundle of process information which can be read for introspective purposes
• it’s call stack
• an incoming message Q
• an optional name, i.e. an atom; such processes are said to be registered
• an optional globally registered name.
• a rarely used associative store known as a dictionary
• symmetric links to zero or more other processes which configure how to tear things down when a processes exits
• zero or more monitors, when a monitored process exits a message is send to any process monitoring it.
• zero or more timers
• a rarely used flag, trap_exit, that converts exit signals into a message to the process.
• a rarely used flag, error_handler, which provides a hook to catch and resolve errors prior to the stack unwinding.  By default this is used to load code.
• a min_heap_size (another flag)
• the integer save_calls flag provides for capturing a debug trace
• a priority, one of low, normal, high (another flag)

A simple erlang node has 24 processes, 15 of which are named; but no globally registered processes.

The inter-process links are bidirectional and my cross between Erlang nodes.   If a process exits these links work to assure that all the reachable processes also exit, but that can be frustrated by setting the trap_exit flag.  A simple spawn creates a process with no links, but such processes are rare; in practice nearly all processes will have one or more links.

Monitors are directional and only trigger a message when the monitored process exits.  They also can cross between Erlang nodes.

At first blush the dictionary looks to be Erlang’s version of thread local storage.  But since most Erlang programs strive to written in a functional style the dictionary is rarely used.  Instead, the typical long lived Erlang process will have it’s persistent state kept in a data structure and at each heart beat of the process that data structure will be replaced with a incrementally modified fresh version which is then passed on to the next cyle.  For example:

server(State) ->
    receive
	{From, Info} ->
            NextState = process_info(From, Info, State),
            server(NextState);
	{From, Ref, Query} ->
             {Reply, NextState} = process_query(From, Query,State),
             From ! {Ref, Reply},
             server(NextState)
    end.

In that example the server’s state to the variable State. The state for the following heartbeat is bound to NextState. The server function is crafted to assure that it loops to the next heartbeat by virtue of a tail recursive call.

Patterns like this are common enough that Erlang’s support libraries have abstractions to for them, see for example the generic server behavior.  These abstractions along with the supervisor tree design pattern mean that Erlang programmers rarely use links, monitors, spawn, etc. directly, except when debugging.

A real world Erlang system will have far more moving parts than most programmers are used to.

• Multiple machines
• Multiple Operation system processes on those Machines
  • One epmd process on each machine
  • One or more erlang nodes
    • Most nodes are running the erlang bytecode interpreter known as beam.
      • An optional monitoring process (a heartbeat) for each beam node.
    • Nodes in other languages which implement the inter-node protocol.
      • A common example is inet_gethost used for DNS lookups
• Inside of each beam node are many erlang processes, often thousands.

For example, on one of my machines I’m running two Erlang systems: ejabberd, and rabbitmq.  This machine happens to be running FreeBSD for it’s operating system.   Two FreeBSD users (known as ejabberd and rabbitmq) run these two systems.  So if we look at their processes we see this:

[bhyde@elm ~]$ ps -axugwwU rabbitmq,ejabberd | sort
USER       PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
ejabberd   919  0.0  0.1  1448   880  ??  S    Sun05PM   0:01.67 /usr/local/lib/erlang/erts-5.6.5/bin/epmd -daemon
ejabberd   920  0.0  2.8 37984 28796  ??  S    Sun05PM   0:45.78 [beam]
ejabberd  1165  0.0  0.1  1428   680  ??  Is   Sun05PM   0:00.04 inet_gethost 4
ejabberd  1166  0.0  0.1  1532   864  ??  I    Sun05PM   0:00.03 inet_gethost 4
ejabberd  1167  0.0  0.1  1532   868  ??  I    Sun05PM   0:00.03 inet_gethost 4
ejabberd  3082  0.0  0.1  1532   888  ??  I    Sun06PM   0:00.05 inet_gethost 4
ejabberd 49972  0.0  0.1  1532   912  ??  I     7:22AM   0:00.00 inet_gethost 4
rabbitmq   969  0.0  1.2 17600 12112  ??  Ss   Sun05PM   0:30.35 [beam]
rabbitmq   982  0.0  0.1  1432   680  ??  Is   Sun05PM   0:00.01 inet_gethost 4
rabbitmq   983  0.0  0.1  1468   848  ??  I    Sun05PM   0:00.00 inet_gethost 4
[bhyde@elm ~]$

There is only a single epmd which everybody uses to rondevous around; presumably it’s user just happened to be started first.

My guess for why there are so many inet_gethost processes? Well I’m assuming they never die; and when ejabberd and rabbitmq started up they had a few DNS lookups they needed to do.

Each of those beam Erlang nodes have about a dozen Erlang processes inside of them.

[bhyde@elm ~]$ echo 'i().' | erl -sname foo1 -remsh rabbitmq | grep -c '^<'
28
[bhyde@elm ~]$ echo 'i().' | erl -sname foo1 -remsh ejabberd | grep -c '^<'
28
[bhyde@elm ~]$ echo 'i().' | erl -sname foo1 -remsh ejabberd | tail
<0.30.0>              kernel_config:init/1                   233       49    0
                      gen_server:loop/6                        9
<0.31.0>              supervisor:kernel/1                    233       65    0
kernel_safe_sup       gen_server:loop/6                        9
<0.35.0>              erlang:apply/2                        1597    10318    0
                      rpc:local_call/3                        49
Total                                                      31487   279423    0
                                                             255
ok
(foo1@elm)2> *** Terminating erlang (foo1@elm)
[bhyde@elm ~]$

By the way, in that last transcript I invoked erl erl -sname foo1 -remsh … a few times. That fired up yet another erlang beam process each time. In this case those ran under my Freebsd user account (bhyde). They chatted with empd to using the node name given after the -remsh argument, ejabberd for example. At that point they created a remote shell into that process. The ‘i().’ command asked for a list of the erlang processes in the node.