AstRecipes

Command-line apps with Clojure and GraalVM: 300x better start-up times

Fri, 20 Jul 2018 00:00:00 +0200

I am a fond Clojure user, and love to use it for both server-side projects and small command-line tools. I love its expressivity and concision, and I like the fact that I can have a small “script” and hack on it. The problem I have is that while this is fine for interactive use (I don’t mind waiting a second to see results) or for server side projects (where initialization happens just once), it is not a sweet spot for a command-line tool that I want to deploy to automate something.

In this case, the cost of firing up the JVM, Clojure and the script itself, vastly outnumbers the actual processing happening, that is maybe scanning some files; so I see CPU spikes when my cron jobs run those tasks. Nothing bad, but it’s quite useless to load a complex environment to process a 100k text file and then throw everything away.

Enter GraalVM

GraalVM is an extension of the Java virtual machine to support more languages and execution modes. The Graal project includes a new high performance Java compiler which can be used in a just-in-time configuration on the HotSpot VM, or in an ahead-of-time configuration on the Substrate VM.

Substrate VM is a framework that allows ahead-of-time compilation of Java applications under closed-world assumption into executable images or shared objects.

This means that we can create a Clojure command-line app, compile it as :aot :all into a set of Java class files, and then compile those into a single, standalone binary file.

As all initialization is run before compiling into a binary file, it is already present in the generated image and therefore does not require running again! for Clojure, with its complex initialization phase, this is a major win.

Limits of GraalVM / Substrate VM

Substrate VM does not support all features of Java to keep the implementation small and concise, and also to allow aggressive ahead-of-time optimizations (though it does some pretty impresive things, e.g. most of Java reflection).

The biggest issue that we will face is that dynamic class loading is not allowed. We may not care much about that if we have a closed project where all classes are defined at compile time, but as Clojure itself uses it in clojure.lang.RT.resourceAsStream, at the moment we have to tell Graal not to abort at compile time but to defer such exceptions to runtime. Not very elegant, but it works.

This also means that not all valid Java libraries will work out-of-the box in GraalVM - see e.g. Akka and Graal’s native image tool.

The second limit of GraalVM is that in order to compile an even moderately simple executable, it will require a lot of RAM - 6G to 8G not being uncommon from my own experiments.

Our first GraalVM Clojure application

To get started, we want to implement the simple “toycalc” app that ships with CLI-matic - a handy library that lets you define declaratively the CLI parameters your application accepts, and takes care of generating online help, validating paramaters, casting values and all of those nitty-gritty tasks you really do not want to handle when writing a command-line app.

The Clojure side

First, we create a new Leiningen project:

lein new testgraal

Its main class will be testgraal.core.

So we edit its project.clj to include our dependencies and set everything up for AOT compilation:

(defproject testgraal "0.1"
  :description "My first test with GraalVM"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.9.0"]
                 [cli-matic "0.1.14"]]
  :main testgraal.core
  :aot  :all)

We then edit src/testgraal/core.clj; copy and paste the source from https://github.com/l3nz/cli-matic/blob/master/examples/toycalc.clj and modify the namespace declaration so that this namespace is compiled AOT.

(ns testgraal.core
  (:require [cli-matic.core :refer [run-cmd]])
  (:gen-class))

(defn add_numbers
  "Sums A and B together, and prints it in base `base`"
  [{:keys [a1 a2 base]}]
  (println
   (Integer/toString (+ a1 a2) base)))

(defn subtract_numbers
  "Subtracts B from A, and prints it in base `base` "
  [{:keys [pa pb base]}]
  (println
   (Integer/toString (- pa pb) base)))

(def CONFIGURATION
  {:app         {:command     "toycalc"
                 :description "A command-line toy calculator"
                 :version     "0.1"}
   :global-opts [{:option  "base"
                  :as      "The number base for output"
                  :type    :int
                  :default 10}]
   :commands    [{:command     "add" :short "a"
                  :description ["Adds two numbers together"
                                ""
                                "Looks great, doesn't it?"]
    :opts        [{:option "a1" :short "a" :env "AA" :as "First addendum" :type :int :default 0}
                  {:option "a2" :short "b" :as "Second addendum" :type :int :default 0}]
                  :runs        add_numbers}
                 {:command     "sub"  :short "s"
                  :description "Subtracts parameter B from A"
                  :opts        [{:option "pa" :short "a" :as "Parameter A" :type :int :default 0}
                                {:option "pb" :short "b" :as "Parameter B" :type :int :default 0}]
                  :runs        subtract_numbers}]})

(defn -main
  "This is our entry point.
  Just pass parameters and configuration.
  Commands (functions) will be invoked as appropriate."
  [& args]
  (run-cmd args CONFIGURATION))                                

At this point we can have Leiningen generate everything we need:

lein compile && lein uberjar

And we get a single, standalone JAR file under target:

$ ls -l target/
total 5480
drwxr-xr-x 7 ll staff     224 Jul 19 21:55 classes
drwxr-xr-x 3 ll staff      96 Jul 19 21:55 stale
-rw-r--r-- 1 ll staff 4698514 Jul 19 21:55 testgraal-0.1-standalone.jar
-rw-r--r-- 1 ll staff  298095 Jul 19 21:55 testgraal-0.1.jar

And we can test it:

$ java -jar target/testgraal-0.1-standalone.jar --help
NAME:
 toycalc - A command-line toy calculator

Creating a standalone executable on Linux

Now for the fun part: create a CentOS 7 Linux VM with at least 4-6G of RAM; log in and download the latest version of GraalVM Community Edition from GitHub:

wget https://github.com/oracle/graal/releases/download/vm-1.0.0-rc4/graalvm-ce-1.0.0-rc4-linux-amd64.tar.gz
tar zxvf graalvm-ce-1.0.0-rc4-linux-amd64.tar.gz

There is no actual installation to do - for the moment we will just run files within the uncompresed archive.

We will also need zlib for compiling:

yum install zlib-devel

Believe it or not, that’s all we have to do.

Now we copy our testgraal-0.1-standalone.jar uberjar to our work directory, and run the native-image executable of GraalVM:

# ./graalvm-ce-1.0.0-rc4/bin/native-image \
                             -H:+ReportUnsupportedElementsAtRuntime \
                             -J-Xmx3G -J-Xms3G --no-server \
                             -jar testgraal-0.1-standalone.jar
   classlist:   1,397.10 ms
       (cap):     673.76 ms
       setup:   1,087.60 ms
   (objects):  14,990.15 ms
  (features):   4,894.45 ms
    analysis:  78,338.83 ms
    universe:   7,191.33 ms
     (parse):   5,807.91 ms
    (inline):   4,237.25 ms
   (compile):  25,749.35 ms
     compile:  37,432.01 ms
       image:   3,398.08 ms
       write:     662.61 ms
     [total]: 129,597.44 ms

Please note that I specify:

-H:+ReportUnsupportedElementsAtRuntime so Graal won’t abort for dynamic class loading;
-J-Xmx3G -J-Xms3G so we can specify the Java heap (not really needed here, but you may need this option to compile larger projects);
--no-server so it runs as one single process without spawning a compiler daemon.

And after a couple of minutes and a lot of CPU under the bridge….

# ls -l
total 265392
drwxr-xr-x  8 root root      4096 Jul 19 21:38 graalvm-ce-1.0.0-rc4
-rw-r--r--  1 root root 236474318 Jul 17 18:02 graalvm-ce-1.0.0-rc4-linux-amd64.tar.gz
-rwxr-xr-x  1 root root  25873378 Jul 19 22:12 testgraal-0.1-standalone
-rw-r--r--  1 root root   4698514 Jul 19 21:55 testgraal-0.1-standalone.jar

There it is! the testgraal-0.1-standalone executable, ~25M that we can compress to ~7M with gzip.

Results

If a normal execution of out testgraal takes a couple of seconds on our VM, using Graal’s own Java VM…..

# time ./graalvm-ce-1.0.0-rc4/bin/java -jar testgraal-0.1-standalone.jar -?
NAME:
 toycalc - A command-line toy calculator

USAGE:
 toycalc [global-options] command [command options] [arguments...]

VERSION:
 0.0.1

COMMANDS:
   add, a               Adds two numbers together
   sub, s               Subtracts parameter B from A

GLOBAL OPTIONS:
       --base N  10  The number base for output
   -?, --help


real    0m1.563s
user    0m2.755s
sys     0m0.156s

If we run the compiled image we get the very same result….

# time ./testgraal-0.1-standalone -?
NAME:
 toycalc - A command-line toy calculator

USAGE:
 toycalc [global-options] command [command options] [arguments...]

VERSION:
 0.0.1

COMMANDS:
   add, a               Adds two numbers together
   sub, s               Subtracts parameter B from A

GLOBAL OPTIONS:
       --base N  10  The number base for output
   -?, --help


real    0m0.005s
user    0m0.003s
sys     0m0.002s

Same result, but 300x faster in real time, and 900x faster in term of user time. Not half bad, I’d say!

So you have no excuses not to build your CLI tools in Clojure now.

Happy hacking!

Learning Clojure: Transducers how-to

Thu, 24 Nov 2016 00:00:00 +0100

Clojure 1.8 introduced transducers to the world. They are quite an interesting idea that has many possible uses; but as they are very flexible and have a scary name, it is sometimes hard to get started with them.

So - what are transducers?

Transducers are funny “objects” that receive values, one by one, and for each value “in” can return zero or more values “out”. They decouple the logic of computing new values from the logic that iterates through a set of them and feeds the transducers - so they don’t really care if it’s a lazy sequence, a vector, a channel or whatever you want.

And why should I care?

This means transducers can…..

work like a “map”
work like a “filter”
work like a fancy “reducer”
filter-and-aggregate elements
be composed together without a need for intermediate sequences
work on anything, be it a sequence or a core.async channel

Writing your first transducer

Transducers have a fixed form: and this is what you need to copy and paste to get started.

(defn template-transducer [xf]
  (fn 
    	;; SET-UP
    	([] 	         (xf))
      	;; PROCESS
    	([result input]  (xf result input))))
      	;; TEAR-DOWN
    	([result]        (xf result))

This is the basic form:

A function of an external function “xf” - you don’t know and do not care about it. This is what makes them composable. This function allows composition - you call it to “move forward” with processing.
A zero-arity call: starting a processing chain
A two-arity call: receives a current result and a value to add to it. Returns a call to “xf” passing the current result and (usually) some transformation on the input. If you don’t want to return any new value, return “result” as it is.
A one-arity call: transducer is terminating. Return an application of “xf” to the final result, or whatever the “result” should be.

As you can see this is very simple, but you’re left wondering. But practice makes perfect, so let’s start.

So, say we want to write a transducer that retains only odd numbers in a sequence, and returns them incremented. Something like:

(->> (range 10)
     (filter odd?)
     (map inc))

; '(2 4 6 8 10)

So here is what we do:

(defn plain-filter-odd [xf]
  (fn 
    	;; START
    	([]          (xf))
      	;; STOP
    	([result]    (xf result))
      	;; PROCESS
    	([result input] 
         (cond 
           (odd? input) (xf result (inc input))      ;; apply xf to result adding 1 to input
           :ELSE        result )                     ;; do nothing - return result
           )))


(into [] plain-filter-odd (range 10))
; [2 4 6 8 10]

In this case we do not need the set-up and tear-down parts, so we are basically write some logic only in the two arity version. Look how we return “result” as-is if we do not want to change it, while we “return” values by calling xf with a (new) intermediate value.

Using transducers

There are a few simple ways to “run” transducers:

into: as the vanilla version does, appends to an existing collection the transducer applied to a collection
transduce: applies transducer to a collection, and then using a “reducing” function to create our final result
sequence: returns a new sequence by applying the transducer to a sequence. This sequence is lazy, so if it’s not needed, it’s not computed either.
educe: captures the transduction process into a function.

(into [] plain-filter-odd (range 10))
; [2 4 6 8 10]

(transduce plain-filter-odd conj (range 10))
; [2 4 6 8 10]

(sequence plain-filter-odd (range 10))
; '(2 4 6 8 10)

Note that as you have no idea of how many items of the source sequence need to be consumed to produce your final sequence, there is no 1:1 equivalence.

Duplicating values

As we said above, you can return 0 or more values from a transducer:

To return no new values, just return “result” as it is
To return one value, return “(xf result value)”
To return multiple values, just call “(xf result value1)”, then “(xf result value2)”, and so on.

To show how thos works, this transducer duplicates odd values and removes even ones:

(defn duplicates-odd-vals [xf]
  (fn 
    	([] (xf))
    	([result] (xf result))
    	([result input] 
         (cond 
           (odd? input) 
           	(-> result               
              (xf input)             ; first odd value
              (xf input))            ; second odd value
           :ELSE        result ))))  ; do nothing

(transduce duplicates-odd-vals conj (range 11))
; [1 1 3 3 5 5 7 7 9 9]

Stateful transducers

Having the ability to control when a value is emitted is especially important for trasducers that must keep track of a current (inner) state.

Our transducer will return a sequence of odd numbers intertwined with their unique sequence number for each item in our sequence, starting from 100 onwards (we count all numbers for a change).

(defn stateful-transducer [xf]
  (let [state (atom 100)]
	 (fn 
    	([] (xf))
    	([result] (xf result))
    	([result input] 
         (swap! state inc)
         (cond 
           (odd? input) 
	           	(-> result
	              (xf input)
	              (xf (str "S:" @state)))
	           :ELSE        result )                     ;; do nothing - return result
         ))))

(transduce stateful-transducer conj (range 10))
;[1 "S:102" 3 "S:104" 5 "S:106" 7 "S:108" 9 "S:110"]

To do this we use an atom (or possibly a transient) that we use as a keeper of current state. See how this atom is unique per function invocation, so it’s local mutable state (and how this starts to look a bit like an “object” encapsulating state).

Of course, a stateful transducer depends on its internal state and is not inherently parallelizable if run in parallel, so beware!

Transducers as reducers

You can easily turn a stateful transducer into a reducer, just like this:

(defn reducer-transducer [xf]
  (let [state (atom 1)]
	 (fn 
    	([] (xf))
      	([result] (xf @state))
      	([result input] 
         (swap! state #(* input %))
          result ) )))

(transduce reducer-transducer conj [2 3 7])
; 42

See how we basically ignore result and use a result of our own making at the end of the eduction.

Parametrized transducers

You can easily create your own parametrized transducers with “partial”, so that you capture all parameters but “xf”:

(defn string-transducer [prefix xf]
  (fn 
    ([] (xf))
    ([result] (xf result))
    ([result input] 
		(xf result (str prefix ":" input)) )))

(transduce (partial string-transducer "x") conj (range 5))
; ["x:0" "x:1" "x:2" "x:3" "x:4"]

Composing transducers

One of the best things abut transducers is that they are composable; you can create a new transducer out of a set of existing ones.

This is more efficient than mapping/filtering over multiple sequences in turn.

(transduce (comp (partial string-transducer "B") 
                 (partial string-transducer "A"))
           conj 
           (range 3))

; ["A:B:0" "A:B:1" "A:B:2"]

See how B was applied first, and then A.

Putting it all together - separating text paragraphs

We want to create a transducer that receives a sequence of lines (maybe over a core.async channel) and separates them into paragraphs.

This is a typical stateful transducer, and something that we cannot do easily with just map and filter.

(defn line-wrapper [xf]
  (let [state (atom [])]
    (fn 
      ([] (xf))
      ([result] 
       	(let [final-result (xf result @state)]     ; (A)
          (xf final-result)))                      ; (C)
      ([result input]
        ;; if empty line, push accumlated lines at once
       	(cond 
          (= input "") 
			  (let [v @state]
              (reset! state [])
              (xf result v))                       ; (B)
        
          :ELSE 
			   (do 
            	(swap! state #(conj % input))
            	result) )))))       
       
(transduce line-wrapper 
           conj 
           ["A" "B" "" "C" "" "" "D" ""])
;[["A" "B"] ["C"] [] ["D"] []]

In this case it is important that if we have open stanzas when the sequence terminates, we reprocess them (see A, and see it’s the very same thing we’re doing in B) and then we generate the final result, that will in turn be sent down the pipeline (C).

This is pretty cool, as a lot of problems can be expressed in a way that is similar to this; and IMHO this alone makes transducers worth learning.

But the very best thing about transducers is…..

The really best thing about transducers is that (often) you don’t need to write them. If you had to write transducers from scratch every time like in the examples above, they would be still be cool, but in 99% of the cases you would prefer the old map/filter/reduce - it might not be as efficient or powerful, but is concise and compact and easy to read.

Truth is that you can have the best of both worlds - you can use your classic friends map&co. and use transducers. This is because many functions in core that used to have a collection as their last parameter now have an arity that produces a transducer by just omoiiting that collection. So if you say:

(map inc)

you get a transducer for free. In Clojure 1.8, it’s actually the other way around - when you run your plainest Miss Apple Pie function:

(map inc some-seq)

what Clojure internally does is:

(sequence (map inc) some-seq)

So you can use transducers everywhere and create highly-efficient and easy to read pipelines by composing some custom transducers and plenty of common Clojure functions, like in:

; Remove empty stanzas
(transduce (comp 
            	line-wrapper
            	(filter (complement empty?)) )
           conj 
           ["A" "B" "" "C" "" "" "D" ""] )
; [["A" "B"] ["C"] ["D"]]

In the example above, for example, we could add a check within our newly created transducer line-wrapper that makes sure that if we have multiple empty lines, then only the first one is counted. Sure, it would be easy. But filtering in the pipeline is easier.

; Remember: order counts! This is likely *not* what you want.
(transduce (comp 
				(filter (complement empty?))
				line-wrapper)
           conj 
           ["A" "B" "" "C" "" "" "D" ""] )
; [["A" "B" "C" "D"]]

Final thoughts

Transducers take a while getting used to; so make sure you open your REPL and experiment a bit.

The following functions have a transducer arity: map cat mapcat filter remove take take-while take-nth drop drop-while replace partition-by partition-all keep keep-indexed map-indexed distinct interpose dedupe random-sample.

By looking at their source, they can be used as a reference as hot to write your own transducers if you get lost.

Play endless random sounds for testing

Sat, 20 Dec 2014 21:05:32 +0100

This Extension will play random sounds from asterisk standard sound set forever. You might have to adjust the path uses. This example works on Debian:

exten => endless,1,Answer()
exten => endless,2,Set(RANDSOUND=${SHELL(ls /var/lib/asterisk/sounds/en | shuf -n 1 | head -n1 | cut -f1 -d"." | tr -d "\n")})
exten => endless,3,NoOp(*${RANDSOUND}*)
exten => endless,4,Playback(${RANDSOUND})
exten => endless,5,Goto(2)
exten => endless,n,Hangup()

Compiling Asterisk13 on Centos65 64-bit

Fri, 12 Sep 2014 13:08:43 +0200

This is a quick tutorial to get started with Asterisk 13 (currently beta) on Centos 6.5 64-bit.

Luckily the installation procedure is very similar to Asterisk 12 and it is very easy to go through.

First we disable selinux and update the system and install binary dependencies - it may take a while.

Disable selinux:

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
reboot

Note that if you copy the commands below that you must either 1) make single commands all on one line, or 2) add back slashes to carry over to a new line (this wiki does not display the back slashes).

yum update
yum install -y screen lsof mlocate wget
yum install -y gcc-c++ make gnutls-devel kernel-devel libxml2-devel ncurses-devel \
  subversion doxygen texinfo curl-devel net-snmp-devel neon-devel \
  uuid-devel libuuid-devel sqlite-devel sqlite \
  git speex-devel gsm-devel

Installing Jansson

wget http://www.digip.org/jansson/releases/jansson-2.5.tar.gz
tar zxvf jansson-2.5.tar.gz
cd jansson-2.5
./configure --libdir=/usr/lib64
make & make install
cd ..

Installing SRTP

wget http://srtp.sourceforge.net/srtp-1.4.2.tgz
tar zxvf srtp-1.4.2.tgz
cd srtp
autoconf
./configure --enable-pic --libdir=/usr/lib64
make && make install
cd ..

Installing PJSIP

git clone https://github.com/asterisk/pjproject pjproject
cd pjproject
./configure --libdir=/usr/lib64 --prefix=/usr --enable-shared \
             --disable-sound --disable-resample --disable-video --disable-opencore-amr \
             --with-external-speex --with-external-srtp --with-external-gsm 
make dep && make && make install
cd ..

Downloading and installing Asterisk 13

wget http://downloads.asterisk.org/pub/telephony/asterisk/releases/asterisk-13.0.0-beta1.tar.gz
tar zxvf asterisk-13.0.0-beta1.tar.gz
cd asterisk-13.0.0-beta1
./configure --libdir=/usr/lib64
make menuselect

Note that as of 2014 November this works with Asterisk 13.0.0 too now that it is out of beta. Just substitute the current URL for the wget and other commands above, for example:

wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-13-current.tar.gz

Under Channels you chack that there is “

chan_pjsip”

make && make install && make samples cd ..

Running Asterisk

asterisk 
asterisk -vvvr

And you get

localhost*CLI> core show version
Asterisk 13.0.0-beta1 built by root @ localhost.localdomain on a x86_64 running Linux on 2014-09-12 08:28:08 UTC

Happy hacking!

..and of course, if you implement call-centers in Asterisk, do not forget to check out QueueMetrics for reporting and WombatDialer for implementing outbound added value services. :)

Getting started with ARI

Mon, 30 Dec 2013 16:07:50 +0100

Setting up the Asterisk REST Interface on an Asterisk 12 system for an introductory test-drive is quite straightforward. The idea behind ARI is that you have a RESTful part where you send commands and a websocket to receive events.

Asterisk configuration

Edit /etc/asterisk/http.conf so that:

[general]
enabled=yes
bindaddr=0.0.0.0
bindport=8088

And then create an ARI user in /etc/asterisk/ari.conf:

[general]
enabled = yes

[aritest]
type = user
read_only = no
password = testme
password_format = plain

This creates a new user called aritest with password testme.

Now have Asterisk reload and test that the configuration was picked up.

localhost*CLI> reload
...
localhost*CLI> ari show users
r/o?  Username
----  --------
No    aritest
localhost*CLI> ari show status
ARI Status:
Enabled: Yes
Output format: compact
Auth realm: Asterisk REST Interface
Allowed Origins:
User count: 1

Testing ARI - the websocket

You can install the wscat tool to test the ARI. It is a part of the EPEL repository that, though not technically a part of CentOS 6, it surely complements it (see https://fedoraproject.org/wiki/EPEL if you do not have it available).

yum install nodejs nodejs-options nodejs-commander
yum install nodejs-ws

Then type:

# wscat --connect 'ws://localhost:8088/ari/events?app=hello&api_key=aritest:testme'
connected (press CTRL+C to quit)

If so far everything is okay, we can now run a test that is a little more meaningful.

First edit your /etc/asterisk/extensions.conf and add the following stanza:

[testari]
exten => 1,1,Noop()
same => n,Stasis(hello,world)
same => n,Hangup()

This just creates a new context that sends calls to the Stasis (ARI) app called “hello” with a parameter of “world”.

We reload Asterisk so that the dialplan is picked up and in a new window we run again the wscat command above.

While wscat is running, go to the Asterisk CLI and type:

localhost*CLI> channel originate Local/1@testari application wait 100

On the websocket window you’ll see:

< {
  "type": "StasisStart",
  "timestamp": "2013-12-30T15:21:45.688+0100",
  "args": [
    "world"
  ],
  "channel": {
    "id": "1388413305.5",
    "name": "Local/1@testari-00000002;2",
    "state": "Ring",
    "caller": {
      "name": "",
      "number": ""
    },
    "connected": {
      "name": "",
      "number": ""
    },
    "accountcode": "",
    "dialplan": {
      "context": "testari",
      "exten": "1",
      "priority": 2
    },
    "creationtime": "2013-12-30T15:21:45.688+0100"
  },
  "application": "hello"
}

< {
  "type": "StasisEnd",
  "timestamp": "2013-12-30T15:22:15.716+0100",
  "channel": {
    "id": "1388413305.5",
    "name": "Local/1@testari-00000002;2",
    "state": "Ring",
    "caller": {
      "name": "",
      "number": ""
    },
    "connected": {
      "name": "",
      "number": ""
    },
    "accountcode": "",
    "dialplan": {
      "context": "testari",
      "exten": "1",
      "priority": 2
    },
    "creationtime": "2013-12-30T15:21:45.688+0100"
  },
  "application": "hello"
}

(Ok I turned on pretty printing in ari.conf in order to make this readable).

Testing ARI - the RESTful part

Running REST commands through the interface is very straightforward:

[root@localhost ~]# curl -v -u aritest:testme -X GET "http://localhost:8088/ari/asterisk/info"
* About to connect() to localhost port 8088 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8088 (#0)
* Server auth using Basic with user 'aritest'
> GET /ari/asterisk/info HTTP/1.1
> Authorization: Basic YXJpdGVzdDp0ZXN0bWU=
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8088
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Asterisk/12.0.0
< Date: Wed, 01 Jan 2014 13:20:15 GMT
< Connection: close
< Cache-Control: no-cache, no-store
< Content-Length: 535
< Content-type: application/json
<
{
  "build": {
    "os": "Linux",
    "kernel": "2.6.32-431.el6.i686",
    "machine": "i686",
    "options": "LOADABLE_MODULES, OPTIONAL_API",
    "date": "2013-12-30 13:25:41 UTC",
    "user": "root"
  },
  "system": {
    "version": "12.0.0",
    "entity_id": "52:54:00:26:9d:07"
  },
  "config": {
    "name": "",
    "default_language": "en",
    "setid": {
      "user": "",
      "group": ""
    }
  },
  "status": {
    "startup_time": "2013-12-30T14:30:39.344+0100",
    "last_reload_time": "2013-12-30T15:21:39.354+0100"
  }
* Closing connection #0

You’ll have to look up each command - so better using a library.

See also:

Getting started with ARI on which this tutorial is based
The ari4java library wraps ARI calls so that it is accessible from Java.

Compiling Asterisk 12 on CentOS 6.5

Mon, 30 Dec 2013 14:45:44 +0100

Compiling Asterisk 12 (with PJSIP support) on a brand-new CentOS 6 system is pretty straightforward. Most of the packages come prebuilt so it’s not very complex to do.

Prerequisites

Let’s check the current version.

[root@localhost ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.32-431.el6.i686 #1 SMP Fri Nov 22 00:26:36 UTC 2013 i686 i686 i386 GNU/Linux

First we update the system so that we have everything needed to compile plus the packages we need.

yum update
yum install gcc-c++ make gnutls-devel kernel-devel libxml2-devel ncurses-devel subversion doxygen texinfo curl-devel net-snmp-devel neon-devel
yum install uuid-devel libuuid-devel sqlite-devel sqlite git speex-devel gsm-devel

Compiling PJSIP

We have everything for PJSIP but the SRTP library.

wget http://srtp.sourceforge.net/srtp-1.4.2.tgz
tar zxvf srtp-1.4.2.tgz
cd srtp
autoconf
./configure
make
make install
cp /usr/local/lib/libsrtp.a /lib
cd ..

Now we are ready for PJSIP itself. Make sure you download the patched version that works with Asterisk.

git clone https://github.com/asterisk/pjproject pjproject
cd pjproject/
./configure --prefix=/usr --enable-shared --disable-sound --disable-resample --disable-video --disable-opencore-amr --with-external-speex --with-external-srtp --with-external-gsm
make dep
make
make install
cd ..

Note: if compiling on a 64-bit CentOS system, remember to add –libdir=/usr/lib64 to have Asterisk find it later (Thanks Jakub!)

Compiling Asterisk

We start by compiling Jansson - it is available in the CentOS repos, but it’s an old version.

wget http://www.digip.org/jansson/releases/jansson-2.5.tar.gz
tar zxvf jansson-2.5.tar.gz
cd jansson-2.5
./configure --prefix=/
make
make install
cd ..

Note: I had to set –prefix=/usr/ to stop Asterisk borking when detecting the Jansson library. (Centos 6.5 (Final) 64-bit)

Now it’s time for Asterisk itself:

wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-12-current.tar.gz
cd asterisk-12.0.0/
./configure
make menuselect

Under Channel Drivers check that chan_pjsip is checked (and disable chan_sip is you really feel brave!). If building on a KVM box, better uncheck BUILD_NATIVE under Compiler Flags (press x to save).

make 
make install
make samples
cd ..

If all went well…

[root@localhost asterisk-12.0.0]# asterisk
[root@localhost asterisk-12.0.0]# asterisk -vvvvr
Asterisk 12.0.0, Copyright (C) 1999 - 2013 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 12.0.0 currently running on localhost (pid = 18101)

See also

Getting started with ARI

WombatDialer

Tue, 20 Nov 2012 13:21:21 +0100

WombatDialer is a next-generation dialer platform for the Asteriskâ„¢ PBX.

It is meant as a way to implement a number of functions that are not easily handled in an Asterisk-based call center, like:

Queue call-backs: allow callers to leave their numbers instead of waiting on a queue, and then being calls them back
Web-based contact-us forms: allow customers to be called back by using a web form
Automated surveys / Reverse IVRs: call a list of numbers and gather information that is entered as DTMF
Number verification services: automatically check the validity of phone numbers
Telecasting of pre-recorded messages: send pre-recorded messages to thousands of recipents quickly and efficiently
Automated appointment reminders: remind and take appointments, with our without human interventions
Automated subscription expiration: automatically call expiring subscriptions, and have clients renew their subscription
Automated quality surveys: run quality/performance interviews and feed them back into the QueueMetrics QA module
Generic progressive dialing: keep your outbound agents busy by having a dialer compose numbers, handle retries and pre-qualify leads
…and many many more!

WombatDialer is a platform, meaning that it lets you develop specific solutions that are custom-tailored for the task at hand. It hides away the complexisties of tracking calls, handling redials, getting data to and from extenal servers, scaling up to multiple Asterisk servers, and more.

WombatDialer was built from the scratch to work well with QueueMetrics, so that you can use QueueMetrics to report on its activities together with your existing inbound/outbound activities; it may work with agents on queues therefore leveraging the way you currently work with QueueMetrics, and can use the QueueMetrics agent and Real-time pages as a control point for its activities.

See also:

Compiling Asterisk 1.8 on CentOS 5.5 64-bit

Fri, 29 Oct 2010 18:23:14 +0200

After the first official release of Asterisk 1.8, I decided to test how it is and compiled it on a clean-slate 64-bit CentOS system. The experience went very smoothly and the configure script was able to pick up all the packages needed to activate various interesting extensions (it was not always so with previous releases).

Our target is compiling Asterisk 1.8.0 with the following features/extensions:

MySQL storage and dialplan
DAHDI drivers (I still call them Zaptel!) for analog lines
Full docs
CURL available for usage and dialplan
HTTP integration
SNMP agents
Calendars (they look yummy!)
Google Talk integration

Setting up the environment

The following packages have to be installed in order to allow compilation of all required modules. Some of these packages may or may not be already installed on your system. I use as a reference a CentOS 5.5 x86_64 with kernel 2.6.18-194.17.4.el5-x86_64.

[root@Centos64]# uname -a
Linux Centos64 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

The following packages should all be present to allow compiling Asterisk:

yum install gcc-c++ make gnutls-devel kernel-devel
yum install libxml2-devel ncurses-devel subversion doxygen texinfo

And these are optional and required for specific modules to build:

yum install curl-devel net-snmp-devel neon-devel

We will compile under /usr/src .

Compiling DAHDI and tools and libpri

Since the days of Zaptel, we now have to install both DAHDI and its tools. Luckily both are packaged together and can be compiled in a single pass.

I immediately relize that the symbolic link used to find the kernel top .configure is broken; in my freshly-installed CentOS system /lib/modules/2.6.18-194.el5/build is a link to a non-existent ../../../usr/src/kernels/2.6.18-194.el5-x86_64. Finding the correct location for the kernel build is easy-peasy anyway; instead of correcting the link, I prefer to override the check using an environment variable:

export KSRC=/usr/src/kernels/2.6.18-194.17.4.el5-x86_64
wget http://downloads.asterisk.org/pub/telephony/dahdi-linux-complete/dahdi-linux-complete-2.4.0+2.4.0.tar.gz
tar zxvf dahdi-linux-complete-2.4.0+2.4.0.tar.gz
cd dahdi-linux-complete-2.4.0+2.4.0
make
make install
cd ..

Internet connectivity is needed to download the firmwares for the various modules to be built - wouldn’t it be better to package them with the main files?

Compiling libpri is - as always - a breeze:

tar zxvf libpri-1.4.12-beta2.tar.gz
cd libpri-1.4.12-beta2
make
make install
cd ..

Compiling optional dependencies

A couple of libraries are not availble in the CentOS repository and need to be installed manually to enjoy the benefits of 1.8.

The first is iksemel, an XML parser library for Jabber applications that is needed for GTalk integration:

wget http://iksemel.googlecode.com/files/iksemel-1.4.tar.gz
tar zxvf iksemel-1.4.tar.gz
cd iksemel-1.4
./configure
make
make install
cd ..

The other one is libical, an iCal library used to activate integration with iCal and WebDAV calendars:

wget http://sourceforge.net/projects/freeassociation/files/libical/libical-0.44/libical-0.44.tar.gz/download
tar zxvf libical-0.44.tar.gz
cd libical-0.44
./configure --enable-shared
make
make install
cd ..

I used version 0.44 because the 0.46 version does not compile out-of-the-box - it misses the template Makefiles from the downloadable archive. By the way, how many projects on the Internet bear the very name name libical? please… On the other side, by adding this library, we get res_calendar_caldav, res_calendar_exchange and res_calendar_icalendar - that’s not too bad.

If you also want to build the PDF manual (but you don’t need to, you will need an optional tool called rubber that compiles Latex into a PDF:

wget http://launchpad.net/rubber/trunk/1.1/+download/rubber-1.1.tar.gz
tar zxvf rubber-1.1.tar.gz
cd rubber-1.1
./configure
make
make install
cd ..

Run target make pdf after building Asterisk.

Compiling Asterisk 1.8

Nothing could be easier than building Asterisk now.

wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-1.8.0.tar.gz
tar zxvf asterisk-1.8.0.tar.gz
cd asterisk-1.8.0
./configure
make menuselect
make

#this is ony for format MP3 - SVN required
contrib/scripts/get_mp3_source.sh
  
make install
make samples
make progdocs

After running menuselect, choose the modules you want to build.

I always build docs so that I can copy them to an intranet webserver sand use them as a local reference in case it’s needed. Asterisk 1.8 comes with an almost 300-page User Manual available under doc/tex/asterisk.pdf.

Starting Asterisk

Start Asterisk by typing:

asterisk

This will start Asterisk in the background.

[root@Centos64]# asterisk -rx "core show version"
Asterisk 1.8.0 built by root @ Centos64 on a x86_64 running Linux on 2010-10-28 16:12:05 UTC

Will display the Asterisk version.

asterisk -r

Will connect you to the Asterisk shell. Good luck!

See also

Exporting queue_log data

Fri, 19 Feb 2010 11:59:03 +0100

This query lets you export the contents of the queue_log table in SQL format for inspection. Instead of exporting the whole table, it exports only the period specified.

/usr/bin/mysqldump \\
  -h localhost \\
  -u queuemetrics -p \\
  --databases queuemetrics \\
  --tables queue_log \\
  --where="time_id between \\
           unix_timestamp('2008-10-23 00:00:00') \\
       and unix_timestamp('2008-10-28 00:00:00')" \\
  > outfile.sql

When you run an export, make sure that you express the time period in GMT time, as that is usually the one used by Asterisk.

If you have to send the file over for inspection, you can greatly reduce its size by compressing it, e.g.

bzip2 outfile.sql

This will create a file called ‘oufile.sql.bz2’.

New server 100210

Wed, 10 Feb 2010 13:35:51 +0100

We are now running on a new web server that is located at Rackspace.

It should be better than the old one, and likely way better!

We took some measures to avoid referer spam, an deleted over 150,000 fake refer entries.