RDTN Simulation Specs

For my research activities I regularly need to run simulations of various networking scenarios. I use RDTN for this which I extended to include a simulation module where all nodes in a virtual network are separate instances of the RDTN daemon running in one interpreter. The connections are simulated by copying the data (bundles in DTN slang) between the instances.

The simulator itself runs quite fine (except that it could use some more optimizations for consuming less memory). But I had the problem that I need to run simulations with many variations of various parameters, e.g. different routing algorithms, different buffer sizes, different network environments, different what-have-yous, for scientific comparison. The individual simulation runs can take some time so starting them manually one after the other would be tedious, which clearly calls for scripting the stuff.

Simulation Specs

Of course, a run script is trivial. The issue I’m getting at here, is encoding the variations of simulation parameters in a way that’s easy to write, read, and that allow the run script to start one variation after the other.

The solution, I came up with, was to create specs for the simulations. The specs contain ruby code that use a specific API (a DSL if you will). There is one spec for one set of experiments. The variations are listed in the spec, but the run script takes care of computing all combinations for the subsequent runs.

Okay, this explanation is abstract in an incomprehensible way. Let’s clarify with an example: Say we want to test the behavior of a network when we send bundles of different sizes and with different rates. More specifically, let’s say we want to see the effect of 1K payloads and 1000K payloads, and also the impact of sending one bundle per hour and ten bundles per hour. To be really thorough, we want to simulate all possible combinations of these parameters: 1K payloads at 1 bundle per hour, 1K at 10 bundles per hour, 1000K at 1 bundle per hour, and 1000K at 10 bundle per hour. But we don’t want to write down these combinations ourselves, because that’s tedious work (especially as the number of combination grows quickly for actual experiments) and that’s what computers are for. Our simulation spec looks like this:

class Example < Sim::Specification

  def execute(sim)
    g = Sim::Graph.new
    g.edge 1 => 2
    sim.events = g.events
    sim.nodes.router :epidemic

    data = 'a' * variants(:size, 1024, 1024000)

    sim.at(variants(:sendRate, 3600, 360)) do |time|
      sim.node(1).sendDataTo(data, 'dtn://kasuari2/')
      # run for one day
      time < 3600*24
    end
  end
end

Lines 1 - 3: All specs contain a class that inherits from Sim::Specification and implements the method execute.

Lines 4 - 6: For the sake of this explanation we use a dead simple network with two permanently connected nodes.

Line 7: We let the nodes use epidemic routing.

Line 9: Here it starts to get interesting, as we declare our first variable parameter: the payload size. By calling Specification#variants, we define a list of variants with the identifier :size. When we run the simulations, this function will either return the first or the second value varying between the runs. We use this size value to create a string of that length to define the actual payload data.

Line 11: Here, we define a time-dependent action that is executed either after one hour (3600 seconds) or six minutes (360 seconds) depending on which variant is currently being executed.

Line 12: We let node 1 send a bundle with the payload data we assigned in line 9 to node 2 (all nodes are called kasuari(n) for historical reasons).

Line 13: When the block executed by the at method returns true, it will be called again after the same amount of time. So in this line, we define that we want the simulator to stop after one day.

Running the Simulations

So how do we run this thing? The code above must be saved under simulations/specs in your RDTN directory in a ruby file named like class with only lowercase characters, i.e. simulations/specs/example.rb. The run script is called sim/run and must be called with the class name of the spec that should be run. Without any other parameters, the run script executes only the first variant. To get all of our variants executed, we need the -v switch. So we run our simulations by calling

$ ruby sim/run -v Example

But how do we see what happened? The run script doesn’t write any output to the standard output, but it writes statistics into a directory under simulations/results. We can analyze these statistics with tools/analyze which gets the class name of the spec as a command line argument. By default the analyze tool only displays the most recent results. To see the results from all variants, we need the -a switch

$ ruby tools/analyze -a Example

Behind the Scenes

The results of this example simulation experiment are not really interesting, so let’s move on to how RDTN actually runs the combinations. To compute the combinations of variants, we need to go through the spec before starting the actual simulations in dry run. The dry run calls the execute method passing in an empty Sim::Core object. The only thing that really different between a dry run and the actual execution, is the behavior of the variants method.

In dry run mode, variants takes the parameter list which contains all the variants and stores it in a hash that serves as a template for generating the combinations. The variants are indexed by the id that is passed as the first parameter to variants (:size and :sendRate in our example). The template hash of our example looks like this:

{:size => [1024, 1024000], :sendRate => [3600, 360]}

After the dry run, we compute a list of hashes in which each variant id is mapped to exactly one variant from the template hash that we populated in the dry run. Together, all the hashes in the list cover all combinations of variants. In our example this list contains the following entries:

{:size => 1024,    :sendRate => 3600}
{:size => 1024,    :sendRate => 360}
{:size => 1024000, :sendRate => 3600}
{:size => 1024000, :sendRate => 360}

For each combinations, the run script creates one simulator core instance that is passed to the execute method call for the spec. When those executions call the variants method, the value from the current hash is returned. So, when running the first variant, the variant method for :sendRate returns 3600, in the second run, we get 360 and so on.

Parallelization

As I mentioned earlier, the simulations take quite some time to run, and in their current form, only use one processor core. So I figured, it would be nice to use some of the resources sitting idly around to speed up my simulations.

First, I thought about building multi-threading support for the simulations. But that would not have had any effect when running on the Ruby interpreter which only gives to green threads (I admit, that I thought for a while about a grandiose rewrite of the whole simulator in Erlang, but that would have been another silly yak-shave that I really could not justify). Also parallelizing the simulations is conceptually extremely difficult, so I gave up on that line of thought.

Fortunately, there is an easy kind of parallelization: just run two unrelated thing in two separate processes. And this kind is also applicable to my simulations, as I need to run many of them with varying parameters. So my run scripts now takes a parallelization parameter (-p) that tells it how many simulations to run at a time (this should be the number of processor cores of the host machine since the simulator can use all the CPU time it can get). For each simulation run the script forks a new instance of the ruby interpreter, so they all run in splendid isolation.