apsis on the BRML cluster

Generally, apsis consists of a server, whose task it is to generate new candidates and receive updates, and several worker processes, who evaluate the actual machine learning algorithm and update the server. Right now, it's best if you start the server on your own computer, and the worker processes as jobs on the cluster.

To start with, you need to install apsis and one requirement. To do so, first clone the apsis repo.

git clone https://github.com/FrederikDiehl/apsis.git

And add it to the python path (or call sys.path.append(YOUR_PATH) everytime you need it).

Additionally, you'll need a newer requests version; locally at least.

pip install --upgrade --user requests

Now, change to the cloned apsis directory, and change to the brml_dev branch

git checkout brml_dev

I'll keep the current mostly-stable version with some hacks for brml there. You can then either start the server in a python shell, or with the REST_start_script in code/webservice. In the python shell (don't do this here, because it blocks the shell), do

from apsis.webservice.REST_start_script import start_rest
start_rest(port=5000)

Or whichever port you want to use. You can do the rest here, now. But, first of all, try to access HOSTNAME:5000 via browser. You should see an overview page.

Congratulations; that means the server is working.

Now, first of all, let's look at the experiments page. The site to access (also via browser) is simply HOSTNAME:5000/experiments, the result should look like this:

{
  "result": []
}

This means the result of our request (getting all experiment ids) was successful, but we have no experiments started. Let's change that!

This is the Connection object, which we'll use to interface with the server. I've used PC-HIWI6:5116 as my hostname (and yes, the http is important); yours will vary. We can do the same we've done before, and look for experiment ids.

Not surprisingly, there still aren't any experiments. Time to change that; let's build a simple experiment. We need to define several parameters for that:

• name: The human-readable name of the experiment.
• optimizer: The string defining the optimizer, can be either RandomSearch or BayOpt
• param_defs: The parameter definition dictionary, we'll come back to that in a bit.
• optimizer_arguments: The parameters for how the optimizer is supposed to work.
• minimization: Bool whether the problem is one of minimization or maximization. Let's begin defining them.

Now, parameter definitions is interesting. It is a dictionary with string keys (the parameter names) and a dictionary defining the parameter. The latter dictionary contains the type field (defining the type of parameter definition). The other entries are the kwargs-like field to initialize the parameter definitions.

For example, let's say we have two parameters. x is a numeric parameter between 0 and 10, and class is one of "A", "B" or "C". This, we define like this:

param_defs = {
    "x": {
        "type": "MinMaxNumericParamDef",
        "lower_bound": 0,
        "upper_bound"; 10
    },
    "class": {
        "type": "NominalParamDef",
        "values": ["A", "B", "C"]
    }
}

And that's it!

For our example, we'll use the BraninHoo function, so we need two parameters, called x and y (or, sometimes, called x_0 and x_1, but that's ugly to type). x is between -5 and 10, y between 0 and 15.

We'll ignore optimizer_params for now. Usually, you could use it to set the number of samples initially evaluated via RandomSearch instead of BayOpt, or the optimizer used for the acquisition function, or the acquisition function etc.

Instead, we'll start with the initialization:

The experiment id is important for specifiying the experiment which you want to update, from which you want to get results etc. It can be set in init_experiment, but in doing so you have to be extremely careful not to use one already in use. If not specified, it's a newly generated uuid4 hex, and is guarenteed not to occur multiple times.

Now, we had looked at all available experiment IDs before (when no experiment had been initialized). Let's do it again now.

As you can see, the experiment now exists. Are there candidates already evaluated? Of course now, which the following can show us:

This function shows us three lists of candidates (currently empty). finished are all candidates that have been evaluated and are, well, finished. pending are candidates which have been generated, have possibly begun evaluating and then been paused. working are candidates currently in progress.

How do we get candidates? Simple, via the get_next_candidate function:

A candidate is nothing but a dictionary with the following fields:

• cost: The cost of evaluating this candidate. Is currently unused, but can be used for statistics or - later - for Expected Improvement Per Second.
• params: The parameter dictionary. This contains one entry for each parameter, with each value being the parameter value for this candidate.
• id: The id of the candidate. Not really important for you.
• worker_information: This can be used to specify continuation information, for example. It will never be changed by apsis.
• result: The interesting field. The result of your evaluation.

Now it's time for your work; for evaluating the parameters. Here, let's use the BraninHoo function.

And let's extract the parameters. Depending on your evaluation function, you can also just use the param entry dictionary directly (for example for sklearn functions).

Now, we'll just update the candidate with the result, and update the server:

And let's look at the candidates again:

Yay, it worked! And that's basically it. Every worker only has to use a few of the lines above (initializing the connection, getting the next candidate, evaluating and update).