How are PPAs used?

PPAs can be added by hand, but it is easier using the command add-apt-repository. That command is provided by the package python-software-properties.

Listing 2.1: Install python-software-properties for the command add-apt-repository.

sudo apt-get install python-software-properties The command add-apt-repository is used with the PPA name preceded by ppa:. An important thing to know is, that by installing such software you trust the packager that created the packages. It is advised to make sure that the packages won’t harm your system. The packager signsthe packages with his private key and also provides a public key. That public key is used by add-apt-repository to make sure that the packages have not been modified/manipulated since the packager created them. This adds security but as it was said before it does not protect you from malicious software that the packager might have included in the software.

Listing 2.2: Add the ppa repository.

sudo add-apt-repository ppa:<ppa name> After adding the repository you have to call apt-get update to update the package database. If you skip this step the software of the PPA won’t be available for installation.

Listing 2.3: Update the package database.

sudo apt-get update Now the software can be installed using apt-get insatll. It will also receive updated when apt-get upgrade is called. There is no extra step required to update software from PPAs.

Listing 2.4: Install the desired packages.

sudo apt-get install <packages>

Requirements

The requirements for the Qlustar OS are:

• A DVD or a USB flash-drive (minimum size 2GB) loaded with the Qlustar installer
• A 64bit x86 server/PC (or virtual machine) with
  • at least two network adapters
  • at least one disk with a minimum size of 160GB
  • optionally a second (or more) disk(s) with a minimum size of 160GB
  • CPU supporting virtualization (for virtual front-end and demo nodes)
  • Working Internet connection

Installation

Thanks to the provided PPA the installation was rather simple. There was only one minor nuisance. The official guide </ref> for installation using packages for Ubuntu suggested to install Icinga like this:

Listing 4.1 Suggested order of packages to install Icinga on Ubuntu.

apt-get install icinga icinga-doc icinga-idoutils postgresql libdbd-pgsql postgresql-client Unfortunately this failed, since the package installation of icinga-idoutils required a working database (either PostgreSQL or MySQL). So one has to switch the order of packages or just install PostgreSQL before Icinga.

Configuration

After the installation of Icinga the provided web interface was accessable right away (using port forwarding to access the virtual machine). Some plugins were enabled to monitor the host on which Icinga was installed by default.

Figure 4.2: The plugins (services) used to monitor a local host by default.

Figure 4.2 shows the service status details for these plugins on the master node. Getting Icinga to monitor remote hosts (the worker nodes) required much more configuration. A look into the configuration folder of Icinga revealed how the master node was configured to display the information of figure [fig:default_plugins]. Information is split into two parts: host identification and service specification. The host identification consists of host_name, address and alias. A service is specified by a host_name, service_description and a check_command. The check_command accepts a Nagios plugin or a custom plugin which has to be configured in another Icinga configuration file: commands.cfg.

Figure 4.3: Node configuration consists of two parts: host and services specification.

Figure 4.3 shows some important parts of the modified default configuration file used for the master node. As it can be seen both the host and service section start with a use statement which stands for the template that is going to be used. Icinga ships with a default (generic) template for hosts and services which is sufficient for us.

Figure 4.4: Overview of used ICINGA components.

The question of how to achieve a setup as presented in figure 4.4 now arises. We want to use Icinga to monitor our worker nodes. For that purpose Icinga provides two different methods, which work the same way but use different techniques. In either case the Icinga running on the master node periodically asks the worker nodes for data. The other approach would have been, that Icinga just listens for data and the worker nodes initiate the communication themselves. The two different methods are SSH^[7] and NRPE.^[8] In the manuals both methods are compared and NRPE is recommended at the cost of increased configuration effort. NRPE causes less CPU overhead, SSH on the other hand is available on nearly every Linux machine and thus does not need to be configured. For our purpose decreased CPU overhead is a selling point and therefore NRPE is used. The next sections describe how Icinga has to be configured to monitor remote hosts with NRPE.

Master

In order to use NRPE additional software has to be installed on the master node. The package nagios-nrpe-plugin provides Icinga with the possibility to use NRPE to gather data from remote hosts. Unfortunately that package is part of Nagios and thus upon installation the whole Nagios project is supposed to be installed as a dependency. Luckily using the option –no-install-recommends for apt-get we can skip the installation of those packages. The now installed package provides a new check_command that can be used during the service definition for a new host: check_nrpe. That command can be used to execute a Nagios plugin or a custom command on a remote host. As figure [fig:flo_ahc_icinga_components] shows, we want to be able to check “gmond” (a deamon of the next monitoring solution: Ganglia) and if two NFS folders (/opt and /home) are mounted correctly. For that purpose we create a new configuration file /etc/icinga/objects, in this case worker1.cfg, and change the host section presented in figure [fig:host_service_config] to the hostname and IP of the desired worker. The check_command in the service section has to be used like this:

Listing 4.2 NRPE check command in worker configuration file.

check_command check_nrpe_1arg!check-nfs-opt The NRPE command accepts one argument (thus _1arg): a command that is going to be executed on the remote host specified in the host section. In this case that command is check-nfs-opt which not part of the Nagios plugin package, it is a custom shell script. The next section describes the necessary configuration on the remote host that has to be done before check-nfs-opt works.

Worker

Additional software has to be installed on the worker as well. In order to be able to respond to NRPE commands from the master the package nagios-nrpe-server has to be installed. That package provides Nagios plugins and a service that is answering the NRPE requests from the master. We are not going to use a Nagios plugin, instead we write three basic shell scripts that will make sure that (as shown in figure [fig:flo_ahc_icinga_components]):

1. The gmond service of Ganglia is running.
2. Both \opt and \home are correctly mounted using NFS from the master.

Before we can define those commands we have to allow the master to connect to our worker nodes:

Listing 4.3 Add the IP address of the master to //etc/nagios/nrpe.cfg.

allowed_hosts=127.0.0.1,!\colorbox{light-gray}{10.0.1.100}! After that we can edit the file /etc/nagios/nrpe_local.cfg and add an alias and a path for the three scripts. The commands will be available to the master under the name of the specified alias.

Listing 4.4 Add custom commands to /etc/nagios/nrpe_local.cfg

command[check-gmond-worker]=/opt/check-gmond.sh command[check-nfs-home]=/opt/check-nfs-home.sh command[check-nfs-opt]=/opt/check-nfs-opt.sh This is all that has to be done on the worker. One can check if everything is setup correctly with a simple command from the master as listing [lst:nrpe_check] shows:

Listing 4.5 Check if NRPE is setup correctly with check_nrpe.

ehmke@master:/etc/icinga/objects$ /usr/lib/nagios/plugins/check_nrpe -H 10.0.1.2 CHECK_NRPE: Error - Could not complete SSL handshake. Unfortunately in our case some extra steps were needed as the above command returned an error from every worker node. After turning on (and off again) the debug mode on the worker nodes (debug=1 in /etc/nagios/nrpe.cfg) the command returned the NRPE version and everything worked as expected. That is some strange behaviour, especially since it had to be done on every worker node.

Listing 4.6 check_nrpe success!.

ehmke@master:/etc/icinga/objects$ /usr/lib/nagios/plugins/check_nrpe -H 10.0.1.2 NRPE v2.12

Usage

Figure 4.5 shows the service status details for all hosts. Our custom commands are all working as expected. If that would not be the case they would appear as the ido2db process. The status of that service is critical which is visible at first glance. The Icinga plugin api^[9] allows 4 different return statuses:

• OK
• WARNING
• CRITICAL
• UNKNOWN

Additionally to the return code it is possible to return some text output. In our example we only return “Everything ok!”. The plugin which checks the ido2db process uses that text output to give a reason for the critical service status which is quite self-explanatory.

Figure 4.5: Overview over monitored services by our test setup.

Installation

The installation of Ganglia was pretty straightforward. We downloaded the latest packages for Ganglia^[10] and RRDTool^[11] which is used to generate the nice graphs. RRDTool itself also needed libconfuse to be installed. After the compilation (no special configure flags were set) and installation we had to integrate RRDTool into the environment such that Ganglia is able to use it. This usually means adjusting the environment variables PATH and LD_LIBRARY_PATH. Out of personal preference we choose another solution as listing [lst:rrd_env] shows.

Listing 4.7 Integrating RRDTool into the environment..

echo '/opt/rrdtool-1.4.7/lib' >> /etc/ld.so.conf.d/rrdtool.conf ldconfig ln -s /opt/rrdtool-1.4.7/bin/rrdtool /usr/bin/rrdtool Ganglia also needs libconfuse and additionally libapr. Both also have to be installed on the worker nodes. It was important to specify –with-gmetad during the configuration.

Listing 4.8 Installation of Ganglia.

./configure --with-librrd=/opt/rrdtool-1.4.7 --with-gmetad --prefix=/opt/ganglia-3.5.0 make sudo make install

Configuration

Figure 4.6: Overview of used Ganglia components.

Ganglia consists of two major components: gmond and gmetad. Gmond is a monitoring daemon that has to run on every node that is supposed to be monitored. Gmetad is a daemon that polls other gmond daemons and stores their data in rrd databases which are then used to visualize the data in the Ganglia web interface. The goal was to configure Ganglia as shown in figure [fig:flo_ahc_ganglia_components]. The master runs two gmond daemons, one specifically for collecting data from the master, and the other one just to gather data from the gmond daemons running on the worker nodes. We installed Ganglia to /opt which is mounted on every worker via NFS. In order to start the gmond and gmetad processes on the master and worker nodes init scripts were used. The problem was, that there were no suitable init scripts provided by the downloaded tar ball. Our first idea was to extract the init script of the (older) packages of the Ubuntu repositories. That init script didn’t work as expected. Restarting and stopping the gmond service caused problems on the master node, since 2 gmond processes were running there. Instead of using the pid of the service they were killed by name, obviously no good idea. We tried to change the behaviour manually, but unfortunately that didn’t work. After the gmond process is started, the init systems reads the pid of the started service and stores it in a gmond.pid file. The problem was, that the gmond process demonizes after starting and changes the running user (from root to nobody). Those actions also changed the pid which means the .pid file is no longer valid and stopping and restarting the service won’t work. After a lot of trial and error we found a working upstart (the new init system used by Ubuntu) script in the most recent (not yet released) Ubuntu version 13.04. In that script we only had to adjust service names and make sure that the NFS partition is mounted before we start the service (start on (mounted MOUNTPOINT=/opt and runlevel [2345])). For some magical reason that setup even works on the master node with two gmond processes.

Master

At first we configured the gmetad daemon. We specified two data sources: “Infrastructure” (the master node) and “Cluster Nodes” (the workers). Gmetad gathers the data for these sources from the two running gmond processes on the master. To prevent conflicts both are accepting connections on different ports: 8649 (Infrastructure) and 8650 (Cluster Nodes). We also adjusted the grid name and the directory in which the rrd databases are stored.

Listing 4.9 Interesting parts of gmetad.conf.

data_source "Infrastructure" localhost:8649 data_source "Cluster Nodes" localhost:8650 gridname "AHC Cluster" rrd_rootdir "/opt/ganglia/rrds" The next step was to configure the gmond processes on the master: gmond_master and gmond_collector. Since the gmond_master process doesn’t communicate with other gmond’s no communication configuration was necessary. We only had to specify a tcp_accept_channel on which the gmond responds to queries of gmetad. Additionally one can specify names for the host, cluster and owners and provide a location (for example the particular rack).

Listing 4.10 Configuration of gmond_master.conf.

tcp_accept_channel {
    port = 8649
}

The gmond_collector process needs to communicate with the four gmond_worker processes. There are two different communications methods present in ganglia: unicast and multicast. We choose unicast and the setup was easy. The gmond_collector process additionally has to accept queries from the gmetad process which is why we specified another tcp_accept_channel. On the specified udp_recv_channel the gmond_collector waits for data from the gmond_worker processes.

Listing 4.11 Configuration of gmond_collector.conf.

tcp_accept_channel {
    port = 8650
}
udp_recv_channel {
    port = 8666
}

Worker

The gmond_worker processes neither listens to other gmond processes nor accepts queries from a gmetad daemon. Thus the only interesting part in the configuration file is the sending mechanism of that gmond daemon.

Listing 4.12 Configuration of gmond_worker.conf.

udp_send_channel {
    host = master
    port = 8666
    ttl = 1
}

Usage

Ganglia already gathers and visualizes data about the cpu, memory, network and storage by default. It is also possible to extend the monitoring capabilities with custom plugins. The gathered data can be viewed in many small graphs each only featuring one data source, or in larger aggregated “reports”.

Figure 4.7: Front page of Ganglia.

The front page of ganglia shows many of those aggregated reports for the whole grid and the “sub clusters”. Figure 4.7 shows that front page from where it is possible to navigate to the separate sub clusters and also to specific nodes. The reports on that page also show some interesting details. The master node for example has some outgoing network traffic every 5 minutes. By default all reports show data from the last hour, but it is also possible to show the data over the last 2/4 hours, week, month or year.

Graph aggregation

An especially interesting feature is the custom graph aggregation. Let’s say there is a report available that visualizes the cpu utilization of all (for example 10) available nodes. If you run a job that requires four of these nodes, you are likely not interested in the data of the other 6 nodes. With Ganglia you can create a custom report that only matches nodes that you specified with a regular expression.

Figure 4.8: Enter a host regular expression to only visualize data of interesting nodes.

Figure 4.9: A custom aggregated graph with with the nodes specified in figure 4.8.

If that is not enough it is also possible to create entirely custom aggregated graphs where you can specify the used metrics, axis limits and labels, graph type (line or stacked) and nodes. In figure [fig:graph_aggregation] we specified such a graph. We choose a custom title, set the Y-axis label to percent, set the lower and upper axis limits to 0 and 100 and the system cpu utilization as a metric. It is also possible to choose more than one metric as long as the composition is meaningful.

Figure 4.10: Dialogue to create custom aggregated graphs.

Figure 4.11: A custom aggregated graph created by the dialogue presented in figure 4.10.

Interactive

The key command here is srun. Thus one can perform the jobs interactively and allocate resources. This is followed by options that are passed to the batch system. There are options whose values are set for the environment variables in SLURM (see Section 5.3).

Jobscript

A jobscript is a file with for example shell-commands. There are no in- or output parameters. In a jobscript, the environment variables are set directly. Therefore the lines need to marked with #SBATCH. Other lines, which start with a hash are comments. The main part of a job script is the program call. Optionally, however, additional parameters can be passed like in the interactive mode.

Listing 5.7 Example for a jobscript.

#!/bin/sh

srun hostname The script is called as usual and runs all included commands. A job script can contain several program calls also from different programs. Each program call can have redundant options attached, which can override the environment variables. If there are no further details the in the beginning of the script specified options apply.

Package

The prefabricated package contains the version 2.3.2. With

$ apt-get install slurm-llnl you download the package. In the second step, a configuration file must be created. There is a website that accepts manual entries and generates the file automatically - https://computing.llnl.gov/linux/slurm/configurator.html. In this case, only the name of the machine has to be adjusted. Most importantly, this file must be identical on all clients and on the server. The daemons are started automatically. With sinfo you can quickly check if everything went well.

Manually

Via a link you download the archive with the latest version - 2.5.3. The archive must be unpacked.

$ wget http://schedmd.com/download/latest/slurm-2.5.3.tar.bz2 $ tar --bzip -x -f slurm*tar.bz2 Furthermore the configure must be called. It would suffice to indicate no options, however, one should make sure that the directory is the same on the server and the clients, because you would have to install it on each client individually otherwise. If gcc and make are available, you install SLURM.

$ ./configure --prefix=/opt/slurm_2.5.3 $ make $ make install As in the last section, the configuration file must be created (see link above) even with manual installation. The name of the machine have to be adjusted. The option ReturnToService should get the value of 2. After the failure of a node it is otherwise no longer recruited, even if he is available again. In addition, ${\displaystyle Munge}$ should be selected.

Munge

Munge is a highly scalable authentication service. It is needed for the client node to respond only to requests from the “real” server, not from any. For this, a key must be generated and copied to all associated client nodes and the server. Munge can be installed as usual. In order that authentication requests are performed correctly, the clocks must be synchronized on all machines. For this the ntp-service is enough. The relative error is within the tolerance range of Munge. As server the SLURM server should also be registered.
After Munge was installed, a system user has been added, which was specified in the configuration file (by default slurm).

$ sudo adduser --system slurm The goal is that each user can submit jobs and execute SLURM commands from his working directory. This requires that the PATH is customized in /etc/environment. Add this following path:

/opt/slurm_2.5.3/sbin: The path refers of course to the directory where SLURM has been installed and can also differ. Now it is possible for the user to call from any directory SLURM commands. However, this does not include sudo commands, because they do not read from the environment variable path. For this you use the following detour:

$ sudo $(which <SLURM-Command>) With which one gets the full path of the command that is read from the modified environment variables. This path you pass to the sudo command. It may be useful, because after manual installation both daemons must be started manually. On the server the slurmctld and the slurmd on the client machines is executed. With additional options -D -vvvv you can see the error messages in more detail if something went wrong. -D stands for debugging and -v for ${\displaystyle verbose}$. The more “v”s are strung together, the more detailed is the output.

Internal Scheduler

In the configuration file three methods can be defined - ${\displaystyle builtin,backfill}$ and ${\displaystyle gang.}$

Builtin

This method works on the FIFO principle without further intervention.

Backfill

The backfill process is a kind of FIFO with efficient allocation. Once a job requires currently free resources and is queued behind other jobs, whose resource claims currently can not be satisfied, the “minor” job is prefered. The by the user defined time limit is relevant.

Figure 5.2: Auswirkungen von FIFO und Backfill Strategien.

In the graphic [fig:schedul] left is shown a starting situation in which there are three jobs. Job1 and job3 just need one node, while job2 needs two. Both nodes are not busy in the beginning. Following the FIFO strategy job1 would be carried out and block a node. The other two have to wait in the queue, although a node is still available. Following the backfill strategy job3 would be preferred to the job2 since the required resources for job3 are currently free. Very important here is the prescribed time limit. Job3 is finished before job1 and would thus not prevent job2’s execution, if both nodes would become available. If job3 had a longer time-out, this job would not be preferred.

Gang

The Gang-scheduling has only one application, if accepted, that the resources are not allocated exclusively (option –shared must be specified). At this it jumps between time slices. Gang-scheduling causes that in the same time slots, as possible, belonging processes are handled and thus the context jumps are reduced. The SchedulerTimeSlice option specifies the length of a time slot in seconds. Within this time slice builtin- or backfill-Scheduling can be used if multiple processes compete for resources. In older versions (lower than 2.1) the distribution follows the Round-Robin principle.
To use the Gang-scheduling at least three options have to be set:

PreemptMode = GANG SchedulerTimeSlice = 10 SchedulerType = sched/builtin By default, 4 processes would be able to allocate the same resources. With the option

Shared=FORCE:xy the number ${\displaystyle xy}$ can be defined.

Policy

However, the scheduling capabilities of SLURM are not limited to the three strategies. With the help of ${\displaystyle Policy}$ options each strategy can be refined and adjusted to the needs of users and administrators. The Policy is a kind of house rules, which are subject to any job, user, or group or project. In large systems, it will quickly become confusing when each user has his own set of specific rules. Therefore SLURM also supports database connections via MySQL or PortgreSQL. For the use of databases those need to be explicitly configured for SLURM. On the SLURM part certain options need to be set so that the policy rules can be applied to the defined groups. Detailed description - https://computing.llnl.gov/linux/slurm/accounting.html.

Most options in scheduling are targeted at the priority of jobs. SLURM uses a complex calculation method to determine the priority - ${\displaystyle priority-multifactor}$. Five factors play a role in the calculation - ${\displaystyle Age}$ (waiting time of a waiting jobs), ${\displaystyle Fair-share}$ (the difference between allocated and used resources), ${\displaystyle Jobsize}$ (number of allocated nodes), ${\displaystyle Partition}$ (a factor that has been assigned to a node group), ${\displaystyle QOS}$ (a factor of service quality). Each of these factors will also receive a weighting. This means that some factors are defined as more dominant. The overall priority is the sum of the weighted factors (${\displaystyle float}$ values between 0.0 and 1.0):

Job_priority =

   (PriorityWeightAge) * (age_factor) +
   (PriorityWeightFairshare) * (fair-share_factor) +
   (PriorityWeightJobSize) * (job_size_factor) +
   (PriorityWeightPartition) * (partition_factor) +
   (PriorityWeightQOS) * (QOS_factor)

The detailed descriptions of the factors and their composition can be found here: https://computing.llnl.gov/linux/slurm/priority_multifactor.html.
Particularly interesting is the QOS (${\displaystyle QualityOfService}$)-factor. The prerequisite for it’s usage is the ${\displaystyle multi-factor-plugin}$ and that PriorityWeightQOS is nonzero. A user can specify a QOS for each job. This affects the scheduling, context jumps and limits. The allowed QOSs be specified as a comma-separated list in the database. QOSs in that list can be used by users of the associated group. The default value normal does not affect the calculations. However, if the user knows that his job is particularly short, he could define his job script as follows:

#SBATCH --qos=short This option increases the priority of the job (if properly configured), but cancels it after the time limit described in his QOS. Thus, one should take into account realistically. The available QOSs can be display with the command:

$ sacctmgr show qos The default values of the QOS look like this:^[14]

These values can be changed by the administrator in the configuration. Example:

MaxNodesPerJob=12 Complete list - https://computing.llnl.gov/linux/slurm/resource_limits.html

External Scheduler

SLURM is compatible with various other schedulers - this includes Maui, Moab, LSF and Catalina. In the configuration file Builtin should be selected if an external scheduler should be integrated.

Maui

Maui is a freely available scheduler by Adaptive Computing.^[15]) The development of Maui has been set of 2005. The package can be downloaded after registration on the manufacturing side. The version requires Java for installation. Maui features numerous policy and scheduling options, however, are now being offered by SLURM itself.

Moab

Moab is the successor of Maui. Since the project Maui was set, Adaptive Computing has developed the package under the name of Moab and under commercial license. Moab shell scale better than Maui. Also paid support is available for the product.

LSF

LSF - Load Sharing Facility - is a commercial software from IBM. The product is suitable not only for IBM machines, but also for systems with Windows or Linux operating systems.

Catalina

Catalina is a on going project for years, of which there is currently a pre-production release. It includes many features of Maui, supports grid-computing and allows guaranteeing of available nodes after as certain time. For the use Python is required.

Predecoding

The predecoding step only extracts static information like time, date, hostname, program name and the log message. These are well-known fields of several used protocols. There are a lot of standards for computer data logging - like the Apple System Log (ASL) and syslog - which use a different formatting to handle and to store log messages. However, OSSEC is able to distinguish between these types of protocols. As an example, the following log message shows an invalid login of an unknown user using the ssh command (syslog standard).

Feb 27 13:11:18 master sshd[13030]: Invalid user evil from 136.172.13.56

OSSEC now extracts these information and classify the fields of this message. Table 7.1 shows the fields that are picked by the predecoding process and their description.

Table 7.1: Example of a predecoded ssh event.

As mentioned before, there are several protocols that store log messages in different ways. The log messages have to be normalized so that the same rule can be applied for differently formatted log files. This is the task of the decoding phase that is described in the next section.

Decoding and Rule Matching

Following the predecoding phase, the decoding step extracts the nonstatic information of the events, e.g. the IP address, usernames and similar data, that can change from event to event. To get this data, a special XML file is used as a collection of predefined and user-defined decoders. They are matched with the current event using regular expressions. A decoder is specified by several options. These options define on which conditions a decoder will be executed. First of all, a decoder is delimited by a <decoder></decoder> tag, where the name of the decoder is specified. Within this tag all the possible options can be applied to fully refine the characteristics of the decoder. When working with syslog, the program name has to be set so that the decoder is only called if the program name appears in the log message. Another important option is the <prematch></prematch> tag. It can optionally be used jointly with the program name as an additional condition or as a single condition for non-syslog messages. To extract nonstatic information the decoder needs a <regex></regex> tag. This tag describes a regular expression which defines a pattern for matching the information included in the log message. As there are several fields in a message, the order of the fields also has to be specified, so that OSSEC knows how to parse out the fields of the message. This is done by the <order></order> tag.
Listing 7.1 gives an example of a decoder for an ssh event. It extracts the name and the source IP address of a user who tries to get access without any permissions via ssh corresponding to the given example.

Listing 7.1 Example of a decoder for ssh.

<decoder name="ssh-invalid-user">

        <program_name>sshd</program_name>
        <prematch>^Invalid user</prematch>
        <regex> (\S+) from (\S+)$</regex>
        <order>user,srcip</order>

</decoder> This decoder will only be called when the program name sshd has been extracted in the predecoding phase. Additionally, the prematch tag has to be satisfied. In this example the prematch option searches for the expression Invalid user in the log message. When this pattern will match what is in the log, the regular expression will be called to extract the user and source IP address. So only the decoder relevant for finding a rule associated with an invalid ssh success will be executed.
There are a lot of other options to specify the decoders. Here, the most important ones were explained. See OSSEC’s homepage^[17] for more explanations on decoder options.
By now OSSEC has all the information it needs to find a rule that matches an event. Rules are typically stored as XML files in /var/ossec/rules. A rule is identified by a unique number and a severity level. Depending on this level, OSSEC decides if this event should be escalated to an alert.
Two types of rules exist: atomic and composite. Atomic rules are based on single events, whereas composite rules are related to multiple events. For example an ssh event can be escalated to an alert already after one single ssh authentication failure (atomic) or can be suspended until three authentications have failed (composite).

Setup of the Server

The setup of the server is straight forward. One simply follows the instructions of the install script which include:

1. setting the language (default is english)
2. choosing Server as install type
3. choosing the directory where OSSEC will be installed (default is /var/ossec)
4. configuring the email notification (default is no email notification, see section 7.6.1 for how to set up the email notification)
5. enabling/disabling the integrity check, the rootkit detection engine, active response and the firewall-drop response (enabled by default)
6. adding IP addresses to the white list (localhost is the only IP address on the list by default)
7. enabling/disabling remote syslog (allows redirecting the syslog messages from one host to another host)

When the setup is successful, the installer script provides some final information. If it was not successful, the script shows ”Building error. Unable to finish the installation” at the end of the compiler output. Maybe a prerequisite is missing; the OSSEC homepage might be a good help.
OSSEC is to be started with the command ossec-control located at the directory of OSSEC’s binary files (Listing 7.4). This command starts all the services as specified in the setup, e.g. rootkit detection and integrity check.

Listing 7.4 Starting OSSEC.

/var/ossec/bin/ossec-control start Whenever starting the server, OSSEC will be started automatically. OSSEC is active until the user stops it with ossec_control stop.
It recommended not to run any other intrusion detection systems in parallel with OSSEC. It is possible that they mutually influence each other. For example OSSEC could recognize the other intrusion detection and its actions as an “attack” to the system, when it modifies files that are monitored by OSSEC.

Email Notification

Some SMTP servers permit emails to be relayed from the host running the OSSEC HIDS. Because of that, it is better to install a local mail server. For this, all the settings are done on the OSSEC server. First a mail server is needed, for example postfix.^[18] To process the emails, an email program has to be installed. The bsd-mailx, a command-line based email program is a possible choice for it (Listing 7.5).

Listing 7.5 Installing the mail server and an email program.

sudo apt-get install postfix sudo apt-get install bsd-mailx Finally, OSSEC needs to know where to send the emails to. In the main configuration file ossec.conf, the following entry has to be added:

Listing 7.6 Configuring the email notification in OSSEC.

<global>

   <email_notification>yes</email_notification> 
   <email_to>root@master</email_to> 
   <smtp_server>localhost</smtp_server> 
   <email_from>ossec-notification</email_from> 

</global> This enables the email notification. All emails will be send to the user root who has an account on the localhost (the OSSEC server master in this example), where the mail server has been installed. The <email_from></email_from> tag specifies the sender of the email notification.
Subsequent to the base configuration, it is possible to personalize the email notification. At default, OSSEC will send an email for every alert. This can be customized by setting the email alert level. Events are rated by their severity ranging from 0 to 15. Listing 7.7 shows how to set the email alert level, that means the minimum level an event must have so that an email will be sent. In the above example (Listing 7.6) there will only be sent an email if the event has an severity level greater than 7. The same option can be specified for writing the log files. In the below example (Listing 7.7) every event will be logged and saved in a log file, because the log alert level is set on the smallest possible severity level 0. Both options are located in the <alert></alert> tag in the main configuration file ossec.conf.

Listing 7.7 Setting the email alert level.

<alerts>

   <log_alert_level>0</log_alert_level>
   <email_alert_level>7</email_alert_level>

</alerts>

Setup of the Agent

Before installing the agents, the server needs to know the agents from which it has to collect the data. The agents must be able to identify themselves to the server and the server must be able to validate the identity of an agent.
The communication between server and agent is handled by key management. Keys are generated on the server and are then imported on each agent. OSSEC offers a tool, to perform the key management: manage_agents (Listing 7.8). First this tool has to be executed on the server to add the agents.

Listing 7.8 Starting the key management.

/var/ossec/bin/manage_agents After starting the tool, there are five options to choose:

(A)

for adding an agent

(E)

for extracting the key for an agent

(L)

for listing already added agents

(R)

for removing an agent and

(Q)

to quit the service

When adding an agent (option (A)), the user is prompted for host details and identifier for the agent. This includes a name, the IP address of the agent and a unique identifier which is a three-digit number (Listing 7.9]).

Listing 7.9 Adding an agent.

- Adding a new agent (use '\q' to return to the main menu).

 Please provide the following:
 * A name for the new agent: worker1
 * The IP address of the new agent: 10.0.0.1
 * An ID for the new agent[001]: 001

 ID: 001
 Name: worker1
 IP Address: 10.0.0.1

Confirm adding it? (y/n): y Agent added. Then OSSEC has to be installed on each agent separately using the installer script. The setup is very similar to the setup of the server (see section 7.6.1), the only difference is that the agents need to know the IP address of the server (Listing 7.10). Here the server’s IP address in the subnetwork is needed (Hint: ifconfig eth1).

Listing 7.10 Setup of OSSEC on the agent’s side. The only difference is the setting of the server’s IP address.

3 - Configuring the OSSEC HIDS.

   3.1 - What's the IP Address of the OSSEC HIDS server?: 10.0.0.250
       - Adding Server IP 10.0.0.250

After the installation has finished, a key for each agent has to be generated. This is done by starting the manage_agents tool again on the server’s side and then choosing option (E) for extracting a key for the agent. By selecting an agent by its three-digit identification number, a key will be generated (Listing 7.11). This key can be copied to the clipboard.

Listing 7.11 Generating the key for an agent (worker1).

Available agents:

   ID: 001, Name: worker1, IP: 10.0.0.1

   MDAxIG1hcnMgMTkyLjE2OC42NS40MCBmY2UzMjM4OT
   c1ODgzYTU4ZWM3YTRkYWJiZTJmMjQ2Y2ViODhmMzlm
   YjE3MmI4OGUzMTE0MDczMzVhYjk2OTRh

• • Press ENTER to return to the main menu.

To import the key, the manage_agents tool has to be executed on the corresponding agent. There are only two options to choose: (I) for importing a key from the server and (Q) to quit the service. After choosing the option (I), the key value is pasted from the clipboard into the terminal behind the prompt Paste it here. The tool provides some information about the agent for verifying the agent.

Listing 7.12 Importing the key of the agent (worker1) generated on the server.

Paste it here:

   MDAxIG1hcnMgMTkyLjE2OC42NS40MCBmY2UzMjM4OT
   c1ODgzYTU4ZWM3YTRkYWJiZTJmMjQ2Y2ViODhmMzlm
   YjE3MmI4OGUzMTE0MDczMzVhYjk2OTRh

   ID:001
   Name:worker1
   IP Address:10.0.0.1
   Confirm adding it?(y/n): y

Added. Then the service is quitted and OSSEC is restarted with ossec_control restart. When the restart and the connection to the server are successful, you can find a similar entry - seen in Listing 7.13 - in the log file ossec.log of the agent.

Listing 7.13 Success of the setup of the agent.

2007/10/10 23:25:48 ossec-agentd: Connecting to server (<IP-address-server>:1514). 2007/10/10 23:25:48 ossec-agentd(4102): Connected to the server.

Setup

The WUI runs on a fixed OSSEC HIDS server which collects and manages the agent’s data. The WUI cannot be installed on an agent, because all collected alerts are sent back to an OSSEC HIDS server for further processing. Before installing the WUI, there are some basic prerequisites that have to be met. The OSSEC HIDS server needs to be set as an HTTP web server - choosing apache2 here - and a package including the interpreter for the server-side scripting language PHP has to be installed (Listing 7.14). With this, the code containing the web page with php code can be interpreted by the web server.

Listing 7.14 Installing the apache server and php5.

sudo apt-get install apache2 php5 After downloading and extracting the archive containing the sources (Listing 7.15), the extracted directory is moved to /var/www, so as the web server can access the files.

Listing 7.15 Getting the WUI sources.

wget http://www.ossec.net/files/ossec-wui-0.3.tar.gz tar -xvf http://www.ossec.net/files/ossec-wui-0.3.tar.gz The directory contains a installer script guiding the user through the setup (Listing 7.16).

Listing 7.16 Extracting the wui files, moving it to /var/www and starting the setup.

mv ossec-wui-0.3 /var/www cd /var/www/ossec-wui-0.3 ./setup.sh The installation prompts for a username and a password (Listing 7.17). After this, the installation starts and the installer reports whether the setup was successful.

Listing 7.17 Installing the WUI.

Setting up ossec ui... Username: schroeder New password: Re-type new password: Adding password for user schroeder ...#Setup output Setup completed successfully. To complete the setup, the web server user has to be added to the ossec group (Listing 7.18). This user is typically apache, www or www-data.

Listing 7.18 Adding user www-data to ossec group.

adduser www-data ossec After restarting the whole system, the WUI is available at http://<ip-address-of-webserver>/ossec-wui-0.3/.

Functionality

The WUI offers a powerful functionality that helps the user to view the events that are happening within the system. Figure [fig:overview] shows the main window. It gives an overview about the available server(s) and agents, the latest modified files and the latest events. The available agents are described by their name and the associated IP address. This information is shown when clicking on the corresponding agent. If the agent is inactive or unable to connect to the server, the word Inactive is displayed beside the agent’s name. In the figure, the agent named worker1 is currently set inactive. Only the OSSEC server is available.

Figure 7.3: The WUI’s main window.

Moreover the main window contains four tabs, where each of them provides a specific functionality. These are

• Main, shows the main window
• Search, allows searching through collected alerts
• Integrity Checking, allows searching through collected syscheck alerts and
• Stats, shows aggregated statistics about collected alerts.

The fifth tab About does not provide a special service and just shows information about the WUI version and its contributors.

The search window (Figure 7.4) offers a query interface with a lot of possible options to specify the kind of alerts the user is searching for. Options of interest may be the minimum severity level of an alert, the category (e.g authentication control) and the log formats (e.g sshd). A specific search period can be set. Alternatively, by choosing the real time monitoring, the WUI shows all the upcoming events.
There are some more options to reduce the search results. A rule ID can be specified, so that only alerts and events are shown that are allocated to this rule number. The field Pattern searches for a pattern in an event described by regular expressions. To show only events from a particular agent (for example worker1) or from an user account (for example root), the fields Location and User are used. When there are too many search results, the maximal number of alerts to display can be restricted with the field Max Alerts.
When results are found, the WUI divides the types of alerts into three classes:

Severity breakdown

Alerts are sorted by their severity level

Rules breakdown

Informs about the alert’s corresponding rule ID

Src IP breakdown

Informs about the alert’s corresponding source ID, i.e. the user who triggered this alert.

Figure 7.4: Searching for alerts in a period from 16th February 2013 to 25th march 2013. Only alerts with a minimum severity level of seven are shown.

The WUI also shows files that have been modified on the server and on all agents. Clicking on a specific file, for example /etc/resolv.conf in Figure 7.5, gives further information about where the file is located - specified by the server’s or agent’s name and a directory path - and when it has been modified.

Figure 7.5: Latest modified files for all agents and servers sorted by date. Figure

Figure 7.6 shows a detailed view of modified files for a particular agent. This view can be selected by choosing an agent via the drop down menu offering all the agents that are part of the OSSEC system. Clicking Dump database gives the overview of all files that have changed for this agent.

Figure 7.6: Modified files for a particular agent (worker1). The configuration files ossec.conf and internal_options.conf have changed.

Figure 7.7 shows the statistics of a selected date. It shows values aggregated over all agents sorted by severity level of events and by rules (described by their rule ID), respectively. Moreover it counts the total and the averaged number of events and how many events are classified as alerts.

Figure 7.7: Statistics for the 25th March 2013. It shows which and how many rules were found for specific events and how much rules were found with a certain severity level.

An event is represented as in Figure 7.8. This representation is used in the main window and when showing the search results. It contains the following information: The date and time when the OSSEC agent has recognized the event are shown. The date and time stamps are displayed at the beginning of every event. The rule ID (Rule Id) that is associated with the event has a link to pieces of information about the rule. Clicking on it will redirect the user to the web page containing rule specific information. The severity level of an event is also shown in this representation (level). The Location field gives information on the agent that reported the event and what file is associated with the event. Description summarizes the event in a short description.

Figure 7.8: Representation of an event with in the OSSEC WUI.

Usage

Snort has the the ability to perform real-time traffic analysis and packet logging on Internet Protocol networks and performs protocol analysis, content searching, and content matching. Snort can also be used to detect probes or attacks, including operating system fingerprinting attempts, common gateway interface, buffer overflows, server message block probes and stealth port scans.

Snort has three main configuration modes: sniffer, packet logger and network intrusion detection. In sniffer mode, Snort will read the network packets and display them on the console. Snort will log packets to the disk in the packet logger mode and in the intrusion detection mode, Snort will monitor the network traffic and analyze it against a rule set the user has defined. Snort will then alarm the user or it will perform a specific action based on what has happened.

LAMP-Server

LAMP is a combination of free, open source software, LAMP refers to Linux, Apache HTTP Server, MySQL and PHP, Perl or Python. The exact Software may vary in the LAMP-Package and it is not designed by its original authors as one Package. But it is developed in close conjunction, therefore it is easy adaptable. When used together, they support web application servers.

BASE

BASE is the Basic Analysis and Security Engine. It is based on the code from the Analysis Console for Intrusion Databases (ACID) project. This application provides a web front-end to query and analyze the alerts coming from a SNORT IDS system.

BASE is a web interface to perform analysis of intrusions that snort has detected on your network. It uses a user authentication and role-base system, so that you as the security admin can decide what and how much information each user can see. It also has a simple to use, web-based setup program for people not comfortable with editing files directly.

LAMP-Server

root@master:/# sudo tasksel install lamp-server tasksel: Starts a simple Interface for the user to select general tasks for the installation.

You will be prompted to select a MySQL root password. This will be needed further in the installation.

Figure 9.1: Setting MySQL root password

Snort-Mysql

First you create a database.

root@master:/# mysql -u root -p
mysql> create database snort;
mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON snort.* TO 'masterusername'@'localhost' IDENTIFIED BY 'password';
mysql> FLUSH PRIVILEGES;
mysql> quit
root@master:/# _

Further on you can now install SNORT

root@master:/# sudo apt-get -y install snort-mysql You will be prompted to enter the IP address for your local network. It is your cluster-network, that you want to protect. The Classless Inter-Domain Routing (CIDR) format have to be used. 10.0.x.0/24

Figure 9.2: setting IP address for the network to protect

You will be asked, if you wish to set up a database for use with Snort.

Figure 9.3: Database setup for Snort

Choose no. You already created the database, you only need to configure it. With this you will update the Snort table structure.

root@master:/# pushd /usr/share/doc/snort-mysql root@master:/# sudo zcat create_mysql.gz | mysql -u snort -p snort

root@master:/# popd Now modify the Snort configuration file to include your MySQL specific information.

root@master:/# sudo sed -i "s/output\ log_tcpdump:\ tcpdump.log/#output\ log_tcpdump:\ tcpdump.log\noutput\ database:\ log,\ mysql, user=snort password=password dbname=snort host=localhost/" /etc/snort/snort.conf The above line was located at line number 786 /etc/snort/snort.conf. The snippet simply places a comment in front of the output for the log, and appends the output line for our database.

Now remove the pending Snort database configuration file.

root@master:/# sudo rm -rf /etc/snort/db-pending-config Start Snort.

root@master:/# sudo /etc/init.d/snort start To check the status of the Snort daemons use:

root@master:/# sudo /etc/init.d/snort status

Acidbase

To check the output of Snort, you will use ACID, a web front-end.

root@master:/# sudo apt-get -y install acidbase You will be asked to configure a database for acidbase. Choose yes and use MySQL.

Figure 9.4: BASE configuration

You will be prompted for the password of the database administrator. This is the same password you used when MySQL was initially installed with LAMP.

Figure 9.5: Setting Passwort for the database administrator

To get access for the acidbase web front-end, you have to edit the apache.conf

root@master:/# sudo sed -i "s#allow\ from\ 127.0.0.0/255.0.0.0#allow\ from\ 127.0.0.0/255.0.0.0\ 'your IP'/255.255.255.0#" /etc/acidbase/apache.conf This will allow your and the machine to get access to the front-end.

You can also allow access for all

Figure 9.6: setting front-end access

To take affect, you have to restart apache.

root@master:/# sudo /etc/init.d/apache2 restart Browse to: http://’IP of the machine with SNORT’/acidbase

For testing purpose perform a portscan of the Snort host. You will need the nmap-package for this.

root@master:/# sudo nmap -p1-65535 -sV -sS -O 'IP of your SNORT-machine' If something is not working properly, try to reconfigure SNORT. With the following command, you can reset the IP´s.

root@master:/# sudo dpkg-reconfigure snort-mysql

Manager

Figure 10.1: Interaction between managers and sensors

The Prelude-Manager is the heart of the software, here all messages of the sensors come together and then are written into the database. The sensors can observe very different places such as various logs, or the kernel itself. In this way, a sensor is arranged for each relevant interface, which then being conntected to the manager. The manager collects, sorts and normalizes the incoming data and stores it in IDMEF format in the created database.

In addition to the sensors Prelude brings from home, it is also possible to register third-party sensors to the Manager.

Figure 10.2: Possible third party sensors

Requirements

This installation has been tested on a virtual system with one master and several workers. On all machines the 64 bit version of Ubuntu Server was used. For ease of passage of the installation, it is recommended to give appropriate “root” rights once with the command
# sudo su
.

Step by Step Installation

Step 1

Install all software packages: (All packets are solely installed on the master)

# apt-get install prelude-lml prelude-manager prelude-correlator prelude-notify apache2 postgresql-9.1 prewikka

The user is eventually asked if he wants to set up the database using a wizard, this is recommended here for inexperienced users.

Step 2

Now various configuration files must be set. We start with the Prelude configuration.

# nano /etc/prelude/default/global.conf

Figure 10.3: IMPORTANT [NODE-ADDRESS] must be commented.

Here, the “node-name” must be entered, as well as the right address. When on a virtual machine, use the following address for the local environment “127.0.0.1”. It is important that [NODE ADDRESS] is also commented here!

Step 3

Start the Prelude Manager:
# /etc/init.d/prelude-manager start

Here we are told that the prelude-manager has yet to be activated. The matching file is given in the error message.

prelude-manager disabled, please adjust the configuration to your needs * and then set RUN to ’yes’ in /etc/default/prelude-manager to enable it.

After activating the Prelude Manager in the following file, we start it again with the command from above.

# nano /etc/default/prelude-manager

Step 4

The Prelude sensors must be registered in the manager, so that they can communicate with each other. For this we register the Prelude in this step - LML Sensor to the Manager. On the Manager, we run the following command to register the prelude-lml sensor.

# prelude-admin register prelude-lml ’idmef:w admin:r’ 127.0.0.1 –uid 0 –gid 0

Figure 10.4: The manager now waits for the password

This now awaits the entry of a on the server generated password. Since our Prelude-lml tool is running on the same master as the manager, a second terminal window should be used.

# prelude-admin registration-server prelude-manager

Figure 10.5: The by the sensor generated password must now be entered in the manager

This command generates for our sensor a password, which is necessary to enter at the registration, after successfully entering the password in the manager, the registration must again be confirmed. Now Manager and Sensor (Prelude-lml) are connected to each other.

Figure 10.6: Registration successful!

Step 5

We configured our managers already in step 2, now the sensor (Prelude-lml) must be set. For this, we edit the file:

# nano /etc/prelude-lml/prelude-lml.conf

Here are two lines need to be commented, so that the sensor has set the correct server address.

[prelude]

server-addr = 127.0.0.1

Step 6

Prelude is now fully functional, but we still lack the graphical representation of the results, we get this with Prewikka displayed in the browser. Therefore it is recommended once again to check the databases.

# nano /etc/prewikka/prewikka.conf

Listing 10.1 Database name and user (example)

type: pgsql

host: localhost user: prelude pass: prelude name: prelude [database] type: pgsql host: localhost user: prewikka pass: prewikka name: prewikka

Step 7

Lastly the Apache server can be configured so that it also takes the necessary files from Prewikka.

# nano /etc/apache2/apache2.conf

At the end of this file, the following code must be added:

Listing 10.2 Apache configuration for Prewikka

Alias /prewikka/prewikka/ /usr/share/prewikka/htdocs/ ScriptAlias /prewikka/ /usr/share/prewikka/cgi-bin/prewikka.cgi <Directory /usr/share/prewikka/htdocs/>

Options None AllowOverride None Order allow,deny Allow from all

</Directory> <Directory /usr/share/prewikka/cgi-bin/>

AllowOverride None Options ExecCGI <IfModule mod_mime.c> AddHandler cgi-script .cgi </IfModule> Order allow,deny Allow from all

</Directory>

Apache has to be restarted once after reconfiguration.

# /etc/init.d/apache2 restart

Step 8

Prewikka must be started (after every reboot) with the following daemon:

# /usr/bin/prewikka-httpd

Step 9

Now Prewikka can be accessed in the browser. (Of course, replace the IP adress with your eth0 inet adress)
192.168.178.56/prewikka/
Name: admin
Password: admin

Issues during installation

When you install a few problems might occur, a few of them, I would like to briefly discuss.

lack of access rights in step 8

eventually an error that reports a lack of privileges of prewikka.conf file appears when starting Prewikka in step 8. If this error occurs the rights to the file must be again be adjusted with the command, .

# chmod 755 /etc/prewikka/prewikka.conf

PreSQL instead of mySQL

The reason we use PreSQL and not mySQL is that Prelude uses outdated database settings. It uses the outdated command ’TYPE = InnoDB’, however, mySQL 5.0+ accepts only the command ENGINE = InnoDB ’. This can indeed be corrected in the file:

# nano /usr/share/libpreludedb/classic/mysql.sql

It then accepts this file also, but then fails to start from the prelude-manager due to an error in the libpreludedb.
It is therefore advisable to use a PreSQL database for Prelude.

Installation of Firestarter

With
sudo apt-get install firestarter
you install the firewall Firestarter. The command
sudo gksu gedit /etc/firestarter/user-pre
installs in addition the interface. Furthermore
sudo /etc/init.d/firestarter restart
restarts the system, so the installation can be completed.
Applications -> Internet -> Firestarter
activates opens the Firestarter gui. Then the Firestarter settings will be adapted using a Setup menu.

Adaption of Firestarter

Step 1:
- Network device setup
For this one must make the following settings:
- Set detected device ethernet to eth1
- IP Adress is assigned via DHCP checkmark
- click forward

Step 2
- Ready to start your firewall
- Start firewall now, restart system
- click save, Firestarter should start now

Step 3
Internet connection sharing setup
- internet connection sharing setup menu
- internet connection sharing setup
- change to IP tunnel
- click DHCP Server details
- forward
(Example of https://help.ubuntu.com/community/Firestarter)

Firestarter Errors and Cleanup

With Firestarter a problem is that the firewall is engaged too deeply into the operating system. Linux is not quite so often attacked as Windows, because of the underlying infrastructure of the system. A firewall should be installed yet. The packets of each software require a very large number of MB and requires corresponding time before the system unpacked it. The VPN configuration is somewhat complicated for the user. It requires appropriate exercise time until you get familiar with the system. The codes are initially easily mistyped in the beginning, if you are not so familiar. The firewall should start when booting the system to protect it accordingly.

Set Directory

By default make install installs all files in in /usr/local/bin, /usr/local/lib, /usr/local/sbin, /usr/local/include, and /usr/local/man.
You can also specify a different folder where the files should be stored by putting -–prefix=$directoryname behind ./configure. So If you don’t want to change anything, you do not need to consider this step.

Set Library Folder

Create a new file: /etc/ld.so.conf.d/torque.conf

$ sudo nano /etc/ld.so.conf.d/torque.conf

There you write the path to the libraries. In the standard setting, it would be /usr/local/lib (is home defined as a directory it would be /home/lib). Then enter the following command:

$ sudo ldconfig

Perform Configure

To execute configure you have to install build-essentials, libssl-devel and libxml2-devel with this command:

$ sudo apt-get install build-essentials libssl-dev libxml2-dev

If you excute ./configure you will get an error that libxml2-devel isn’t installed. This is a bug in Torque and can be fixed with following steps:
Firstly two lines in the configure.ac file need to be changed (see screenshot).

$ sudo nano configure.ac

Figure 12.1: Configure Bug Fix

The minus describes the line that needs to be changed, the plus describes how the line should read after the change. It is best to look for a keyword for the line to be changed because the file has a lot of lines.
After that execute autoconf:

$ sudo autoconf

and change the configure file:

sudo nano configure

Figure 12.2: Configure Bug Fix 2

Again, you look for the yellow marked line and change in the end (red rectangle) the -1 in a -l.
Now you can run ./configure and it should finish without errors.

sudo ./configure

In the end also run make and make install.

sudo make

sudo make install

By default, make install creates the directory /var/spool/torque. This directory is referred to as TORQUE_HOME. There, various subfolders are created that are used to configure and run the program.

Create packages

Torque has the function to create the packages, which uses the configurations and then can be installed on the nodes. Use the command make for this.

make packages

The packages are stored in the torque-4.1.4/ and must be copied from there in a shared directory the nodes have access to. In our case it would be the /home directory.
For example:

cp torque-package-mom-linux-i686.sh /home

On the nodes only the mom-linux package is needed. All others are optional.

Install Package

On the node you navigate to the directory in which you have copied the package and install it with the following command:

./torque-package-mom-linux-i686.sh –install

Initialise serverdb

In the directory TORQUE_HOME/server_priv are configurations and information located that the pbs_server Service uses. To initialise the file serverdb run following command:

sudo ./torque.setup

Then the pbs_server needs a restart.

sudo qterm

sudo pbs_server

The server properties can be see by the following command:

sudo qmgr -c ’p s’

Specify Nodes

Thus, the pbs_server recognizes which computers in the network are the nodes. For this create in the directory TORQUE_HOME/server_priv a new file nodes:

sudo nano nodes

In this file, the nodes will be stored with their name. Normally it is sufficient to write the names in the file, you may set special properties for each node. The syntax is:
NodeName[:ts] [np=] [gpus=] [properties]
[:ts]: This option sets the node as timeshared. These nodes are indeed listed by the server, but do not get jobs allocated.
[np=] This option is used to specify how many virtual processors are located on the nodes.
[gpus=] This option is used to specify how many CPUs are on the node.
[properties] This option allows to enter a name to identify the node. However, it must start with a letter.
One can detect the number of processors also automatically:

sudo qmgr -c set server auto_node_np = True

As a result, properties in the server auto_node_np are set to True.

Configure Nodes

To configure the nodes, the file config in the directory TORQUE_HOME/mom_priv has to be created:

sudo nano config

This file is created the same on all nodes and should read the following:

Figure 12.3: Config file

Furthermore, one must write the line $usecp*:/home /home write into it. This ensures that the file of the finished jobs is stored in a specific directory (here the shared /home). Otherwise the following error will occur when running the command tracejob:

Figure 12.4: Tracejob error

Run Job

Figure 12.5: Bash file example

The command qsub [file name], executed on the master, starts a job. To run a job, you need a Bash file. In the example above, the date is displayed, wait 10 seconds and then again output the date. The result is then stored in the directory on the master from which the job was started.

Useful Commands

There are some commands in Torque with which you can trace the running jobs and which are very useful for troubleshooting.

The command

pbsnodes -a

, executed on the master, shows if a node is active or not. With the command

qstat

a list of running or finished jobs is displayed.

Figure 12.6: qstat display

There you can see which number a job has which node is used and whether the job is started, in progress or has already ended.
A very useful command for debugging is

tracejob [job number]

This is a command from Torque which searches and summarizes the log files in the pbs_server, mom and scheduler. With this one gets a quick overview

Board

Als Erstes mussten wir uns Gedanken über die Wahl des Einplatinen-ARM-Computers machen. Dazu haben wir die verschiedenen Boards mit einander verglichen:

Mit vier Kernen á 2 GHz, 2 GiB DDR3 RAM und Gigabit-Ethernet ist das Cubieboard4 unschwer als das Leistungsstärkste der vier aufgeführten zu erkennen. Der Preis von 100 ist für die Leistung zwar komplett angemessen, für unser Projekt jedoch etwas zu hoch, sodass das Cubieboard4 leider aus der engeren Auswahl fiel. Anders beim BeagleBone Black, hier passt der Preis von nur 55,00Euro, leider kann das Board leistungstechnisch nicht mit den beiden anderen verbliebenden Boards mithalten. Mit nur einem Kern und 512 MB RAM, steht es klar hinten an. Der Raspberry Pi 2 (Modell B) besitzt zwar einen Preisvorteil gegenüber dem Banana Pi M2, hat dafür aber auch nur DDR2 statt DDR3 RAM und eine 100 MHz niedrigere Taktfrequenz. Ausschlaggebend war am Ende die höhere Netzwerkdurchsatzrate des Banana Pi M2, der mit 1000Mbit/s aufwarten kann. Wir haben uns für fünf Banana Pis entschieden, vier Compute-Nodes und ein Head-Node.

Weitere Komponenten

• SD-Karten - Da die meisten nötigen Komponenten des Clusters nur auf dem Head installiert werden sollten, haben wir für diesen eine 32GB Micro-SD-Karte besorgt und für die Nodes, auf denen eigentlich nur die Berechnungen stattfinden sollen und nicht viel installiert sein muss, vier 8 GB Karten.
• Switch - Wir haben uns für einen D-Link DGS-10008D Switch entschieden, ausschlaggebend waren hierbei das Gigabit LAN sowie die 8 Ports, so dass alle Compute-Nodes + Head-Node gleichzeitig am Switch, und dieser auch noch am Internet angeschlossen werden kann.
• Stromversorgung - Ein Banana Pi soll bei Höchstleistung nicht mehr als 5 Watt benötigen, deshalb haben wir uns bei der Stromversorgung für einen USB-Port von Logilink entschieden, der 50 Watt Leistung bringt und sechs USB-Anschlüsse hat. Damit hatten wir über die Dauer des Projekts auch keine Schwierigkeiten, was den Energieverbrauch angeht.
• Case - Unser Cluster hat noch kein Case in dem die ganze Hardware unterkommt, allerdings gibt es ein 3D-gedrucktes Case in dem sich die Banana Pis befinden, sodass diese nicht lose herumliegen. Das Case ist ursprünglich für Raspberry Pis gedacht, die Maße des Banana Pis sollen laut Hersteller aber gleich dem der Raspberrys sein. Wie wir leider herausfinden mussten entspricht dies nicht ganz der Wahrheit, die Banana Pis sind etwas größer. Das Case musste daher von Hand angepasst werden. Die Banana Pis passen nun hinein, sitzen aber nicht so gut und fest wie es Raspberry Pis tun würden.

Figure 13.1: Das Herz unseres Clusters: 5 Banana Pis in einem 3D-gedruckten Case

OS

Eines unserer primären Ziele war es, die Anbindung neuer Compute Nodes so einfach wie möglich zu machen. Zusätzlich sollte der Cluster für eine breite Menge an Anwendungen zur Verfügung stellen. Für solche Anforderungen bietet sich CoreOS besonders an:

• Verteiltes,automatisches Konfigurationsmanagement über etcd (inklusiveIP-Adressen- Vergabe)
• Isolierte Umgebungen mit individuellen Abhängigkeiten für jede Anwendung über Container (rkt)
• Anwendungsverwaltung á la Slurm über fleet

Leider wird die ARM-Architektur, wie wir schnell herausfanden, (noch) nicht von CoreOS unterstützt. Somit entschieden wir uns gegen CoreOS und beschlossen, die Lösung für unsere Anforderungen auf einer anderen Ebene als dem Betriebssystem zu suchen. Auf den Banana Pis läuft nun die Raspbian-Distribution, zu finden auf der offiziellen Banana-Pi-Homepage.^[19]

Verkabelung

Der Aufbau des Clusters ist verhältnismäßig simpel. Die 5 Banana Pis werden, sowohl Head als auch Compute Nodes, direkt mit dem Switch verbunden. Vom Switch geht ein Netzwerkkabel in das umliegende Netzwerk. Der USB-Port liefert über 6 Ausgänge Strom, durch 5 davon werden die Banana Pis versorgt, an den 6. wird der Switch angeschlossen.

Netzwerk

Die Topologie unseres Clusters ist, wie im vorigen Abschnitt bereits zu erahnen, etwas ungewöhnlich. Bei den üblichen Clustern ist die Head Node über ein Interface an das umliegende Netzwerk und über ein anderes an den Switch und damit die restlichen Compute Nodes angebunden. Das bedeutet, dass jede Kommunikation von den Compute Nodes nach außen über die Head Node geht. Oft wird diese Topologie genutzt, um ein lokales Netzwerk zwischen den Compute Nodes aufzubauen und Kommunikation der Compute Nodes nach außen zu unterbinden. Die Head Node vergibt in diesem Fall über DHCP lokale IP-Adressen an die Compute Nodes. Diese Herangehensweise vereinfacht sowohl das Management der Compute Nodes als auch die Zugriffskontrolle auf den Cluster. Da ein Banana Pi allerdings nur einen Netzwerk-Anschluss hat, verwendet unser Cluster eine leicht abgewandelte Version dieser Topologie, wie die Abbildung zeigt. Über den Switch sind alle Nodes mit dem umliegenden Netzwerk verbunden. Statt zwei echten Netzwerk-Interfaces hat die Head Node ein virtuelles Interface, das die Verbindung zum lokalen Netzwerk darstellt. Dieses Setup funktioniert zwar, verlässt sich allerdings darauf, dass die Compute Nodes sich nicht zufällig über die direkte Verbindung an das umliegende Netz von einem anderen DHCP-Server eine IP-Adresse besorgen. Um dem Vorzubeugen, sind die Compute Nodes in unserem Fall auf der Blacklist des äußeren DHCP-Servers eingetragen und werden von ihm ignoriert. Als DHCP-Server auf der Head Node verwenden wir dnsmasq, in dessen Konfigurationsdateien die einzelnen Compute Nodes über die Option dhcp-host per MAC-Adresse mit einer festen IP assoziiert werden.

Figure 13.2: Netzwerktopologie des Clusters

NFS

Da wir fast alle Daten mölichst zentralisiert auf der Head-Node speichern, verwenden wir NFS um diese den Compute-Nodes über das Netzwerk als POSIX-kompatibles Dateisystem zugänglich zu machen:

• Auf der Head-Node läuft der NFS-Server, der /srv und /home bereitstellt.
• Auf jeder Compute-Node läuft ein NFS-Client, sodass die home-Verzeichnisse auf dem Head per fstab auf den Compute-Nodes gemountet werden können und die Compute-Nodes ihre lokalen Packages mit dem Golden Image auf dem Head synchronisieren können.

# /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5).

/srv *(rw,sync,no_subtree_check,no_root_squash) /home *(rw,sync,no_subtree_check) proc /proc /dev/mmcblk0p1 /boot /dev/mmcblk0p2 / proc defaults 0 0 vfat defaults 0 2 ext4 defaults,noatime 0 1 bpi-head:/home

-> async,rw,relatime,rsize=1048576,

wsize=1048576,proto=tcp,intr,nfsvers=3

Golden Image

Cluster setzen sich aus mehreren einzelnen Rechnern zusammen und sind daher in ihrer Administration deutlich aufwändiger als ein einzelnes System. Um diese dennoch möglichst einfach zu halten, wird für alle Compute-Nodes nach Möglichkeit die gleiche Hardware verwendet und sämtliche Software der Compute-Nodes (inklusive Betriebssystem) zentral gespeichert und übers Netzwerk bereitgestellt. Das auf der Head-Node gespeicherte Systemabbild wird als “Golden Image” bezeichnet und stellt den gemeinsamen aktuellen Zustand aller Compute-Nodes dar. Durch das Golden Image müssen effektiv nur noch zwei verschiedene Systeme administriert werden, was die Skalierbarkeit des Cluster-Setups drastisch steigert indem es den Aufwand für die Verteilung von neuer Software und deren Konfiguration konstant hält. Die Compute-Nodes der meisten handelsüblichen Cluster booten per PXE direkt das, im Netzwerk bereitgestellte, Golden Image. Banana Pis können jedoch nicht per PXE über das Netzwerk gebootet werden, weil ihr BIOS so konfiguriert ist, dass es den Bootloader auf der SD-Karte benutzt, weswegen wir einen kleinen Umweg gehen mussten:

• Die SD-Karte einer jeden Compute-Node enthält ein komplett lauffähiges System, das beim Einschalten gebootet wird.
• Alle Daten auf der SD-Karte stammen aus dem Golden Image. Die Installationen unterscheiden sich nur in ihrem hostname.
• Wir verteilen Änderungen am Golden Image nicht direkt beim Reboot, sondern indem wir semiautomatisch per Skript die lokale Installation mit dem Golden Image synchronisieren.
• Die SD-Karten neuer Compute-Nodes können ebenfalls per Skript mit dem aktuellen Zustand des Golden Image geflasht werden, sodass der Cluster einfach erweiterbar bleibt.

Skripte

Im folgenden werden unsere drei selbstentwickelten Skripte zur Verwaltung und Nutzung des Golden Image vorgestellt:

create-local-installation.sh

• Komplett neue Installation des aktuellen Golden Image für eine neue/kaputte Node
• Ausführung auf einem extra Gerät (z.B. Laptop) mit SD-Karten-Slot
• Funktionsweise: erzeugt neue Partitionstabelle, erzeugt Dateisysteme, kopiert Bootloader, kopiert Inhalt des Golden Image mit rsync über NFS von der Head-Node

update-local-installation.sh

• Aktualisierung der lokalen Installation auf einer bereits installierten Node
• Ausführung auf der laufenden Compute-Node
• Funktionsweise: kopiert alle geänderten neuen Dateien mit rsync über NFS aus dem Golden Image

start-chroot.sh

• Wrapper-Skript für die Administration des Golden Image per chroot-Environment
• Ausführung auf der Head-Node, nur eine Instanz gleichzeitig möglich
• Funktionsweise: mounten aller benötigten Partitionen, starten einer bash im chroot für den Nutzer

Container

Trotz unserer Entscheidung gegen CoreOS gefiel uns die Idee der Anwendungsisolierung über Container. Im Gegensatz zu virtuellen Maschinen bringen Container weniger Performanceeinbußen mit sich, was uns bei der ohnehin schon unterdurchschnittlichen Leistung der Banana Pis besonders gelegen kam.

Docker

Da Docker momentan die beliebteste Container-Implementation darstellt, begannen wir hier mit unseren Installationsversuchen. Docker verwendet einige neue Features des Linux-Kernels, die in der offiziellen Raspbian- Distribution für den Banana Pi nicht enthalten sind. Die exzellente Arch-Linux-Distribution für ARM2 unterstützt diese out of the box, da der restliche Cluster des DKRZ allerdings komplett auf Debian-basierten Systemen läuft, wurde beschlossen, für einfachere Maintenance auch auf unserem Cluster ein solches System zu verwenden. Auch auf Raspbian soll es möglich sein, Docker zum laufen zu bringen, allerdings braucht man dort einen selbst kompilierten, neueren Kernel. Im Prinzip kein Problem - leider brauchen die Banana Pis einen speziellen, eigenen Kernel vom Hersteller LeMaker3. Der Quelltext wurde erst vor kurzem von LeMaker bereitgestellt^[20] und ist noch bei Version 3.4, während Docker mindestens Version 3.10 braucht. Nach einer Menge Recherche und einem fehlgeschlagenen Versuch, einen aktuellen Linux-Kernel auf einem Banana Pi zu kompilieren, gaben wir also neben CoreOS auch Docker auf.

rkt

rkt ist eine alternative Container-Implementierung, entwickelt vom CoreOS-Team. rkt hat zwar keine offizielle Unterstützung für die ARM-Architektur - das hinderte uns allerdings nicht daran, das Projekt selber zu kompilieren. Dabei kam dann heraus, dass rkt leider auch (noch) keine 32-bit-Systeme unterstützt, was das Projekt auf den ARMv7-Prozessoren der Banana Pis nicht lauffähig macht.

systemd-nspawn

Nach unseren Versuchen mit Docker und rkt legten wir unsere letzte Hoffnung auf das systemd-initsystem. Das hat seit einiger Zeit eine minimale Container-Implementierung, genannt systemd-nspawn.^[21] Ursprünglich für schnelle Tests von systemd selber gedacht, enthält es inzwischen die grundlegendsten Features und hätte für unsere Zwecke vollkommen gereicht. Gegen Ende unseres Projekts fingen wir damit an, systemd testweise auf einem Banana Pi zu installieren (unsere Version von Raspbian verwendete noch klassische initscripts). Die Installation endete jedoch jedes mal mit einem nicht bootenden Banana Pi, einer unbrauchbaren Raspbian-Installation und einem zeitaufwändigen neuflashen der SD-Karte. An diesem Punkt des Projekts hatten wir leider nicht mehr genug Zeit, um diesen Fehler zu beheben.

MPI

Der “Message Passing Interface”-Standard (MPI) beschreibt den Nachrichtenaustausch zwischen einzelnen parallel Prozessen, die gemeinsam an der Lösung eines Problems arbeiten. MPI legt dabei kein konkretes Protokoll oder Implementierung fest, sondern beschreibt die Semantik der verschiedenen Arten von Kommunikations-Operationen und ihre API, sodass die eigentliche Nutzung des Standards eine MPI-Implementierung benötigt. Anfänglich wollten wir OpenMPI oder MVAPICH2 nutzen, die jedoch aus verschiedenen Gründen beide nicht funktionierten:

• OpenMPI ließ sich nicht vollständig kompilieren, da es auf ARM nicht lauffähigen Assembler-Code enthielt und der Portierungsaufwand sich vermutlich nicht gelohnt hätte.
• MVAPICH2 funktionierte bis zu einem bestimmten Systemupdate einwandfrei, danach ließ sich jedoch der ”libpmi“-Header nicht mehr finden.

Aus der Not heraus entschieden wir uns daher dazu ein schon vorkompiliertes MPICH2 aus den Paketquellen unserer Distribution zu benutzen, da wir an dieser Stelle davon ausgingen, dass sich die Wahl der MPI-Implementierung für unseren Cluster wenn überhaupt nur sehr gering in der gemessenen Leistung widerspiegeln würde.

OpenBLAS

Die “Basic Linear Algebra Subprograms” (BLAS) sind eine Sammlung von Routinen für grundlegende Vektor- und Matrix-Operationen. Das von uns genutzte OpenBLAs^[23] ist eine optimierte BLAS-Bibliothek und musste zur Nutzung auf den Banana Pis neu kompiliert werden. Das Kompilieren von OpenBLAS auf dem Pi hat ca. 1 Stunde gedauert. Die besten Messergebnisse haben wir mit einer Version ohne Threading (mittels USE THREADING=0) erzielt.

HPL

Die “High-Performance Linpack”-Benchmark ist ein Programm zur Messung der Fließkomma-Rechenleistung eines (verteilten) Computer-Systems. HPL misst die Leistung des Systems in GFLOPS indem es ein dichtes ${\displaystyle n*n}$-System von linearen Gleichungen ${\displaystyle (Ax=B)}$ löst und errechnet die Leistung in GFLOPS. Wie OpenBLAS mussten wir HPL speziell für unser System kompilieren, damit es möglichst effizient an die verfügbare Hardware angepasst ist.

Messergebnisse

HPL hat vier verschiedene Hauptparameter:

• ${\displaystyle N}$ gibt die Höhe und Breite der Matrix an, das Problem wächst quadratisch mit ${\displaystyle N}$
• ${\displaystyle NB}$ ist die Nachrichtengröße bei der Kommunikation zwischen den Prozessen
• ${\displaystyle P}$ und ${\displaystyle Q}$ beschreiben die Aufteilung der Matrix auf das Prozessgitter

Figure 13.3: Messergebnisse mit HPL

Die vier Kurven zeigen das Verhältnis der gemessenen Performance mit verschiedenen vielen Compute-Nodes und Problemgrößen: Der schwache Speedup (bei gleich großem Problem pro Kern) ist in Anbetracht der verwendeten Hardware überraschend gut (ungefähr 2.8 bei 3 verwendeten Compute-Nodes) Die Entwicklung des starken Speedups (bei fester Problemgröße trotz steigender Anzahl der Kerne) zeigt jedoch, dass sich der verwendete handelsübliche Switch nicht für HPC eignet, da mit steigender Zahl an Compute-Nodes die Leistung einbricht. Leider konnten wir keine Leistungsdaten für weitere HPL-Runs mit größerem ${\displaystyle N}$ auf 4 Nodes messen, da es technische Probleme gab:

• bpi4 schaltete sich bereits nach wenigen Minuten unter Volllast ab.
• bpi-head konnte nicht als Ersatz verwendet werden, da er die gleichen Symptome zeigte.

Die Instabilität der Pis bei großer Hitze stellt ein großes Problem für einen tatsächlich intensiv genutzten Pi-Cluster dar.

SLURM

SLURM^[24] steht für Simple Linux Utility for Resource Management und ist ein Open Source Workload Manager, der auf vielen Clustern und Supercomputern rund um die Welt verwendet wird. SLURM wird dazu verwendet, einzelne Jobs möglichst effizient auf die Knoten des Clusters zu verteilen. Obwohl wir uns sehr bemüht haben, gab es im Laufe des Projekt wesentlich mehr Komplikationen als wir erwartet haben und so fehlte uns am Ende die Zeit dafür, SLURM komplett in den Cluster zu integrieren.

Aktueller Stand

Zum jetzigen Zeitpunkt haben wir einen zwar nicht optimalen aber funktionierenden Cluster. Wir benutzen ein Golden Image, mit dessen Hilfe es ein leichtes ist, alle Knoten auf den selben Stand zu bringen, und durch welches der Cluster einfach um zusätzliche Knoten erweiterbar ist. Die Netzwerkeinbindung ist noch suboptimal, da Compute-Nodes in manchen fällen eine IP aus dem Internet, statt von der Head-Node beziehen, wo durch sie von dieser nicht ansprechbar sind. Auf der positiven Seite, kann der Cluster mit nutzbaren MPI-Libraries, sowie lauffähigem HPL aufwarten, womit ausführliche Leistungstests möglich sind. Die Banana Pis befinden sich in einem eigens dafür 3D-gedrucktem Case, welches leider ein bisschen klein geraten ist.

Weiterführende Arbeit

Die Lösung Netzwerk-Problematik wären statische IPs, sodass keine Knoten mehr eigenwillig auf IP-Suche gehen können. Für eine möglichst vollständige Integration in den WR-Cluster müssten nur noch ein paar wenige Softwarepakete hinzugefügt werden:

• miit SLURM bleibt das (vom WR-Cluster) gewohnte UI zur Nutzung des Clusters erhalten.
• auf dem WR-Cluster wird LDAP zum Zugriff auf die Home-Verzeichnisse der Nutzer verwendet und wird daher zwingend auf dem Pi-Cluster benötigt um die Problematik mehrerer Home-Verzeichnisse pro Nutzer zu umgehen.
• das Monitoring-Tool Ganglia ist zwar im allgemeinen optional, sollte aber auch vorhanden sein, da es wie SLURM ebenfalls zur gewohnten UI gehört.

Außerdem fehlt für den Einbau in diesen Cluster ein Plexiglas-Gehäuse in dem die ARM-Computer sowie das gesamte Zubehör, einige LEDs und Lüfter Platz finden.

Der Banana Pi

Nach einiger Erfahrung mit dem Modell “Banana Pi” möchten wir hier einen kurzen Überblick über Vor- und Nachteile des Modells geben. Da wir aus der Perspektive des HPC an dieses Projekt gegangen sind, waren für uns die Hardware-Aspekte bei der Planung besonders wichtig. Im Besonderen versprach der Banana Pi durch das Gigabit-Ethernet eine problemfreie Kommunikation zwischen den Compute Nodes, auch mit den für HPC üblichen großen Datenmengen. Tatsächlich konnten wir in unseren Benchmarks auch keine Probleme mit der Kommunikationslast erkennen. Zusätzlich ergaben unsere Vergleiche mit einem Raspberry Pi 2 Modell B gerade mal einen Performanceunterschied von etwa 300 MFLOPS. Schnell mussten wir jedoch feststellen, dass die eher kleine Community undd as Ökosystem um den Banana Pi einige Probleme mit sich bringen. Zum einen mussten wir feststellen, dass es tatsächlich verschiedene Hersteller gibt, die alle von sich behaupten, den “offiziellen” Banana Pi zu fabrizieren. Mit jedem dieser Hersteller kam eine leicht andere Dokumentation mit vielen fehlenden Stellen. SD-Karten-Images für den Raspberry Pi funktionieren mit dem Banana Pi nicht; Es bleibt unklar, ob es eine tatsächlich offizielle Seite mit offiziellen Linux-Distributionen gibt. Lange Zeit war der modifizierte Linux-Kernel für den Banana Pi nicht quelloffen; Nach viel Druck aus der Community wurde der Code dann doch veröffentlicht.^[25] Leider ist das letzte stabile Release dieses Kernels noch bei Version 3.4. Das machte unter Anderem das kompilieren eines eigenen Kernels für die Docker-Installation unmöglich.

Andere Vor- und Nachteile

Eine der Fragen, die wir mit unserem Projekt beantworten wollten, war die Frage nach der Stromeffizienz der Banana Pis, insbesondere im Kontext des High Performance Computing. Leider enttäuscht der ARM-Cluster in dieser Hinsicht. Wenn wir eher konservativ ^[26] von einem Verbrauch von 5 Watt (unter voller Last) ausgehen, entspräche das bei unseren Benchmarks ungefähr ${\displaystyle 1/4}$ GFLOPS pro Watt. Laut der TOP500-Liste sind Werte im Bereich von 2 GFLOPS pro Watt der aktuelle Stand der Technik.^[27] Trotz des geringen Preis eignet sich ein Cluster aus Banana Pis nichtmal gut für das Testen von Programmen in kleinem Maßstab, da die ARM-CPU nicht alle Befehlssätze einer x86(64)-CPU unterstützt und daher für Probleme bei der Kompilierung sorgt - die Nutzung eines virtualisierten Clusters (z.B. via Vagrant) ist deutlich komfortabler. Im Allgemeinen kann man also sagen, dass sich ein ARM-Cluster unserer Erfahrung nach hauptsächlich lohnt, um in einem möglichst realistischen Szenario Aufbau und Installation eines Clusters zu üben. Die Skripte zur Verwaltung der verschiedenen Nodes funktionieren exzellent und können ohne Probleme in jedem Projekt, das mit mehreren Pi-artigen Computern arbeitet, zur einfachen Administration verwendet werden.

create-local-installation.sh

#!/bin/bash
# JG/RE/LB 2015

if [[ $# -ne 4 ]]; then
        echo "Usage: $0 bpi_head_ip dist device hostname"
        exit 1
fi

if [ "$USER" != "root" ]; then
        echo "MUST be run as root! Try 'sudo $0 ...'"
    exit 1 
fi

BPI_HEAD_IP="$1"
DIST="$2"
DEV="$3"
HOSTNAME="$4"

# sd card formatting
echo "Partitioning $DEV"
parted -s $DEV mklabel msdos || exit 1
parted -s $DEV unit s mkpart primary fat32 8192s 122879s || exit 1
parted -s -- $DEV unit s mkpart primary 122880s -1s || exit 1
echo "Report:"
fdisk -l $DEV

echo "Filesystems:"
mkfs.vfat -F 32 ${DEV}p1 || exit 1
mkfs.ext4 ${DEV}p2 || exit 1

echo "Bootloader"
dd if="$(dirname $0)"/bootloader_ohne_table.img count=2048 of=/dev/mmcblk0 seek=8 bs=1024

# mounting
mkdir /tmp/target/{b,r}oot -p
mount ${DEV}p1 /tmp/target/boot
mount ${DEV}p2 /tmp/target/root

# rsync magic
rsync -axzh --stats root@$BPI_HEAD_IP:/srv/nodes/$DIST/boot/ /tmp/target/boot || exit 1
rsync -axzh --stats root@$BPI_HEAD_IP:/srv/nodes/$DIST/root/ /tmp/target/root || exit 1

echo $HOSTNAME > /tmp/target/root/etc/hostname

# Clean up
umount /tmp/target/*

update-local-installation.sh

#!/bin/bash
# JG/RE/LB 2015

test -z "$1" && echo "Please specify a distribution." && exit 1

DIST="$1"

#####
# mounting the source, this could be done outside of this script to use arbitrary sources.
mkdir -p /tmp/source_root
umount /tmp/source_root
mount -o ro,nfsvers=3 bpi-head:/srv/nodes/$DIST/root /tmp/source_root || exit 1

mkdir -p /tmp/source_boot
umount /tmp/source_boot
mount -o ro,nfsvers=3 bpi-head:/srv/nodes/$DIST/boot /tmp/source_boot || exit 1

####
# Sanity checks to check if the ip is 10.0.0.X where X below 100, then the script is allowed to run.

IP=$(ip addr show dev eth0 |grep "inet " |sed "s/.* inet \([0-9.]*\)\/.*/\1/")

if [[ ${#IP} -lt 8 ]] ; then
        echo "Could not determine IP!"
        exit 1
fi

LAST=${IP#10.0.0.}
if [[ $IP == $LAST || $LAST -gt 99 ]] ; then
        echo "Invalid host with IP: $IP"
        echo "I won't run the script on this machine!"
        exit 1
fi

rsync -axzh --delete --exclude="/home" --ignore-errors --progress /tmp/source_root/ /
rsync -axzh --delete --exclude="/home" --ignore-errors --progress /tmp/source_boot/boot

hostname > /etc/hostname

# cleanup and restoring the previous state
umount /tmp/source_root
umount /tmp/source_boot

start-chroot.sh

#!/bin/bash
# JG/RE/LB 2015

(
flock -n 200
CHROOTDIR="/srv/nodes/default/root"

usage ()
{
    echo "USAGE: ${0} [-h|--help] [<DIR>]"
    echo
    echo "-h"
    echo "  --help     print this message"
    echo "<DIR>  directory with common file system"
    echo
}

if [[ "$1" != "" ]] ; then
        CHROOTDIR="/srv/nodes/${1}/root"
fi
[ -d "${CHROOTDIR}" ] || \
        { echo "Directory for chroot ${CHROOTDIR} not found!" && exit 1; }

echo "Starting chroot environment in ${CHROOTDIR}"

# mount dev
[ -d ${CHROOTDIR}/dev ] &&
mount -o bind /dev ${CHROOTDIR}/dev

# mount dev/pts
[ -d ${CHROOTDIR}/dev/pts ] &&
mount -o bind /dev/pts ${CHROOTDIR}/dev/pts

# mount /run for resolv.conf
[ -d ${CHROOTDIR}/run ] &&
mount -o bind /run ${CHROOTDIR}/run

# mount boot 'partition'
mount -o bind ${CHROOTDIR}/../boot ${CHROOTDIR}/boot

# mount proc
[ -d ${CHROOTDIR}/proc ] &&
mount -t proc proc_chroot ${CHROOTDIR}/proc

# mount sysfs
[ -d ${CHROOTDIR}/sys ] &&
mount -t sysfs sysfs_chroot ${CHROOTDIR}/sys

#JK:
#sed -i "s/10.0.0.250/129.206.100.126/" ${CHROOTDIR}/etc/resolv.conf if [ -f ${CHROOTDIR}/usr/sbin/invoke-rc.d ]
then
        echo '#!/bin/sh' > ${CHROOTDIR}/usr/sbin/policy-rc.d
        echo 'exit 101' >> ${CHROOTDIR}/usr/sbin/policy-rc.d
        chmod +x ${CHROOTDIR}/usr/sbin/policy-rc.d
fi

/usr/sbin/chroot ${CHROOTDIR} /bin/bash

# YOU ARE IN CHROOT HERE AND THIS SCRIPT IS STOPPED UNTIL YOU QUIT BASH!

echo "Closing chroot environment!"

[ -f ${CHROOTDIR}/usr/sbin/policy-rc.d ] &&
rm ${CHROOTDIR}/usr/sbin/policy-rc.d

# umount sysfs
mountpoint -q ${CHROOTDIR}/sys && umount sysfs_chroot

# umount proc
mountpoint -q ${CHROOTDIR}/proc && umount proc_chroot

# umount boot 'partition'
# 'mountpoint' doesn't work here for some reason umount ${CHROOTDIR}/boot

# umount dev/pts
mountpoint -q ${CHROOTDIR}/dev/pts && umount ${CHROOTDIR}/dev/pts

# umount /run
mountpoint -q ${CHROOTDIR}/run && umount ${CHROOTDIR}/run

# umount dev/pts
mountpoint -q ${CHROOTDIR}/dev && umount ${CHROOTDIR}/dev
) 200>/var/lock/start-chroot.sh

SLURM Konfiguration

ControlMachine=bpi-head ControlAddr=10.0.0.250
#
MailProg=/usr/bin/mail
MpiDefault=none
#MpiParams=ports=#-# 
ProctrackType=proctrack/cgroup
ReturnToService=1 
SlurmctldPidFile=/var/run/slurmctld.pid 
#SlurmctldPort=6817 
SlurmdPidFile=/var/run/slurmd.pid 
#SlurmdPort=6818 
SlurmdSpoolDir=/var/spool/slurmd 
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/spool/slurm 
SwitchType=switch/none 
TaskPlugin=task/none
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
#
# SCHEDULING
FastSchedule=1
SchedulerType=sched/backfill
#SchedulerPort=7321
SelectType=select/linear
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=pi-cluster
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
#SlurmctldDebug=3
SlurmctldLogFile=/etc/slurm-llnl/slurmctldLog.log
#SlurmdDebug=3
SlurmdLogFile=/etc/slurm-llnl/slurmdLog.log
#
# COMPUTE NODES
NodeName=bpi[1-4] CPUs=4 State=UNKNOWN
PartitionName=debug Nodes=bpi[1-4] Default=YES MaxTime=INFINITE State=UP