ROSE uses Jenkins to implement a contiguous integration software development process. It leverages a range of software packages to test its correctness, robustness, and performance.
make check rules
we leverage make check rules to do internal testing.
check exist status of pipeline command
In bash scripting, we can use pipelines | as follows:
- command1 | command2 : the output of each command in the pipeline is connected to the input of the next command
each command is executed in its own subshell, exit status: the last command's exit status
To catch any command's return code, please use ${PIPESTATUS[0]}
For example: Using pipeline will only return the last command 'folds status. we add a test to catch the first command's return status
../autoPar -c $(srcdir)/$(@:.o=.c) | fold >$(@:.o=.out); test ${PIPESTATUS[0]} = 0
Benchmarks
The software used by the ROSE's Jenkins include:
- SPEC CPU 2006 benchmark: a subset is supported for now
- SPEC OMP benchmark: a subset is supported for now
- NAS parallel benchmark: developed by NASA Ames Research Center. Both C (customized version) and OpenMP versions are used
- Plum Hall C and C++ Validation Test Suites: a subset is supported for now
- Jt++ - Java conformance testing: http://modena.us/
Modena Jt++ Test Suite
1. Clone the Modena test suite repository:
$ git clone ssh://rose-dev@rose-git/modena
2. Autotools setup
$ cd modena $ ./build.sh + libtoolize --force --copy --ltdl --automake + aclocal -I ./acmacros -I ./acmacros/ac-archive -I /usr/share/aclocal + autoconf + automake -a -c configure.ac:4: installing `./install-sh' configure.ac:4: installing `./missing'
3. Environment bootstrap
$ source /nfs/apps/python/latest/setup.sh
4. Build and test!
$ mkdir buildTree
$ cd buildTree
$ ../configure \
--with-sqlalchemy=${HOME}/opt/python/sqlalchemy/0.7.5/lib64/python2.4/site-packages \
--with-target-java-interpreter=java \
--with-target-java-compiler=testTranslator \
--with-target-java-compiler-flags="-ecj:1.6" \
--with-host-java-compiler-flags="-source 1.6"
Jenkins
Using External Benchmarks
The way we set it up is to
- In the benchmark, we change the benchmark's build system to call the ROSE tool (identityTranslator or your RTED tool) installed.
- In the Jenkins test job,
- Build and install the tested ROSE, prepare environment variables.
- Go to the benchmark with modified build system. Build and run the benchmark.
Basically, the test job should simulate how a ROSE tool would be used by end-users, not by tweaking ROSE for each different benchmarks.
NAS Parallel Benchmarks
We have NPB as part of our regression tests.
To get the benchmark:
git clone rose-dev@rosecompiler1.llnl.gov:testsuite/npb-c-parallel.git
Within the benchmark, there is a make.def file with configuration about which compiler and options to use. The configuration should be correct but some path may be changed to point to your version of ROSE.
After that, type "make suite" to build the entire benchmark suite. Or type "make mg class=A" to just build the benchmark in question.