Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
akmaxsat_ls 876 0 0 2.275 2100 1247.79 2100 2100 1002.22 0.803192
akmaxsat 876 0 0 2.24 2100 1247.8 2100 2100 1002.22 0.803194
DSWPM1_924 876 0 0 12.295 2100 1200.4 2100 2100 995.552 0.829352
pwbo2.1 876 0 0 1.825 53.985 718.23 2100 2100 957.566 1.33323
qmaxsat0.21comp 876 0 0 1.105 22.92 574.664 1334.94 2100 888.381 1.54591
qmaxsat0.21g2comp 876 0 0 0.9175 15.65 534.922 564.725 2100 873.368 1.6327

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
akmaxsat 42.808 57.192 0.000 0.000 0.000 0.000
akmaxsat_ls 42.808 57.192 0.000 0.000 0.000 0.000
DSWPM1_924 46.575 53.425 0.000 0.000 0.000 0.000
pwbo2.1 67.922 32.078 0.000 0.000 0.000 0.000
qmaxsat0.21comp 75.799 24.201 0.000 0.000 0.000 0.000
qmaxsat0.21g2comp 76.941 23.059 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-4

Estimated densitities of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-5

Performance values without imputation.
plot of chunk unnamed-chunk-6

Estimated cumulative distribution functions of performance values


Performance values without imputation.
plot of chunk unnamed-chunk-7

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Performance values with imputation.
plot of chunk unnamed-chunk-8

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-9