Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
MPhaseSAT_2011.02.15 296 0 0.001999 95.4812 5000 3074.28 5000 5000 2301.06 0.748487
Sol_2011.04.04 296 0 0.06199 36.6904 5000 3195.55 5000 5000 2329.74 0.72906
QuteRSat_2011.05.12..fixed.. 296 0 0.033994 357.553 5000 3368.53 5000 5000 2230.9 0.662277
CryptoMiniSat_Strange.Night2.st..fixed.. 296 0 0.001998 328.722 5000 3347.71 5000 5000 2241.35 0.669517
PicoSAT_941 296 0 0.000999 398.757 5000 3318.98 5000 5000 2214.13 0.667112
glucose_2 296 0 0.006998 339.607 5000 3266.98 5000 5000 2213.57 0.677558
clasp_2.0.R4092.crafted 296 0 0.002998 176.913 5000 2997.07 5000 5000 2227.31 0.743164
SAT07referencesolverminisat_SAT2007 296 0 0.002999 352.099 5000 3326.16 5000 5000 2206.4 0.663347
jMiniSat_2011 296 0 0.12498 816.456 5000 3542.41 5000 5000 2169.38 0.612403
RestartSAT_B95 296 0 0.001999 357.771 5000 3353.7 5000 5000 2239.18 0.667676
SAT09referencesolverclasp_1.2.0.SAT09.32 296 0 0.003999 343.978 4957.31 3089.27 5000 5000 2166.04 0.70115
sathys_2011.04.01 296 0 0.008997 350.969 5000 3486.45 5000 5000 2241.23 0.64284
SApperloT2010_2011.05.15..fixed.. 296 0 0.001999 342.034 5000 3370.59 5000 5000 2236.34 0.663485
sattime._2011.03.02 296 0 0.000999 51.6829 5000 3299.18 5000 5000 2336.21 0.708118
sattime_2011.03.02 296 0 0.001999 17.0434 5000 3269.91 5000 5000 2341.4 0.716044

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
clasp_2.0.R4092.crafted 49.662 50.338 0.000 0.000 0.000 0.000
CryptoMiniSat_Strange.Night2.st..fixed.. 36.824 63.176 0.000 0.000 0.000 0.000
glucose_2 41.554 58.446 0.000 0.000 0.000 0.000
jMiniSat_2011 32.770 67.230 0.000 0.000 0.000 0.000
MPhaseSAT_2011.02.15 44.257 55.743 0.000 0.000 0.000 0.000
PicoSAT_941 40.541 59.459 0.000 0.000 0.000 0.000
QuteRSat_2011.05.12..fixed.. 36.824 63.176 0.000 0.000 0.000 0.000
RestartSAT_B95 37.500 62.500 0.000 0.000 0.000 0.000
SApperloT2010_2011.05.15..fixed.. 36.486 63.514 0.000 0.000 0.000 0.000
SAT07referencesolverminisat_SAT2007 40.878 59.122 0.000 0.000 0.000 0.000
SAT09referencesolverclasp_1.2.0.SAT09.32 50.000 50.000 0.000 0.000 0.000 0.000
sathys_2011.04.01 32.095 67.905 0.000 0.000 0.000 0.000
sattime_2011.03.02 36.149 63.851 0.000 0.000 0.000 0.000
sattime._2011.03.02 35.135 64.865 0.000 0.000 0.000 0.000
Sol_2011.04.04 38.851 61.149 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-4

Estimated densitities of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-5

Performance values without imputation.
plot of chunk unnamed-chunk-6

Estimated cumulative distribution functions of performance values


Performance values without imputation.
plot of chunk unnamed-chunk-7

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Performance values with imputation.
plot of chunk unnamed-chunk-8

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-9