Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
ebglucose 767 0 0.04 191.37 1200 841.976 1200 1200 513.832 0.610269
ebminisat 767 0 0 109.32 1200 805.473 1200 1200 529.119 0.656904
glucose2 767 0 0.04 102.395 1200 800.815 1200 1201 531.365 0.66353
glueminisat 767 0 0 92.84 1200 816.648 1200 1200 532.494 0.652048
lingeling 767 0 0 88.755 1200 810.449 1200 1200 533.204 0.657912
lrglshr 767 0 0 169.73 1200 862.039 1200 1200 512.79 0.594857
minisatpsm 767 0 0.03 146.405 1200 831.399 1200 1200 520.744 0.626346
mphaseSAT64 767 0 0 40.635 1200 720.16 1200 1200 551.389 0.765648
precosat 767 0 0 31.35 1200 779.531 1200 1200 547.94 0.702909
qutersat 767 0 0.03 118.905 1200 803.079 1200 1200 531.793 0.662193
rcl 767 0 0.04 295.275 1200 869.775 1200 1200 505.399 0.581068
restartsat 767 0 0 101.74 1200 810.717 1200 1200 522.2 0.644121
cryptominisat2011 767 0 0 94.14 1200 798.821 1200 1200 531.537 0.665401
spear.sw 767 0 0 1200 1200 978.954 1200 1200 440.77 0.450246
spear.hw 767 0 0 564.965 1200 912.923 1200 1200 478.859 0.524534
eagleup 767 0 0 1200 1200 1120.09 1200 1200 296.875 0.265045
sparrow 767 0 0 1200 1200 942.238 1200 1200 477.33 0.506591
marchrw 767 0 0 1200 1200 950.55 1200 1200 464.611 0.488781
mphaseSATm 767 0 0 32.075 1200 698.976 1200 1200 559.036 0.799793
satime11 767 0 0 1200 1200 931.074 1200 1200 483.076 0.518838
tnm 767 0 0 1200 1200 956.811 1200 1200 467.89 0.48901
mxc09 767 0 0 98.485 1200 792.439 1200 1200 530.81 0.669844
gnoveltyp2 767 0 0 1200 1200 1018.9 1200 1200 416.356 0.408634
sattime 767 0 0 1200 1200 933.532 1200 1200 484.989 0.519521
sattimep 767 0 0 1200 1200 931.69 1200 1200 484.728 0.520267
clasp2 767 0 0 60.48 1200 744.853 1200 1200 540.88 0.726156
clasp1 767 0 0 79.515 1200 775.364 1200 1201 536 0.691288
picosat 767 0 0 148.41 1200 824.398 1200 1200 519.805 0.630526
mphaseSAT 767 0 0 35.165 1200 692.487 1200 1200 559.788 0.808374
sapperlot 767 0 0 206.035 1200 839.063 1200 1200 510.469 0.60838
sol 767 0 0.06 86.315 1200 818.74 1200 1200 533.722 0.651883

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
clasp1 41.460 58.540 0.000 0.000 0.000 0.000
clasp2 44.459 55.541 0.000 0.000 0.000 0.000
cryptominisat2011 38.331 61.669 0.000 0.000 0.000 0.000
eagleup 6.910 93.090 0.000 0.000 0.000 0.000
ebglucose 34.811 65.189 0.000 0.000 0.000 0.000
ebminisat 38.201 61.799 0.000 0.000 0.000 0.000
glucose2 38.201 61.799 0.000 0.000 0.000 0.000
glueminisat 36.245 63.755 0.000 0.000 0.000 0.000
gnoveltyp2 16.688 83.312 0.000 0.000 0.000 0.000
lingeling 36.636 63.364 0.000 0.000 0.000 0.000
lrglshr 32.595 67.405 0.000 0.000 0.000 0.000
marchrw 24.511 75.489 0.000 0.000 0.000 0.000
minisatpsm 35.593 64.407 0.000 0.000 0.000 0.000
mphaseSAT 47.718 52.282 0.000 0.000 0.000 0.000
mphaseSAT64 45.763 54.237 0.000 0.000 0.000 0.000
mphaseSATm 47.458 52.542 0.000 0.000 0.000 0.000
mxc09 40.287 59.713 0.000 0.000 0.000 0.000
picosat 37.158 62.842 0.000 0.000 0.000 0.000
precosat 39.635 60.365 0.000 0.000 0.000 0.000
qutersat 37.679 62.321 0.000 0.000 0.000 0.000
rcl 32.203 67.797 0.000 0.000 0.000 0.000
restartsat 38.853 61.147 0.000 0.000 0.000 0.000
sapperlot 36.375 63.625 0.000 0.000 0.000 0.000
satime11 24.772 75.228 0.000 0.000 0.000 0.000
sattime 24.120 75.880 0.000 0.000 0.000 0.000
sattimep 24.381 75.619 0.000 0.000 0.000 0.000
sol 35.463 64.537 0.000 0.000 0.000 0.000
sparrow 23.729 76.271 0.000 0.000 0.000 0.000
spear.hw 28.683 71.317 0.000 0.000 0.000 0.000
spear.sw 22.164 77.836 0.000 0.000 0.000 0.000
tnm 22.295 77.705 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-4

Estimated densitities of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-5

Performance values without imputation.
plot of chunk unnamed-chunk-6

Estimated cumulative distribution functions of performance values


Performance values without imputation.
plot of chunk unnamed-chunk-7

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Performance values with imputation.
plot of chunk unnamed-chunk-8

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-9