Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
ebglucose 1614 0 0.04 41.7475 1200 720.336 1200 1201 547.994 0.760748
ebminisat 1614 0 0 36.3375 1086.5 675.155 1200 1201 551.183 0.81638
glucose2 1614 0 0.04 28.615 1200 678.497 1200 1201 553.755 0.81615
glueminisat 1614 0 0 32.255 1200 698.474 1200 1201 557.054 0.79753
lingeling 1614 0 0 35.6775 1065.75 667.019 1200 1201 556.459 0.834248
lrglshr 1614 0 0 51.2975 1200 736.569 1200 1201 550.09 0.746827
minisatpsm 1614 0 0.03 34.82 1200 706.185 1200 1201 553.969 0.784454
mphaseSAT64 1614 0 0 7.745 105.94 446.75 1200 1201 520.87 1.16591
precosat 1614 0 0 26.5325 1200 673.872 1200 1201 558.832 0.829284
qutersat 1614 0 0 46.62 890.785 666.37 1200 1201 547.648 0.821838
rcl 1614 0 0.04 44.8425 1200 749.209 1200 1201 545.078 0.727539
restartsat 1614 0 0 43.2575 1102.25 680.226 1200 1201 548.194 0.8059
cryptominisat2011 1614 0 0 45.2775 892.205 668.57 1200 1201 545.228 0.815515
spear.sw 1614 0 0 336.298 1200 884.227 1200 1201 495.879 0.560805
spear.hw 1614 0 0 169.235 1200 806.39 1200 1201 517.925 0.642275
eagleup 1614 0 0 417.925 1200 902.13 1200 1200 508.803 0.564002
sparrow 1614 0 0 18.335 1200 784.524 1200 1200 553.642 0.705705
marchrw 1614 0 0 358.07 1200 882.262 1200 1201 500.572 0.567374
mphaseSATm 1614 0 0 5.495 84.93 409.998 1167.88 1201 506.556 1.23551
satime11 1614 0 0 36.0075 1200 794.86 1200 1200 540.925 0.68053
tnm 1614 0 0 45.735 1200 822.444 1200 1200 539.227 0.655639
mxc09 1614 0 0 32.3925 845.27 657.87 1200 1201 549.902 0.835882
gnoveltyp2 1614 0 0 187.952 1200 870.547 1200 1200 518.818 0.595968
sattime 1614 0 0 45.1625 1200 831.131 1200 1200 538.143 0.647483
sattimep 1614 0 0 174.925 1200 870.62 1200 1201 519.307 0.59648
clasp2 1614 0 0 41.9825 734.415 649.774 1200 1201 544.044 0.837281
clasp1 1614 0 0 40.47 842.32 664.355 1200 1201 543.385 0.817914
picosat 1614 0 0 42.9175 1200 691.522 1200 1201 545.962 0.789508
mphaseSAT 1614 0 0 5.4175 95.235 430.679 1200 1201 516.632 1.19958
sapperlot 1614 0 0 48.36 1200 705.47 1200 1201 542.694 0.769266
sol 1614 0 0.06 114.693 1200 799.974 1200 1200 523.042 0.653824

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
clasp1 53.470 46.530 0.000 0.000 0.000 0.000
clasp2 54.399 45.601 0.000 0.000 0.000 0.000
cryptominisat2011 51.797 48.203 0.000 0.000 0.000 0.000
eagleup 26.022 73.978 0.000 0.000 0.000 0.000
ebglucose 46.097 53.903 0.000 0.000 0.000 0.000
ebminisat 50.805 49.195 0.000 0.000 0.000 0.000
glucose2 49.628 50.372 0.000 0.000 0.000 0.000
glueminisat 47.026 52.974 0.000 0.000 0.000 0.000
gnoveltyp2 29.988 70.012 0.000 0.000 0.000 0.000
lingeling 50.248 49.752 0.000 0.000 0.000 0.000
lrglshr 44.052 55.948 0.000 0.000 0.000 0.000
marchrw 30.855 69.145 0.000 0.000 0.000 0.000
minisatpsm 46.716 53.284 0.000 0.000 0.000 0.000
mphaseSAT 72.862 27.138 0.000 0.000 0.000 0.000
mphaseSAT64 71.933 28.067 0.000 0.000 0.000 0.000
mphaseSATm 75.279 24.721 0.000 0.000 0.000 0.000
mxc09 53.098 46.902 0.000 0.000 0.000 0.000
picosat 49.628 50.372 0.000 0.000 0.000 0.000
precosat 49.876 50.124 0.000 0.000 0.000 0.000
qutersat 51.115 48.885 0.000 0.000 0.000 0.000
rcl 43.185 56.815 0.000 0.000 0.000 0.000
restartsat 50.558 49.442 0.000 0.000 0.000 0.000
sapperlot 49.009 50.991 0.000 0.000 0.000 0.000
satime11 37.794 62.206 0.000 0.000 0.000 0.000
sattime 33.147 66.853 0.000 0.000 0.000 0.000
sattimep 29.678 70.322 0.000 0.000 0.000 0.000
sol 40.087 59.913 0.000 0.000 0.000 0.000
sparrow 37.237 62.763 0.000 0.000 0.000 0.000
spear.hw 39.591 60.409 0.000 0.000 0.000 0.000
spear.sw 30.979 69.021 0.000 0.000 0.000 0.000
tnm 34.201 65.799 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-4

Estimated densitities of performance values


Performance values with imputation.
plot of chunk unnamed-chunk-5

Performance values without imputation.
plot of chunk unnamed-chunk-6

Estimated cumulative distribution functions of performance values


Performance values without imputation.
plot of chunk unnamed-chunk-7

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Performance values with imputation.
plot of chunk unnamed-chunk-8

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-9