Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.853	3127.236	0.000
baseline	singleBest	0.428	12057.041	903.778
baseline	singleBestByPar	0.769	4893.141	190.904
baseline	singleBestBySuccesses	0.769	4893.141	190.904
classif	meta/AdaBoostM1	0.805	4130.846	97.338
classif	bayes/BayesNet	0.816	3888.785	71.030
classif	lazy/IBk	0.826	3670.967	47.390
classif	rules/OneR	0.807	4092.553	102.196
classif	trees/RandomTree	0.824	3724.295	57.568
classif	trees/J48	0.821	3792.070	60.617
classif	rules/JRip	0.808	4059.149	90.367
classif	classif.ctree	0.811	4008.208	82.577
classif	classif.ksvm	0.813	3960.941	78.460
classif	classif.naiveBayes	0.689	6575.873	363.255
classif	classif.randomForest	0.822	3767.973	58.095
classif	classif.rpart	0.807	4080.946	90.589
regr	regr.lm	0.822	3777.872	67.994
regr	regr.rpart	0.821	3812.837	81.384
regr	regr.randomForest	0.842	3345.319	23.798
regr	regr.earth	0.828	3663.769	61.768
cluster	EM	0.805	4133.977	100.469
cluster	FarthestFirst	0.769	4893.244	190.901
cluster	SimpleKMeans	0.806	4118.855	106.923

The following default feature steps were used for model building:

group_basics

Number of presolved instances: 0

The cost for using the feature steps (adapted for presolving) is: 132.54 or on average: 0.1513014

The feature steps correspond to the following 37 / 37 features:

numVars, numClauses, perc_soft, soft_mean, soft_std,
soft_min, soft_max, var_clauses_ratio, vcg_var_mean, vcg_var_std,
vcg_var_min, vcg_var_max, vcg_var_spread, vcg_cls_mean, vcg_cls_std,
vcg_cls_min, vcg_cls_max, vcg_cls_spread, pnr_var_mean, pnr_var_std,
pnr_var_min, pnr_var_max, pnr_var_spread, pnr_cls_mean, pnr_cls_std,
pnr_cls_min, pnr_cls_max, pnr_cls_spread, unary, binary,
trinary, horn_mean, horn_std, horn_min, horn_max,
horn_spread, horn,