Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.988	241.318	0.000
baseline	singleBest	0.372	7562.963	677.035
baseline	singleBestByPar	0.753	3079.886	302.509
baseline	singleBestBySuccesses	0.753	3079.886	302.509
classif	meta/AdaBoostM1	0.665	4105.794	338.401
classif	bayes/BayesNet	0.805	2440.883	192.448
classif	lazy/IBk	0.878	1585.068	119.534
classif	rules/OneR	0.706	3629.007	303.249
classif	trees/RandomTree	0.871	1674.075	128.243
classif	trees/J48	0.861	1789.989	137.093
classif	rules/JRip	0.695	3742.276	302.763
classif	classif.ctree	0.828	2172.054	171.204
classif	classif.ksvm	0.874	1647.655	135.280
classif	classif.naiveBayes	0.766	2920.918	244.231
classif	classif.randomForest	0.921	1064.053	66.919
classif	classif.rpart	0.787	2657.716	215.229
regr	regr.lm	0.880	1572.421	126.961
regr	regr.rpart	0.836	2088.158	174.296
regr	regr.randomForest	0.926	1006.477	62.875
regr	regr.earth	0.885	1510.379	118.450
cluster	EM	0.631	4532.084	396.661
cluster	FarthestFirst	0.372	7576.431	643.982
cluster	SimpleKMeans	0.436	6839.792	596.562

The following default feature steps were used for model building:

Pre, Basic, KLB, CG, DIAMETER, cl, sp, ls_saps, ls_gsat, lobjois

Number of presolved instances: 20

The cost for using the feature steps (adapted for presolving) is: 8.560377 × 10⁴ or on average: 53.0382714

The feature steps correspond to the following 115 / 115 features:

nvarsOrig, nclausesOrig, nvars, nclauses, reducedVars,
reducedClauses, vars_clauses_ratio, POSNEG_RATIO_CLAUSE_mean, POSNEG_RATIO_CLAUSE_coeff_variation, POSNEG_RATIO_CLAUSE_min,
POSNEG_RATIO_CLAUSE_max, POSNEG_RATIO_CLAUSE_entropy, VCG_CLAUSE_mean, VCG_CLAUSE_coeff_variation, VCG_CLAUSE_min,
VCG_CLAUSE_max, VCG_CLAUSE_entropy, UNARY, BINARYp, TRINARYp,
VCG_VAR_mean, VCG_VAR_coeff_variation, VCG_VAR_min, VCG_VAR_max, VCG_VAR_entropy,
POSNEG_RATIO_VAR_mean, POSNEG_RATIO_VAR_stdev, POSNEG_RATIO_VAR_min, POSNEG_RATIO_VAR_max, POSNEG_RATIO_VAR_entropy,
HORNY_VAR_mean, HORNY_VAR_coeff_variation, HORNY_VAR_min, HORNY_VAR_max, HORNY_VAR_entropy,
horn_clauses_fraction, VG_mean, VG_coeff_variation, VG_min, VG_max,
CG_mean, CG_coeff_variation, CG_min, CG_max, CG_entropy,
cluster_coeff_mean, cluster_coeff_coeff_variation, cluster_coeff_min, cluster_coeff_max, cluster_coeff_entropy,
DIAMETER_mean, DIAMETER_coeff_variation, DIAMETER_min, DIAMETER_max, DIAMETER_entropy,
cl_num_mean, cl_num_coeff_variation, cl_num_min, cl_num_max, cl_num_q90,
cl_num_q10, cl_num_q75, cl_num_q25, cl_num_q50, cl_size_mean,
cl_size_coeff_variation, cl_size_min, cl_size_max, cl_size_q90, cl_size_q10,
cl_size_q75, cl_size_q25, cl_size_q50, SP_bias_mean, SP_bias_coeff_variation,
SP_bias_min, SP_bias_max, SP_bias_q90, SP_bias_q10, SP_bias_q75,
SP_bias_q25, SP_bias_q50, SP_unconstraint_mean, SP_unconstraint_coeff_variation, SP_unconstraint_min,
SP_unconstraint_max, SP_unconstraint_q90, SP_unconstraint_q10, SP_unconstraint_q75, SP_unconstraint_q25,
SP_unconstraint_q50, saps_BestSolution_Mean, saps_BestSolution_CoeffVariance, saps_FirstLocalMinStep_Mean, saps_FirstLocalMinStep_CoeffVariance,
saps_FirstLocalMinStep_Median, saps_FirstLocalMinStep_Q10, saps_FirstLocalMinStep_Q90, saps_BestAvgImprovement_Mean, saps_BestAvgImprovement_CoeffVariance,
saps_FirstLocalMinRatio_Mean, saps_FirstLocalMinRatio_CoeffVariance, gsat_BestSolution_Mean, gsat_BestSolution_CoeffVariance, gsat_FirstLocalMinStep_Mean,
gsat_FirstLocalMinStep_CoeffVariance, gsat_FirstLocalMinStep_Median, gsat_FirstLocalMinStep_Q10, gsat_FirstLocalMinStep_Q90, gsat_BestAvgImprovement_Mean,
gsat_BestAvgImprovement_CoeffVariance, gsat_FirstLocalMinRatio_Mean, gsat_FirstLocalMinRatio_CoeffVariance, lobjois_mean_depth_over_vars, lobjois_log_num_nodes_over_vars,