Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.821	2221.497	0.000
baseline	singleBest	0.650	4290.527	218.130
baseline	singleBestByPar	0.736	3266.048	128.355
baseline	singleBestBySuccesses	0.736	3266.048	128.355
classif	meta/AdaBoostM1	0.703	3698.375	128.786
classif	bayes/BayesNet	0.723	3448.569	101.088
classif	lazy/IBk	0.751	3109.392	67.310
classif	rules/OneR	0.710	3604.087	108.534
classif	trees/RandomTree	0.757	3041.031	54.476
classif	trees/J48	0.762	2984.956	53.927
classif	rules/JRip	0.732	3341.906	86.970
classif	classif.ctree	0.719	3488.727	94.974
classif	classif.ksvm	0.735	3297.047	79.129
classif	classif.naiveBayes	0.690	3851.831	143.425
classif	classif.randomForest	0.766	2924.683	39.927
classif	classif.rpart	0.701	3706.252	118.154
regr	regr.lm	0.751	3115.674	73.592
regr	regr.rpart	0.724	3444.112	105.886
regr	regr.randomForest	0.768	2907.608	41.361
regr	regr.earth	0.738	3278.042	87.887
cluster	EM	0.710	3615.047	119.494
cluster	FarthestFirst	0.718	3513.068	110.060
cluster	SimpleKMeans	0.713	3575.510	116.976

The following default feature steps were used for model building:

Pre, Basic, KLB, CG, DIAMETER, cl, sp, ls_saps, ls_gsat, lobjois

Number of presolved instances: 43

The cost for using the feature steps (adapted for presolving) is: 1.2003082 × 10⁵ or on average: 102.8541731

The feature steps correspond to the following 115 / 115 features:

nvarsOrig, nclausesOrig, nvars, nclauses, reducedVars,
reducedClauses, vars_clauses_ratio, POSNEG_RATIO_CLAUSE_mean, POSNEG_RATIO_CLAUSE_coeff_variation, POSNEG_RATIO_CLAUSE_min,
POSNEG_RATIO_CLAUSE_max, POSNEG_RATIO_CLAUSE_entropy, VCG_CLAUSE_mean, VCG_CLAUSE_coeff_variation, VCG_CLAUSE_min,
VCG_CLAUSE_max, VCG_CLAUSE_entropy, UNARY, BINARYp, TRINARYp,
VCG_VAR_mean, VCG_VAR_coeff_variation, VCG_VAR_min, VCG_VAR_max, VCG_VAR_entropy,
POSNEG_RATIO_VAR_mean, POSNEG_RATIO_VAR_stdev, POSNEG_RATIO_VAR_min, POSNEG_RATIO_VAR_max, POSNEG_RATIO_VAR_entropy,
HORNY_VAR_mean, HORNY_VAR_coeff_variation, HORNY_VAR_min, HORNY_VAR_max, HORNY_VAR_entropy,
horn_clauses_fraction, VG_mean, VG_coeff_variation, VG_min, VG_max,
CG_mean, CG_coeff_variation, CG_min, CG_max, CG_entropy,
cluster_coeff_mean, cluster_coeff_coeff_variation, cluster_coeff_min, cluster_coeff_max, cluster_coeff_entropy,
DIAMETER_mean, DIAMETER_coeff_variation, DIAMETER_min, DIAMETER_max, DIAMETER_entropy,
cl_num_mean, cl_num_coeff_variation, cl_num_min, cl_num_max, cl_num_q90,
cl_num_q10, cl_num_q75, cl_num_q25, cl_num_q50, cl_size_mean,
cl_size_coeff_variation, cl_size_min, cl_size_max, cl_size_q90, cl_size_q10,
cl_size_q75, cl_size_q25, cl_size_q50, SP_bias_mean, SP_bias_coeff_variation,
SP_bias_min, SP_bias_max, SP_bias_q90, SP_bias_q10, SP_bias_q75,
SP_bias_q25, SP_bias_q50, SP_unconstraint_mean, SP_unconstraint_coeff_variation, SP_unconstraint_min,
SP_unconstraint_max, SP_unconstraint_q90, SP_unconstraint_q10, SP_unconstraint_q75, SP_unconstraint_q25,
SP_unconstraint_q50, saps_BestSolution_Mean, saps_BestSolution_CoeffVariance, saps_FirstLocalMinStep_Mean, saps_FirstLocalMinStep_CoeffVariance,
saps_FirstLocalMinStep_Median, saps_FirstLocalMinStep_Q10, saps_FirstLocalMinStep_Q90, saps_BestAvgImprovement_Mean, saps_BestAvgImprovement_CoeffVariance,
saps_FirstLocalMinRatio_Mean, saps_FirstLocalMinRatio_CoeffVariance, gsat_BestSolution_Mean, gsat_BestSolution_CoeffVariance, gsat_FirstLocalMinStep_Mean,
gsat_FirstLocalMinStep_CoeffVariance, gsat_FirstLocalMinStep_Median, gsat_FirstLocalMinStep_Q10, gsat_FirstLocalMinStep_Q90, gsat_BestAvgImprovement_Mean,
gsat_BestAvgImprovement_CoeffVariance, gsat_FirstLocalMinRatio_Mean, gsat_FirstLocalMinRatio_CoeffVariance, lobjois_mean_depth_over_vars, lobjois_log_num_nodes_over_vars,