Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.764	2872.843	0.000
baseline	singleBest	0.673	3965.424	109.322
baseline	singleBestByPar	0.731	3271.137	49.396
baseline	singleBestBySuccesses	0.731	3271.137	49.396
classif	meta/AdaBoostM1	0.730	3299.719	48.461
classif	bayes/BayesNet	0.670	4020.127	126.577
classif	lazy/IBk	0.726	3346.852	55.945
classif	rules/OneR	0.729	3306.539	47.351
classif	trees/RandomTree	0.717	3452.140	66.079
classif	trees/J48	0.722	3398.065	59.581
classif	rules/JRip	0.725	3350.552	51.716
classif	classif.ctree	0.729	3305.831	46.643
classif	classif.ksvm	0.731	3291.267	47.938
classif	classif.naiveBayes	0.710	3542.215	76.860
classif	classif.randomForest	0.730	3299.171	47.913
classif	classif.rpart	0.728	3317.410	50.293
regr	regr.lm	0.733	3254.533	42.922
regr	regr.rpart	0.724	3372.812	58.118
regr	regr.randomForest	0.739	3188.678	40.503
regr	regr.earth	0.719	3439.484	69.282
cluster	EM	0.730	3299.719	48.461
cluster	FarthestFirst	0.730	3299.719	48.461
cluster	SimpleKMeans	0.730	3299.719	48.461

The following default feature steps were used for model building:

Pre, Basic, KLB, CG, DIAMETER, cl, sp, ls_saps, ls_gsat, lobjois

Number of presolved instances: 0

The cost for using the feature steps (adapted for presolving) is: 2.500177 × 10⁴ or on average: 18.3566593

The feature steps correspond to the following 115 / 115 features:

nvarsOrig, nclausesOrig, nvars, nclauses, reducedVars,
reducedClauses, vars_clauses_ratio, POSNEG_RATIO_CLAUSE_mean, POSNEG_RATIO_CLAUSE_coeff_variation, POSNEG_RATIO_CLAUSE_min,
POSNEG_RATIO_CLAUSE_max, POSNEG_RATIO_CLAUSE_entropy, VCG_CLAUSE_mean, VCG_CLAUSE_coeff_variation, VCG_CLAUSE_min,
VCG_CLAUSE_max, VCG_CLAUSE_entropy, UNARY, BINARYp, TRINARYp,
VCG_VAR_mean, VCG_VAR_coeff_variation, VCG_VAR_min, VCG_VAR_max, VCG_VAR_entropy,
POSNEG_RATIO_VAR_mean, POSNEG_RATIO_VAR_stdev, POSNEG_RATIO_VAR_min, POSNEG_RATIO_VAR_max, POSNEG_RATIO_VAR_entropy,
HORNY_VAR_mean, HORNY_VAR_coeff_variation, HORNY_VAR_min, HORNY_VAR_max, HORNY_VAR_entropy,
horn_clauses_fraction, VG_mean, VG_coeff_variation, VG_min, VG_max,
CG_mean, CG_coeff_variation, CG_min, CG_max, CG_entropy,
cluster_coeff_mean, cluster_coeff_coeff_variation, cluster_coeff_min, cluster_coeff_max, cluster_coeff_entropy,
DIAMETER_mean, DIAMETER_coeff_variation, DIAMETER_min, DIAMETER_max, DIAMETER_entropy,
cl_num_mean, cl_num_coeff_variation, cl_num_min, cl_num_max, cl_num_q90,
cl_num_q10, cl_num_q75, cl_num_q25, cl_num_q50, cl_size_mean,
cl_size_coeff_variation, cl_size_min, cl_size_max, cl_size_q90, cl_size_q10,
cl_size_q75, cl_size_q25, cl_size_q50, SP_bias_mean, SP_bias_coeff_variation,
SP_bias_min, SP_bias_max, SP_bias_q90, SP_bias_q10, SP_bias_q75,
SP_bias_q25, SP_bias_q50, SP_unconstraint_mean, SP_unconstraint_coeff_variation, SP_unconstraint_min,
SP_unconstraint_max, SP_unconstraint_q90, SP_unconstraint_q10, SP_unconstraint_q75, SP_unconstraint_q25,
SP_unconstraint_q50, saps_BestSolution_Mean, saps_BestSolution_CoeffVariance, saps_FirstLocalMinStep_Mean, saps_FirstLocalMinStep_CoeffVariance,
saps_FirstLocalMinStep_Median, saps_FirstLocalMinStep_Q10, saps_FirstLocalMinStep_Q90, saps_BestAvgImprovement_Mean, saps_BestAvgImprovement_CoeffVariance,
saps_FirstLocalMinRatio_Mean, saps_FirstLocalMinRatio_CoeffVariance, gsat_BestSolution_Mean, gsat_BestSolution_CoeffVariance, gsat_FirstLocalMinStep_Mean,
gsat_FirstLocalMinStep_CoeffVariance, gsat_FirstLocalMinStep_Median, gsat_FirstLocalMinStep_Q10, gsat_FirstLocalMinStep_Q90, gsat_BestAvgImprovement_Mean,
gsat_BestAvgImprovement_CoeffVariance, gsat_FirstLocalMinRatio_Mean, gsat_FirstLocalMinRatio_CoeffVariance, lobjois_mean_depth_over_vars, lobjois_log_num_nodes_over_vars,