Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.820	9186.441	0.000
baseline	singleBest	0.547	22744.229	1257.789
baseline	singleBestByPar	0.603	19916.356	979.915
baseline	singleBestBySuccesses	0.603	19916.356	979.915
classif	meta/AdaBoostM1	0.775	11451.395	216.063
classif	bayes/BayesNet	0.802	10164.973	129.641
classif	lazy/IBk	0.720	14180.973	470.641
classif	rules/OneR	0.777	11350.091	189.759
classif	trees/RandomTree	0.735	13448.270	412.938
classif	trees/J48	0.722	14085.930	450.598
classif	rules/JRip	0.725	13913.139	427.807
classif	classif.ctree	0.760	12143.159	232.827
classif	classif.ksvm	0.772	11561.921	176.589
classif	classif.naiveBayes	0.740	13267.213	456.881
classif	classif.randomForest	0.783	10999.269	138.937
classif	classif.rpart	0.778	11286.157	200.825
regr	regr.lm	0.778	11275.392	190.059
regr	regr.rpart	0.762	12092.478	257.146
regr	regr.randomForest	0.795	10420.574	85.242
regr	regr.earth	0.778	11268.265	182.933
cluster	EM	0.760	12161.929	251.597
cluster	FarthestFirst	0.607	19774.930	964.598
cluster	SimpleKMeans	0.615	19389.062	953.730

The following default feature steps were used for model building:

Pre, Basic, KLB, CG, DIAMETER, cl, sp, ls_saps, ls_gsat, lobjois

Number of presolved instances: 0

The cost for using the feature steps (adapted for presolving) is: 1.884118 × 10⁴ or on average: 31.4019667

The feature steps correspond to the following 115 / 115 features:

nvarsOrig, nclausesOrig, nvars, nclauses, reducedVars,
reducedClauses, vars_clauses_ratio, POSNEG_RATIO_CLAUSE_mean, POSNEG_RATIO_CLAUSE_coeff_variation, POSNEG_RATIO_CLAUSE_min,
POSNEG_RATIO_CLAUSE_max, POSNEG_RATIO_CLAUSE_entropy, VCG_CLAUSE_mean, VCG_CLAUSE_coeff_variation, VCG_CLAUSE_min,
VCG_CLAUSE_max, VCG_CLAUSE_entropy, UNARY, BINARYp, TRINARYp,
VCG_VAR_mean, VCG_VAR_coeff_variation, VCG_VAR_min, VCG_VAR_max, VCG_VAR_entropy,
POSNEG_RATIO_VAR_mean, POSNEG_RATIO_VAR_stdev, POSNEG_RATIO_VAR_min, POSNEG_RATIO_VAR_max, POSNEG_RATIO_VAR_entropy,
HORNY_VAR_mean, HORNY_VAR_coeff_variation, HORNY_VAR_min, HORNY_VAR_max, HORNY_VAR_entropy,
horn_clauses_fraction, VG_mean, VG_coeff_variation, VG_min, VG_max,
CG_mean, CG_coeff_variation, CG_min, CG_max, CG_entropy,
cluster_coeff_mean, cluster_coeff_coeff_variation, cluster_coeff_min, cluster_coeff_max, cluster_coeff_entropy,
DIAMETER_mean, DIAMETER_coeff_variation, DIAMETER_min, DIAMETER_max, DIAMETER_entropy,
cl_num_mean, cl_num_coeff_variation, cl_num_min, cl_num_max, cl_num_q90,
cl_num_q10, cl_num_q75, cl_num_q25, cl_num_q50, cl_size_mean,
cl_size_coeff_variation, cl_size_min, cl_size_max, cl_size_q90, cl_size_q10,
cl_size_q75, cl_size_q25, cl_size_q50, SP_bias_mean, SP_bias_coeff_variation,
SP_bias_min, SP_bias_max, SP_bias_q90, SP_bias_q10, SP_bias_q75,
SP_bias_q25, SP_bias_q50, SP_unconstraint_mean, SP_unconstraint_coeff_variation, SP_unconstraint_min,
SP_unconstraint_max, SP_unconstraint_q90, SP_unconstraint_q10, SP_unconstraint_q75, SP_unconstraint_q25,
SP_unconstraint_q50, saps_BestSolution_Mean, saps_BestSolution_CoeffVariance, saps_FirstLocalMinStep_Mean, saps_FirstLocalMinStep_CoeffVariance,
saps_FirstLocalMinStep_Median, saps_FirstLocalMinStep_Q10, saps_FirstLocalMinStep_Q90, saps_BestAvgImprovement_Mean, saps_BestAvgImprovement_CoeffVariance,
saps_FirstLocalMinRatio_Mean, saps_FirstLocalMinRatio_CoeffVariance, gsat_BestSolution_Mean, gsat_BestSolution_CoeffVariance, gsat_FirstLocalMinStep_Mean,
gsat_FirstLocalMinStep_CoeffVariance, gsat_FirstLocalMinStep_Median, gsat_FirstLocalMinStep_Q10, gsat_FirstLocalMinStep_Q90, gsat_BestAvgImprovement_Mean,
gsat_BestAvgImprovement_CoeffVariance, gsat_FirstLocalMinRatio_Mean, gsat_FirstLocalMinRatio_CoeffVariance, lobjois_mean_depth_over_vars, lobjois_log_num_nodes_over_vars,