Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.875	6344.251	0.000
baseline	singleBest	0.858	7201.556	79.143
baseline	singleBestByPar	0.858	7201.556	79.143
baseline	singleBestBySuccesses	0.858	7201.556	79.143
classif	meta/AdaBoostM1	0.866	6802.677	35.995
classif	bayes/BayesNet	0.867	6753.415	31.199
classif	lazy/IBk	0.867	6753.529	31.313
classif	rules/OneR	0.865	6827.434	38.518
classif	trees/RandomTree	0.868	6704.023	26.274
classif	trees/J48	0.868	6701.576	23.827
classif	rules/JRip	0.866	6802.945	36.263
classif	classif.ctree	0.865	6848.988	37.840
classif	classif.ksvm	0.865	6849.037	37.889
classif	classif.naiveBayes	0.787	10775.342	473.581
classif	classif.randomForest	0.869	6629.103	18.053
classif	classif.rpart	0.864	6874.495	41.114
regr	regr.lm	0.869	6633.481	22.431
regr	regr.rpart	0.863	6927.993	50.145
regr	regr.randomForest	0.871	6541.289	19.173
regr	regr.earth	0.867	6761.599	39.383
cluster	EM	0.855	7358.226	102.413
cluster	FarthestFirst	0.858	7201.556	79.143
cluster	SimpleKMeans	0.858	7201.556	79.143

The following default feature steps were used for model building:

all_feats

Number of presolved instances: 0

The cost for using the feature steps (adapted for presolving) is: 0 or on average: NA

The feature steps correspond to the following 86 / 86 features:

stats_varcount, stats_var_bool, stats_var_discrete, stats_var_bound, stats_var_sparsebound,
stats_dom_0, stats_dom_25, stats_dom_50, stats_dom_75, stats_dom_100,
stats_dom_mean, stats_dom_not2_2_ratio, stats_discrete_bool_ratio, stats_branchingvars, stats_auxvars,
stats_auxvar_branching_ratio, stats_conscount, stats_arity_0, stats_arity_25, stats_arity_50,
stats_arity_75, stats_arity_100, stats_arity_mean, stats_arity_mean_normalised, stats_cts_per_var_mean,
stats_cts_per_var_mean_normalised, stats_alldiff_count, stats_alldiff_proportion, stats_sums_count, stats_sums_proportion,
stats_or_atleastk_count, stats_or_atleastk_proportion, stats_ternary_count, stats_ternary_proportion, stats_binary_count,
stats_binary_proportion, stats_reify_count, stats_reify_proportion, stats_table_count, stats_table_proportion,
stats_lex_count, stats_lex_proportion, stats_unary_count, stats_unary_proportion, stats_nullary_count,
stats_nullary_proportion, stats_element_count, stats_element_proportion, stats_minmax_count, stats_minmax_proportion,
stats_occurrence_count, stats_occurrence_proportion, stats_multi_shared_vars, stats_edge_density, stats_Local_Variance,
standard_deviation_of_node_degree, normalised_standard_deviation_of_node_degree, clustering_coefficient, minimum_degree, normalised_minimum_degree,
maximum_degree, normalised_maximum_degree, median_degree, normalised_median_degree, mean_degree,
normalised_mean_degree, width_of_ordering, normalised_width_of_ordering, width_of_graph, normalised_width_of_graph,
SAC_literals, normalised_SAC_literals, stats_tightness_0, stats_tightness_25, stats_tightness_50,
stats_tightness_75, stats_tightness_100, stats_tightness_mean, stats_tightness_mean_normalised, stats_literal_tightness_0,
stats_literal_tightness_25, stats_literal_tightness_50, stats_literal_tightness_75, stats_literal_tightness_100, stats_literal_tightness_mean,
stats_literal_tightness_coeff_of_variation,