Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.937	400.179	0.000
baseline	singleBest	0.859	880.547	58.884
baseline	singleBestByPar	0.859	880.547	58.884
baseline	singleBestBySuccesses	0.859	880.547	58.884
classif	meta/AdaBoostM1	0.859	881.326	58.426
classif	bayes/BayesNet	0.884	722.196	37.008
classif	lazy/IBk	0.871	795.643	39.513
classif	rules/OneR	0.873	782.371	38.760
classif	trees/RandomTree	0.883	728.557	35.023
classif	trees/J48	0.882	731.406	33.699
classif	rules/JRip	0.879	751.048	40.822
classif	classif.ctree	0.876	770.383	39.291
classif	classif.ksvm	0.880	742.112	36.059
classif	classif.naiveBayes	0.887	704.988	36.493
classif	classif.randomForest	0.890	680.119	28.317
classif	classif.rpart	0.885	714.117	33.102
regr	regr.lm	0.902	614.977	25.770
regr	regr.rpart	0.906	600.676	32.335
regr	regr.randomForest	0.917	526.633	16.716
regr	regr.earth	0.899	635.373	29.474
cluster	EM	0.845	956.776	62.933
cluster	FarthestFirst	0.845	961.716	63.700
cluster	SimpleKMeans	0.850	929.896	61.091

The following default feature steps were used for model building:

Static, Dynamic.1, Dynamic.2, Dynamic.3, Dynamic.4

Number of presolved instances: 204

The cost for using the feature steps (adapted for presolving) is: 1720.11 or on average: 1.3292968

The feature steps correspond to the following 138 / 138 features:

Frac_Neg_Body, Frac_Pos_Body, Frac_Unary_Rules, Frac_Binary_Rules, Frac_Ternary_Rules,
Frac_Integrity_Rules, Tight, Problem_Variables, Free_Problem_Variables, Assigned_Problem_Variables,
Constraints, Constraints.Vars, Created_Bodies, Program_Atoms, SCCS,
Nodes_in_Positive_BADG, Rules, Normal_Rules, Cardinality_Rules, Choice_Rules,
Weight_Rules, Frac_Normal_Rules, Frac_Cardinality_Rules, Frac_Choice_Rules, Frac_Weight_Rules,
Equivalences, Atom.Atom_Equivalences, Body.Body_Equivalences, Other_Equivalences, Frac_Atom.Atom_Equivalences,
Frac_Body.Body_Equivalences, Frac_Other_Equivalences, Binary_Constraints, Ternary_Constraints, Other_Constraints,
Frac_Binary_Constraints, Frac_Ternary_Constraints, Frac_Other_Constraints, Choices.1, Conflicts.Choices.1,
Avg_Conflict_Levels.1, Avg_LBD_Levels.1, Learnt_from_Conflict.1, Learnt_from_Loop.1, Frac_Learnt_from_Conflict.1,
Frac_Learnt_from_Loop.1, Literals_in_Conflict_Nogoods.1, Literals_in_Loop_Nogoods.1, Frac_Literals_in_Conflict_Nogoods.1, Frac_Literals_in_Loop_Nogoods.1,
Removed_Nogoods.1, Learnt_Binary.1, Learnt_Ternary.1, Learnt_Others.1, Frac_Removed_Nogood.1,
Frac_Learnt_Binary.1, Frac_Learnt_Ternary.1, Frac_Learnt_Others.1, Skipped_Levels_while_Backjumping.1, Avg_Skipped_Levels_while_Backjumping.1,
Longest_Backjumping.1, Running_Avg_Conflictlevel.1, Running_Avg_LBD.1, Choices.2, Conflicts.Choices.2,
Avg_Conflict_Levels.2, Avg_LBD_Levels.2, Learnt_from_Conflict.2, Learnt_from_Loop.2, Frac_Learnt_from_Conflict.2,
Frac_Learnt_from_Loop.2, Literals_in_Conflict_Nogoods.2, Literals_in_Loop_Nogoods.2, Frac_Literals_in_Conflict_Nogoods.2, Frac_Literals_in_Loop_Nogoods.2,
Removed_Nogoods.2, Learnt_Binary.2, Learnt_Ternary.2, Learnt_Others.2, Frac_Removed_Nogood.2,
Frac_Learnt_Binary.2, Frac_Learnt_Ternary.2, Frac_Learnt_Others.2, Skipped_Levels_while_Backjumping.2, Avg_Skipped_Levels_while_Backjumping.2,
Longest_Backjumping.2, Running_Avg_Conflictlevel.2, Running_Avg_LBD.2, Choices.3, Conflicts.Choices.3,
Avg_Conflict_Levels.3, Avg_LBD_Levels.3, Learnt_from_Conflict.3, Learnt_from_Loop.3, Frac_Learnt_from_Conflict.3,
Frac_Learnt_from_Loop.3, Literals_in_Conflict_Nogoods.3, Literals_in_Loop_Nogoods.3, Frac_Literals_in_Conflict_Nogoods.3, Frac_Literals_in_Loop_Nogoods.3,
Removed_Nogoods.3, Learnt_Binary.3, Learnt_Ternary.3, Learnt_Others.3, Frac_Removed_Nogood.3,
Frac_Learnt_Binary.3, Frac_Learnt_Ternary.3, Frac_Learnt_Others.3, Skipped_Levels_while_Backjumping.3, Avg_Skipped_Levels_while_Backjumping.3,
Longest_Backjumping.3, Running_Avg_Conflictlevel.3, Running_Avg_LBD.3, Choices.4, Conflicts.Choices.4,
Avg_Conflict_Levels.4, Avg_LBD_Levels.4, Learnt_from_Conflict.4, Learnt_from_Loop.4, Frac_Learnt_from_Conflict.4,
Frac_Learnt_from_Loop.4, Literals_in_Conflict_Nogoods.4, Literals_in_Loop_Nogoods.4, Frac_Literals_in_Conflict_Nogoods.4, Frac_Literals_in_Loop_Nogoods.4,
Removed_Nogoods.4, Learnt_Binary.4, Learnt_Ternary.4, Learnt_Others.4, Frac_Removed_Nogood.4,
Frac_Learnt_Binary.4, Frac_Learnt_Ternary.4, Frac_Learnt_Others.4, Skipped_Levels_while_Backjumping.4, Avg_Skipped_Levels_while_Backjumping.4,
Longest_Backjumping.4, Running_Avg_Conflictlevel.4, Running_Avg_LBD.4,