Some LLAMA benchmark results

LLAMA results

All results were produced by using the cross-validation splits in the repository with 10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.

The performance is measured in three different ways.

Percentage solved records the percentage of problem instances in the data set for which the selector selected an algorithm that was able to solve it with runstatus "ok" and the algorithm time plus the feature computation time was at most the timeout.
The penalized average runtime score (PAR10) measures the time required to run on all problem instances. If an instance was solved within the timeout by the algorithm the selector chose, the actual runtime is taken. If a timeout occurred, the timeout value was multiplied by 10.
The misclassification penalty (mcp) measures the additional time required to run on all problems if sub-optimal algorithms were used. That is, if an algorithm is run on a problem instance that is not the best, a performance loss is incurred. There are no additional penalties or factors for timeouts. The virtual best solver always has a misclassification penalty of zero.

algo	model	succ	par10	mcp
baseline	vbs	0.770	8337.099	0.000
baseline	singleBest	0.396	21884.319	1420.904
baseline	singleBestByPar	0.577	15330.171	716.756
baseline	singleBestBySuccesses	0.577	15330.171	716.756
classif	meta/AdaBoostM1	0.577	15330.171	716.756
classif	lazy/IBk	0.692	11159.665	288.355
classif	rules/OneR	0.651	12657.656	436.346
classif	trees/RandomTree	0.716	10305.316	215.585
classif	trees/J48	0.707	10637.255	239.629
classif	rules/JRip	0.700	10878.806	268.023
classif	classif.ctree	0.691	11224.438	305.760
classif	classif.ksvm	0.691	11221.558	302.879
classif	classif.naiveBayes	0.491	18410.085	1025.617
classif	classif.randomForest	0.730	9800.574	160.843
classif	classif.rpart	0.689	11288.085	298.354
regr	regr.lm	0.714	10388.712	227.928
regr	regr.rpart	0.714	10405.535	244.751
regr	regr.randomForest	0.749	9152.113	104.487
regr	regr.earth	0.721	10132.304	208.362
cluster	EM	0.641	13022.871	493.666
cluster	FarthestFirst	0.577	15330.171	716.756
cluster	SimpleKMeans	0.577	15330.171	716.756

The following default feature steps were used for model building:

all_feats

Number of presolved instances: 0

The cost for using the feature steps (adapted for presolving) is: 0 or on average: NA

The feature steps correspond to the following 46 / 46 features:

EXIST_VARS, FORALL_VARS, TOTAL_VARS, CLAUSES, LITERALS,
EXIST_SET, FORALL_SET, TOTAL_SET, UNARY_CLAUSES, BINARY_CLAUSES,
TERNARY_MORE_CLAUSES, POS_HORN, NEG_HORN, EXIST_LIT_PER_CLAUSE, FORALL_LIT_PER_CLAUSE,
EXIST_VARS_PER_SET, FORALL_POS_LITS_PER_CLAUSE, FORALL_NEG_LITS_PER_CLAUSE, OCCS_POS_NO_PER_VAR, OCCS_FORALL_NO_PER_VAR,
OCCS_FORALL_POS_NO_PER_VAR, W_OCCS_POS_NO_PER_VAR, W_OCCS_FORALL_NO_PER_VAR, W_OCCS_FORALL_POS_NO_PER_VAR, W_PRODUCTS,
LITN_LIT, LITEP_LIT, LITEN_LITE, LITEN_LITN, LITFN_LIT,
LITFP_LITFN, OCCP_OCCN, OCCE_OCC, OCCEN_OCC, OCCFP_OCCF,
OCCEN_OCCE, OCCEN_OCCN, OCCFP_OCCFN, TERMORE_CLAUSE, NEG_HORN_CLAUSE,
WOCCN_WOCC, WOCCEP_WOCC, WOCCFN_WOCC, WOCCEP_WOCCE, WOCCEP_WOCCP,
WOCCFN_WOCCN,