×

ASlib: a benchmark library for algorithm selection. (English) Zbl 1357.68202

Summary: The task of algorithm selection involves choosing an algorithm from a set of algorithms on a per-instance basis in order to exploit the varying performance of algorithms over a set of instances. The algorithm selection problem is attracting increasing attention from researchers and practitioners in AI. Years of fruitful applications in a number of domains have resulted in a large amount of data, but the community lacks a standard format or repository for this data. This situation makes it difficult to share and compare different approaches effectively, as is done in other, more established fields. It also unnecessarily hinders new researchers who want to work in this area. To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature. Our format has been designed to be able to express a wide variety of different scenarios. To demonstrate the breadth and power of our platform, we describe a study that builds and evaluates algorithm selection models through a common interface. The results display the potential of algorithm selection to achieve significant performance improvements across a broad range of problems and algorithms.

MSC:

68T20 Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.)
68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Amadini, R.; Gabbrielli, M.; Mauro, J., An enhanced features extractor for a portfolio of constraint solvers, (Symposium on Applied Computing. Symposium on Applied Computing, SAC 2014, Gyeongju, Republic of Korea - March 24-28 (2014)), 1357-1359
[2] Amadini, R.; Gabbrielli, M.; Mauro, J., SUNNY: a lazy portfolio approach for constraint solving, Theory Pract. Log. Program., 14, 509-524 (2014) · Zbl 1307.68077
[3] Ansótegui, C.; Malitsky, Y.; Sellmann, M., MaxSAT by improved instance-specific algorithm configuration, (Proceedings of the Twenty-Eighth National Conference on Artificial Intelligence (2014)), 2594-2600
[4] Ansótegui, C.; Sellmann, M.; Tierney, K., A gender-based genetic algorithm for the automatic configuration of algorithms, (Proceedings of the Fifteenth International Conference on Principles and Practice of Constraint Programming (CP’09) (2009)), 142-157
[5] Arbelaez, A.; Hamadi, Y.; Sebag, M., Continuous search in constraint programming, (Proceedings of the Twenty-Second IEEE International Conference on Tools with Artificial Intelligence (2010)), 53-60
[6] Argelich, J.; Berre, D. L.; Lynce, I.; Marques-Silva, J.; Rapicault, P., Solving Linux upgradeability problems using Boolean optimization, (Proceedings of the International Workshop on Logics for Component Configuration (2010)), 11-22
[7] Argelich, J.; Li, C.; Manyà, F.; Planes, J., Seventh MaxSAT evaluation (2012)
[8] Baral, C., Knowledge Representation, Reasoning and Declarative Problem Solving (2003), Cambridge University Press · Zbl 1056.68139
[9] Biere, A., Yet another local search solver and Lingeling and friends entering the SAT competition 2014, (Proceedings of SAT Competition 2014: Solver and Benchmark Descriptions (2014)), 39-40
[10] (Biere, A.; Heule, M.; van Maaren, H.; Walsh, T., Handbook of Satisfiability. Handbook of Satisfiability, Frontiers in Artificial Intelligence and Applications, vol. 185 (2009), IOS Press) · Zbl 1183.68568
[11] Birattari, M.; Yuan, Z.; Balaprakash, P.; Stützle, T., F-race and iterated F-race: an overview, (Empirical Methods for the Analysis of Optimization Algorithms (2010), Springer) · Zbl 1204.68280
[12] Bischl, B.; Kotthoff, L.; Lindauer, M.; Malitsky, Y.; Fréchette, A.; Hoos, H.; Hutter, F.; Kerschke, P.; Leyton-Brown, K.; Vanschoren, J., Algorithm selection format specification (2014), Technical report, available at
[13] Bischl, B.; Lang, M.; Mersmann, O.; Rahnenführer, J.; Weihs, C., BatchJobs and BatchExperiments: abstraction mechanisms for using R in batch environments, J. Stat. Softw., 64, 1-25 (2015)
[14] Bischl, B.; Lang, M.; Richter, J.; Bossek, J.; Judt, L.; Kuehn, T.; Studerus, E.; Kotthoff, L.; Jones, Z., mlr: machine learning in R (2015), R package version 2.7
[15] Bischl, B.; Mersmann, O.; Trautmann, H.; Preuss, M., Algorithm selection based on exploratory landscape analysis and cost-sensitive learning, (Proceedings of the Fourteenth Annual Conference on Genetic and Evolutionary Computation (2012)), 313-320
[16] Bischl, B.; Mersmann, O.; Trautmann, H.; Weihs, C., Resampling methods for meta-model validation with recommendations for evolutionary computation, Evol. Comput., 20, 249-275 (2012)
[17] Brazdil, P.; Giraud-Carrier, C.; Soares, C.; Vilalta, R., Metalearning: Applications to Data Mining (2008), Springer · Zbl 1173.68625
[18] Cicirello, V. A.; Smith, S. F., The max k-armed bandit: a new model of exploration applied to search heuristic selection, (Proceedings of the Twentieth National Conference on Artificial Intelligence (2005), AAAI Press), 1355-1361
[19] Cook, D. J.; Varnell, R. C., Maximizing the benefits of parallel search using machine learning, (Proceedings of the Fourteenth National Conference on Artificial Intelligence (1997), AAAI Press), 559-564
[20] Crawford, J. M.; Baker, A. B., Experimental results on the application of satisfiability algorithms to scheduling problems, (Proceedings of the Twelfth National Conference on Artificial Intelligence (1994)), 1092-1097
[21] Dan Pelleg, A. M., X-means: extending k-means with efficient estimation of the number of clusters, (Proceedings of the Seventeenth International Conference on Machine Learning (2000), Morgan Kaufmann: Morgan Kaufmann San Francisco), 727-734
[22] Demmel, J.; Dongarra, J.; Eijkhout, V.; Fuentes, E.; Petitet, A.; Vuduc, R.; Whaley, R. C.; Yelick, K., Self-adapting linear algebra algorithms and software, Proc. IEEE, 93, 293-312 (2005)
[23] Domhan, T.; Springenberg, J. T.; Hutter, F., Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves, (Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI) (2015))
[24] Eggensperger, K.; Hutter, F.; Hoos, H. H.; Leyton-Brown, K., Efficient benchmarking of hyperparameter optimizers via surrogates, (Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015))
[25] Fawcett, C.; Vallati, M.; Hutter, F.; Hoffmann, J.; Hoos, H.; Leyton-Brown, K., Improved features for runtime prediction of domain-independent planners, (Proceedings of the International Conference on Automated Planning and Scheduling (2014))
[26] Feurer, M.; Klein, A.; Eggensperger, K.; Springenberg, J.; Blum, M.; Hutter, F., Efficient and robust automated machine learning, (Advances in Neural Information Processing Systems, vol. 28 (2015)), 2944-2952
[27] Feurer, M.; Springenberg, J. T.; Hutter, F., Initializing Bayesian hyperparameter optimization via meta-learning, (Bonet, B.; Koenig, S., Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA (2015), AAAI Press), 1128-1135
[28] Gagliolo, M.; Schmidhuber, J., Learning restart strategies, (Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI) (2007)), 792-797
[29] Gagliolo, M.; Zhumatiy, V.; Schmidhuber, J., Adaptive online time allocation to search algorithms, (Proceedings of European Conference on Machine Learning (2004), Springer), 134-143 · Zbl 1132.68546
[30] Gebser, M.; Kaminski, R.; Kaufmann, B.; Schaub, T., Answer Set Solving in Practice, Synthesis Lectures on Artificial Intelligence and Machine Learning (2012), Morgan and Claypool Publishers
[31] Gebser, M.; Kaminski, R.; Kaufmann, B.; Schaub, T.; Schneider, M. T.; Ziller, S., A portfolio solver for answer set programming: preliminary report, (Eleventh International Conference on Logic Programming and Nonmonotonic Reasoning (2011), Springer), 352-357
[32] Gebser, M.; Kaufmann, B.; Schaub, T., Multi-threaded ASP solving with clasp, Theory Pract. Log. Program., 12, 525-545 (2012) · Zbl 1260.68061
[33] Gent, I.; Jefferson, C.; Kotthoff, L.; Miguel, I.; Moore, N.; Nightingale, P.; Petrie, K., Learning when to use lazy learning in constraint solving, (Proceedings of the Nineteenth European Conference on Artificial Intelligence (2010), IOS Press), 873-878
[34] Gent, I. P.; Jefferson, C. A.; Miguel, I., MINION: a fast, scalable, constraint solver, (Proceedings of the European Conference on Artificial Intelligence (2006)), 98-102
[35] Gent, I. P.; Miguel, I.; Moore, N. C.A., Lazy explanations for constraint propagators, (Proceedings of the Twelfth International Symposium on Practical Aspects of Declarative Languages (2010)), 217-233
[36] Gomes, C.; Selman, B.; Crato, N.; Kautz, H., Heavy-tailed phenomena in satisfiability and constraint satisfaction problems, J. Autom. Reason., 24, 67-100 (2000) · Zbl 0967.68145
[37] Gomes, C. P.; Selman, B., Algorithm portfolios, Artif. Intell., 126, 43-62 (2001) · Zbl 0969.68047
[38] Grasso, G.; Iiritano, S.; Leone, N.; Lio, V.; Ricca, F.; Scalise, F., An ASP-based system for team-building in the Gioia-Tauro seaport, (Proceedings of the Twelfth International Symposium on Practical Aspects of Declarative Languages (2010)), 40-42
[39] Guerri, A.; Milano, M., Learning techniques for automatic algorithm portfolio selection, (Proceedings of the Sixteenth European Conference on Artificial Intelligence (2004), IOS Press), 475-479
[40] Hall, M.; Frank, E.; Holmes, G.; Pfahringer, B.; Reutemann, P.; Witten, I. H., The WEKA data mining software: an update, ACM SIGKDD Explor. Newsl., 11, 10-18 (2009)
[41] Hastie, T. J.; Tibshirani, R. J.; Friedman, J. H., The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer Series in Statistics (2009), Springer: Springer New York · Zbl 1273.62005
[42] Helmert, M.; Röger, G.; Karpas, E., Fast downward stone soup: a baseline for building planner portfolios, (Proceedings of the Workshop on Planning and Learning at the Twenty-First International Conference on Automated Planning and Scheduling (2011)), 28-35
[43] Hoos, H.; Lindauer, M.; Schaub, T., Claspfolio 2: advances in algorithm selection for answer set programming, Theory Pract. Log. Program., 569-585 (2014) · Zbl 1307.68016
[44] Hoos, H. H.; Kaminski, R.; Lindauer, M.; Schaub, T., aspeed: solver scheduling via answer set programming, Theory Pract. Log. Program., 1-26 (2014)
[45] Howe, A. E.; Dahlman, E.; Hansen, C.; Scheetz, M.; von Mayrhauser, A., Exploiting competitive planner performance, (Proceedings of the Fifth European Conference on Planning (1999), Springer), 62-72
[46] Hurley, B.; Kotthoff, L.; Malitsky, Y.; O’Sullivan, B., Proteus: a hierarchical portfolio of solvers and transformations, (Proceedings of the Eleventh International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (2014)), 301-317 · Zbl 06298800
[47] Hutter, F.; Babić, D.; Hoos, H. H.; Hu, A. J., Boosting verification by automatic tuning of decision procedures, (Formal Methods in Computer Aided Design (2007), IEEE Computer Society), 27-34
[48] Hutter, F.; Hamadi, Y.; Hoos, H. H.; Leyton-Brown, K., Performance prediction and automated tuning of randomized and parametric algorithms, (Proceedings of the Twelfth International Conference on Principles and Practice of Constraint Programming (2006)), 213-228 · Zbl 1160.68551
[49] Hutter, F.; Hoos, H. H.; Leyton-Brown, K., Automated configuration of mixed integer programming solvers, (Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (2010)), 186-202
[50] Hutter, F.; Hoos, H. H.; Leyton-Brown, K., Sequential model-based optimization for general algorithm configuration, (Proceedings of the International Conference on Learning and Intelligent Optimization (2011)), 507-523
[51] Hutter, F.; Hoos, H. H.; Leyton-Brown, K., Identifying key algorithm parameters and instance features using forward selection, (LION 7 (2013))
[52] Hutter, F.; Hoos, H. H.; Leyton-Brown, K.; Stützle, T., ParamILS: an automatic algorithm configuration framework, J. Artif. Intell. Res., 36, 267-306 (2009) · Zbl 1192.68831
[53] Hutter, F.; López-Ibáñez, M.; Fawcett, C.; Lindauer, M.; Hoos, H.; Leyton-Brown, K.; Stützle, T., Aclib: a benchmark library for algorithm configuration, (Proceedings of the International Conference on Learning and Intelligent Optimization (2014)), 36-40
[54] Hutter, F.; Xu, L.; Hoos, H. H.; Leyton-Brown, K., Algorithm runtime prediction: methods & evaluation, Artif. Intell., 206, 79-111 (2014) · Zbl 1334.68185
[55] Ishebabi, H.; Mahr, P.; Bobda, C.; Gebser, M.; Schaub, T., Answer set vs. integer linear programming for automatic synthesis of multiprocessor systems from real-time parallel programs, Int. J. Reconfigurable Comput. (2009)
[56] Kadioglu, S.; Malitsky, Y.; Sabharwal, A.; Samulowitz, H.; Sellmann, M., Algorithm selection and scheduling, (Proceedings of the International Conference on Principles and Practice of Constraint Programming. Proceedings of the International Conference on Principles and Practice of Constraint Programming, Lecture Notes in Computer Science, vol. 6876 (2011), Springer), 454-469
[57] Kadioglu, S.; Malitsky, Y.; Sellmann, M.; Tierney, K., ISAC - instance-specific algorithm configuration, (Proceedings of Nineteenth European Conference on Artificial Intelligence (2010), IOS Press), 751-756
[58] Karatzoglou, A.; Smola, A.; Hornik, K.; Zeileis, A., kernlab - an S4 package for kernel methods in R, J. Stat. Softw., 11, 1-20 (2004)
[59] Kautz, H.; Selman, B., Unifying SAT-based and graph-based planning, (Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (1999), Morgan Kaufmann), 318-325
[60] Kerschke, P.; Preuss, M.; Hernández, C.; Schütze, O.; Sun, J. Q.; Grimme, C.; Rudolph, G.; Bischl, B.; Trautmann, H., Cell mapping techniques for exploratory landscape analysis, (Proceedings of the EVOLVE 2014: A Bridge Between Probability, Set Oriented Numerics, and Evolutionary Computation (2014), Springer), 115-131 · Zbl 1306.90175
[61] Kohavi, R.; John, G. H., Wrappers for feature subset selection, Artif. Intell., 97, 273-324 (1997) · Zbl 0904.68143
[62] Kotthoff, L., LLAMA: leveraging learning to automatically manage algorithms (2013), Technical report
[63] Kotthoff, L., Algorithm selection for combinatorial search problems: a survey, AI Mag., 35, 48-60 (2014)
[64] Lagoudakis, M.; Littman, M., Algorithm selection using reinforcement learning, (Proceedings of the Seventeenth International Conference on Machine Learning (2000)), 511-518
[65] Lagoudakis, M.; Littman, M., Learning to select branching rules in the DPLL procedure for satisfiability, (Proceedings of the International Conference on Satisfiability (2001)), 344-359
[66] Le Berre, D.; Lynce, I., CSP2SAT4J: a simple CSP to SAT translator, (Proceedings of the Second International CSP Solver Competition (2008)), 43-54
[67] Leite, R.; Brazdil, P.; Vanschoren, J., Selecting classification algorithms with active testing, (Machine Learning and Data Mining in Pattern Recognition (2012), Springer), 117-131
[68] Leyton-Brown, K.; Nudelman, E.; Andrew, G.; McFadden, J.; Shoham, Y., A portfolio approach to algorithm selection, (Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (2003), Morgan Kaufmann), 1542-1543
[69] Liaw, A.; Wiener, M., Classification and regression by randomForest, R News, 2, 18-22 (2002)
[70] Lindauer, M.; Hoos, H.; Hutter, F.; Schaub, T., Autofolio: an automatically configured algorithm selector, J. Artif. Intell., 53, 745-778 (2015)
[71] Malitsky, Y.; Mehta, D.; O’Sullivan, B., Evolving instance specific algorithm configuration, (The Sixth Annual Symposium on Combinatorial Search (2013))
[72] Malitsky, Y.; O’Sullivan, B.; Previti, A.; Marques-Silva, J., A portfolio approach to enumerating minimal correction subsets for satisfiability problems, (Proceedings of the Eleventh International Conference on Integration of Artificial Intelligence and Operations Research Techniques in Constraint Programming (2014)) · Zbl 06298804
[73] Malitsky, Y.; Sabharwal, A.; Samulowitz, H.; Sellmann, M., Non-model-based algorithm portfolios for SAT, (Proceedings of the Fourteenth International Conference on Theory and Applications of Satisfiability Testing (2011), Springer), 369-370
[74] Mersmann, O.; Bischl, B.; Trautmann, H.; Wagner, M.; Bossek, J.; Neumann, F., A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem, Ann. Math. Artif. Intell., 1-32 (2013)
[75] Nikolić, M.; Marić, F.; Janičić, P., Instance-based selection of policies for SAT solvers, (Proceedings of the Twelfth International Conference on Theory and Applications of Satisfiability Testing (2009), Springer), 326-340
[76] Nogueira, M.; Balduccini, M.; Gelfond, M.; Watson, R.; Barry, M., An A-Prolog decision support system for the Space Shuttle, (Proceedings of the Third International Symposium on Practical Aspects of Declarative Languages (2001), Springer), 169-183
[77] Nudelman, E.; Leyton-Brown, K.; Andrew, G.; Gomes, C.; McFadden, J.; Selman, B.; Shoham, Y., Satzilla 0.9 (2003)
[78] Nudelman, E.; Leyton-Brown, K.; Hoos, H. H.; Devkar, A.; Shoham, Y., Understanding random SAT: beyond the clauses-to-variables ratio, (Principles and Practice of Constraint Programming - CP 2004 (2004), Springer), 438-452 · Zbl 1152.68569
[79] O’Mahony, E.; Hebrard, E.; Holland, A.; Nugent, C.; O’Sullivan, B., Using case-based reasoning in an algorithm portfolio for constraint solving, (Proceedings of the Nineteenth Irish Conference on Artificial Intelligence and Cognitive Science (2008))
[80] Pfahringer, B.; Bensusan, H.; Giraud-Carrier, C., Meta-learning by landmarking various learning algorithms, (Proceedings of the Seventeenth International Conference on Machine Learning (2000)), 743-750
[81] Prasad, M. R.; Biere, A.; Gupta, A., A survey of recent advances in SAT-based formal verification, Int. J. Softw. Tools Technol. Transf., 7, 156-173 (2005)
[82] Pulina, L.; Tacchella, A., A multi-engine solver for quantified Boolean formulas, (Proceedings of the Thirteenth International Conference on Principles and Practice of Constraint Programming (2007), Springer), 574-589
[83] Pulina, L.; Tacchella, A., A self-adaptive multi-engine solver for quantified Boolean formulas, Constraints, 14, 80-116 (2009) · Zbl 1183.68589
[84] R: A Language and Environment for Statistical Computing (2014), R Foundation for Statistical Computing: R Foundation for Statistical Computing Vienna, Austria
[85] Rice, J. R., The algorithm selection problem, Adv. Comput., 15, 65-118 (1976)
[86] Roberts, M.; Howe, A., Learning from planner performance, Artif. Intell. J., 173, 536-561 (2009) · Zbl 1191.68640
[87] Roberts, M.; Howe, A. E., Learned models of performance for many planners, (Proceedings of the Workshop on AI Planning and Learning at the Seventeenth International Conference on Automated Planning and Scheduling (2007))
[88] Roberts, M.; Howe, A. E.; Wilson, B.; desJardins, M., What makes planners predictable?, (ICAPS (2008)), 288-295
[89] Sabharwal, A.; Samulowitz, H.; Sellmann, M.; Malitsky, Y., Boosting sequential solver portfolios: knowledge sharing and accuracy prediction, (LION 7 (2013))
[90] Samulowitz, H.; Memisevic, R., Learning to solve QBF, (Proceedings of the Twenty-Second National Conference on Artificial Intelligence (2007), AAAI Press), 255-260
[91] Serban, F.; Vanschoren, J.; Kietz, J. U.; Bernstein, A., A survey of intelligent assistants for data analysis, ACM Comput. Surv., 45, 1-35 (2013)
[92] Silverthorn, B.; Miikkulainen, R., Latent class models for algorithm portfolio methods, (Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (2010)), 167-172
[93] Smith-Miles, K. A., Cross-disciplinary perspectives on meta-learning for algorithm selection, ACM Comput. Surv., 41, 6:1-6:25 (2008)
[94] Smith-Miles, K. A.; Baatar, D.; Wreford, B. J.; Lewis, R., Towards objective measures of algorithm performance across instance space, Comput. Oper. Res., 45, 12-24 (2014) · Zbl 1348.90646
[95] Soininen, T.; Niemelä, I., Developing a declarative rule language for applications in product configuration, (Proceedings of the First International Workshop on Practical Aspects of Declarative Languages (1999), Springer), 305-319
[96] Stahlbock, R.; Voß, S., Operations research at container terminals: a literature update, OR Spektrum, 30, 1-52 (2008) · Zbl 1133.90313
[97] Stergiou, K., Heuristics for dynamically adapting propagation in constraint satisfaction problems, AI Commun., 22, 125-141 (2009) · Zbl 1185.90191
[98] Streeter, M. J.; Golovin, D.; Smith, S. F., Combining multiple heuristics online, (Proceedings of the Twenty-Second National Conference on Artificial Intelligence (2007), AAAI Press), 1197-1203
[99] Streeter, M. J.; Golovin, D.; Smith, S. F., Restart schedules for ensembles of problem instances, (Proceedings of the Twenty-Second National Conference on Artificial Intelligence (2007), AAAI Press), 1204-1210
[100] Stuckey, P. J.; Feydy, T.; Schutt, A.; Tack, G.; Fischer, J., The MiniZinc challenge 2008-2013, AI Mag., 35, 55-60 (2014)
[101] Tamura, N.; Tanjo, T.; Banbara, M., System description of a SAT-based CSP solver sugar, (Proceedings of the Third International CSP Solver Competition (2008)), 71-75
[102] Tanjo, T.; Tamura, N.; Banbara, M., Azucar: a SAT-based CSP solver using compact order encoding, (Theory and Applications of Satisfiability Testing - SAT 2012 (2012), Springer), 456-462
[103] Therneau, T.; Atkinson, B.; Ripley, B., rpart: recursive partitioning and regression trees (2014), R package version 4.1-8
[104] Thornton, C.; Hutter, F.; Hoos, H.; Leyton-Brown, K., Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms, (Proceedings of the Nineteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2013), ACM), 847-855
[105] Tierney, K.; Malitsky, Y., An algorithm selection benchmark of the container pre-marshalling problem, (Dhaenens, C.; Jourdan, L.; Marmion, M. E., Learning and Intelligent Optimization (2015), Springer International Publishing), 17-22
[106] Tierney, K.; Pacino, D.; Voß, S., Solving the pre-marshalling problem to optimality with A* and IDA* (2014), Decision Support & Optimization Lab, University of Paderborn, Technical Report Working Paper #1401
[107] Vallati, M.; Chrpa, L.; Grzes, M.; McCluskey, T. L.; Roberts, M.; Sanner, S., The 2014 international planning competition: progress and trends, AI Mag., 36, 90-98 (2015)
[108] Vallati, M.; Chrpa, L.; Kitchin, D., Portfolio-based planning: state of the art, common practice and open challenges, AI Commun., 28, 717-733 (2015)
[109] Vallati, M.; Fawcett, C.; Gerevini, A.; Hoos, H. H.; Saetti, A., Automatic generation of efficient domain-optimized planners from generic parametrized planners, (International Symposium on Combinatorial Search (SoCS) (2013))
[110] Van Gelder, A., Another look at graph coloring via propositional satisfiability, Discrete Appl. Math., 156, 230-243 (2008) · Zbl 1131.05090
[111] Vanschoren, J., Understanding machine learning performance with experiment databases (2010), University of Leuven, Ph.D. thesis
[112] Vanschoren, J.; Blockeel, H.; Pfahringer, B.; Holmes, G., Experiment databases. A new way to share, organize and learn from experiments, Mach. Learn., 87, 127-158 (2012)
[113] Vanschoren, J.; van Rijn, J. N.; Bischl, B.; Torgo, L., OpenML: networked science in machine learning, ACM SIGKDD Explor. Newsl., 15, 49-60 (2013)
[114] Ward, J., Hierarchical grouping to optimize an objective function, J. Am. Stat. Assoc., 22, 236-244 (1963)
[115] Xu, H.; Rutenbar, R.; Sakallah, K., Sub-SAT: a formulation for relaxed Boolean satisfiability with applications in routing, (IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2003)), 814-820
[116] Xu, L.; Hoos, H. H.; Leyton-Brown, K., Hierarchical hardness models for SAT, (Proceedings of the Thirteenth International Conference on Principles and Practice of Constraint Programming (2007), Springer), 696-711 · Zbl 1145.68534
[117] Xu, L.; Hoos, H. H.; Leyton-Brown, K., Hydra: automatically configuring algorithms for portfolio-based selection, (Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (2010), AAAI Press), 210-216
[118] Xu, L.; Hutter, F.; Hoos, H. H.; Leyton-Brown, K., SATzilla: portfolio-based algorithm selection for SAT, J. Artif. Intell. Res., 32, 565-606 (2008) · Zbl 1182.68272
[119] Xu, L.; Hutter, F.; Hoos, H. H.; Leyton-Brown, K., Hydra-MIP: automated algorithm configuration and selection for mixed integer programming, (Proceedings of the RCRA Workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion at the Twenty-Second International Joint Conference on Artificial Intelligence (2011))
[120] Xu, L.; Hutter, F.; Hoos, H. H.; Leyton-Brown, K., Evaluating component solver contributions to portfolio-based algorithm selectors, (Proceedings of the Fifteenth International Conference on Theory and Applications of Satisfiability Testing (2012), Springer), 228-241
[121] Xu, L.; Hutter, F.; Shen, J.; Hoos, H. H.; Leyton-Brown, K., Satzilla2012: improved algorithm selection based on cost-sensitive classification models, (Proceedings of SAT Challenge 2012: Solver and Benchmark Descriptions (2012)), 55-58
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.