Joblib HTCondor Backend
The Joblib HTCondor library provides an HTCondor backend for joblib to easily queue jobs on high throughput computational clusters using HTCondor. With this tool, users can parallelize a large number of compute-intensive tasks and specify their execution right inside of their analyses. This allows to parallelize software that uses joblib, for example scikit-learn. Joblib HTCondor also supports nested parallel calls in which the backend is used inside a parallel call. This is useful when a large number of tasks can be parallelized, and each of these tasks can be parallelized as well - a common pattern in machine learning, e.g., with several folds in a cross-validation where the training of each fold can be parallelized (e.g. hyperparameter tuning).