AutoML using Asynchronous PySpark

Maggy is a framework for efficient asynchronous optimization of expensive black-box functions in Hopsworks.

In Hopsworks, Maggy is mostly used to run experiments for:

  • Directed Hyperparameter Search (ASHA, Bayesian) on TensorFlow, PyTorch, ScikitLearn, XGBoost;
  • Parallel Ablation Studies. Without changing your inner training loop in TensorFlow, evaluate (in parallel) the effect of regularization techniques;
  • Unified Logging in Jupyter notebooks. In real-time, view logs for parallel HParam/Ablation study tasks from your Jupyter notebook.

For more details, please read Maggy’s own documentation: