这四种超参调试技术，你都了解了吗？-FlyAI

这四种超参调试技术，你都了解了吗？

深度学习

最后更新 2020-11-16 15:30 阅读 4552

最后更新 2020-11-16 15:30

阅读 4552

深度学习

##FlyAI文献翻译
**英文原文：**[：4 Hyper-Parameter Tuning Techniques](https://medium.com/swlh/4-hyper-parameter-tuning-techniques-924cb188d199)
> **标签** ：深度学习

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599790968625656.png)

>图片来自[SigOpt](https://sigopt.com/blog/common-problems-in-hyperparameter-optimization/ "SigOpt")

##Introduction

- Wikipedia states that “Hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm”

One of the toughest parts in ML workflow is finding the best hyperparameters for the model. Performance of ML models is directly related with Hyper-parameter’s. The more you tune the hyperparameters, the better model you get. Tuning Hyper-parameters could be really tedious, difficult and is more of an art than science.

##介绍

>- Wikipedia  指出：“超参数优化或调整是为学习算法选择一组最佳超参数的问题”

>ML工作流程中最困难的部分之一是为模型找到最佳的超参数。 ML模型的性能与超参数的性能直接相关。 您对超参数的调整越多，获得的模型就越好。 调整超参数可能真的很繁琐，困难，并且比科学更是一门艺术。

##Hyper-parameters

Hyper-parameters are the parameters used to control the behavior of the algorithm while building the model. These parameters cannot be learned from the regular training process. They need to be assigned before training of the model.

>##超参数

>超参数是用于在构建模型时控制算法行为的参数。 这些参数无法从常规培训过程中学习。 在训练模型之前，需要先分配它们。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599790968280260.png)

A sample list of Hyper-parameters by Dr.Mukesh Rao

>Dr.Mukesh Rao的超参数示例列表

##Table of Content

1. Traditional or Manual Tuning

3. Grid Search

5. Random Search

7. Bayesian Search

>##目录

>. 传统或手动调优

>. 网格搜索

>. 随机搜索

>. 贝叶斯搜索

## Traditional or Manual Tuning

In Traditional Tuning, we manually check with random sets of hyper-parameters by training the algorithm and select the best set of parameters that fits our objective.

Let's see with code

>##传统或手动调优

>统调优中，我们通过训练算法来手动检查随机的超参数集，并选择适合我们目标的最佳参数集。

>我们看一下代码

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599791099576858.png)

Drawbacks:

1. No guarantee to get the best parameter combination.

3. It’s a trial and error approach. So, it consumes more time.

>缺点：

>. 无法保证获得最佳的参数组合。

>. 这是一种反复试验的方法。 因此，它消耗更多时间。

##Grid Search

Grid Search is a basic hyper-parameter tuning technique. It is similar to manual tuning where it builds a model for each permutation of all the given hyperparameter values specified in the grid, evaluates and selects the best model. Consider the above example where it has two hyperparameters k_value =[2,3,4,5,6,7,8,9,10] & algorithm = [‘auto’ , ’ball_tree’ , ’kd_tree’ , ’brute’], in this case it builds a total of 9*4 = 36 different models.

##网络搜索

>搜索是一种基本的超参数调整技术。 它类似于手动调整，在手动调整中，它为网格中指定的所有给定超参数值的每个排列构建一个模型，评估并选择最佳模型。 考虑上面的示例，它有两个超参数k_value = [2,3,4,5,6,7,8,9,10]＆算法= ['auto'，'ball_tree'，'kd_tree'，'brute'] ，在这种情况下，它总共构建了9 * 4 = 36个不同的模型。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599790969374365.png)

Image from SigOpt

Let's understand how sklearn’s GridSearchCV works,

>图片来自 SigOpt

>我们了解sklearn的GridSearchCV的工作原理

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599791143203831.png)

Drawback:

As it tries every combination of the hyper-parameters and choose the best combination based on the cross-validation score, this makes GridsearchCV extremely slow.

> 缺点：

>当它尝试超参数的每种组合并根据交叉验证得分选择最佳组合时，这会使GridsearchCV极其缓慢。

##3. Random Search

The motivation to use random search in place of grid search is that for many cases, all the hyperparameters may not be equally important. Random Search selects random combination of parameters from the hyper-parameter space, parameters will be selected with fixed numbers of iterations given by n_iter. Random search has empirically been demonstrated to give better results than Grid Search.

##3.随机搜索

>使用随机搜索代替网格搜索的动机是，在许多情况下，所有超参数可能都没有同等重要。 随机搜索从超参数空间中选择参数的随机组合，参数将以n_iter给出的固定迭代次数进行选择。 实验证明，随机搜索比网格搜索可提供更好的结果。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599790968203842.png)

Image from SigOpt

Let's understand how sklearn’s RandomizedSearchCV works,

>图片来自 SigOpt

>让我们了解sklearn的RandomizedSearchCV的工作原理，

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599791195518035.png)

Drawback:

The problem with random search is it doesn’t guarantee to give the best parameter combination.

>缺点：
>
随机搜索的问题在于不能保证给出最佳的参数组合。

##Bayesian Search

Bayesian optimization falls in a class of optimization algorithms called sequential model-based optimization (SMBO) algorithms. These algorithms use previous observations of the loss f , to determine the next (optimal) point to sample f for. The algorithm can roughly be outlined as follows.

1. Using previously evaluated points X1:n, compute a posterior expectation of what the loss f looks like.

3. Sample the loss f at a new point Xnew, that maximizes some utility of the expectation of f. The utility specifies which regions of the domain of f are optimal to sample from.

These steps are repeated until some convergence criterion is met.

>##贝叶斯搜索

>叶斯优化属于一类优化算法，称为基于顺序模型的优化（SMBO）算法。 这些算法使用对损耗f的先前观测值来确定采样f的下一个（最佳）点。 该算法大致可概述如下。

>. 使用先前评估的点X1：n，计算损失f的后验期望。
>
3. 在新点Xnew上对损耗f进行采样，这将最大化f期望值的效用。 实用程序指定f的哪个区域是最佳样本。

>重复这些步骤，直到满足某些收敛标准为止。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599790968662481.png)

Image from SigOpt

Let's understand this with scikit-optimize’s BayesSearchCV

Installation : pip install scikit-optimize

>图片来自 SigOpt

>让我们通过scikit-optimize的BayesSearchCV了解这一点。

>安装：pip install scikit-optimize

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599791270391649.png)

Another similar library to implement Bayesian Search is bayesian-optimization

Installation : pip install bayesian-optimization

Drawback:

It takes a dozen or so samples to get a good surrogate surface in 2 or 3 dimensions of search space; increasing dimensionality of the search space requires yet more samples.

>另一个实现贝叶斯搜索的相似库是贝叶斯优化
>
安装：
>
pip install bayesian-optimization
>
缺点：
>
在2到3维的搜索空间中，需要十几个样本才能获得良好的代理曲面； 搜索空间维数的增加需要更多的样本。

##Conclusion

There is always a trade-off between the guarantee to identify the best combination of parameters and the computation time. If your hyper-parameter space (number of hyper-parameters) is very large, then use Random Search to find the potential combination of hyper-parameters and then use Grid Search in that locality (potential combination of hyper-parameters) to select the optimal features.

>##结论

>在确定参数的最佳组合和计算时间之间始终要进行取舍。如果您的超参数空间（超参数的数量）非常大，请使用随机搜索找到超参数的潜在组合，然后在该位置使用网格搜索（超参数的潜在组合）选择最佳特征。

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://flyai.com/article/758

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除

这四种超参调试技术， 你都了解了吗？

这四种超参调试技术，你都了解了吗？