Grid search with SSPOC

[janezlapajne]

Hello, as I have seen in example pysensor supports sklearn based parameters optimization. In the example, it is shown how the parameter tuning is performed for SSPOR. However, I would like to perform it for SSPOC, but for now I was unable to do so. At the end of optimization it throws me an error: ValueError: w must be a 1D vector; received a vector of dimension 0 which is implemented in pysensors\utils\_optimizers.py. Furthermore, scores which are calculated during grid search are all nan values.

I am attaching simple code snippet in case anyone could help. I hope I will not have to code grid search manually.

model_clf = LinearDiscriminantAnalysis()
model_cs = SSPOC(classifier=model_clf)

param_grid = {
    "basis": [ps.basis.Identity(), ps.basis.SVD(), ps.basis.RandomProjection()],
    "basis__n_basis_modes": [1,2,3,4,5],
    "n_sensors": [5,2,3]
}

cross_val = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1)
search = GridSearchCV(estimator = model_cs, 
                        param_grid = param_grid, 
                        scoring = "balanced_accuracy", 
                        n_jobs = -1, 
                        cv = cross_val,
                        refit = True,
                        verbose = 0,
                    )

search.fit(X_train, y_train)

print("---------------------------------------")
print("Best parameters:")
for k, v in search.best_params_.items():
    print(f"{k}:  {v}")
print("")

[briandesilva]

I'll look into this and get back to you in the next few days.

[janezlapajne]

Perfect, thank you! If you need anything else (data, more code etc.) let me know.

[briandesilva]

I was able to reproduce the error, but I am not yet sure what the cause is. It might take me some time to get to the bottom of this. If you are blocked in the meantime, I'd recommend using a tool like Tune for hyperparameter optimization.

[briandesilva]

@janezlapajne, try downloading the latest pysensors release—I just pushed a minor fix—and then replacing

search.fit(X_train, y_train)

with

search.fit(X_train, y_train, refit=False)

I think what's going wrong is the SSPOC model is failing to actually refit the classifier for new parameter combinations, and so it passes measurements of varying dimension to the same fixed classifier (SSPOC typically picks a different number of relevant sensors for each set of hyperparameters).

[janezlapajne]

Hello, thank you for your response. I installed a new version and the score is now available. Grid search seems to accurately find the best set of hyperparameters.

Also, thank you for a recommendation of the ray tune library. I personally use it when the set of hyperparameters is extensive enough. In case of simple tuning tasks and smaller datasets, I usually just directly use sklearn grid search (I find it easier to use). Do you think I should always just use tune? Anyways, if you have any additional recommendations about the tuning, I will really appreciate it! For the time being, I have an article on my bucket list, titled: PySensors: A Python Package for Sparse Sensor Placement, for which I am still waiting for the time to be able to read it.

[briandesilva]

I think you've got it right—use sklearn grid search when the search space is small enough that grid search is feasible and use ray tune when the search space is larger. I'd recommend taking a look at section 5 of our paper for some practical tips (only a few paragraphs long).