Change HP Name & Include Text example #1410
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,79 +1,84 @@ | ||||||
| # -*- encoding: utf-8 -*- | ||||||
| """ | ||||||
| ================== | ||||||
| Text preprocessing | ||||||
| ================== | ||||||
|
|
||||||
| The following example shows how to fit a simple NLP problem with | ||||||
| *auto-sklearn*. | ||||||
|
|
||||||
| For an introduction to text preprocessing you can follow these links: | ||||||
| 1. https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html | ||||||
| 2. https://machinelearningmastery.com/clean-text-machine-learning-python/ | ||||||
| """ | ||||||
| from pprint import pprint | ||||||
|
|
||||||
| import pandas as pd | ||||||
| import sklearn.metrics | ||||||
| from sklearn.datasets import fetch_20newsgroups | ||||||
|
|
||||||
| import autosklearn.classification | ||||||
|
|
||||||
| ############################################################################ | ||||||
| # Data Loading | ||||||
| # ============ | ||||||
| cats = ["comp.sys.ibm.pc.hardware", "rec.sport.baseball"] | ||||||
| X_train, y_train = fetch_20newsgroups( | ||||||
| subset="train", # select train set | ||||||
| shuffle=True, # shuffle the data set for unbiased validation results | ||||||
| random_state=42, # set a random seed for reproducibility | ||||||
| categories=cats, # select only 2 out of 20 labels | ||||||
| return_X_y=True, # 20NG dataset consists of 2 columns X: the text data, y: the label | ||||||
| ) # load this two columns separately as numpy array | ||||||
|
|
||||||
| X_test, y_test = fetch_20newsgroups( | ||||||
| subset="test", # select test set for unbiased evaluation | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| categories=cats, # select only 2 out of 20 labels | ||||||
| return_X_y=True, # 20NG dataset consists of 2 columns X: the text data, y: the label | ||||||
| ) # load this two columns separately as numpy array | ||||||
|
|
||||||
| ############################################################################ | ||||||
| # Creating a pandas dataframe | ||||||
| # =========================== | ||||||
| # Both categorical and text features are often strings. Python Pandas stores python stings | ||||||
| # in the generic `object` type. Please ensure that the correct | ||||||
| # `dtype <https://pandas.pydata.org/docs/user_guide/basics.html#dtypes>`_ is applied to the correct | ||||||
| # column. | ||||||
|
|
||||||
| # create a pandas dataframe for training labeling the "Text" column as sting | ||||||
| X_train = pd.DataFrame({"Text": pd.Series(X_train, dtype="string")}) | ||||||
|
|
||||||
| # create a pandas dataframe for testing labeling the "Text" column as sting | ||||||
| X_test = pd.DataFrame({"Text": pd.Series(X_test, dtype="string")}) | ||||||
|
|
||||||
| ############################################################################ | ||||||
| # Build and fit a classifier | ||||||
| # ========================== | ||||||
|
|
||||||
| # create an autosklearn Classifier or Regressor depending on your task at hand. | ||||||
| automl = autosklearn.classification.AutoSklearnClassifier( | ||||||
| time_left_for_this_task=60, | ||||||
| per_run_time_limit=30, | ||||||
| tmp_folder="/tmp/autosklearn_text_example_tmp", | ||||||
| ) | ||||||
|
|
||||||
| automl.fit(X_train, y_train, dataset_name="20_Newsgroups") # fit the automl model | ||||||
|
|
||||||
| ############################################################################ | ||||||
| # View the models found by auto-sklearn | ||||||
| # ===================================== | ||||||
|
|
||||||
| print(automl.leaderboard()) | ||||||
|
|
||||||
| ############################################################################ | ||||||
| # Print the final ensemble constructed by auto-sklearn | ||||||
| # ==================================================== | ||||||
|
|
||||||
| pprint(automl.show_models(), indent=4) | ||||||
|
|
||||||
| ########################################################################### | ||||||
| # Get the Score of the final ensemble | ||||||
| # =================================== | ||||||
|
|
||||||
| predictions = automl.predict(X_test) | ||||||
| print("Accuracy score:", sklearn.metrics.accuracy_score(y_test, predictions)) | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.