Model Building: Preprocessing Methods: Difference between revisions
imported>Jeremy |
imported>Scott No edit summary |
||
(10 intermediate revisions by 2 users not shown) | |||
Line 5: | Line 5: | ||
==Preprocessing Methods== | ==Preprocessing Methods== | ||
Data preprocessing describes any type of processing procedures that are performed on raw data to prepare it for another processing procedure and ultimately, analysis. Preprocessing linearizes the relationships among the variables in your DataSet and removes extraneous sources of variation that are of no interest to the analysis. A variety of preprocessing methods are available in Solo. | Data [[preprocess|preprocessing]] describes any type of processing procedures that are performed on raw data to prepare it for another processing procedure and ultimately, analysis. Preprocessing linearizes the relationships among the variables in your DataSet and removes extraneous sources of variation that are of no interest to the analysis. A variety of preprocessing methods are available in Solo. | ||
The '''[[#Preprocessing window|Preprocessing window]]''' section describes the basic steps for setting up preprocessing rules for an analysis and verifying that the rules that you have set up are as you want them. | The '''[[#Preprocessing window|Preprocessing window]]''' section describes the basic steps for setting up preprocessing rules for an analysis and verifying that the rules that you have set up are as you want them. | ||
Line 15: | Line 15: | ||
* [[Advanced Preprocessing: Noise, Offset, and Baseline Filtering | Noise, Offset, and Baseline Filtering]] (e.g. Baselining, Smoothing, Derivatives, Detrending) | * [[Advanced Preprocessing: Noise, Offset, and Baseline Filtering | Noise, Offset, and Baseline Filtering]] (e.g. Baselining, Smoothing, Derivatives, Detrending) | ||
* [[Advanced Preprocessing: Sample Normalization | Sample Normalization]] (e.g. Normalization, MSC, SNV) | * [[Advanced Preprocessing: Sample Normalization | Sample Normalization]] (e.g. Normalization, MSC, SNV) | ||
* [[Advanced Preprocessing: Simple Mathematical Operations | Simple Mathematical Operations]] | * [[Advanced Preprocessing: Simple Mathematical Operations | Simple Mathematical Operations]] (e.g. Absolute Value, Log10, log(1/T) ) | ||
* [[Advanced Preprocessing: Variable Centering | Variable Centering]] (e.g. Mean Centering, Median Centering) | * [[Advanced Preprocessing: Variable Centering | Variable Centering]] (e.g. Mean Centering, Median Centering, Class Centering) | ||
* [[Advanced Preprocessing: Variable Scaling| Variable Scaling]] (e.g. Autoscaling) | * [[Advanced Preprocessing: Variable Scaling| Variable Scaling]] (e.g. Autoscaling, Pareto, Poisson, Log Decay) | ||
* [[Advanced Preprocessing: Multiway Centering and Scaling| Multiway Centering and Scaling]] | * [[Advanced Preprocessing: Multiway Centering and Scaling| Multiway Centering and Scaling]] | ||
* [[Advanced Preprocessing: Multivariate Filtering| Multivariate Filtering]] (e.g. GLSW, EPO, OSC) | * [[Advanced Preprocessing: Multivariate Filtering| Multivariate Filtering]] (e.g. GLSW, EPO, OSC) | ||
* [[Variable_Alignment_Settings| Variable Alignment]] (e.g. Registerspec, Correlation Optimized Warping) | |||
* [[Image Preprocessing Methods| Image-Specific Methods]] (e.g. Background-Subtraction, Erosion, Smoothing) | |||
===Preprocessing Window=== | |||
===Preprocessing | |||
You use the Preprocessing window to specify the preprocessing methods that you want to carry out for your data and in what order. The available methods are grouped by type (Filtering, Normalization, and so on) in the Available Methods (left) pane of the window. The methods that you select for preprocessing your data are displayed in the Selected Methods (right) pane. The methods are carried out in the order in which they are listed in the pane. The default selected method is Autoscale. | You use the Preprocessing window to specify the preprocessing methods that you want to carry out for your data and in what order. The available methods are grouped by type (Filtering, Normalization, and so on) in the Available Methods (left) pane of the window. The methods that you select for preprocessing your data are displayed in the Selected Methods (right) pane. The methods are carried out in the order in which they are listed in the pane. The default selected method is Autoscale. | ||
Line 31: | Line 32: | ||
:: | :: | ||
::[[Image:Preprocessing_window.png| | ::[[Image:Preprocessing_window.png|500px]] | ||
:: | :: | ||
Line 176: | Line 177: | ||
::[[Image:Preprocessing_icon_tooltip_text.png|498x122px]] | ::[[Image:Preprocessing_icon_tooltip_text.png|498x122px]] | ||
:: | ===Customizing Preprocessing Window=== | ||
The preprocessing selections can be customized by adding frequently used methods to the "favorites" list or by "hiding" unused methods. | |||
* Add a method to "Favorites" list by selecting it on the Selected Methods (right) panel and clicking the "Favorite" button at the top of the panel. | |||
* Remove a method from "Favorites" by selecting it on the Available Methods panel and clicking the "Delete" button (same as the Hide button in image below). | |||
* Hide a method by selecting it in the Available Methods panel and clicking the "Hide" button [see image below]. | |||
* Unhide a method expanding (clicking) the "Hidden" list, selecting the method, and clicking "Unhide" button (same as the Hide button in image below). | |||
::[[Image:CustomizePP.jpg| |Customize Preprocessing]] |
Latest revision as of 12:24, 25 September 2014
Table of Contents | Previous | Next
Preprocessing Methods
Data preprocessing describes any type of processing procedures that are performed on raw data to prepare it for another processing procedure and ultimately, analysis. Preprocessing linearizes the relationships among the variables in your DataSet and removes extraneous sources of variation that are of no interest to the analysis. A variety of preprocessing methods are available in Solo.
The Preprocessing window section describes the basic steps for setting up preprocessing rules for an analysis and verifying that the rules that you have set up are as you want them.
The Advanced Preprocessing: Introduction page gives a technical discussion of Preprocessing theory.
A thorough discussion of the objectives, theory, and equations associated with specific preprocessing methods can be found in the individual topics listed below:
- Noise, Offset, and Baseline Filtering (e.g. Baselining, Smoothing, Derivatives, Detrending)
- Sample Normalization (e.g. Normalization, MSC, SNV)
- Simple Mathematical Operations (e.g. Absolute Value, Log10, log(1/T) )
- Variable Centering (e.g. Mean Centering, Median Centering, Class Centering)
- Variable Scaling (e.g. Autoscaling, Pareto, Poisson, Log Decay)
- Multiway Centering and Scaling
- Multivariate Filtering (e.g. GLSW, EPO, OSC)
- Variable Alignment (e.g. Registerspec, Correlation Optimized Warping)
- Image-Specific Methods (e.g. Background-Subtraction, Erosion, Smoothing)
Preprocessing Window
You use the Preprocessing window to specify the preprocessing methods that you want to carry out for your data and in what order. The available methods are grouped by type (Filtering, Normalization, and so on) in the Available Methods (left) pane of the window. The methods that you select for preprocessing your data are displayed in the Selected Methods (right) pane. The methods are carried out in the order in which they are listed in the pane. The default selected method is Autoscale.
- Preprocessing window
1. | To open this window, do one of the following: |
|
|
|
2. | In the Available Methods pane, select the method by which you want to preprocess your data, and then click Add. |
- Note: Some preprocessing methods, for example, the Savitzky-Golay method, require you to specify values for method-specific parameters before you can add the method to the Selected Methods pane.
3. | Repeat Step 2 until you have selected all of the necessary preprocessing methods. |
- Note: Regardless of the other preprocessing steps selected, Autoscale or Mean Centering should typically be the final preprocessing step in the series.
4. | Optionally, after you have selected all of the necessary preprocessing methods, you can do one or more of the following: |
|
|
|
|
Note: The figure below shows preprocessing using default preprocessing methods on a 200 x 30 DataSet for a simple PCA model.
- Preprocessing using default preprocessing methods on a 200 x 30 DataSet for a simple PCA model
|
After you have selected preprocessing methods for your data, you can point your mouse cursor on the Preprocessing icon for a control in which you have loaded data. Tooltip text opens with preprocessing information.
- Tooltip text with preprocessing information
Customizing Preprocessing Window
The preprocessing selections can be customized by adding frequently used methods to the "favorites" list or by "hiding" unused methods.
- Add a method to "Favorites" list by selecting it on the Selected Methods (right) panel and clicking the "Favorite" button at the top of the panel.
- Remove a method from "Favorites" by selecting it on the Available Methods panel and clicking the "Delete" button (same as the Hide button in image below).
- Hide a method by selecting it in the Available Methods panel and clicking the "Hide" button [see image below].
- Unhide a method expanding (clicking) the "Hidden" list, selecting the method, and clicking "Unhide" button (same as the Hide button in image below).