DeleteAnomalies
DeleteAnomalies[{example1,example2,…}]
gives a list in which examplei that are considered anomalous have been dropped.
DeleteAnomalies[fun,data]
drops anomalies in data using the given AnomalyDetectorFunction[…] or LearnedDistribution[…].
Details and Options
- DeleteAnomalies can be used on many types of data, including numerical, nominal and images.
- Each examplei can be a single data element, a list of data elements or an association of data elements. Examples can also be given as a Dataset object.
- DeleteAnomalies attempts to model the distribution of non-anomalous data in order to detect anomalies (i.e. "out-of-distribution" examples). Examples are considered anomalous when their RarerProbability is below the value specified for AcceptanceThreshold.
- In DeleteAnomalies[AnomalyDetectorFunction[…],data], if data comes from the same distribution as the training examples for the detector, AcceptanceThreshold corresponds to the anomaly detection false-positive rate.
- The following options can be given:
-
AcceptanceThreshold 0.001 RarerProbability threshold to consider an example anomalous FeatureExtractor Identity how to extract features from which to learn FeatureNames Automatic feature names to assign for input data FeatureTypes Automatic feature types to assume for input data Method Automatic which modeling algorithm to use PerformanceGoal Automatic aspects of performance to optimize RandomSeeding 1234 what seeding of pseudorandom generators should be done internally TimeGoal Automatic how long to spend training the detector TrainingProgressReporting Automatic how to report progress during training ValidationSet Automatic the set of data on which to evaluate the model during training - Possible settings for PerformanceGoal include:
-
"Quality" maximize the modeling quality of the detector "Speed" maximize speed for detecting anomalies Automatic automatic tradeoff among speeds, quality and memory {goal1,goal2,…} automatically combine goal1, goal2, etc. - Possible settings for Method are as given in LearnDistribution[…].
- The following settings for TrainingProgressReporting can be used:
-
"Panel" show a dynamically updating graphical panel "Print" periodically report information using Print "ProgressIndicator" show a simple ProgressIndicator "SimplePanel" dynamically updating panel without learning curves None do not report any information - DeleteAnomalies[…,FeatureExtractor"Minimal"] indicates that the internal preprocessing should be as simple as possible.
Examples
open allclose allBasic Examples (3)
Scope (2)
Train an AnomalyDetectorFunction on a two-dimensional array of pseudorandom real numbers:
Use the trained AnomalyDetectorFunction with DeleteAnomalies to drop anomalous examples:
Obtain a random sample of training and test datasets of images:
Add anomalous examples to corrupt the datasets:
Train a distribution on images:
Use the trained distribution to drop anomalous examples in the test set:
Options (2)
AcceptanceThreshold (1)
Specify the AcceptanceThreshold for dropping anomalies from a list of colors:
Applications (4)
Find the statistical mean of numeric values with an anomaly:
Plot a list of numeric values that contains anomalies:
Plot a list of numeric values after removing anomalies:
Delete anomalies before computing the mean:
Find the linear fit for numerical data:
Delete outliers before modeling the fit:
Text
Wolfram Research (2019), DeleteAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/DeleteAnomalies.html (updated 2020).
CMS
Wolfram Language. 2019. "DeleteAnomalies." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/DeleteAnomalies.html.
APA
Wolfram Language. (2019). DeleteAnomalies. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/DeleteAnomalies.html