TransformAnomalies

TransformAnomalies[data,tspec]

transforms anomalies in data using the transformation tspec.

TransformAnomalies[data,dspectspec]

detects the anomalies using the specification dspec.

TransformAnomalies[tab,{col1spec1,…}]

transforms tabular data tab using the specification speci for column coli.

Details

  • TransformAnomalies is also known as anomaly detection or outlier treatment.
  • TransformAnomalies helps maintain data quality by effectively identifying and managing anomalies, ensuring reliable analysis.
  • Possible forms of data include:
  • {el1,el2,…}a list of indivudual elements
    tabtabular data
  • Possible forms of tabular data tab include:
  • Tabulara tabular object
    Dataseta dataset object
    {assoc1,assoc2,…}a list of associations
    matrixa matrix
  • Possible transformations tspec for managing the anomalies include:
  • "Clip"limit extreme values to non-anomalies range
    "Replace"replace anomalies with a calculated value
    {"Replace",method}replace anomalies using method
    "Trim"drop the anomalous elements
    "Flag"replace anomalies with Missing["Anomalous"]
    tfunuse the function tfun to transform the anomalies
  • Possible "Replace" methods are the same as in TransformMissing:
  • " Mean"the mean value
    "Median"the median value
    "Mode"the most frequent value
    {"Constant", val}a constant value val
    "RandomElement"randomly selected value
    {"RandomVariate",dist}value sampled from dist
    "NearestElement"use the closest non-missing element
    "NextElement"use the next non-missing element
    "PreviousElement"use the previous non-missing element
    "Interpolation"use interpolation
    {"Interpolation",InterpolationOrderdegree}specify the interpolation order degree
  • Excluding the anomalies minimizes their influence on analysis. Clipping or replacing minimizes the impact of anomalies while preserving the dataset's structure. Flagging allows keeping all data points for further analysis while highlighting anomalies for review.
  • Available methods dspec to detect the anomalies include:
  • "IQR"interquartile range thresholds
    "ZScore"standard deviations from the mean
    "ModifiedZScore"robust version using median and MAD
    {spec,t}specifies the threshold t for anomaly detection
    {spec,{t1,t2}}specifies the lower and upper thresholds
  • Additionally, the following specifications are supported:
  • Scaled[f]a fraction f of the smallest and largest elements
    Scaled[{f1,f2}]separate thresholds for smallest and largest elements
    dfuna custom function fun
  • A custom anomaly detector dfun is applied to each element individually and must return a Boolean value.
  • When specifying the desired columns, coli can have the following format:
  • nthe n^(th) column
    "name"the column "name"
    TypeSpecifier[…]columns matching the specified type
    Allall the columns
    Automaticautomatically find columns based on spec (default)

Examples

open allclose all

Basic Examples  (2)

Handle outliers by removing large and small values:

Clip the data above the 95^(th) percentile:

Scope  (16)

Data Types  (5)

Detect anomalies in a list:

Detect anomalies in a TabularColumn:

Detect anomalies along matrix columns:

Detect anomalies in a Tabular object:

Detect anomalies in a list of associations across the same key:

Detection Methods  (6)

Detect anomalies using the interquartile range:

Detect anomalies using the z-score:

Detect anomalies using the modified z-score:

Specify a custom threshold:

Specify upper and lower thresholds individually:

Flag a fraction of the smallest and largest elements:

Specify separate smallest and largest fractions:

Use a custom function to detect anomalies:

Transformation Methods  (5)

Remove the anomalies:

Clip the anomalies with the closest valid value:

Replace the anomalies with missing elements:

Apply any TransformMissing specification:

Apply a custom transformation:

Applications  (2)

Take a Tabular object of sales data:

The "Sales" column contains outlier zero values, which are standing in for missing values:

Transform the zero anomalies by removing the relevant rows from the original data:

Transform the zero anomalies by converting them to Missing objects:

Check the new data:

Take a Tabular object of weather observations:

Plot the temperature:

It appears the temperature data contains outliers, possibly resulting from entering the data in wrong units. Similarly, the "Pressure" column contains outliers, that is number 99, which is often used in place of missing values. Specify the temperature unit conversion function:

Transform anomalies using the above function for the "Temperature" column and in the "Pressure" column replace the anomaly with the nearest valid value:

Plot the observations:

Wolfram Research (2025), TransformAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/TransformAnomalies.html.

Text

Wolfram Research (2025), TransformAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/TransformAnomalies.html.

CMS

Wolfram Language. 2025. "TransformAnomalies." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TransformAnomalies.html.

APA

Wolfram Language. (2025). TransformAnomalies. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TransformAnomalies.html

BibTeX

@misc{reference.wolfram_2025_transformanomalies, author="Wolfram Research", title="{TransformAnomalies}", year="2025", howpublished="\url{https://reference.wolfram.com/language/ref/TransformAnomalies.html}", note=[Accessed: 08-May-2025 ]}

BibLaTeX

@online{reference.wolfram_2025_transformanomalies, organization={Wolfram Research}, title={TransformAnomalies}, year={2025}, url={https://reference.wolfram.com/language/ref/TransformAnomalies.html}, note=[Accessed: 08-May-2025 ]}