WOLFRAM

Wolfram Language & System Documentation Center
Wolfram Language Home Page »

TransformAnomalies
✖
TransformAnomalies

New in 14.2[Experimental]

✖
TransformAnomalies[data,tspec]

transforms anomalies in data using the transformation tspec.

✖
TransformAnomalies[data,dspectspec]

detects the anomalies using the specification dspec.

✖
TransformAnomalies[tab,{col1spec1,…}]

transforms tabular data tab using the specification speci for column coli.

Details

  • TransformAnomalies is also known as anomaly detection or outlier treatment.
  • TransformAnomalies helps maintain data quality by effectively identifying and managing anomalies, ensuring reliable analysis.
  • Possible forms of data include:
  • {el1,el2,…}a list of indivudual elements
    tabtabular data
  • Possible forms of tabular data tab include:
  • Tabulara tabular object
    Dataseta dataset object
    {assoc1,assoc2,…}a list of associations
    matrixa matrix
  • Possible transformations tspec for managing the anomalies include:
  • "Clip"limit extreme values to non-anomalies range
    "Replace"replace anomalies with a calculated value
    {"Replace",method}replace anomalies using method
    "Trim"drop the anomalous elements
    "Flag"replace anomalies with Missing["Anomalous"]
    tfunuse the function tfun to transform the anomalies
  • Possible "Replace" methods are the same as in TransformMissing:
  • " Mean"the mean value
    "Median"the median value
    "Mode"the most frequent value
    {"Constant", val}a constant value val
    "RandomElement"randomly selected value
    {"RandomVariate",dist}value sampled from dist
    "NearestElement"use the closest non-missing element
    "NextElement"use the next non-missing element
    "PreviousElement"use the previous non-missing element
    "Interpolation"use interpolation
    {"Interpolation",InterpolationOrderdegree}specify the interpolation order degree
  • Excluding the anomalies minimizes their influence on analysis. Clipping or replacing minimizes the impact of anomalies while preserving the dataset's structure. Flagging allows keeping all data points for further analysis while highlighting anomalies for review.
  • Available methods dspec to detect the anomalies include:
  • "IQR"interquartile range thresholds
    "ZScore"standard deviations from the mean
    "ModifiedZScore"robust version using median and MAD
    {spec,t}specifies the threshold t for anomaly detection
    {spec,{t1,t2}}specifies the lower and upper thresholds
  • Additionally, the following specifications are supported:
  • Scaled[f]a fraction f of the smallest and largest elements
    Scaled[{f1,f2}]separate thresholds for smallest and largest elements
    dfuna custom function fun
  • A custom anomaly detector dfun is applied to each element individually and must return a Boolean value.
  • When specifying the desired columns, coli can have the following format:
  • nthe n^(th) column
    "name"the column "name"
    TypeSpecifier[…]columns matching the specified type
    Allall the columns
    Automaticautomatically find columns based on spec (default)

Examples

open allclose all

Basic Examples  (2)Summary of the most common use cases

Handle outliers by removing large and small values:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-bgffpd
Out[1]=1

Clip the data above the 95^(th) percentile:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-8iazn9
In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-vj1mjz
Out[2]=2

Scope  (16)Survey of the scope of standard use cases

Data Types  (5)

Detect anomalies in a list:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-5qw5ki
Out[1]=1

Detect anomalies in a TabularColumn:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-3skyhj
In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-wo232c
Out[2]=2
In[3]:=3
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-8kwc1q
Out[3]=3
In[4]:=4
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-zc874m
Out[4]=4

Detect anomalies along matrix columns:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-i31bl4
Out[1]=1

Detect anomalies in a Tabular object:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-4u2u8i
In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-xq3pwz
Out[2]=2

Detect anomalies in a list of associations across the same key:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-hxlxl6
Out[1]=1

Detection Methods  (6)

Detect anomalies using the interquartile range:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-1wnbug
Out[1]=1

Detect anomalies using the z-score:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-cx9jx4
Out[1]=1

Detect anomalies using the modified z-score:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-6ws6qo
Out[1]=1

Specify a custom threshold:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-cmyee3
Out[1]=1

Specify upper and lower thresholds individually:

In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-q63xa2
Out[2]=2

Flag a fraction of the smallest and largest elements:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-zm5drg
Out[1]=1

Specify separate smallest and largest fractions:

In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-0oi5in
Out[2]=2

Use a custom function to detect anomalies:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-ezv21d
Out[1]=1

Transformation Methods  (5)

Remove the anomalies:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-lwf4ma
Out[1]=1

Clip the anomalies with the closest valid value:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-y8sj87
Out[1]=1

Replace the anomalies with missing elements:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-74aib9
Out[1]=1

Apply any TransformMissing specification:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-vvx7e5
Out[1]=1

Apply a custom transformation:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-fb9oxh
Out[1]=1

Applications  (2)Sample problems that can be solved with this function

Take a Tabular object of sales data:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-h39wd1

The "Sales" column contains outlier zero values, which are standing in for missing values:

In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-cre4sv
Out[2]=2

Transform the zero anomalies by removing the relevant rows from the original data:

In[3]:=3
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-y3xwc
Out[3]=3

Transform the zero anomalies by converting them to Missing objects:

In[4]:=4
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-itutb4
Out[4]=4

Check the new data:

In[5]:=5
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-qmp448
Out[5]=5

Take a Tabular object of weather observations:

In[1]:=1
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-fha1be

Plot the temperature:

In[2]:=2
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-2hkg3k
Out[2]=2

It appears the temperature data contains outliers, possibly resulting from entering the data in wrong units. Similarly, the "Pressure" column contains outliers, that is number 99, which is often used in place of missing values. Specify the temperature unit conversion function:

In[3]:=3
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-mks3kb

Transform anomalies using the above function for the "Temperature" column and in the "Pressure" column replace the anomaly with the nearest valid value:

In[4]:=4
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-rtj7di
Out[4]=4

Plot the observations:

In[5]:=5
✖
https://wolfram.com/xid/0bzqjb04t4ot63dhw-wnfy3e
Out[5]=5
Wolfram Research (2025), TransformAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/TransformAnomalies.html.
✖
Wolfram Research (2025), TransformAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/TransformAnomalies.html.

Text

Wolfram Research (2025), TransformAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/TransformAnomalies.html.

✖
Wolfram Research (2025), TransformAnomalies, Wolfram Language function, https://reference.wolfram.com/language/ref/TransformAnomalies.html.

CMS

Wolfram Language. 2025. "TransformAnomalies." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TransformAnomalies.html.

✖
Wolfram Language. 2025. "TransformAnomalies." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TransformAnomalies.html.

APA

Wolfram Language. (2025). TransformAnomalies. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TransformAnomalies.html

✖
Wolfram Language. (2025). TransformAnomalies. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TransformAnomalies.html

BibTeX

@misc{reference.wolfram_2025_transformanomalies, author="Wolfram Research", title="{TransformAnomalies}", year="2025", howpublished="\url{https://reference.wolfram.com/language/ref/TransformAnomalies.html}", note=[Accessed: 13-May-2025 ]}

✖
@misc{reference.wolfram_2025_transformanomalies, author="Wolfram Research", title="{TransformAnomalies}", year="2025", howpublished="\url{https://reference.wolfram.com/language/ref/TransformAnomalies.html}", note=[Accessed: 13-May-2025 ]}

BibLaTeX

@online{reference.wolfram_2025_transformanomalies, organization={Wolfram Research}, title={TransformAnomalies}, year={2025}, url={https://reference.wolfram.com/language/ref/TransformAnomalies.html}, note=[Accessed: 13-May-2025 ]}

✖
@online{reference.wolfram_2025_transformanomalies, organization={Wolfram Research}, title={TransformAnomalies}, year={2025}, url={https://reference.wolfram.com/language/ref/TransformAnomalies.html}, note=[Accessed: 13-May-2025 ]}