--- title: "ImageContents" language: "en" type: "Symbol" summary: "ImageContents[image] gives a dataset of identified entities in image. ImageContents[image, category] gives a dataset that only contains entities in the specified category. ImageContents[image, category, prop] includes the properties prop for each identified object. ImageContents[video, ...] gives a time series of detected objects in frames of video." keywords: - object detection - find subimage - subimage position - template matching - find template - find image template - locate image template - image recognition - identify image canonical_url: "https://reference.wolfram.com/language/ref/ImageContents.html" source: "Wolfram Language Documentation" related_guides: - title: "Computer Vision" link: "https://reference.wolfram.com/language/guide/ComputerVision.en.md" - title: "Converting between Expressions & Strings" link: "https://reference.wolfram.com/language/guide/ConvertingBetweenExpressionsAndStrings.en.md" - title: "Segmentation Analysis" link: "https://reference.wolfram.com/language/guide/SegmentationAnalysis.en.md" - title: "Video Computation: Update History" link: "https://reference.wolfram.com/language/guide/VideoComputation-UpdateHistory.en.md" - title: "Image Computation: Update History" link: "https://reference.wolfram.com/language/guide/ImageComputation-UpdateHistory.en.md" - title: "Video Analysis" link: "https://reference.wolfram.com/language/guide/VideoAnalysis.en.md" related_functions: - title: "ImageContainsQ" link: "https://reference.wolfram.com/language/ref/ImageContainsQ.en.md" - title: "ImageCases" link: "https://reference.wolfram.com/language/ref/ImageCases.en.md" - title: "ImageIdentify" link: "https://reference.wolfram.com/language/ref/ImageIdentify.en.md" - title: "ImageInstanceQ" link: "https://reference.wolfram.com/language/ref/ImageInstanceQ.en.md" - title: "ImagePosition" link: "https://reference.wolfram.com/language/ref/ImagePosition.en.md" - title: "ImageBoundingBoxes" link: "https://reference.wolfram.com/language/ref/ImageBoundingBoxes.en.md" - title: "FacialFeatures" link: "https://reference.wolfram.com/language/ref/FacialFeatures.en.md" - title: "TextRecognize" link: "https://reference.wolfram.com/language/ref/TextRecognize.en.md" - title: "TextContents" link: "https://reference.wolfram.com/language/ref/TextContents.en.md" --- [EXPERIMENTAL] # ImageContents ImageContents[image] gives a dataset of identified entities in image. ImageContents[image, category] gives a dataset that only contains entities in the specified category. ImageContents[image, category, prop] includes the properties prop for each identified object. ImageContents[video, …] gives a time series of detected objects in frames of video. ## Details and Options * ``ImageContents`` is also known as object detection and attempts to find instances of an object in an image and returns a dataset of identified properties such as name, bounding box and probability. * Possible forms for ``category`` include: | | | | --------------------------- | --------------------------------------------- | | "concept" | named concept, as used in "Concept" entities | | "word" | English word, as used in WordData | | wordspec | word sense specification, as used in WordData | | Entity[…] | any appropriate entity | | category1 \| category2 \| … | any of the categoryi | * The property ``prop`` can be any of the following: | | | | ----------------- | ------------------------------------- | | "BoundingBox" | the bounding box given as a Rectangle | | "Confidence" | confidence of the identification | | "Dimensions" | width and height of the subimage | | "Image" | the identified subimage | | "Position" | center of the identified bounding box | | {prop1, prop2, …} | a list of properties | * Masked subimage properties: | | | | ----------------- | -------------------------------------- | | "Mask" | binary mask indicating the object | | "MaskBoundingBox" | the bounding box of the masked image | | "MaskCentroid" | centroid of the mask | | "MaskedImage" | subimage masked by the component shape | | "MaskMedoid" | medoid of the mask | | "MaskPosition" | center of the mask bounding box | * The following options can be given: | | | | | -------------------- | ---------- | ----------------------------------------- | | AcceptanceThreshold | Automatic | identification acceptance threshold | | MaxFeatures | Automatic | maximum number of subimages to return | | MaxOverlapFraction | Automatic | maximum bounding box overlap | | PerformanceGoal | "Balanced" | aspects of performance to try to optimize | | TargetDevice | "CPU" | the target device on which to compute | * When no content is found at an acceptable threshold, ``Missing["NotRecognized"]`` is returned. * ``ImageContents`` uses machine learning. Its methods, training sets and biases included therein may change and yield varied results in different versions of the Wolfram Language. * ``ImageContents`` may download resources that will be stored in your local object store at ``\$LocalBase``, and can be listed using ``LocalObjects[]`` and removed using ``ResourceRemove``. --- ## Examples (20) ### Basic Examples (1) Identify contents of an image: ```wl In[1]:= ImageContents[[image]] Out[1]= Dataset[«1»] ``` Get results only in the specified category: ```wl In[2]:= ImageContents[[image], Entity["Concept", "Zebra::nx5qr"]] Out[2]= Dataset[{Association["Image" -> Image[CompressedData["«40808»"], "Byte", ColorSpace -> "RGB", Interleaving -> True], "Concept" -> Entity["Concept", "Zebra::nx5qr"], "BoundingBox" -> Rectangle[{105, 8}, {169, 164}], "Confidence" -> 0.91359 ... "Image" -> Image[CompressedData["«103352»"], "Byte", ColorSpace -> "RGB", Interleaving -> True], "Concept" -> Entity["Concept", "Zebra::nx5qr"], "BoundingBox" -> Rectangle[{28, 38}, {212, 174}], "Confidence" -> 0.8059133540540415]}] ``` ### Scope (11) #### Data (2) Detect objects in an image: ```wl In[1]:= ImageContents[[image]] Out[1]= Dataset[{Association["Image" -> Image[CompressedData["«28526»"], "Byte", ColorSpace -> "RGB", Interleaving -> True], "Concept" -> Entity["Concept", "Auto::p735c"], "BoundingBox" -> Rectangle[{456, 73}, {569, 160}], "Confidence" -> 0.933286 ... ation["Image" -> Image[CompressedData["«4072»"], "Byte", ColorSpace -> "RGB", Interleaving -> True], "Concept" -> Entity["Concept", "Auto::p735c"], "BoundingBox" -> Rectangle[{147, 236}, {187, 266}], "Confidence" -> 0.5895928022428052]}] ``` --- Detect objects in frames of video: ```wl In[1]:= ts = ImageContents[\!\(\*VideoBox["![Embedded Video Player](video://content-k8ee4)"]\)] Out[1]= TemporalData[TimeSeries, «1»] ``` Objects at a given time: ```wl In[2]:= ts[.3] Out[2]= Dataset[{Association["Image" -> Image[CompressedData["«29740»"], "Byte", ColorSpace -> "RGB", Interleaving -> True], "Concept" -> Entity["Concept", "Auto::p735c"], "BoundingBox" -> Rectangle[{473, 50}, {591, 142}], "Confidence" -> 0.936365 ... iation["Image" -> Image[CompressedData["«20780»"], "Byte", ColorSpace -> "RGB", Interleaving -> True], "Concept" -> Entity["Concept", "Auto::p735c"], "BoundingBox" -> Rectangle[{348, 172}, {436, 241}], "Confidence" -> 0.799910353777733]}] ``` #### Categories (3) Find all the detectable objects in an image: ```wl In[1]:= ImageContents[[image]] Out[1]= Dataset[{Association["Image" -> Image[CompressedData["«35944»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Person::93r37"], "BoundingBox" -> Rectangle[{9, 5}, {98, 145 ... essedData["«33688»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Bird::56ny3"], "BoundingBox" -> Rectangle[{223, 55}, {350, 182}], "Confidence" -> 0.9111570762783572]}] ``` --- Find all birds in an image: ```wl In[1]:= ImageContents[[image], Entity["Species", "Class:Aves"]] Out[1]= Dataset[{Association["Image" -> Image[CompressedData["«33688»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Bird::56ny3"], "BoundingBox" -> Rectangle[{223, 55}, {350, 182}], "Confidence" -> 0.9111570762783572]}] ``` --- Group together the detection for multiple categories: ```wl In[1]:= ImageCases[[image], Entity["Species", "Class:Aves"] | Entity["Concept", "Person::93r37"] -> {"Name", "Image"}] Out[1]= {{"person", [image]}, {"bird", [image]}} ``` #### Properties (6) Return all properties for all classes of objects: ```wl In[1]:= ImageContents[[image]] Out[1]= Dataset[{Association["Image" -> Image[CompressedData["«35944»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Person::93r37"], "BoundingBox" -> Rectangle[{9, 5}, {98, 145 ... essedData["«33688»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Bird::56ny3"], "BoundingBox" -> Rectangle[{223, 55}, {350, 182}], "Confidence" -> 0.9111570762783572]}] ``` --- Return one property for all classes: ```wl In[1]:= ImageContents[[image], All, "BoundingBox"] Out[1]= Dataset[{Association["BoundingBox" -> Rectangle[{9, 5}, {98, 145}]], Association["BoundingBox" -> Rectangle[{223, 55}, {350, 182}]]}] ``` --- Return one property for a specified class: ```wl In[1]:= ImageContents[[image], Entity["Species", "Class:Aves"], "BoundingBox"] Out[1]= Dataset[{Association["BoundingBox" -> Rectangle[{223, 55}, {350, 182}]]}] ``` --- Return multiple properties for all objects: ```wl In[1]:= ImageContents[[image], All, {"BoundingBox", "Confidence"}] Out[1]= Dataset[{Association["BoundingBox" -> Rectangle[{9, 5}, {98, 145}], "Confidence" -> 0.9129925923951987], Association["BoundingBox" -> Rectangle[{223, 55}, {350, 182}], "Confidence" -> 0.9111570762783572]}] ``` --- Return multiple properties for the specified class: ```wl In[1]:= ImageContents[[image], Entity["Species", "Class:Aves"], {"BoundingBox", "Confidence"}] Out[1]= Dataset[{Association["BoundingBox" -> Rectangle[{223, 55}, {350, 182}], "Confidence" -> 0.9111570762783572]}] ``` --- Query properties of the concept: ```wl In[1]:= ImageContents[[image], Entity["Species", "Class:Aves"], {"Name", "Definition"}] Out[1]= Dataset[{Association["Name" -> "bird", "Definition" -> "warm-blooded egg-laying vertebrates characterized by feathers and forelimbs modified as wings"]\ }] ``` ### Options (4) #### AcceptanceThreshold (1) Objects with low probability are not returned: ```wl In[1]:= i = [image]; In[2]:= ImageContents[i] Out[2]= Dataset[«1»] ``` Allowing a lower probability may result in more objects being recognized: ```wl In[3]:= ImageContents[i, AcceptanceThreshold -> .2] Out[3]= Dataset[«1»] ``` #### MaxFeatures (1) By default, all the detected objects are returned: ```wl In[1]:= i = [image]; In[2]:= ImageContents[i] Out[2]= Dataset[«1»] ``` Specify a maximum number of results: ```wl In[3]:= ImageContents[i, MaxFeatures -> 2] Out[3]= Dataset[{Association["Image" -> Image[CompressedData["«39882»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {300, 300}, Interleaving -> True, MetaInformation -> Association["Exif" -> Association["Make" -> "NIKON CORPORATION", "Mod ... True, "RedEyeCorrection" -> False], "FocalLength" -> Quantity[60., "Millimeters"]]]], "Concept" -> Entity["Concept", "Wineglass::w5dj5"], "BoundingBox" -> Rectangle[{143, 4}, {205, 215}], "Confidence" -> 0.9179262391257836]}] ``` #### MaxOverlapFraction (1) The detected bounding boxes may overlap each other: ```wl In[1]:= i = [image]; In[2]:= ImageContents[i, All, "BoundingBox"] Out[2]= Dataset[{Association["BoundingBox" -> Rectangle[{21, 20}, {93, 162}]], Association["BoundingBox" -> Rectangle[{143, 4}, {205, 215}]], Association["BoundingBox" -> Rectangle[{101, 43}, {147, 223}]], Association["BoundingBox" -> Rectangle[{2 ... ingBox" -> Rectangle[{302, 108}, {344, 229}]], Association["BoundingBox" -> Rectangle[{382, 128}, {407, 233}]], Association["BoundingBox" -> Rectangle[{0, 3}, {480, 138}]], Association["BoundingBox" -> Rectangle[{448, 111}, {480, 186}]]}] In[3]:= HighlightImage[i, Normal@%[All, "BoundingBox"]] Out[3]= [image] ``` Find only non-intersecting objects: ```wl In[4]:= ImageContents[i, MaxOverlapFraction -> 0]; In[5]:= HighlightImage[i, Normal[%[[All, "BoundingBox"]]]] Out[5]= [image] ``` #### TargetDevice (1) By default, the function is evaluated on CPU: ```wl In[1]:= i = [image]; In[2]:= RepeatedTiming[ImageContents[i];] Out[2]= {0.38483, Null} ``` Use the ``TargetDevice`` option to specify a different device: ```wl In[3]:= RepeatedTiming[ImageContents[i, TargetDevice -> "GPU"];] Out[3]= {0.0829374, Null} ``` ### Applications (4) Get a summary of all the identified entities in an image: ```wl In[1]:= ImageContents[[image]] Out[1]= Dataset[«1»] ``` --- Get a summary only in the specified category: ```wl In[1]:= ImageContents[[image], Entity["Concept", "Bowl::8mv4z"]] Out[1]= Dataset[{Association["Image" -> Image[CompressedData["«26350»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Bowl::8mv4z"], "BoundingBox" -> Rectangle[{320, 99}, {4 ... pressedData["«30458»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Concept" -> Entity["Concept", "Bowl::8mv4z"], "BoundingBox" -> Rectangle[{124, 17}, {296, 96}], "Confidence" -> 0.5622660766163108]}] ``` --- Specify which properties to include in the summary: ```wl In[1]:= ImageContents[[image], Entity["Concept", "Bowl::8mv4z"], {"Image", "Name", "Confidence", "Position"}] Out[1]= Dataset[{Association["Image" -> Image[CompressedData["«26350»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Name" -> "bowl", "Confidence" -> 0.91193854214184, "Position" -> {397.5, 130.}], Associa ... " -> {264., 166.}], Association["Image" -> Image[CompressedData["«30458»"], "Byte", ColorSpace -> "RGB", ImageResolution -> {72, 72}, Interleaving -> True], "Name" -> "bowl", "Confidence" -> 0.5622660766163108, "Position" -> {210., 56.5}]}] ``` --- Perform object detection and segmentation: ```wl In[1]:= ImageContents[[image], All, {"Concept", "Mask", "MaskedImage"}] Out[1]= Dataset[<>] ``` ## See Also * [`ImageContainsQ`](https://reference.wolfram.com/language/ref/ImageContainsQ.en.md) * [`ImageCases`](https://reference.wolfram.com/language/ref/ImageCases.en.md) * [`ImageIdentify`](https://reference.wolfram.com/language/ref/ImageIdentify.en.md) * [`ImageInstanceQ`](https://reference.wolfram.com/language/ref/ImageInstanceQ.en.md) * [`ImagePosition`](https://reference.wolfram.com/language/ref/ImagePosition.en.md) * [`ImageBoundingBoxes`](https://reference.wolfram.com/language/ref/ImageBoundingBoxes.en.md) * [`FacialFeatures`](https://reference.wolfram.com/language/ref/FacialFeatures.en.md) * [`TextRecognize`](https://reference.wolfram.com/language/ref/TextRecognize.en.md) * [`TextContents`](https://reference.wolfram.com/language/ref/TextContents.en.md) ## Related Guides * [Computer Vision](https://reference.wolfram.com/language/guide/ComputerVision.en.md) * [Converting between Expressions & Strings](https://reference.wolfram.com/language/guide/ConvertingBetweenExpressionsAndStrings.en.md) * [Segmentation Analysis](https://reference.wolfram.com/language/guide/SegmentationAnalysis.en.md) * [Video Computation: Update History](https://reference.wolfram.com/language/guide/VideoComputation-UpdateHistory.en.md) * [Image Computation: Update History](https://reference.wolfram.com/language/guide/ImageComputation-UpdateHistory.en.md) * [Video Analysis](https://reference.wolfram.com/language/guide/VideoAnalysis.en.md) ## History * [Introduced in 2019 (12.0)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn120.en.md) \| [Updated in 2024 (14.0)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn140.en.md) ▪ [2025 (14.2)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn142.en.md)