--- title: "VideoMapTimeSeries" language: "en" type: "Symbol" summary: "VideoMapTimeSeries[f, video] applies f to each frame of the Video object video, returning a time series. VideoMapTimeSeries[f, video, n] applies f to overlapping partitions of n video frames. VideoMapTimeSeries[f, video, n, d] applies f to partitions with offset d. VideoMapTimeSeries[f, {video1, video2, ...}, ...] applies f to a list of inputs extracted from each videoi." keywords: - video partition - block map - video block map - video analysis - video apply - video local filtering - video local analysis - apply function to video images - apply function to video frames - video measurements canonical_url: "https://reference.wolfram.com/language/ref/VideoMapTimeSeries.html" source: "Wolfram Language Documentation" related_guides: - title: "Video Computation: Update History" link: "https://reference.wolfram.com/language/guide/VideoComputation-UpdateHistory.en.md" - title: "Video Analysis" link: "https://reference.wolfram.com/language/guide/VideoAnalysis.en.md" - title: "Video Processing" link: "https://reference.wolfram.com/language/guide/VideoProcessing.en.md" --- [EXPERIMENTAL] # VideoMapTimeSeries VideoMapTimeSeries[f, video] applies f to each frame of the Video object video, returning a time series. VideoMapTimeSeries[f, video, n] applies f to overlapping partitions of n video frames. VideoMapTimeSeries[f, video, n, d] applies f to partitions with offset d. VideoMapTimeSeries[f, {video1, video2, …}, …] applies f to a list of inputs extracted from each videoi. ## Details and Options * ``VideoMapTimeSeries`` can be used to detect temporal or spatial events in videos, such as object detection, motion detection or activity recognition. [image] * ``VideoMapTimeSeries`` returns a ``TimeSeries`` whose values are the results of ``f`` applied to an association including partial video data and their properties, such as video frames, audio data and time. * The function ``f`` can access video and audio data using the following arguments: | | | | ---------------- | ------------------------------------------------------- | | #Image | video frames as Image objects | | #Audio | a chunk of the audio as an Audio object | | #Time | time from the beginning of the video | | #TimeInterval | beginning and end time stamps for the current partition | | #FrameIndex | index of the current output frame | | #InputFrameIndex | index of the current input frame | * In ``VideoMapTimeSeries[f, {video1, video2, …}, …]``, data provided to each of the arguments is a list where the $$\text{\textit{$i$}}^{\text{th}}$$ element corresponds to the data extracted from ``videoi``. * Using ``VideoMapTimeSeries[f, video, n]``, the partition slides by one image frame. * Frame variables ``n`` and ``d`` can be given as a scalar specifying the number of frames or a time ``Quantity`` object. * ``VideoMapTimeSeries`` supports video containers and codecs specified by ``\$VideoDecoders``. * The following options can be given: | | | | | ----------------- | --------------- | -------------------------------------------- | | Alignment | Center | alignment of the time stamps with partitions | | MetaInformation | None | include additional metainformation | | MissingDataMethod | None | method to use for missing values | | ResamplingMethod | "Interpolation" | the method to use for resampling paths | --- ## Examples (11) ### Basic Examples (2) Compute the mean of the RGB colors for every video frame and plot them: ```wl In[1]:= v = Video["ExampleData/Caminandes.mp4"]; In[2]:= ts = VideoMapTimeSeries[ImageMeasurements[#Image, "Mean"]&, v] Out[2]= TemporalData[TimeSeries, {CompressedData["«59922»"], {{0.020833333333333332, 89.97916666666666, 0.041666666666666664}}, 1, {"Discrete", 1}, {"Discrete", 1}, 3, {MetaInformation -> None, MissingDataMethod -> None, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0}, ValueDimensions -> 3}}, False, 12.2] In[3]:= DateListPlot[ts, PlotStyle -> {Red, Green, Blue}] Out[3]= [image] ``` --- Compute image distance between consecutive frames: ```wl In[1]:= v = Video["ExampleData/Caminandes.mp4"]; In[2]:= ts = VideoMapTimeSeries[ImageDistance@@#Image&, v, 2] Out[2]= TemporalData[TimeSeries, {CompressedData["«11040»"], {{0.041666666666666664, 45., 0.04166666666666667}}, 1, {"Discrete", 1}, {"Discrete", 1}, 1, {MetaInformation -> None, MissingDataMethod -> None, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0}, ValueDimensions -> 1}}, False, 12.2] ``` Plot the result, showing times with significant scene change: ```wl In[3]:= DateListPlot[ts, PlotRange -> All] Out[3]= [image] ``` ### Scope (4) #### Function Specification (2) The function ``f`` receives an ``Association`` holding data for each partition: ```wl In[1]:= v = Video["ExampleData/bullfinch.mkv"]; ``` Check the keys of the provided association: ```wl In[2]:= Quiet[VideoMapTimeSeries[(Print[Keys[#]];Return[])&, v]]; During evaluation of In[2]:= {"Time", "TimeInterval", "InputFrameIndex", "FrameIndex", "Image", "Audio"} ``` Process individual video frames: ```wl In[3]:= ts = VideoMapTimeSeries[ImageMeasurements[#Image, "StandardDeviation"]&, v] Out[3]= TemporalData[TimeSeries, {CompressedData["«13338»"], CompressedData["«1854»"], 1, {"Discrete", 1}, {"Discrete", 1}, 3, {MetaInformation -> None, MissingDataMethod -> None, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0}, ValueDimensions -> 3}}, False, 12.2] ``` Plot the result: ```wl In[4]:= DateListPlot[ts, PlotRange -> All] Out[4]= [image] ``` --- The function ``f`` can operate on the audio data, provided as an ``Audio`` object: ```wl In[1]:= v = Video["ExampleData/bullfinch.mkv"]; ``` Compute time-synchronous measurements on both image and audio data: ```wl In[2]:= ts = VideoMapTimeSeries[Append[ImageMeasurements[#Image, "StandardDeviation"], AudioMeasurements[#Audio, "StandardDeviation"]]&, v] Out[2]= TemporalData[TimeSeries, {CompressedData["«18050»"], CompressedData["«1852»"], 1, {"Discrete", 1}, {"Discrete", 1}, 4, {MetaInformation -> None, MissingDataMethod -> None, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0}, ValueDimensions -> 4}}, False, 12.2] ``` Plot the result: ```wl In[3]:= DateListPlot[ts, ...] Out[3]= [image] ``` #### Partition Specification (2) Process partitions corresponding to four frames: ```wl In[1]:= v = Import["http://exampledata.wolfram.com/popcorn.mov"]; In[2]:= VideoMapTimeSeries[ImageMeasurements[Blend[#Image], "Mean"]&, v, 4]//DateListPlot Out[2]= [image] ``` Specify a partition size using a time ``Quantity`` : ```wl In[3]:= VideoMapTimeSeries[ImageMeasurements[Blend[#Image], "Mean"]&, v, Quantity[200, "Milliseconds"]]//DateListPlot Out[3]= [image] ``` --- By default, the offset of one frame is used: ```wl In[1]:= v = Import["http://exampledata.wolfram.com/popcorn.mov"]; In[2]:= VideoMapTimeSeries[ImageMeasurements[Blend[#Image], "Mean"]&, v, 4]//DateListPlot Out[2]= [image] ``` Use an offset of four frames: ```wl In[3]:= VideoMapTimeSeries[ImageMeasurements[Blend[#Image], "Mean"]&, v, 4, 4]//DateListPlot Out[3]= [image] ``` Specify an offset using a time ``Quantity`` : ```wl In[4]:= VideoMapTimeSeries[ImageMeasurements[Blend[#Image], "Mean"]&, v, 4, Quantity[500, "Milliseconds"]]//DateListPlot Out[4]= [image] ``` Specify an offset proportional to the partition size by a ``Scaled`` amount: ```wl In[5]:= VideoMapTimeSeries[ImageMeasurements[Blend[#Image], "Mean"]&, v, 4, Scaled[1 / 2]]//DateListPlot Out[5]= [image] ``` ### Options (1) #### Alignment (1) By default, the time stamps are aligned with the center of each partition and correspond to the value of the ``"Time"`` key: ```wl In[1]:= v = Video["ExampleData/bullfinch.mkv"]; In[2]:= Normal[VideoMapTimeSeries[#Time&, v]][[ ;; 3]] Out[2]= {{0.0165, 0.0165}, {0.0495, 0.0495}, {0.0835, 0.0835}} ``` Use ``Alignment -> Right`` to return the computed property at the end of each partition: ```wl In[3]:= Normal[VideoMapTimeSeries[#Time&, v, Alignment -> Right]][[ ;; 3]] Out[3]= {{0.033, 0.033}, {0.066, 0.066}, {0.1, 0.1}} ``` Use a custom alignment ranging from –1 (left) to 1 (right): ```wl In[4]:= Normal[VideoMapTimeSeries[#Time&, v, Alignment -> -.5]][[ ;; 3]] Out[4]= {{0.00825, 0.00825}, {0.04125, 0.04125}, {0.07525, 0.07525}} ``` The boundaries of each partition are the start time for the first frame and the end time for the last frame of the partition. They can be queried using the ``"TimeInterval"`` key: ```wl In[5]:= Normal[VideoMapTimeSeries[#TimeInterval&, v]][[ ;; 3]] Out[5]= {{0.0165, {0., 0.033}}, {0.0495, {0.033, 0.066}}, {0.0835, {0.067, 0.1}}} ``` ### Applications (2) Find portions of a video with constant images: ```wl In[1]:= v = Video["ExampleData/Caminandes.mp4"]; ``` Define a function to detect whether an image has constant pixel values: ```wl In[2]:= constantQ[img_] := Boole[Total[Subtract@@ImageMeasurements[img, {"Max", "Min"}]] < .1] ``` Apply the function to each frame and plot the result: ```wl In[3]:= VideoMapTimeSeries[constantQ[#Image]&, v]//ListLinePlot Out[3]= [image] ``` --- Count the number of cars in each frame: ```wl In[1]:= v = Video["http://exampledata.wolfram.com/cars.avi"] Out[1]= [image] In[2]:= VideoMapTimeSeries[Length@ImageCases[#Image, Entity["Concept", "Auto::p735c"]]&, v]//ListLinePlot Out[2]= [image] ``` ### Properties & Relations (1) ``VideoMapTimeSeries`` returns the results along with corresponding times in a ``TimeSeries`` : ```wl In[1]:= v = Video["ExampleData/bullfinch.mkv"]; In[2]:= VideoMapTimeSeries[ImageMeasurements[First[#Image], "MeanIntensity"]&, v, Quantity[2, "Seconds"], Quantity[2, "Seconds"]] Out[2]= TemporalData[TimeSeries, {{{0.4660439133987596, 0.47610896650332235, 0.4479285641340334, 0.45557245710788835, 0.4526614583333747, 0.46148115808828766, 0.46806838916126475, 0.46968368736387345}}, {{{1., 3., 5., 7., 9., 11., 13., 14.991999999999987}}}, 1, {"Discrete", 1}, {"Discrete", 1}, 1, {MetaInformation -> None, MissingDataMethod -> None, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0}, ValueDimensions -> 1}}, False, 12.2] In[3]:= Normal[%] Out[3]= {{1., 0.466044}, {3., 0.476109}, {5., 0.447929}, {7., 0.455572}, {9., 0.452661}, {11., 0.461481}, {13., 0.468068}, {14.992, 0.469684}} ``` Use ``VideoMapList`` to get a list of results without time stamps: ```wl In[4]:= VideoMapList[ImageMeasurements[First[#Image], "MeanIntensity"]&, v, Quantity[2, "Seconds"], Quantity[2, "Seconds"]] Out[4]= {0.466044, 0.476109, 0.447929, 0.455572, 0.452661, 0.461481, 0.468068, 0.469684} ``` ### Possible Issues (1) When the function returns a list, all lists should have similar dimensions: ```wl In[1]:= v = Video["ExampleData/bullfinch.mkv"]; In[2]:= VideoMapTimeSeries[DominantColors[First@#Image]&, v, Quantity[1, "Seconds"], Quantity[1, "Seconds"]] ``` VideoMapTimeSeries::dmmtch: The function DominantColors[First[\#Image]]&@\*KeyTake[{Audio,FrameIndex,Image,InputFrameIndex,OutputTime,OutputTimeInterval,Time,TimeInterval}] did not generate time series values of consistent dimensionality. ```wl Out[2]= VideoMapTimeSeries[DominantColors[First[#Image]]&, \!\(\*VideoBox["![Video Player: ExampleData/bullfinch.mkv](video://content-kp5r4)"]\), Quantity[1, "Seconds"], Quantity[1, "Seconds"]] ``` Pad or trim the resulting lists to the same size to store them in the ``TimeSeries`` : ```wl In[3]:= ts = VideoMapTimeSeries[PadRight[DominantColors[First@#Image], 5, Transparent]&, v, Quantity[1, "Seconds"], Quantity[1, "Seconds"]] Out[3]= TemporalData[TimeSeries, {{{{RGBColor[0.4023895626881033, 0.4171054464216182, 0.3737577215922802], RGBColor[0.48457805418480043, 0.551851206970585, 0.30476187884564654], RGBColor[0.7185481314705715, 0.6627848784911373, 0.6808594825251 ... 1.5, 12.5, 13.5, 14.496999999999996}}}, 1, {"Discrete", 1}, {"Discrete", 1}, 5, {MetaInformation -> None, MissingDataMethod -> None, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0}, ValueDimensions -> 5}}, False, 12.2] In[4]:= ChromaticityPlot[Values[ts]] Out[4]= [image] ``` Results may also be wrapped into other containers before being stored in a ``TimeSeries`` : ```wl In[5]:= ts = VideoMapTimeSeries[Point[ImagePosition[First@#Image, "bird"]]&, v, Quantity[1, "Seconds"], Quantity[1, "Seconds"]]; In[6]:= HighlightImage[VideoExtractFrames[v, 5], Values[ts]] Out[6]= [image] ``` Use ``VideoMapList`` to return the list of varying length lists: ```wl In[7]:= VideoMapList[DominantColors[First@#Image]&, v, Quantity[1, "Seconds"], Quantity[1, "Seconds"]]//Column Out[7]= {RGBColor[0.4023895626881033, 0.4171054464216182, 0.3737577215922802], RGBColor[0.48457805418480043, 0.551851206970585, 0.30476187884564654], RGBColor[0.7185481314705715, 0.6627848784911373, 0.680859482525134], RGBColor[0.18321303504915806, 0.21058 ... 00192537, 0.9560122817430247], RGBColor[0.08104721369686124, 0.07068400507389552, 0.09620988501180747], RGBColor[0.25824297859125683, 0.27244328384963173, 0.3957628292931801], RGBColor[0.46863073437139285, 0.16702639976652753, 0.1345011826356992]} ``` ## See Also * [`VideoIntervals`](https://reference.wolfram.com/language/ref/VideoIntervals.en.md) * [`VideoMapList`](https://reference.wolfram.com/language/ref/VideoMapList.en.md) * [`VideoMap`](https://reference.wolfram.com/language/ref/VideoMap.en.md) * [`VideoFrameMap`](https://reference.wolfram.com/language/ref/VideoFrameMap.en.md) * [`HighlightVideo`](https://reference.wolfram.com/language/ref/HighlightVideo.en.md) * [`VideoFrameList`](https://reference.wolfram.com/language/ref/VideoFrameList.en.md) * [`Video`](https://reference.wolfram.com/language/ref/Video.en.md) * [`VideoTrim`](https://reference.wolfram.com/language/ref/VideoTrim.en.md) * [`AudioBlockMap`](https://reference.wolfram.com/language/ref/AudioBlockMap.en.md) * [`MP4`](https://reference.wolfram.com/language/ref/format/MP4.en.md) * [`QuickTime`](https://reference.wolfram.com/language/ref/format/QuickTime.en.md) * [`Matroska`](https://reference.wolfram.com/language/ref/format/Matroska.en.md) ## Related Guides * [Video Computation: Update History](https://reference.wolfram.com/language/guide/VideoComputation-UpdateHistory.en.md) * [Video Analysis](https://reference.wolfram.com/language/guide/VideoAnalysis.en.md) * [Video Processing](https://reference.wolfram.com/language/guide/VideoProcessing.en.md) ## History * [Introduced in 2020 (12.2)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn122.en.md) \| [Updated in 2021 (12.3)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn123.en.md)