NetFoldOperator

[Experimental]

✖

represents a net in which net is folded over a sequence of inputs, maintaining a recurrent state.

✖

NetFoldOperator[net,{"out_i""in₁",…}]

represents a net in which net is folded over its inputs, maintaining a recurrent state by feeding the out_i of each step back to the in_i of the next step.

✖

NetFoldOperator[net,feedback,{"const₁","const₂",…}]

treats the inputs const_i to net as being constant instead of mapped over.

✖

NetFoldOperator[net,feedback,constants,{"out₁",…}]

returns the sequence of values for the specific outputs out_i.

Details and Options

NetFoldOperator[net] is equivalent to NetFoldOperator[net,{"Output"->"State"}] and specifies that the output called "Output" of the folded net will be fed back to the input called "State".
NetFoldOperator[…] represents a net that takes one or more sequences and outputs one or more sequences of the same length.
NetFoldOperator[net,{"out₁""in₁",…}] accepts the same inputs as net, except for those in_i given by one of the feedback connections "out_i"->"in_i".
NetFoldOperator[net,feedback,constants] will map over any inputs that are not specified in feedback or constants. All mapped-over inputs should have the same length.
In the typical case in which net has two inputs named "Input" and "State" and one output named "Output", the single argument form NetFoldOperator[net] accepts a single input sequence {x₁,x₂,…,x_n} named "Input" and produces a sequence of states {s₁,s₂,…,s_n} named "Output", computed by applying the net repeatedly to x_i and s_i-1 to compute s_i, effectively using net[<|"Input"->x_i,"State"->s_i-1|>]. The initial state s₀ is an array of zeros.
Within a NetGraph, a connection of the form src->NetPort[layer,"state"] can be used to provide the initial value of a recurrent state of a NetFoldOperator. Similarly, a connection of the form NetPort[layer,"state"]->dst can be used to obtain the final value of a recurrent state. If left unconnected, these states can be tracked by NetStateObject.
NetFoldOperator[net,{"out₁""in₁",…}] will have state ports named for all the recurrent ports in_i whose values are fed back.
NetFoldOperator[net,…] can be seen as allowing a form of weight sharing between multiple copies of net, one for each sequence element that is being mapped over.
NetFoldOperator[…,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:

	NetEncoder[…]	encoder producing an array
	n	a vector of length n
	{n₁,n₂,…}	an array of dimensions n₁×n₂×…
	"Varying"	a vector whose length is variable
	{"Varying",n₂,n₃,…}	an array whose first dimension is variable and remaining dimensions are n₂×n₃×…

The following training parameter can be included:
LearningRateMultipliers Automatic learning rate multipliers for trainable arrays in the net
Options[NetFoldOperator] gives the list of default options to construct the operator. Options[NetFoldOperator[…]] gives the list of default options to evaluate the operator on some data.
Information[NetFoldOperator[…]] gives a report about the operator.
Information[NetFoldOperator[…],prop] gives the value of the property prop of NetFoldOperator[…]. Possible properties are the same as for NetGraph.

Examples

open allclose all

Basic Examples (1)Summary of the most common use cases

Create a "core" net that takes a state and an input and produces an output:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-jih2ti

Out[1]=1

Evaluate the core on an input:

In[2]:=2

✖

https://wolfram.com/xid/0ywk5twv00ke-xp1bkf

Out[2]=2

Create a NetFoldOperator that uses the core to operate on sequences of arrays:

In[3]:=3

✖

https://wolfram.com/xid/0ywk5twv00ke-55hcye

Out[3]=3

Evaluate the net on an input sequence of length-2 vectors:

In[4]:=4

✖

https://wolfram.com/xid/0ywk5twv00ke-3ke204

Out[4]=4

Scope (5)Survey of the scope of standard use cases

Construct a net that takes a weighted average of the state and input:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-k28iv4

Out[1]=1

Apply the net to an input and state:

In[2]:=2

✖

https://wolfram.com/xid/0ywk5twv00ke-qq7ccq

Out[2]=2

Use NetFoldOperator to create a higher-order net that folds the average over an input sequence:

In[3]:=3

✖

https://wolfram.com/xid/0ywk5twv00ke-7jlbbr

Out[3]=3

Fold the accumulator over a sequence of real numbers:

In[4]:=4

✖

https://wolfram.com/xid/0ywk5twv00ke-zle13v

Out[4]=4

Use a NetMapOperator to map the folder over the color channels of an image, resulting in the rows of the image being accumulated:

In[5]:=5

✖

https://wolfram.com/xid/0ywk5twv00ke-jhxo8n

Out[5]=5

Apply the operator to an image:

In[6]:=6

✖

https://wolfram.com/xid/0ywk5twv00ke-u0y3c6

Out[6]=6

Create a net that can take a constant input that is used during the folding operation:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-el4kre

Out[1]=1

In[2]:=2

✖

https://wolfram.com/xid/0ywk5twv00ke-r1q3ta

Out[2]=2

Apply the net to an input sequence, providing a value for the constant:

In[3]:=3

✖

https://wolfram.com/xid/0ywk5twv00ke-s189q3

Out[3]=3

Create a net that has two independent states that are fed back. These states will additionally be used to provide the final output sequences:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-s4r0xz

Out[1]=1

In[2]:=2

✖

https://wolfram.com/xid/0ywk5twv00ke-bocma1

Out[2]=2

Obtain the output sequences when given the input sequences:

In[3]:=3

✖

https://wolfram.com/xid/0ywk5twv00ke-8de57c

Out[3]=3

Construct a net that combines elements from two sequences with a recurrent state using a simple nonlinear function:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-wq1aun

Out[1]=1

In[2]:=2

✖

https://wolfram.com/xid/0ywk5twv00ke-d8tap4

Out[2]=2

Obtain the output sequence when given the input sequences:

In[3]:=3

✖

https://wolfram.com/xid/0ywk5twv00ke-mf8np1

Out[3]=3

Obtain the final state:

In[4]:=4

✖

https://wolfram.com/xid/0ywk5twv00ke-gubvqw

Out[4]=4

Create a NetGraph that allows the initial and final states of a NetFoldOperator to be set and obtained:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-50odgk

Out[1]=1

In[2]:=2

✖

https://wolfram.com/xid/0ywk5twv00ke-kvurdg

Out[2]=2

Apply the net to an input sequence, providing the initial state:

In[3]:=3

✖

https://wolfram.com/xid/0ywk5twv00ke-ukypzy

Out[3]=3

The final state is the last element of the output sequence:

In[4]:=4

✖

https://wolfram.com/xid/0ywk5twv00ke-v1bxir

Out[4]=4

If the initial and final states do not come from a larger network, it is also possible to use NetPort[All,"States"] to control these without using a NetGraph:

In[5]:=5

✖

https://wolfram.com/xid/0ywk5twv00ke-w5mvpt

Out[5]=5

Applications (1)Sample problems that can be solved with this function

Create training data based on strings containing x's and y's and either Less, Greater or Equal by comparing the number of x's and y's. The training data consists of all possible sentences up to length 8:

In[1]:=1

✖

https://wolfram.com/xid/0ywk5twv00ke-ulr9zg

Define a NetGraph representing the computation performed in a single time step by a basic recurrent layer with a state size of 5 and using a Ramp activation:

In[7]:=7