CompiledLayer

CompiledLayer[func]

represents a net layer whose computation is defined by the compilable function func.

CompiledLayer[func,gradientfunc]

specifies a gradient propagating function allowing the layer to be used in NetTrain.

Details and Options

  • CompiledLayer is typically used to create a net layer that computes a custom function that cannot be implemented with other layers.
  • CompiledLayer[func], such as CompiledLayer[func,Automatic], attempts to automatically differentiate func to produce a gradient propagating function. CompiledLayer[func,None] disables automatic differentiation.
  • The functions func and gradientfunc must be valid Function objects suitable to be processed by FunctionCompile (e.g. their argument specification must be built using Typed and TypeSpecifier).
  • CompiledLayer currently only supports a single input and output port.
  • The function func represents the computation performed by the layer, which is also known as the forward pass.
  • The function func must take a single array as input (representing the layer input) and must produce a single array as output (representing the layer output).
  • The function gradientfunc is used to compute input gradients from output gradients, which is also known as the backward pass.
  • The function gradientfunc should take at least two arrays as input representing the layer input and the output gradients. Optionally, the previously computed output can be given as a third argument. The result of the function gradientfunc is a single array representing the layer input gradient.
  • The function func can be defined to take either a single example or a batch of examples. CompiledLayer[func,"Input"…] can be used to specify the rank of the input to disambiguate between the two cases (by comparing with the rank defined in func). func and gradientfunc should either both take a single example or both take a batch of examples.
  • CompiledLayer exposes the following ports for use in NetGraph etc.:
  • "Input"an array
    "Output"an array
  • When given a NumericArray as input, the output will be a NumericArray.
  • Options[CompiledLayer] gives the list of default options to construct the layer. Options[CompiledLayer[…]] gives the list of default options to evaluate the layer on some data.
  • Information[CompiledLayer[…]] gives a report about the layer.
  • Information[CompiledLayer[…],prop] gives the value of the property prop of CompiledLayer[…]. Possible properties are the same as for NetGraph.
  • CompiledLayer does not yet compile to GPU architectures. When setting TargetDevice"GPU", both training and inference will happen on the CPU.

Examples

open allclose all

Basic Examples  (2)

Create a compiled layer from a function taking a rank-3 array of machine precision:

Create a CompiledLayer from a function taking a rank-2 array of 32-bit precision:

Apply the layer to an input:

Compute the input gradient on an input:

Scope  (3)

Create a CompiledLayer with a gradient propagating function:

With the gradient propagating function, the layer can be used to compute gradients and in NetTrain:

Specify a third argument in the gradient propagating function in order to reuse the previously computed output when computing the gradient:

Compute the input gradient using the layer:

Create a CompiledLayer acting on a batch of elements. Define a function reversing an array of rank 2:

Specify an input of rank 1 in CompiledLayer. The function is now interpreted to act on the batch level:

When acting on a single vector of length 4, the vector is unchanged:

When acting on a batch, the order of the vectors is reversed:

When used in this way, CompiledLayer can make different elements in a batch interact with each other, which is normally not possible to do with ordinary layers.

Possible Issues  (5)

Automatic differentiation can fail:

When specifying a gradient propagating function in the second argument, there is a number of strict compatibility constraints that must be satisfied:

When the layer function acts on batches of inputs, the batch dimension must be preserved:

The output shapes produced by the layer and gradient functions are expected to only depend on their input shapes, not on their values. Create a function whose output shape depends on the value of its input:

During shape inference, CompiledLayer runs its functions on test inputs to obtain an expected output shape:

Subsequent evaluations complete successfully if the output shape matches the expected shape:

If an unexpected shape is produced, an error is generated:

For speed purposes, CompiledLayer disables sanity checks that compiled functions normally enable. This can result in bad code causing unexpected behavior or even kernel crashes. Take, for example, a function that accesses an element outside the boundaries of its input array:

The CompiledCodeFunction generated by FunctionCompile throws an error:

CompiledLayer runs the function without checking the array boundaries, resulting in undefined behavior:

Wolfram Research (2020), CompiledLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/CompiledLayer.html (updated 2021).

Text

Wolfram Research (2020), CompiledLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/CompiledLayer.html (updated 2021).

CMS

Wolfram Language. 2020. "CompiledLayer." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2021. https://reference.wolfram.com/language/ref/CompiledLayer.html.

APA

Wolfram Language. (2020). CompiledLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/CompiledLayer.html

BibTeX

@misc{reference.wolfram_2024_compiledlayer, author="Wolfram Research", title="{CompiledLayer}", year="2021", howpublished="\url{https://reference.wolfram.com/language/ref/CompiledLayer.html}", note=[Accessed: 22-November-2024 ]}

BibLaTeX

@online{reference.wolfram_2024_compiledlayer, organization={Wolfram Research}, title={CompiledLayer}, year={2021}, url={https://reference.wolfram.com/language/ref/CompiledLayer.html}, note=[Accessed: 22-November-2024 ]}