EmbeddingLayer

EmbeddingLayer[size,n]

represents a trainable net layer that embeds integers between 1 and n into a continuous vector space of dimension size.

EmbeddingLayer[size]

leaves the n to be inferred from context.

Details and Options

  • EmbeddingLayer operates on a single integer or rank-k array of integers, producing a vector or rank-k+1 array, respectively.
  • Specifying EmbeddingLayer[size] will produce a net that infers the n when a NetEncoder[…] that produces integers is connected to its input.
  • The embedding performed by EmbeddingLayer is learned during training.
  • The following optional parameters can be included:
  • LearningRateMultipliersAutomaticlearning rate multipliers for the weights
    "Weights" Automaticinitial matrix of weights of dimensions n×size
  • With Automatic settings, the weights are added automatically when NetInitialize or NetTrain is used.
  • If weights have been added, EmbeddingLayer[…][input] explicitly computes the output from applying the layer.
  • EmbeddingLayer[…][{input1,input2,…}] explicitly computes outputs for each of the inputi.
  • When given a NumericArray as input, the output will be a NumericArray.
  • NetExtract can be used to extract weights from an EmbeddingLayer object.
  • EmbeddingLayer is typically used inside NetChain, NetGraph, etc.
  • EmbeddingLayer exposes the following ports for use in NetGraph etc.:
  • "Input"an integer or rank-k array of integers
    "Output"a vector or rank-k+1 array
  • EmbeddingLayer[size,n,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:
  • NetEncoder[…]encoder producing an integer or array of integers
    "Integer"a single integer
    da vector of integers of length d
    {d1,d2,…}an array of integers of dimensions d1×d2×…
    "Varying"a variable-length vector of integers
    {"Varying",d2,d3,…}an array whose first dimension is variable and whose remaining dimensions are d2×d3×…
  • Options[EmbeddingLayer] gives the list of default options to construct the layer. Options[EmbeddingLayer[…]] gives the list of default options to evaluate the layer on some data.
  • Information[EmbeddingLayer[…]] gives a report about the layer.
  • Information[EmbeddingLayer[…],prop] gives the value of the property prop of EmbeddingLayer[…]. Possible properties are the same as for NetGraph.

Examples

open allclose all

Basic Examples  (2)

Create an EmbeddingLayer that will take integers 1, 2 and 3 and produces vectors of size 4:

Create a randomly initialized EmbeddingLayer that will take integers 1, 2 and 3 and produces vectors of size 2:

Apply the layer to an integer:

Apply the layer to a vector of integers:

Scope  (4)

Create a randomly initialized EmbeddingLayer that embeds a variable-length sequence of integers from 1 to 10 into a sequence of vectors of length 2 represented by a matrix whose last dimension has size 2:

Apply the layer to a single sequence of integers:

Apply the layer to multiple sequences simultaneously, in which the sequence lengths are not identical:

Create an EmbeddingLayer with an encoded input, where n is inferred from the encoder:

Randomly initialize and apply the layer:

Create an EmbeddingLayer whose input is a sequence of integers representing the characters of a string:

Apply the layer to a string:

Create an EmbeddingLayer whose input is a sequence of codes representing tokens from a string:

Options  (1)

"Weights"  (1)

Create an EmbeddingLayer with weights explicitly specified:

Apply the layer to some inputs:

Applications  (2)

In text processing applications, networks that operate on entire words typically use an embedding to collapse the large vocabulary size of human languages into manageable vector sizes.

Create a chain that embeds successive words from a string into a sequence of vectors of size 10 and then processes those vectors, using a recurrent layer to produce a final vector. First create an encoder using common English words:

Create and randomly initialize the net, attaching the encoder to its input:

Randomly initialize the net and apply it to some sentences:

Obtain a pre-trained EmbeddingLayer using NetModel:

A NetEncoder has been attached to the input of the net:

Apply the embedding to a sentence containing two tokens:

Visualize the embedding of a longer sentence:

Properties & Relations  (1)

Evaluate an initialized EmbeddingLayer on a list of indices:

This is equivalent to the following:

Wolfram Research (2016), EmbeddingLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/EmbeddingLayer.html (updated 2020).

Text

Wolfram Research (2016), EmbeddingLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/EmbeddingLayer.html (updated 2020).

CMS

Wolfram Language. 2016. "EmbeddingLayer." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/EmbeddingLayer.html.

APA

Wolfram Language. (2016). EmbeddingLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/EmbeddingLayer.html

BibTeX

@misc{reference.wolfram_2024_embeddinglayer, author="Wolfram Research", title="{EmbeddingLayer}", year="2020", howpublished="\url{https://reference.wolfram.com/language/ref/EmbeddingLayer.html}", note=[Accessed: 22-December-2024 ]}

BibLaTeX

@online{reference.wolfram_2024_embeddinglayer, organization={Wolfram Research}, title={EmbeddingLayer}, year={2020}, url={https://reference.wolfram.com/language/ref/EmbeddingLayer.html}, note=[Accessed: 22-December-2024 ]}