Skip to content

embed

Author: Heli Qi Affiliation: NAIST Date: 2022.07

EmbedPrenet

Bases: Module

Create new embeddings for the vocabulary.

Use scaling for the Transformer.

Source code in speechain/module/prenet/embed.py 
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
class EmbedPrenet(Module):
    """Create new embeddings for the vocabulary.

    Use scaling for the Transformer.
    """

    def module_init(
        self, embedding_dim, vocab_size, scale: bool = False, padding_idx: int = 0
    ):
        """

        Args:
            embedding_dim: int
                The dimension of token embedding vectors
            scale: bool
                Controls whether the values of embedding vectors are scaled according to the embedding dimension.
                Useful for the Transformer model.
            vocab_size: int
                The number of tokens in the dictionary.
            padding_idx: int
                The token index used for padding the tail areas of the short sentences.

        """
        # output_size initialization
        self.output_size = embedding_dim

        # para recording
        self.embedding_dim = embedding_dim
        self.vocab_size = vocab_size
        self.scale = scale
        self.padding_idx = padding_idx

        # initialize Embedding layer
        self.embed = torch.nn.Embedding(
            num_embeddings=vocab_size,
            embedding_dim=embedding_dim,
            padding_idx=padding_idx,
        )

    def forward(self, text: torch.Tensor):
        """Perform lookup for input `x` in the embedding table.

        Args:
            text: (batch, seq_len)
            index in the vocabulary

        Returns:
            embedded representation for `x`
        """
        if self.scale:
            return self.embed(text) * math.sqrt(self.embedding_dim)
        else:
            return self.embed(text)

forward(text)

Perform lookup for input x in the embedding table.

Parameters:

Name Type Description Default
text Tensor

(batch, seq_len)

 required

Returns:

Type Description

embedded representation for x
Source code in speechain/module/prenet/embed.py 
53
54
55
56
57
58
59
60
61
62
63
64
65
66
def forward(self, text: torch.Tensor):
    """Perform lookup for input `x` in the embedding table.

    Args:
        text: (batch, seq_len)
        index in the vocabulary

    Returns:
        embedded representation for `x`
    """
    if self.scale:
        return self.embed(text) * math.sqrt(self.embedding_dim)
    else:
        return self.embed(text)

module_init(embedding_dim, vocab_size, scale=False, padding_idx=0)

Parameters:

Name Type Description Default
embedding_dim

int The dimension of token embedding vectors

 required
scale bool

bool Controls whether the values of embedding vectors are scaled according to the embedding dimension. Useful for the Transformer model.

 False
vocab_size

int The number of tokens in the dictionary.

 required
padding_idx int

int The token index used for padding the tail areas of the short sentences.

 0
Source code in speechain/module/prenet/embed.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
def module_init(
    self, embedding_dim, vocab_size, scale: bool = False, padding_idx: int = 0
):
    """

    Args:
        embedding_dim: int
            The dimension of token embedding vectors
        scale: bool
            Controls whether the values of embedding vectors are scaled according to the embedding dimension.
            Useful for the Transformer model.
        vocab_size: int
            The number of tokens in the dictionary.
        padding_idx: int
            The token index used for padding the tail areas of the short sentences.

    """
    # output_size initialization
    self.output_size = embedding_dim

    # para recording
    self.embedding_dim = embedding_dim
    self.vocab_size = vocab_size
    self.scale = scale
    self.padding_idx = padding_idx

    # initialize Embedding layer
    self.embed = torch.nn.Embedding(
        num_embeddings=vocab_size,
        embedding_dim=embedding_dim,
        padding_idx=padding_idx,
    )