Pseudocode

The following is the pseudocode for the general embedding procedure.^[9][7]

algorithm Compute entity and relation embeddings is
    input: The training set ${\displaystyle S=\{(h,r,t)\}}$, 
           entity set ${\displaystyle E}$, 
           relation set ${\displaystyle R}$,     
           embedding dimension ${\displaystyle k}$
    output: Entity and relation embeddings

    initialization: the entities ${\displaystyle e}$ and relations ${\displaystyle r}$ embeddings (vectors) are randomly initialized

    while stop condition do
        ${\displaystyle S_{batch}\leftarrow sample(S,b)}$ // From the training set randomly sample a batch of size b
        for each ${\displaystyle (h,r,t)}$ in ${\displaystyle S_{batch}}$ do
            ${\displaystyle (h',r,t')\leftarrow sample(S')}$  // sample a corrupted fact of triple
            ${\displaystyle T_{batch}\leftarrow T_{batch}\cup \{((h,r,t),(h',r,t'))\}}$ 
        end for
        Update embeddings by minimizing the loss function
    end while

Hits@K

Hits@K or in short, H@K, is a performance index that measures the probability to find the correct prediction in the first top K model predictions.^[10] Usually, it is used ${\displaystyle k=10}$.^[10] Hits@K reflects the accuracy of an embedding model to predict the relation between two given triples correctly.^[10]

Hits@K${\displaystyle ={\frac {|\{q\in Q:q<k\}|}{|Q|}}\in [0,1]}$

Larger values mean better predictive performances.^[10]

Mean rank (MR)

Mean rank is the average ranking position of the items predicted by the model among all the possible items.^[10]

${\displaystyle MR={\frac {1}{|Q|}}\sum _{q\in Q}{q}}$

The smaller the value, the better the model.^[10]

Mean reciprocal rank (MRR)

Mean reciprocal rank measures the number of triples predicted correctly.^[10] If the first predicted triple is correct, then 1 is added, if the second is correct ${\displaystyle {\frac {1}{2}}}$ is summed, and so on.^[10]

Mean reciprocal rank is generally used to quantify the effect of search algorithms.^[10]

${\displaystyle MRR={\frac {1}{|Q|}}\sum _{q\in Q}{\frac {1}{q}}\in [0,1]}$

The larger the index, the better the model.^[10]

Bilinear models

This family of models uses a linear equation to embed the connection between the entities through a relation.^[1] In particular, the embedded representation of the relations is a bidimensional matrix.^[5] These models, during the embedding procedure, only use the single facts to compute the embedded representation and ignore the other associations to the same entity or relation.^[18]

• DistMult^[19]: Since the embedding matrix of the relation is a diagonal matrix,^[5] the scoring function can not distinguish asymmetric facts.^[5][18]
• ComplEx^[20]: As DistMult uses a diagonal matrix to represent the relations embedding but adds a representation in the complex vector space and the hermitian product, it can distinguish symmetric and asymmetric facts.^[5][17] This approach is scalable to a large knowledge graph in terms of time and space cost.^[20]
• ANALOGY^[21]: This model encodes in the embedding the analogical structure of the knowledge graph to simulate inductive reasoning.^[21][5][1] Using a differentiable objective function, ANALOGY has good theoretical generality and computational scalability.^[21] It is proven that the embedding produced by ANALOGY fully recovers the embedding of DistMult, ComplEx, and HolE.^[21]
• SimplE^[22]: This model is the improvement of canonical polyadic decomposition (CP), in which an embedding vector for the relation and two independent embedding vectors for each entity are learned, depending on whether it is a head or a tail in the knowledge graph fact.^[22] SimplE resolves the problem of independent learning of the two entity embeddings using an inverse relation and average the CP score of ${\displaystyle (h,r,t)}$ and ${\displaystyle (t,r^{-1},h)}$.^[7][17] In this way, SimplE collects the relation between entities while they appear in the role of subject or object inside a fact, and it is able to embed asymmetric relations.^[5]

Non-bilinear models

• HolE:^[23] HolE uses circular correlation to create an embedded representation of the knowledge graph,^[23] which can be seen as a compression of the matrix product, but is more computationally efficient and scalable while keeping the capabilities to express asymmetric relation since the circular correlation is not commutative.^[18] HolE links holographic and complex embeddings since, if used together with Fourier, can be seen as a special case of ComplEx.^[1]
• TuckER:^[24] TuckER sees the knowledge graph as a tensor that could be decomposed using the Tucker decomposition in a collection of vectors—i.e., the embeddings of entities and relations—with a shared core.^[24][5] The weights of the core tensor are learned together with the embeddings and represent the level of interaction of the entries.^[25] Each entity and relation has its own embedding dimension, and the size of the core tensor is determined by the shape of the entities and relations that interact.^[5] The embedding of the subject and object of a fact are summed in the same way, making TuckER fully expressive, and other embedding models such as RESCAL, DistMult, ComplEx, and SimplE can be expressed as a special formulation of TuckER.^[24]
• MEI:^[26] MEI introduces the multi-partition embedding interaction technique with the block term tensor format, which is a generalization of CP decomposition and Tucker decomposition. It divides the embedding vector into multiple partitions and learns the local interaction patterns from data instead of using fixed special patterns as in ComplEx or SimplE models. This enables MEI to achieve optimal efficiency—expressiveness trade-off, not just being fully expressive.^[26] Previous models such as TuckER, RESCAL, DistMult, ComplEx, and SimplE are suboptimal restricted special cases of MEI.
• MEIM:^[27] MEIM goes beyond the block term tensor format to introduce the independent core tensor for ensemble boosting effects and the soft orthogonality for max-rank relational mapping, in addition to multi-partition embedding interaction. MEIM generalizes several previous models such as MEI and its subsumed models, RotaE, and QuatE.^[27] MEIM improves expressiveness while still being highly efficient in practice, helping it achieve good results using fairly small model sizes.

Pure translational models

This class of models is inspired by the idea of translation invariance introduced in word2vec.^[7] A pure translational model relies on the fact that the embedding vector of the entities are close to each other after applying a proper relational translation in the geometric space in which they are defined.^[18] In other words, given a fact, when the embedding of head is added to the embedding of relation, the expected result should be the embedding of the tail.^[5] The closeness of the entities embedding is given by some distance measure and quantifies the reliability of a fact.^[17]

TransE embedding model. The vector representation (embedding) of the head plus the vector representation of the relation should be equal to the vector representation of the tail entity.

• TransE^[9]: This model uses a scoring function that forces the embeddings to satisfy a simple vector sum equation in each fact in which they appear: ${\displaystyle h+r=t}$.^[7] The embedding will be exact if each entity and relation appears in only one fact, and, for this reason, in practice does not well represent one-to-many, many-to-one, and asymmetric relations.^[5][7]
• TransH^[28]: It is an evolution of TransE introducing a hyperplane as geometric space to solve the problem of representing correctly the types of relations.^[28] In TransH, each relation has a different embedded representation, on a different hyperplane, based on which entities it interacts with.^[7] Therefore, to compute, for example, the score function of a fact, the embedded representation of the head and tail need to be projected using a relational projection matrix on the correct hyperplane of the relation.^[1][7]
• TransR^[29]: TransR is an evolution of TransH because it uses two different spaces to represent the embedded representation of the entities and the relations,^[1][18] and separate completely the semantic space of entities and relations.^[7] Also TransR uses a relational projection matrix to translate the embedding of the entities to the relation space.^[7]
• TransD:^[30] Given a fact, in TransR, the head and the tail of a fact could belongs to two different types of entities, for example, in the fact${\displaystyle (Obama,president\_of,USA)}$, Obama and USA are two entities but one is a person and the other is a country.^[30][7] The matrix multiplication also is an expensive procedure in TransR to compute the projection.^[7][30] In this context, TransD employs two vector for each entity-relation pair to compute a dynamic mapping that substitutes the projection matrix while reducing the dimensional complexity.^[1][7][30] The first vector is used to represent the semantic meaning of the entities and relations, the second one to compute the mapping matrix.^[30]
• TransA:^[31] All the translational models define a score function in their representation space, but they oversimplify this metric loss.^[31] Since the vector representation of the entities and relations is not perfect, a pure translation of ${\displaystyle h+r}$ could be distant from ${\displaystyle t}$, and a spherical equipotential Euclidean distance makes it hard to distinguish which is the closest entity.^[31] TransA, instead, introduces an adaptive Mahalanobis distance to weights the embedding dimensions, together with elliptical surfaces to remove the ambiguity.^[1][7][31]

Translational models with additional embeddings

It is possible to associate additional information to each element in the knowledge graph and their common representation facts.^[1] Each entity and relation can be enriched with text descriptions, weights, constraints, and others in order to improve the overall description of the domain with a knowledge graph.^[1] During the embedding of the knowledge graph, this information can be used to learn specialized embeddings for these characteristics together with the usual embedded representation of entities and relations, with the cost of learning a more significant number of vectors.^[5]

• STransE:^[32] This model is the result of the combination of TransE and of the structure embedding^[32] in such a way it is able to better represent the one-to-many, many-to-one, and many-to-many relations.^[5] To do so, the model involves two additional independent matrix ${\displaystyle W_{r}^{h}}$ and ${\displaystyle W_{r}^{t}}$ for each embedded relation ${\displaystyle r}$ in the KG.^[32] Each additional matrix is used based on the fact the specific relation interact with the head or the tail of the fact.^[32] In other words, given a fact ${\displaystyle (h,r,t)}$, before applying the vector translation, the head ${\displaystyle h}$ is multiplied by ${\displaystyle W_{r}^{h}}$ and the tail is multiplied by ${\displaystyle W_{r}^{t}}$.^[7]
• CrossE:^[33] Crossover interactions can be used for related information selection, and could be very useful for the embedding procedure.^[33] Crossover interactions provide two distinct contributions in the information selection: interactions from relations to entities and interactions from entities to relations.^[33] This means that a relation, e.g.'president_of' automatically selects the types of entities that are connecting the subject to the object of a fact.^[33] In a similar way, the entity of a fact inderectly determine which is inference path that has to be choose to predict the object of a related triple.^[33] CrossE, to do so, learns an additional interaction matrix ${\displaystyle C}$, uses the element-wise product to compute the interaction between ${\displaystyle h}$ and ${\displaystyle r}$.^[5][33] Even if, CrossE, does not rely on a neural network architecture, it is shown that this methodology can be encoded in such architecture.^[1]

Roto-translational models

This family of models, in addition or in substitution of a translation they employ a rotation-like transformation.^[5]

• TorusE:^[34] The regularization term of TransE makes the entity embedding to build a spheric space, and consequently loses the translation properties of the geometric space.^[34] To address this problem, TorusE leverages the use of a compact Lie group that in this specific case is n-dimensional torus space, and avoid the use of regularization.^[1][34] TorusE defines the distance functions to substitute the L1 and L2 norm of TransE.^[5]
• RotatE:^[35] RotatE is inspired by the Euler's identity and involves the use of Hadamard product to represent a relation ${\displaystyle r}$ as a rotation from the head ${\displaystyle h}$ to the tail ${\displaystyle t}$ in the complex space.^[35] For each element of the triple, the complex part of the embedding describes a counterclockwise rotation respect to an axis, that can be describe with the Euler's identity, whereas the modulus of the relation vector is 1.^[35] It is shown that the model is capable of embedding symmetric, asymmetric, inversion, and composition relations from the knowledge graph.^[35]

Convolutional neural networks

This family of models, instead of using fully connected layers, employs one or more convolutional layers that convolve the input data applying a low-dimensional filter capable of embedding complex structures with few parameters by learning nonlinear features.^[1][5][18]

• ConvE:^[36] ConvE is an embedding model that represents a good tradeoff expressiveness of deep learning models and computational expensiveness,^[17] in fact it is shown that it used 8x less parameters, when compared to DistMult.^[36] ConvE uses a one-dimensional ${\displaystyle d}$-sized embedding to represent the entities and relations of a knowledge graph.^[5][36] To compute the score function of a triple, ConvE apply a simple procedure: first concatenes and merge the embeddings of the head of the triple and the relation in a single data ${\displaystyle {\ce {[h;{\mathcal {r}}]}}}$, then this matrix is used as input for the 2D convolutional layer.^[5][17] The result is then passed through a dense layer that apply a linear transformation parameterized by the matrix ${\displaystyle {\mathcal {W}}}$ and at the end, with the inner product is linked to the tail triple.^[5][18] ConvE is also particularly efficient in the evaluation procedure: using a 1-N scoring, the model matches, given a head and a relation, all the tails at the same time, saving a lot of evaluation time when compared to the 1-1 evaluation program of the other models.^[18]
• ConvR:^[37] ConvR is an adaptive convolutional network aimed to deeply represent all the possible interactions between the entities and the relations.^[37] For this task, ConvR, computes convolutional filter for each relation, and, when required, applies these filters to the entity of interest to extract convoluted features.^[37] The procedure to compute the score of triple is the same as ConvE.^[5]
• ConvKB:^[38] ConvKB, to compute score function of a given triple ${\displaystyle (h,r,t)}$, it produces an input ${\displaystyle {\ce {[h;{\mathcal {r}};t]}}}$of dimension ${\displaystyle d\times 3}$ without reshaping and passes it to series of convolutional filter of size ${\displaystyle 1\times 3}$.^[38] This result feeds a dense layer with only one neuron that produces the final score.^[38] The single final neuron makes this architecture as a binary classifier in which the fact could be true or false.^[5] A difference with ConvE is that the dimensionality of the entities is not changed.^[17]

Capsule neural networks

This family of models uses capsule neural networks to create a more stable representation that is able to recognize a feature in the input without losing spatial information.^[5] The network is composed of convolutional layers, but they are organized in capsules, and the overall result of a capsule is sent to a higher-capsule decided by a dynamic process routine.^[5]

• CapsE:^[39] CapsE implements a capsule network to model a fact ${\displaystyle (h,r,t)}$.^[39] As in ConvKB, each triple element is concatenated to build a matrix ${\displaystyle {\ce {[h;{\mathcal {r}};t]}}}$and is used to feed to a convolutional layer to extract the convolutional features.^[5][39] These features are then redirected to a capsule to produce a continuous vector, more the vector is long, more the fact is true.^[39]

Recurrent neural networks

This class of models leverages the use of recurrent neural network.^[5] The advantage of this architecture is to memorize a sequence of fact, rather than just elaborate single events.^[40]

• RSN:^[40] During the embedding procedure is commonly assumed that, similar entities has similar relations.^[40] In practice, this type of information is not leveraged, because the embedding is computed just on the undergoing fact rather than a history of facts.^[40] Recurrent skipping networks (RSN) uses a recurrent neural network to learn relational path using a random walk sampling.^[5][40]