自动化知识图谱表示学习：从三元组到子图.pdf

自动化知识图谱表示：从三元组到子图张永祺 第四范式资深研究员2022年3月12日|01 背景介绍什么是知识表示学习02 重要方向知识表示学习研究点及什么是AutoML03 模型设计从三元组到子图04 总结Summary and future work目录CONTENT|背景介绍什么是知识表示学习？01|Graph representation: ! = ℰ, ℛ, & .Entities ℰ: real world objects or abstract concepts.Relations ℛ: interactions between/among entities.Fact/triples &: the basic unit in form of (head entity, relation, tail entity), ℎ, #, $ .KG is a semantic graph• Semantic information• Structural information4Knowledge graph (KG)SamTobeyacted_inacted_inacted_indirectedU.S.influencesSpider-1Spider-2acted_inlocated_inhttps://www.4paradigm.com/product/knowledge_base.html5KGQA: Recommendation: Drug discovery:Stock prediction: Representative applications6Representation in KGsSamTobeyacted_inacted_inacted_indirectedU.S.influencesSpider-1Spider-2acted_inlocated_inSamSpider-1⋯ ⋯U.S.acted_in⋯⋯⋯⋯influences⋯⋯⋯⋯directedlocated_insymbolsrepresentationsAdvantages:• Continuous, ease of use in ML pipeline.• Discover latent properties.• Efficient similarity search.Observed triplets !!:maximize scoreUnobserved triplets !": minimize score(Sam, directed, Spider-1)(Tobey, acted_in, Spider-2)(Spider-1, located_in, U.S.)…(Tobey, directed, Spider-1)(Tobey, acted_in, U.S.)(Spider-1, directed, U.S.)…KGmodelLossiteratively updateØObjective: Preserve as much information on original graph as possible.7Representation learningtripletsSamTobeyacted_inacted_inacted_indirectedU.S.influencesSpider-1Spider-2acted_inlocated_in• Knowledge graph completion• Link prediction / triple classification: predicting missing links• Entity alignment• Align the same entities in two KGs, e.g. cross-lingual KGs• Entity classification• Predict the type of entities8Basic tasks• Link prediction as an example.• Evaluation on ℎ, #, $ :• head/tail prediction: ? , #, $ or ℎ, #, ?• get the rank of ℎ, #, $ over &, #, $ : & ∈ ℰ or ℎ, #, & : & ∈ ℰ• Metrics:9"# =1|'#$#| (%∈'!"!rank%"## =1|'#$#| (%∈'!"!1rank%-@/ =0 ∈ '#$#: rank% ≤ /|'#$#|Evaluation重要方向知识表示学习研究点及什么是AutoML02|• As a learning problem, the KG reasoning problem contains the following important components:11negative samplingscoring functionloss functionregularizationoptimizationmin4 ℒ ) ⋅, + , ,5, ,6+ # +General framework12Scoring functionsQuery: (Sam, directed, ?)Answer: Spider-1, Spider-2triplet-based(Sam, directed, Spider-2)path-basedSam #$%&'(&#Spider-1_*'(&#_$+Tobey*'(&#_$+Spider-2Sam $+,-.&+'&/Tobey*'(&#_$+Spider-2subgraph-basedSamTobeyacted_inacted_inacted_indirectedU.S.influencesSpider-1Spider-2acted_inlocated_inRESCAL[Nickel et. al. 2011]TransE[Bordes et. al. 2013]ComplEx[Trouillion et. al. 2017]ConvE[Dettmers et. al. ]AutoSF[Zhang et. al. 2020]PTransE[Lin et. al. 2015] RSN[Dettmers et. al. ]Interestellar[Zhang et. al. 2020]GraIL[Teru et. al. 2019] NBFNet[Zhu et. al. 2021]RED-GNN[Zhang et. al. 2022]• KG is sparse, the full set of negative triples is much larger than the set of positive triples.• Given the positive triple ℎ, #, $ , randomly replacing the head or tail entity from the set* ",$,%&= { ℎ', #, $ ∉ *(: ℎ' ∈ ℰ} ∪ { ℎ, #, $' ∉ *(: $' ∈ ℰ}13*(*&Negative sampling)* !,#,$ ="ℎ, %, & ∉ (|"ℎ ∈ ℰ ∪ℎ, %, ̃& ∉ (| ̃& ∈ ℰℎ, #, $training pairsStandard approaches:GAN-based:training pairsrewardsCache-based:gradientsnot dynamic, quality is poor.training pairs" ℎ$, %$, &$!" 'ℎ0, %0, ̃&0ℰ ×/%ℛ ×*!ℎ&, %&, &&'ℎ0, %0, ̃&0lossKGE!14[Wang et. al. AAAI 2018][Cai et. al. NAACL 2018][Zhang et al. ICDE 2019]Negative samplingNSCaching: Simple and Efficient Negative Sampling for Knowledge Graph Embedding. ICDE 2019.Simple and Automated Negative Sampling for Knowledge Graph Embedding. VLDB-J 2021.• The models may overfit on the training data.• General approaches:• Directly constrain the embedding norm 0 0 ≤ 1, 3 0 ≤ 1• Add Lp regularizers# 4 = 5 4 00• Dropout on the embeddings15expressivenesscomplexitytrade offRegularization• Parameters +: embeddings, neural networks• Stochastic gradient descent (Adam, Adagrad) to optimize the parameters16Algorithm 1 Stochastic training of KGE [34].Require: training set S = {(h, r, t)}, scoring function f and loss function `;1: initialize the embedding parameters e, r for each e 2 E and r 2 R.2: for i = 1, · · · , T do3:sample a mini-batch Sbatch 2 S of size m;4:for each (h, r, t) 2 Sbatch do5:sample ˜m negative triplets ˜S(h,r,t) ⌘ {(˜hj, r, ˜tj)} for the positive triplet (h, r, t);6:compute the scores of positive triplet (h, r, t) and negative triplets {(˜hj, r, ˜tj)} based on thescoring function f;7:update embeddings based on loss ` using selected positive and negative triplets;8:end for9: end for10: return embeddings of entities in E and relations in R.the power of deep networks, NNMs are also introduced for KGE. However, due to the huge modelOptimizationLearning rate, initializer, batch size, dimension size, early stop…17Hyper-parameter optimization!ℳ(# $∗, & , '-'.)search algorithmevaluate onwhole graph 'updateselect&!∗! → #!evaluate on subgraph )"!search algorithmupdate$!selectstage one: efficient evaluation on subgraph→! → !!evaluate on subgraph )ℳ(#...