Search SciencePre-trained Language Model forWeb-Scale Retrieval & Rankingin Baidu SearchShuaiqiang WANGhttp://searchscience.baidu.comwangshuaiqiang@baidu.comSearch ScienceOutline1234BackgroundRetrievalRankingSummarySearch ScienceOutline1234BackgroundRetrievalRankingSummarySearch ScienceBackground• Retrieval and ranking are two crucial stages in web-scale search engineDatabaseRetrievalRankingQueryResultsWeb-scaledocumentsFew hundreds orthousands candidatesSearch Sciencehttps://github.com/PaddlePaddle/ERNIE1.Sun, Y. et al, 2019. Ernie: Enhanced representation through knowledge integration. In arXiv:1904.09223.2.Sun, Y. et al, 2020. Ernie 2.0: A continual pre-training framework for language understanding. In AAAI.百度ERNIESearch Science• Beyond text matching: semantic retrieval& ranking• Representation-based methods• Representation: document semantics aslatent vectors• Retrieval: nearest neighbor search inlatent space• Interaction-based models• Ranking: matching over the local interactions* Picture from: Dai, Andrew M., Christopher Olah, and Quoc V. Le. "Document embedding with paragraph vectors." arXiv preprint arXiv:1507.07998 (2015).QuerySemantically-related candidatesBackgroundSearch Science• Semantic retrieval• Effectively understand the semantics of queries and documents• Large number of low-frequency queries• Web-scale retrieval system• Semantic ranking• Expensive computations• Ranking-agnostic pre-training Challenges!"#"$"%&'"#(..."%&'...")([CLS]*#*$[SEP]*#(...[SEP]...*)(&!+%&#&$&%&'&#(...&%&'...&)(,%'ERNIE-+--+-Masked Sentence AMasked Sentence B•Our contribution：One of the largest application ofPLM for Web-scale Retrieval & Ranking1.Zou L. et al. Pre-trained Language Model based Ranking in Baidu Search. In KDD 2021.2.Liu Y. et al. Pre-trained Language Model for Web-scale Retrieval in Baidu Search. In KDD 2021.Search ScienceOutline1234BackgroundRetrievalRankingSummarySearch Science• Retrieval model• Goal: learning query-document semantic relatedness• Backbone: a bi-encoders (i.e., two-tower) architecture*, with• Query & Doc encoders: transformersMethodology – Retrieval ModelQuery EncoderCLS-pooling![CLS]"#"$[SEP]...Doc EncoderCLS-pooling%[CLS]"#&"'&[SEP]...Query (tokenized)Doc (tokenized)retrieval score (Query embeddingDoc embedding* Chang, Wei-Cheng, et al. "Pre-training tasks for embedding-based large-scale retrieval." arXiv preprint arXiv:2002.03932 (2020).Search Science• Retrieval model• Goal: learning query-document semantic relatedness• Poly-attention: bi-encoders with more query-document interaction*Methodology – Retrieval Model* Humeau, Samuel, et al. "Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring." arXiv preprint arXiv:1905.01969 (2019)....!"#"$!′..."#&..."'&EncoderEncoderAttentionCode 1AttentionCode mCLS-poolingP1Pm...!′score s1 ...score smscore ( = *+,-.#/ (-Produce multiple embeddingson the query sideSearch Science• Positive & negative data mining for different data sources• [Search log]• positives: user-clicked documents; negatives: non-clicked documents• [Manually labeled data]• positives: high-scored documents; negatives: low-scored documents• In-batch negative mining• Introducing random negatives• Benefits:• More aligned with retrieval task• Efficiently scale up the number of negativesMethodology – Retrieval Model!"!#!$!%&"'&#'&$'&%'&"(&#(&$(&%(!)query&)'relevant doc&)(strong negativerelevant (+, -) pairirrelevant (+, -) paircorresponding tostrong negativeirrelevant (+, -) paircorresponding torandom negativeSearch Science• Multi-stage training paradigm• Unsupervised -> Supervised• General corpus -> Task-specific dataMethodology – Training ParadigmSearch ScienceMethodology – Embedding Compression• Mode deployment• Compression• QuantizationDoc EmbeddingDoc EmbeddingQuantizationDoc EmbeddingCompression with additional FC layerSearch ScienceMethodology – System Workflow• Deployment• Integrating term-based & ERNIE-based retrieval• Unifying results with post-retrieval filtering1.2.3.Text MatchingSearch ScienceEvaluation – Online Evaluation• Metrics: DCG & GSB• #Good = #queries that the new system performs better• ResultsSearch ScienceOutline1234BackgroundRetrievalRankingSummarySearch ScienceContent-aware Pre-trained Language ModelPyramid-ERNIEMethodTime ComplexityOriginal ErniePyramid-ErnieO(Lh(Nq + Nt + Ns)2)
AC3icbVDLSsNAFJ34rPUVdelmaBFahJKUi6LblxIrWAf0MYwmU7aoZOHMxOhO7d+CtuXCji1h9w5984SbPQ1gMzHM65l3vcUJGhTSMb21peWV1bT23kd/c2t7Z1f2yKIOCYtHLCAdx0kCKM+aUkqGemGnCDPYaTjC8Sv/NAuKCBfysnIbE8NPSpSzGSrL1Qt9DcoQRi6+npatRqWHfw2PYsGX6i/JdtWzrRaNipICLxMxIEWRo2vpXfxDgyCO+xAwJ0TONUFox4pJiRqb5fiRIiPAYDUlPUR95RFhxesUHilAN2Aq+dLmKq/O2LkCTHxHFWZbC7mvUT8z+tF0j2zYuqHkSQ+ng1yIwZlAJNg4IBygiWbKIwp2pXiEeIyxVfHkVgjl/8iJpVytmrXJyUyvWz7M4cuAQFEAJmOAU1MElaIWwOARPINX8KY9aS/au/YxK13Ssp4D8Afa5w9DY5ifO(Llowh(Nq + Nd)2) + Llowh(Ns)2 + Lhighh(Nq + Nt + Ns)2)
查看更多
发表评论取消回复