从特征交互到数据交互：深度点击率模型的新趋势-张伟楠.pdf

张伟楠 上海交通大学http://wnzhang.netDataFunSummit从 特 征 交 互 到 数 据 交 互 ：深 度 点 击 率 模 型 的 新 趋 势From Feature Interaction to Sample Interaction: the new Trend for Deep CTR ModelsAn Display Ad Example[http://news.ifeng.com]Click or not?User Response Estimation Problem• Click-through rate estimation as an example•Date: 20160320•Hour: 14•Weekday: 7•IP: 119.163.222.*•Region: England•City: London•Country: UK•Ad Exchange: Google•Domain: yahoo.co.uk•URL: http://www.yahoo.co.uk/abc/xyz.html•OS: Windows•Browser: Chrome•Ad size: 300*250•Ad ID: a1890•User tags: Sports, ElectronicsClick (1) or not (0)?Predicted CTR (0.15)CTR and Data• Click-Through Rate (CTR) Estimation is an essential task• It predicts the probability of a user clicking an item based on thelogged behavior data• Applications: e-commerce, online advertising, recommendersystems• Data format: multi-categorical data with binary labels• CTR estimation as a binary classification formulationWeinan Zhang, Jiarui Qin, Wei Guo, Ruiming Tang, Xiuqiang He. Deep Learning for Click-Through Rate Estimation. IJCAI 2021 (Survey Track)CTR Models (2007 to 2021)• Developing trend of CTR estimation models• Feature engineering requirementWeinan Zhang, Jiarui Qin, Wei Guo, Ruiming Tang, Xiuqiang He. Deep Learning for Click-Through Rate Estimation. IJCAI 2021 (Survey Track)Focus of this talk•Model capacityFeature Representation• Binary one-hot encoding of categorical datax=[Weekday=Wednesday, Gender=Male, City=London]x=[0,0,1,0,0,0,0 0,1 0,0,1,0…0]High dimensional sparse binary feature vector• Label representationy = 1 (click) or 0 (non-click)Logistic Regression• Prediction• Cross Entropy Loss• Stochastic Gradient Descent LearningLee et al. Estimating Conversion Rate in Display Advertising from Past Performance Data. KDD 12Factorization Machines• Prediction based on feature embeddingOentaryo et al. Predicting response in mobile advertising with hierarchical importance-aware factorization machine. WSDM 14.Rendle. Factorization machines. ICDM 2010.Logistic RegressionFeature InteractionsFor x=[Weekday=Friday, Gender=Male, City=Shanghai]Combined Models: GBDT + LRHe et al. Practical Lessons from Predicting Clicks on Ads at Facebook . ADKDD 2014.Combined Models: GBDT + FMhttp://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdfNeural Network Models• Difficulty: Impossible to directly deploy neural network models on such data1M500500ME.g., input features 1M, first layer 500, then 500M parameters for first layerReview Factorization Machines• Prediction based on feature embedding• Embed features into a k-dimensional latent space• Explore the feature interaction patterns using vector inner-productOentaryo et al. Predicting response in mobile advertising with hierarchical importance-aware factorization machine. WSDM 14.Rendle. Factorization machines. ICDM 2010.Logistic RegressionFeature InteractionsFactorization Machine is a Neural NetworkZhang et al. Deep Learning over Multi-field Categorical Data – A Case Study on User Response Prediction. ECIR 16.[Factorisation Machine Initialised]Factorization-machine supported Neural Networks (FNN)Zhang et al. Deep Learning over Multi-field Categorical Data – A Case Study on User Response Prediction. ECIR 16.Factorization-machine supported Neural Networks (FNN)• Chain rule to update factorisation machine parametersBut factorization machine is still different from common additive neural networksProduct Operations as Feature InteractionsYanru Qu et al. Product-based Neural Networks for User Response Prediction. ICDM 2016.Feature Interaction• Feature interaction operators• Product operator• PNN (IPNN, OPNN) [1], NFM [2], DeepFM[: inner and outerproduct• PIN [3]: micro neural network as feature interaction operator• DCN [4] (DCN V2 [5]), CIN [6]: cross network[1] Qu et al. Product-based neural networks for user response prediction. ICDM, 2016.[2] He et al. Neural factorization machines for sparse predictive analytics. SIGIR, 2017.[3] Qu et al. Product-based neural networks for user response prediction over multi-field categorical data. TOIS, 2018.[4] Wang et al. Deep & cross network for ad click predictions. ADKDD, 2017.[5] Wang et al. Dcn-m: Improved deep & cross network for feature cross learning in web-scale learning to rank systems.[6] Lian et al. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. KDD 2018.Feature Interaction• Feature interaction operators• Convolutional operators• Use convolution networks to model the feature interaction onthe feature map• CCPM [1], FGCNN [2]• GCN: FiGNN [3][1] Liu et al. A convolutional click prediction model. CIKM, 2015.[2] Liu et al. Feature generation by convolutional neural network for click-through rate prediction. WWW 2019.[3] Li et al. Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction. CIKM, 2019.Feature Interaction• Feature interaction operators• Attention operators• Use attention mechanism to weight different featureinteractions: AFM [1], FiBiNet ...