TiDB as an HTAP Database主讲人：PingCap CEO 刘奇About me• CEO and co-founder of PingCAP• Open source hacker• Infrastructure engineer• Founder of TiDB/TiKV/Codis• Infrastructure software engineer• Wandoulabs/JDWhy a new database?Brief History● Standalone RDBMS● NoSQL● Middleware & Proxy● NewSQL1970s20102015PresentMySQLPostgreSQLOracleDB2...RedisHBaseCassandraMongoDB...Google SpannerGoogle F1TiDBRDBMSNoSQLNewSQLNewSQL Database● Horizontal Scalability● ACID Transaction● High Availability● SQL at ScaleOLTP & OLAP8am2pm6pm2amETLDatabaseERPCRMOLTPOLAPData WarehouseWhere is my data?Is the data out-of-date?Why two separate systems● Huge data size● Complex query logic● Latency VS Throughput● Point query VS Full range scan● Transaction & lsolation levelOLAP + OLTP = HTAPHybrid Transactional / Analytical Processing● ACID Transcation● Real-time analysis● SQLHTAPHow do we build the new databaseWhat is TiDB● Scalability as the first class feature● SQL is necessary● Compatible with MySQL, in most cases● OLTP + OLAP = HTAP (Hybrid Transactional/Analytical Processing)● 24/7 availability, even in case of datacenter outages● Open source, of courseArchitectureTiKVTiKVTiKVTiKVRaftRaftRaftTiDBTiDBTiDB......PlacementDriver (PD)Control flow:Balance / FailoverMetadata / Timestamp request Stateless SQL Layer Distributed Storage Layer...TiKV - Overview•Region: a set of continuous key-value pairs•Data is organized/stored/replicated by Regions•Highly layeredTiKV Key Space[ start_key, end_key)(-∞, +∞)Sorted MapRPC (gRPC)TransactionMVCCRaftRocksDBNode BNode CNode ARaftRaftRaft256MBTiKV - Multi-RaftMultiple raft groups in the cluster, one group for each region.ClientStore 1Region 1Region 3Region 5Region 4Store 3Region 3Region 5Region 2Store 2Region 1Region 3Region 2Region 4Store 4Region 1Region 5Region 2Region 4RPCRPCRPCRPCTiKV node 1TiKV node 2TiKV node 3TiKV node 4Raft GroupTiKV - Horizontal Scale Region 1Region 3Region 1^Region 2Region 1*Region 2Region 2Region 3Region 3Node ANode BNode ENode CNode DAdd ReplicaThree steps to move a leader replica●Transfer Leader●Add Replica●Remove ReplicaPD - Overview•Meta data management•Load balance managementTiKVTiKVTiKVTiKV… ...TiKV ClusterPDNode/Region InfoManagement CommandTiKV ClientRoute InfoPD - TiKV Cluster ManagmentRegion ARegion BNode 1Node 2PDSchedulingStrategeCluster InfoAdminHeartBeatSchedulingCommandRegion CConfigMovementTiDB - Overview•The stateless SQL layerSQLASTLogical PlanOptimizedLogical PlanCost ModelSelectedPhysical PlanTiKVTiKVTiKVTiDB SQL LayerStatisticsTiDB - Distributed SQLSELECT COUNT(c1) FROM t WHERE c1 > 10 AND c2 = ‘shanghai’; Partial AggregateCOUNT(c1)Filterc2 = “shanghai”Read Indexidx1: (10, +∞)Physical Plan on TiKV (index scan)Read Row Databy RowIDRowIDRowRowFinal AggregateSUM(COUNT(c1))DistSQL ScanPhysical Plan on TiDBCOUNT(c1)COUNT(c1)TiKVTiKVTiKVCOUNT(c1) COUNT(c1)TiDB - Cost Based Optimizer• Predicate Pushdown• Column Pruning• Eager Aggregate • Convert Subquery to Join • Statistics framework• CBO Framework–Index Selection–Join Operator Selection• Hash join• Index lookup join• Sort-merge join–Stream Operators VS Hash OperatorsOLTP + OLAPTiDBTiDBTiDBTiDBTiDBOLTP QueryOLAP QueryTiKVTiKVTiKVTiKV....Job QueueWorkerSchedulerWorkerWorkerWorker PoolJobsJobsLow PriorityHigh PriorityHTAPEcosystem: Beyond TiDB and SQLSpark on TiDBTiDBTiDBWorkerSpark DriverTiKV Cluster (Storage)Meta dataTiKVTiKVTiKVApplicationSyncerData locationJobTiSparkDistSQL APITiKVTiDBTSO/Data locationWorkerWorkerSpark ClusterTiDB ClusterTiDB.........DistSQL APIPDPDPDPD ClusterTiKVTiKVTiDBSpark on TiDB• Spark ecosystem• TiKV Connector is better than JDBC connector• Index support• Complex Calculation Pushdown• CBO–Pick up right Access Path–Join Reorder• Priority & Isolation LevelFuture plan• Code Generation• MPP Engine• Mixed storage engine (Columnar / Row-based)• Heterogeneous computing (CPU/GPU/FPGA)THANKS
发表评论取消回复