第1章 數(shù)據(jù)中心概述 ······································································.1
1.1 數(shù)據(jù)中心簡介 ····················································································.2
1.1.1 什么是數(shù)據(jù)中心 ·········································································.2
1.1.2 數(shù)據(jù)中心的需求和挑戰(zhàn) ································································.4
1.2 云計算數(shù)據(jù)中心資源調(diào)度需求分析 ·························································.4
1.2.1 技術(shù)需求 ··················································································.4
1.2.2 技術(shù)目標(biāo) ··················································································.5
1.3 云計算數(shù)據(jù)中心資源調(diào)度研究進展 ·························································.5
1.4 云計算數(shù)據(jù)中心資源調(diào)度方案分析 ·························································.6
1.4.1 Google解決方案 ········································································.6
1.4.2 Amazon解決方案 ·······································································.7
1.4.3 IBM解決方案 ············································································.8
1.4.4 HP解決方案··············································································10
1.4.5 VMware解決方案 ·······································································10
1.4.6 阿里云解決方案 ·········································································12
1.4.7 華為云解決方案 ·········································································14
1.4.8 其他廠家解決方案 ······································································15
1.5 云計算數(shù)據(jù)中心資源調(diào)度標(biāo)準(zhǔn)進展 ·························································17
1.6 云資源管理調(diào)度關(guān)鍵技術(shù)及研究熱點 ······················································18
本章小結(jié) ·································································································20
思考題·····································································································21
參考文獻 ·································································································21
第2章 云計算概述 ········································································.25
2.1 云計算的發(fā)展背景 ··············································································26
2.2 云計算是集大成者 ··············································································28
2.2.1 并行計算 ················································································.28
2.2.2 網(wǎng)格計算 ················································································.29
2.2.3 效用計算 ················································································.29
2.2.4 普適計算 ················································································.30
2.2.5 軟件即服務(wù) ·············································································.30
2.2.6 虛擬化技術(shù) ·············································································.31
2.3 云計算的驅(qū)動因素 ············································································.31
2.3.1 云計算的發(fā)展現(xiàn)狀和趨勢 ···························································.33
2.3.2 云計算應(yīng)用初步分類 ·································································.35
2.4 云計算產(chǎn)業(yè)鏈中的不同角色 ································································.36
2.5 云計算的主要特征和技術(shù)挑戰(zhàn) ·····························································.37
2.5.1 云計算的主要特征 ····································································.37
2.5.2 挑戰(zhàn)性問題 ·············································································.38
本章小結(jié) ································································································.46
思考題 ···································································································.46
參考文獻 ································································································.46
第3章 大數(shù)據(jù)處理 ········································································.51
3.1 大數(shù)據(jù)的發(fā)展背景及定義 ···································································.52
3.2 大數(shù)據(jù)問題 ······················································································.55
3.2.1 速度方面的問題 ·······································································.55
3.2.2 種類及架構(gòu)問題 ·······································································.56
3.2.3 體量及靈活性問題 ····································································.56
3.2.4 成本問題 ················································································.57
3.2.5 價值挖掘問題 ··········································································.57
3.2.6 存儲及安全問題 ·······································································.58
3.2.7 互聯(lián)互通與數(shù)據(jù)共享問題 ···························································.59
3.3 大數(shù)據(jù)與云計算的辯證關(guān)系 ································································.60
3.4 大數(shù)據(jù)技術(shù) ······················································································.61
3.4.1 基礎(chǔ)架構(gòu)支持 ··········································································.62
3.4.2 數(shù)據(jù)采集 ················································································.64
3.4.3 數(shù)據(jù)存儲 ················································································.65
3.4.4 數(shù)據(jù)計算 ················································································.68
3.4.5 數(shù)據(jù)展現(xiàn)與交互 ·······································································.73
本章小結(jié) ································································································.75
思考題 ···································································································.76
參考文獻 ································································································.76
第4章 人工智能平臺的資源調(diào)度概述 ················································.78
4.1 引言 ·································································································79
4.2 深度學(xué)習(xí)的分布式并行訓(xùn)練系統(tǒng)架構(gòu) ······················································79
4.3 深度學(xué)習(xí)的分布式并行策略 ··································································81
4.3.1 深度學(xué)習(xí)的基礎(chǔ)概念 ···································································82
4.3.2 分布式并行訓(xùn)練算法 ···································································82
4.3.3 研究現(xiàn)狀分析 ············································································85
4.4 分布式并行訓(xùn)練的時效分析 ··································································91
本章小結(jié) ·································································································96
思考題·····································································································96
參考文獻 ·································································································97
第5章 基于深度學(xué)習(xí)的云服務(wù)負載預(yù)測方法 ·······································.99
5.1 引言 ······························································································.100
5.2 相關(guān)工作 ························································································.101
5.2.1 基于回歸方法的云服務(wù)負載預(yù)測方法···········································.101
5.2.2 基于學(xué)習(xí)的云服務(wù)負載預(yù)測方法 ·················································.102
5.2.3 討論分析 ···············································································.103
5.3 系統(tǒng)模型 ························································································.104
5.4 esDNN：基于高效監(jiān)督學(xué)習(xí)的深度神經(jīng)網(wǎng)絡(luò) ··········································.106
5.4.1 多元時間序列預(yù)測的滑動窗口 ····················································.106
5.4.2 esDNN算法 ···········································································.109
5.5 性能測試 ························································································.112
5.5.1 數(shù)據(jù)集和環(huán)境配置 ···································································.112
5.5.2 與基于無監(jiān)督學(xué)習(xí)方法的比較 ····················································.113
5.5.3 與基于神經(jīng)網(wǎng)絡(luò)方法的比較 ·······················································.114
5.5.4 與其他方面的比較 ···································································.118
本章小結(jié) ······························································································.119
思考題··································································································.120
參考文獻 ······························································································.120
第6章 云應(yīng)用程序和可再生能源的自適應(yīng)管理方法 ····························.123
6.1 引言 ······························································································.124
6.2 相關(guān)工作 ························································································.125
6.2.1 DVFS和虛擬機整合·································································.125
6.2.2 Brownout ···············································································.126
6.2.3 數(shù)據(jù)中心冷卻系統(tǒng)的整體管理 ····················································.126
6.2.4 可再生能源 ············································································.126
6.3 系統(tǒng)模型 ························································································.127
6.4 問題建模 ························································································.129
6.4.1 功率消耗 ···············································································.129
6.4.2 工作負載模型 ·········································································.130
6.4.3 優(yōu)化目標(biāo) ···············································································.131
6.5 根據(jù)可再生資源進行調(diào)度決策 ····························································.132
6.5.1 Green-Aware 調(diào)度算法 ·····························································.132
6.5.2 交互式工作負載的Brownout算法 ···············································.133
6.5.3 批處理工作負載的延遲算法 ·······················································.134
6.5.4 主機調(diào)度 ···············································································.135
6.5.5 可再生能源預(yù)測 ······································································.136
6.6 原型系統(tǒng)的實現(xiàn) ··············································································.137
6.7 性能評估 ························································································.139
6.7.1 環(huán)境設(shè)置 ···············································································.140
6.7.2 工作負載 ···············································································.140
6.7.3 應(yīng)用 ·····················································································.141
6.7.4 結(jié)果 ·····················································································.141
本章小結(jié) ·······························································································.145
思考題 ··································································································.145
參考文獻 ·······························································································.146
第7章 云計算環(huán)境下基于虛擬機整合的自適應(yīng)節(jié)能調(diào)度 ······················.149
7.1 緒論 ······························································································.150
7.2 虛擬機整合技術(shù) ··············································································.150
7.3 相關(guān)研究工作 ··················································································.152
7.4 問題定義 ························································································.153
7.5 數(shù)據(jù)中心能耗模型 ···········································································.154
7.5.1 服務(wù)器功耗模型 ······································································.154
7.5.2 服務(wù)器能耗模型 ······································································.155
7.5.3 云數(shù)據(jù)中心總能耗模型 ·····························································.156
7.5.4 數(shù)據(jù)中心節(jié)能調(diào)度下限 ·····························································.156
7.6 SAVE算法描述 ···············································································.158
7.6.1 概述 ·····················································································.158
7.6.2 分配算法 ···············································································.159
7.6.3 遷移算法 ···············································································.161
7.7 實驗驗證與分析 ··············································································.165
7.7.1 實驗準(zhǔn)備 ···············································································.165
7.7.2 數(shù)據(jù)準(zhǔn)備 ···············································································.167
7.7.3 基線方法 ···············································································.167
7.7.4 結(jié)果分析 ···············································································.169
本章小結(jié) ······························································································.175
思考題··································································································.175
參考文獻 ······························································································.175
第8章 MapReduce模型中數(shù)據(jù)傾斜問題的算法 ·································.177
8.1 緒論 ······························································································.178
8.1.1 背景及意義 ············································································.178
8.1.2 研究現(xiàn)狀 ···············································································.179
8.1.3 研究內(nèi)容 ···············································································.180
8.2 數(shù)據(jù)傾斜相關(guān)理論研究 ·····································································.181
8.2.1 數(shù)據(jù)傾斜 ···············································································.181
8.2.2 算法介紹 ···············································································.183
8.2.3 算法綜合對比 ·········································································.190
8.3 多任務(wù)數(shù)據(jù)傾斜調(diào)度算法設(shè)計 ····························································.193
8.3.1 問題描述與建模 ······································································.193
8.3.2 Revised Johnson1954算法（RJA） ··············································.195
8.3.3 離線多任務(wù)調(diào)度算法 ································································.199
8.3.4 在線多任務(wù)調(diào)度算法 ································································.201
8.4 單任務(wù)數(shù)據(jù)傾斜算法設(shè)計 ··································································.203
8.4.1 YarnTune概述 ·········································································.203
8.4.2 檢測數(shù)據(jù)傾斜 ·········································································.206
8.4.3 YarnTune核心功能 ··································································.208
8.5 系統(tǒng)測試和分析 ··············································································.212
8.5.1 軟硬件測試環(huán)境 ······································································.212
8.5.2 多任務(wù)數(shù)據(jù)傾斜測試 ································································.212
8.5.3 單任務(wù)數(shù)據(jù)傾斜測試 ································································.217
本章小結(jié) ······························································································.220
思考題··································································································.221
參考文獻 ······························································································.221
第9章 Spark中的數(shù)據(jù)均衡分配算法研究 ·········································.223
9.1 Spark設(shè)計思想 ················································································.224
9.1.1 Spark概述 ·············································································.224
9.1.2 Spark計算模型 ·······································································.225
9.1.3 Spark整體架構(gòu) ·······································································.226
9.2 Spark數(shù)據(jù)存儲體系 ··········································································.227
9.2.1 存儲整體架構(gòu) ·········································································.227
9.2.2 數(shù)據(jù)寫入過程 ·········································································.228
9.2.3 數(shù)據(jù)讀取過程 ·········································································.229
9.3 Spark Shuffle分析 ············································································.230
9.3.1 Shuffle概述 ············································································.230
9.3.2 Shuffle寫操作 ·········································································.231
9.3.3 Shuffle讀操作 ·········································································.232
9.4 Spark分區(qū)方法 ················································································.233
9.4.1 HashPartition分區(qū) ····································································.233
9.4.2 RangePartition分區(qū) ··································································.234
9.5 問題描述與建模 ··············································································.235
9.5.1 相關(guān)定義 ···············································································.235
9.5.2 問題建模 ···············································································.236
9.6 數(shù)據(jù)均衡分配算法整體設(shè)計 ·······························································.237
9.6.1 抽樣算法 ···············································································.238
9.6.2 數(shù)據(jù)均衡分區(qū)算法 ···································································.240
9.6.3 權(quán)重調(diào)整算法 ·········································································.242
9.6.4 任務(wù)分配算法 ·········································································.245
9.7 算法復(fù)雜度分析 ··············································································.246
9.8 MRFair概述 ···················································································.246
9.8.1 MRFair的目標(biāo)與特征 ·······························································.246
9.8.2 MRFair系統(tǒng)架構(gòu) ·····································································.247
9.8.3 MRFair數(shù)據(jù)均衡分配示例 ·························································.248
9.9 MRFair傾斜檢測時機及算法 ······························································.249
9.9.1 MRFair傾斜檢測時機 ·······························································.249
9.9.2 MRFair傾斜檢測算法 ·······························································.250
9.10 MRFair數(shù)據(jù)重新分配時機及算法 ······················································.250
9.10.1 MRFair數(shù)據(jù)重新分配時機 ·······················································.250
9.10.2 MRFair數(shù)據(jù)重新分配算法 ·······················································.251
9.11 MRFair核心模塊 ············································································.253
9.12 系統(tǒng)測試環(huán)境 ················································································.255
9.12.1 軟硬件測試環(huán)境 ·····································································.255
9.12.2 測試數(shù)據(jù) ··············································································.256
9.12.3 對比算法或系統(tǒng) ·····································································.257
9.12.4 評價指標(biāo) ··············································································.257
9.13 Reduce Partition數(shù)據(jù)均衡分配算法測試 ··············································.257
9.13.1 WordCount基準(zhǔn)測試 ·······························································.257
9.13.2 Sort基準(zhǔn)測試 ········································································.260
9.14 MRFair數(shù)據(jù)均衡分配算法測試 ·························································.263
9.14.1 WordCount基準(zhǔn)測試 ·······························································.263
9.14.2 Sort基準(zhǔn)測試 ········································································.265
本章小結(jié) ······························································································.266
思考題··································································································.267
參考文獻 ······························································································.267
第10章 深度學(xué)習(xí)框架TensorFlow的高效分布式并行算法研究 ·············.269
10.1 分布式并行算法的背景及意義 ···························································.270
10.1.1 問題背景 ··············································································.270
10.1.2 研究意義 ··············································································.271
10.2 研究現(xiàn)狀及內(nèi)容 ·············································································.272
10.2.1 研究現(xiàn)狀 ··············································································.272
10.2.2 研究內(nèi)容 ··············································································.272
10.3 深度學(xué)習(xí)理論研究 ··········································································.273
10.3.1 大數(shù)據(jù)與云計算 ····································································.273
10.3.2 機器學(xué)習(xí) ··············································································.274
10.3.3 深度學(xué)習(xí) ··············································································.275
10.4 TensorFlow深度學(xué)習(xí)框架研究 ··························································.277
10.4.1 TensorFlow系統(tǒng)架構(gòu) ······························································.277
10.4.2 TensorFlow數(shù)據(jù)流圖 ······························································.280
10.4.3 TensorFlow會話管理 ······························································.281
10.4.4 TensorFlow分布式執(zhí)行 ···························································.282
10.4.5 TensorFlow數(shù)據(jù)輸入 ······························································.283
10.5 TensorFlow分布式架構(gòu)分析 ·····························································.285
10.5.1 TensorFlow遠程調(diào)用 ······························································.285
10.5.2 現(xiàn)有TensorFlow分布式模型 ····················································.286
10.6 優(yōu)化算法設(shè)計與實現(xiàn) ·······································································.289
10.6.1 數(shù)據(jù)并行上的優(yōu)化 ·································································.289
10.6.2 模型并行上的優(yōu)化 ·································································.297
10.7 實驗環(huán)境配置 ················································································.304
10.7.1 硬件環(huán)境 ··············································································.304
10.7.2 軟件環(huán)境 ··············································································.304
10.7.3 實驗對象 ··············································································.305
10.7.4 實驗數(shù)據(jù) ··············································································.305
10.8 實驗結(jié)果展示與分析 ·······································································.307
10.8.1 數(shù)據(jù)并行算法測試··································································.307
10.8.2 模型并行算法測試··································································.312
本章小結(jié) ·······························································································.314
思考題 ··································································································.314
參考文獻 ·······························································································.314
第11章 基于深度強化學(xué)習(xí)和模仿學(xué)習(xí)的任務(wù)完工時間優(yōu)化調(diào)度 ···········.317
11.1 任務(wù)調(diào)度 ······················································································.318
11.2 相關(guān)研究 ······················································································.321
11.3 云資源調(diào)度問題定義 ·······································································.322
11.4 調(diào)度方案 ······················································································.327
11.4.1 DeepRM_Online介紹 ······························································.327
11.4.2 強化學(xué)習(xí)模型 ········································································.328
11.4.3 深度強化學(xué)習(xí)訓(xùn)練算法····························································.330
11.5 算法分析 ······················································································.333
11.6 實驗分析與驗證 ·············································································.337
11.6.1 實驗準(zhǔn)備 ··············································································.337
11.6.2 數(shù)據(jù)準(zhǔn)備 ··············································································.337
11.6.3 基線算法 ··············································································.338
11.6.4 結(jié)果分析 ··············································································.339
本章小結(jié) ·······························································································.341
思考題 ··································································································.341
參考文獻 ·······························································································.342
第12章 總結(jié)與展望 ······································································.345
12.1 綠色節(jié)能數(shù)據(jù)中心的綜合解決方案·····················································.346
12.2 多數(shù)據(jù)中心（多調(diào)度域）的調(diào)度策略和算法動態(tài)可選擇 ·························.348
12.3 支持深度學(xué)習(xí)模型的分布式并行調(diào)度 ·················································.349
12.4 從基礎(chǔ)資源調(diào)度拓展到應(yīng)用任務(wù)調(diào)度 ·················································.350
參考文獻 ·······························································································.351