杰克·唐加拉：转型中的高性能计算

E级超算是一个重要里程碑，但它不是终点，算力之巅仍有广阔空间。 Exascale is an important milestone, but it's not the final destination. There is still plenty of room at the top. 人工智能并非高性能计算的又一类负载，它正在重塑科学计算所能依托的硬件底座。 AI is not simply another HPC workload. It's changing the hardware that the scientific computers have access to. 我们的目标，不是堆砌算力，而是实现每秒、每瓦功耗下产出更可信的科学成果。 Our goal is not really simply to do more computations, but to do more trusted science per second and also per watt.

长久以来，我们习惯以浮点峰值、榜单排名度量算力的能力，LINPACK与TOP500构筑了高性能计算数十年的评价坐标系。但当AI重构硬件栈、大规模科学仿真日益以稀疏问题为主导，这套沿用已久的标尺，正在显现边界。 在ICAIS 2026国际人工智能科学家大会上，2021年图灵奖得主杰克·唐加拉（Jack Dongarra）带来主旨演讲《转型中的高性能计算》。作为LINPACK和HPCG两大基准测试的主要开创者，他站在算力范式切换的节点，重新叩问计算的本质价值，直面数据移动、精度取舍、能效约束三大底层矛盾，为HPC与生成式AI的交叉融合，给出审慎而富有穿透力的底层叩问：我们究竟应当如何定义一次“有价值的计算”？

观点一 数据移动，才是当代超算真正的成本瓶颈，而非浮点运算 算法设计的经典范式以最小化浮点运算量为优先目标，这一认知正在异构超算体系下失效。现代大规模科学问题多表现为稀疏矩阵结构，和LINPACK基准所依托的稠密计算场景存在显著鸿沟。在HPCG稀疏矩阵测试中，顶尖超算往往仅能释放理论峰值0.8%-3%的效能，数据通信与访存延迟取代浮点计算，成为制约求解效率的核心瓶颈。Benchmark的跑分表现，无法直接映射真实科学负载上的执行效率。 在稀疏矩阵的HPCG基准下，其仅能达到理论峰值性能的0.8%，尚不足百分之一。其中表现最优的超算，也仅能释放理论峰值的3%。根源在于数据搬运：在系统内流转数据，是一项代价极高的操作。 For this benchmark, the sparse matrix computation, it's running at 22 petal flops. It's quite a difference in terms of the time the solution and in terms of the fraction of peak performance, we're getting 0.8% of the peak performance. It's not even 1% of peak performance. One of the best machines here is the machine that gets 3% of its theoretical peak performance, just 3% of its performance. It's because of data movement because of the fact that pushing data through the system is a very expensive operation. 通信是算力作业里成本最高的环节之一。昂贵的不是浮点运算，而是数据在系统组件之间的搬运。 Communication is one of the most expensive things that we do. It's not floating-point operations. It's moving data from one component of the system to another component. 观点二 AI正在重构硬件设计逻辑，低精度算力崛起，HPC必须学会“混合精度解题” AI的产业需求主导芯片架构演进，低精度矩阵算力成为硬件设计重心，传统64位浮点算力资源被持续压缩。混合精度范式是HPC适配新硬件的可行路径：依托迭代精化、随机舍入、分层预条件等数值工具，以低精度承担主体计算实现加速，辅以高精度校验收敛，在案例中可实现8倍加速、能耗压缩至原有的1/8，同时保全高精度求解的正确性。这也为HPC与AI的交叉研究开辟了稳定的问题域。 人工智能不需要64位浮点运算，16位、8位甚至4位浮点就可以完成任务。而科学计算的挑战，是探索我们能把精度下探到什么程度，在获得更高性能的同时，依旧产出可靠可用的解。 AI does not need 64-bit floating-point operations. You can get by with perhaps 16-bit or 8-bits or even 4-bit floating point operations. Scientific computing is trying to understand how far we can push down in that range to get better performance while still coming up with acceptable solutions. 观点三 HPC的破局方向：混合精度与“审慎激进”算法 E级超算是算力发展的关键里程碑，而非终极目标。未来HPC的评价体系将从单一FLOPS指标，转向求解时间、能耗开销、结果可信度的多维度量。“审慎激进（responsibly reckless）”的算法思路，允许采用存在失稳风险的低精度快速算法，同时嵌入误差检测与回退机制，在性能与稳健性之间建立可控的平衡。GPU并非AI算力的唯一载体，面向矩阵运算扩展指令、近存架构的定制化处理器，同样可同时承载HPC与AI负载，算力竞争逐步转向软硬件协同的系统性创新。 关键要点是，E级超算是一个重要的里程碑，但它不是最终目标。下一代高性能计算将会被定义，不会仅仅由运算次数来决定，即机器能执行多少次运算，这取决于其智能程度，我们能否利用精准度，通信、能源和算法，来确保其可靠性？科学作为结果，所以说，这方面仍有很大的发展空间。 The take away here is that exascale is an important milestone, but it's not the final destination. The next era of high-performance computing is not gonna be defined simply by how many operations the machine can execute, but will be defined by how intelligently we can use the precision, the communication, the energy, and the algorithms to get this trusted science as a result. So that's where there's still plenty of room at the Top. 我们的目标不再是完成更多运算，而是实现每秒、每瓦功耗下产出更多可信的科学成果。 Our goal is not really simply to do more computations, but to do more trusted science per second and also per watt. 观点四 TOP500的历史价值仍在，但行业评价体系需要迭代 自1993年面世以来，TOP500记录了超算半个世纪的增长脉络，具备不可替代的历史观测价值。但基准的固有局限，使其难以表征稀疏仿真这类主流科研负载。HPCG等面向真实物理问题的基准，应当作为现有评价体系的重要补充。对基准本身边界的辨析，亦是计算方向值得持续深耕的研究命题。 如果重新来做这份榜单，我可能不会收录500台机器，前50台就足以让我们看清高性能计算的格局。 If I was doing this again, I probably wouldn't have 500 machines. I think we can get by with the Top 50 and get a good handle on high performance computing. 这个基准测试，即LINPACK基准测试，进行密集矩阵计算。如今，我们的应用程序并没有真正地以密集矩阵计算为结构。我们做的，是稀疏矩阵计算，所以这并不典型，已经不符合我们对高性能计算的常规认知。 The Linpack benchmark does dense matrix computations. Today, our applications are not really structured in terms of dense matrix computations. We're doing sparse matrix computations. 附全文演讲High Performance Computing in Transition《转型中的高性能计算》 I'd like to talk about high performance computing and what's happening today in the area of high-performance computing. I view high performance computing or computers as very sophisticated scientific instruments. I sort of think of them on the same level as I do the Webb Telescope. But for computational science, HPC involves aggregating, the computing power for solving complex problems at high rates of execution. 今天，我想谈谈高性能计算，以及当今高性能计算领域正在发生的变化。我将高性能计算或超级计算机视为非常复杂的科学仪器。某种程度上，我觉得它们与韦布空间望远镜处于同一量级。但在计算科学中，HPC的核心在于聚合算力，以极高的执行速率解决复杂问题。 We can think about tackling some of the most important challenges that we have today looking at clean energy, resilient power grids, climate change, and weather prediction, cancer studies, advanced materials, as we just heard a little bit about catalysis, if we take a look at the environment that we have today for high performance computing, it looks something like this. We have a highly parallel computing system that uses distributed memory and uses a programming model that's based on MPI and OpenMP. This is a typical high-performance computer. It's number two on the list of the fastest computers. It's a machine at Lawrence Livermore National Laboratory, and it has 11 million processors in it. 我们可以设想去应对当今一些最重要的挑战，例如清洁能源、弹性能源网、气候变化与天气预报、癌症研究、先进材料——正如我们刚刚听到的关于催化作用的讨论。如果我们审视当今的高性能计算环境，它大致是这样的：我们拥有一个高度并行的计算系统，采用分布式内存，并使用基于 MPI 和 OpenMP 的编程模型。这是一台典型的高性能计算机。它在全球最快超级计算机榜单上排名第二，是位于劳伦斯利弗莫尔国家实验室的机器，拥有 1100 万个处理器内核。 That machine consumes 35 megawatts of power. It gets most of its computational power from GPUs. So it has AMD GPUs in this system which are used for that. High performance computing today is really based on heterogeneous computing where we use commodity processors tied together with accelerators in the form of GPUs to get this processing power. 这台机器消耗 35 兆瓦的电力。它的大部分算力来自 GPU——该系统采用了 AMD GPU 来实现这一目标。如今的高性能计算实际上建立在异构计算的基础之上，我们通过将通用处理器与 GPU 形式的加速器结合在一起来获得这种强大的算力。 One of the most important things or one of the critical resources on these machines is data movement. So communication is one of the most expensive things that we do. It's not floating-point operations. It's moving data from one component of the system to another component. When I was growing up and learning about algorithms, it was ingrained in me that we want to choose algorithms that have the fewest number of floating-point operations because they would cost the time the solution would be best with fewer floating operations. But on today's systems, that's not the metric. In fact, because of the communication costs, an algorithm which has more computations can actually cost us less than an algorithm, a comparable algorithm, which does more communication in for scientific computing, we think about doing the computations using 64 bit or 32-bit floating-point operations. And the operations I'm talking about are adds and multiplies. So that's the common way we think about doing our computations using usually 64-bit, but we can occasionally get 32-bit operations. If you use less precision, your operations go much faster. Today we have a whole range of precisions that we can go for. And that's the result of AI does not need 64-bit floating-point operations. You can get by with perhaps 16-bit or 8-bits or even 4-bit floating point operations. So those operations run very fast on computers. So a challenge for us in scientific computing is trying to understand how far we can push down in that range to get better performance while still coming up with acceptable solutions. 这些机器最关键的资源之一就是数据移动。因此，通信是我们所做事情中成本最高昂的环节，而不是浮点运算，真正的瓶颈是将数据从系统的一个组件传输到另一个组件。当我年轻时开始学习算法，老师深深植入我脑海中的观念是：我们应该选择浮点运算次数最少的算法，因为运算次数越少，求解时间就越短，效果就最好。但在当今的系统上，这不再是衡量标准。事实上，由于通信成本的存在，一个计算量稍多但通信量较小的算法，其实际开销反而可能比一个计算量少但通信量大的同类算法要低。在进行科学计算时，我们习惯于使用 64 位或 32 位浮点运算来进行计算。我所指的运算是加法和乘法。这是我们进行计算的常规方式，通常使用 64 位，偶尔也会使用 32 位运算。如果你使用较低的精度，运算速度会快得多。如今，我们可以使用一整套不同的精度范围。这是人工智能带来的结果——AI 并不需要 64 位浮点运算，使用16 位、8 位甚至 4 位浮点运算就完全足够了。因此，这些低精度运算在计算机上的运行速度极快。对于我们从事科学计算的人来说，面临的挑战是如何在精度范围内尽可能向下探索，以获得更高性能，同时仍能得出可接受的解。 I was responsible for this benchmark. It goes by the name of the Linpack benchmark. Linpack benchmark was started back in 1977 when we developed a package of software called Linpack. Linpack a is software for solving systems of linear equations. My colleague, Hans Meuer and Erich Strohmaier had a list of the fastest computers and they ranked those computers by the theoretical peak performance for a computing system. I had a benchmark in 1977 which used the software and Linpack to measure the computational time for solving Ax=b，a system of linear equations. Hans Meuer suggested that we get together and put our list together and come up with a list of the 500 fastest computers based on that benchmark. That's what we have today. We solve a system of linear equations. It's a dense matrix problem. We used Gaussian elimination with partial pivoting as the algorithm you're able to optimize that algorithm as best you can. We're going to measure the time the solution and convert that into a rate of execution. We're going do this for 64 bit operations. That's the Top 500 list, and that list comes out twice a year. It comes out in November and also in June. So the most recent list we have is from the summer. It's about to be updated with new machines in November. 我曾负责过一个基准测试，名为 Linpack 基准。Linpack基准始于 1977 年，当时我们开发了一套名为 Linpack的软件包。Linpack 是一个用于求解线性方程组的软件。我的同事汉斯·梅尔（Hans Meuer）和埃里希·斯特罗迈尔当时整理了一份全球最快计算机的榜单，他们最初是按计算系统的理论峰值性能对这些计算机进行排名的。我在 1977 年设计了一个基准测试，利用Linpack软件来测量求解线性方程组Ax=b的计算时间。汉斯·梅尔建议我们合作，把各自的榜单合并，推出基于该基准测试的全球前500台最快计算机榜单（Top 500）。这就是我们今天所看到的。我们通过求解一个稠密矩阵问题（线性方程组）来测试。我们采用带部分选主元的髙斯消元法作为算法，你可以尽最大努力去优化该算法。我们将测量求解时间，并将其转换为执行速率。我们针对64位运算来执行此操作。这就是Top500榜单，它每年发布两次——分别在11月和6月。因此，我们手中最新的榜单来自今年夏天，11月即将有新的机器加入并对其进行更新。 This is a list of the Top five computers. So the number one machine is a machine here in China. It's in Shenzhen, it's called the Line Shine computer. It execution rate, the rate at which it solves that benchmark problem is at 2.2 EFLOPS. What's an exa flop? Eflops is 10 to the 18 floating point operations per second. So that's a billion floating point operations per second. That's not 2 billion. It's a billion floating point operations per second. That computer is able to run that benchmark at 80% of its theoretical peak performance. The theoretical peak is a paper and pencil calculation. How many processors do you have? How fast are those processors for doing floating point arithmetic. And we can compute that theoretical peak and we are achieving 80% of that theoretical peak for this problem. When this problem is run, it consumes 42 megawatts of power. It's 42 megawatts. If I in my home in Tennessee, if I use one megawatt for a year, I'll get a bill for $1 million. This machine basically to turn it on is around $42 million per year. I don't know what the cost of power is in China, but I know in my area it would be 1,000,000 per megawatt per year. So they're very expensive in terms of the power consumption of these machines. The Top five machines here, we have three of those machines in the US. Those are at the Department of Energy laboratories, another machine in Germany at the Jülich Supercomputing Centre in Germany itself that just achieved one that just made it on the one. EFLOP a list. You can see those very powerful systems there. 以下是前五名超级计算机的名单。排名第一的机器在中国，位于深圳，被称为“Line Shine”超级计算机。它的执行速率——即求解该基准测试问题的速率达到了 2.2 EFLOPS。什么是 Eflops？Eflops是指每秒1018次浮点运算，也就是每秒百亿亿次。这台计算机在运行该基准测试时，达到了其理论峰值性能的 80%。理论峰值是通过纸笔计算得出的：你有多少个处理器？这些处理器做浮点算术的速度有多快？我们可以计算出这个理论峰值，而我们在解决这个问题时，实现了理论峰值 80% 的效能。当运行这个程序时，它消耗 42 兆瓦的电力——整整 42 兆瓦。如果我在田纳西州的家中一年消耗一兆瓦的电，我会收到一张 100 万美元的电费账单。这台机器光是启动一年的电费大约就是 4200 万美元。我不知道中国的电费成本是多少，但我知道在我那儿，每兆瓦每年的电费大约是 100 万美元。因此，这些机器在功耗方面的开销非常巨大。在前五名机器中，我们在美国拥有三台，分别位于能源部的实验室里；另一台在德国于利希超级计算机中心，它刚刚达到一 EFLOPS 的门槛，成功挤进榜单。你可以看到这些系统都非常强大。 We have a list of 500 computers. And if you take a look at the distribution of the performance for those 500 computers, you get something which looks like this. There are 500 dots here. The Line Shine machine is number one. El Capitan is number two, Frontier is number three, and so forth, and you get this long tail associated with it. If I was doing this again, I probably wouldn't have 500 machines. I think we can get by with the Top 50 and get a good handle on high performance computing. This is a list that tries to give you historical view of the fastest computers. The red line here is charting the computer at the position number one. Today up in the right corner, there is the Line Shine system at 2.2 EFLOPS. The guy at the bottom of the list that just made it on the list in June is this machine which is at 2.7 PetaFLOPS. So it's roughly three orders of magnitude separating the number one and the number 500 machine. You can see that trend has been true for a very long time. We started this list back in 1993 and the fastest computer at that time was at 60 billion floating point operations per second. That computer was a Thinking Machines machine at Los Alamos National Laboratory, which was used for numerical weapons simulations. Today my laptop, I can run the benchmark on my laptop. My laptop runs the benchmark at 426 GigaFLOPS so that my laptop would have been on the list back in 1997. It's faster than that machine that was at Los Alamos National Lab for nuclear weapons simulation. We had this stunning achievement in terms of technology that's happened over this period of time. 我们有一份包含 500 台计算机的榜单。如果你看一下这 500 台计算机的性能分布，就会发现呈现出这样的形态：这里500个点。“Line Shine”机器排名第一，“酋长岩”排名第二，“前沿”排名第三，依此类推，后面拖着长长的尾巴。如果让我重新来过，我可能不会设 500 台机器。我认为我们只需关注前 50 就能很好地掌握高性能计算的全局。这份榜单试图为您提供一个关于最快计算机的历史视角。这里的红线描绘了排名第一的计算机。今天在右上角，是达到 2.2 EFLOPS的Line Shine系统。而榜单底部刚刚在6月勉强上榜的机器，其性能为2.7 PetaFLOPS。因此，排名第一的机器和排名第 500 的机器之间存在着大约三个数量级的差距。你可以看到，这种趋势已经持续了很长时间。我们于1993 年开始发布这份榜单，当时最快的计算机性能为每秒600亿次浮点运算。那台计算机是位于洛斯阿拉莫斯国家实验室的“思考机器”公司制造的机器，当时用于数值武器模拟。如今，我可以在自己的笔记本电脑上运行这个基准测试。我的笔记本电脑运行该基准测试的速度达到了426 GigaFLOPS，这意味着它如果放到1997年的榜单上，速度甚至会比当年洛斯阿拉莫斯国家实验室用于核武器模拟的那台巨型机器还要快。在这段时期内，我们在技术上取得了令人惊叹的成就。 This benchmark that we have was very important in the early days when it came out, it really did a very good job of modeling how most scientific computations behave on Computers. But a lot has changed over the time since 1977. Today our machines are over provision for floating-point computation. They have too much floating-point computation and the real crisis in terms of getting performance out of these machines is moving data. The data communication is one of the most expensive things that we do. This benchmark, the Linpack benchmark does dense matrix computations. Today, our applications are not really structured in terms of dense matrix computations. We're doing sparse matrix computations. It's not typical of what we see for high performance computing. We're getting a distorted view of how these machines are used if you were designing a computer that got a good Linpack number, you probably would not do very well for most applications. Today we're using our high-performance computers for a number of things looking at climate modeling, looking at combustion reactions, looking at electronic fuels, looking at fusion, looking at material design. And those problems are really at the core doing a three-dimensional partial differential equation. The partial differential equation gets discretized, and the discretization leads to a system of linear equations that we have to solve. We're still solving ax equals B, but the structure is quite different. It's no longer a dense matrix. It's a sparse matrix and solving that problem on our machines, we use a different algorithm. 我们拥有的这个基准测试，在早期刚推出时非常重要，它确实极好地模拟了当时绝大多数科学计算在计算机上的行为特征。但自1977 年以来，情况发生了很大变化。如今，我们的机器在浮点计算能力上是“过度配置”的——它们拥有过多的浮点计算能力，而从这些机器中榨取性能的真正危机在于数据移动。数据通信是我们所做事情中成本最高昂的环节之一。Linpack 基准测试执行的是稠密矩阵计算。然而今天，我们的应用程序并不是以稠密矩阵计算为主构架的，我们做的是稀疏矩阵计算。这并不典型代表我们所看到的高性能计算实际应用场景。如果你根据 Linpack 得分来设计一台计算机，你可能会发现它对大多数实际应用来说表现并不理想。今天，我们使用高性能计算来做很多事情：气候模拟、燃烧反应、电子燃料、核聚变研究、材料设计等。这些核心问题本质上都在求解三维偏微分方程。偏微分方程会被离散化，而离散化会导致我们必须求解的大型线性方程组。我们仍然在求解Ax=b，但其结构完全不同。它不再是稠密矩阵，而是一个稀疏矩阵，并且在我们的机器上求解该问题时，我们需要使用一种不同的算法。 We don't use Gaussian elimination with partial pivoting, we end up doing iterative methods. We have another benchmark which models that particular problem. This benchmark goes by the name of HPCG. It's a conjugate gradient algorithm the conjugate gradient algorithm is an iterative process. You make a guess at the solution. And then you iterate. And that iteration will eventually hopefully converge to the solution. The 3D partial differential equation gets discretize using a 27-point stencil that leads to a matrix that has the structure of that matrix at the upper right corner. That's a sparse matrix, has a number, large number of zeroes in the matrix and has the diagonal stripes which really reflect the non-zero structure of the matrix. And then we iterate on that. So the computational kernel is quite different than the previous benchmark. The previous benchmark did a matrix multiply as its computational kernel. This is doing a sparse matrix times a vector that is as its computational kernel. We're going to be moving a lot more data. We're not going to be doing as many floating operations. We're going to stress the system and that stress leads to this performance. This is a list of the Top five computers for this benchmark. So again, the Chinese computer Line Shine is number one on the list. It has 13 million cores in it. For the dense problem, it ran at ExaFLOPS little over ExaFLOPS. It was number one on the list, but for this benchmark, the sparse matrix computation, it's running at 22 PetaFLOPS. It's quite a difference in terms of the time the solution and in terms of the fraction of peak performance, we're getting 0.8% of the peak performance. It's not even 1% of peak performance 8%. So very little of the peak performance. And you can look down the list here and see what the other machines are getting. So we're getting very poor performance out of these computers on typical problems that are being solved on these machines. And again, it's because of data movement because of the fact that pushing data through the system is a very expensive operation. One of the best machines here is the machine in Japan that gets 3% of its theoretical peak performance, just 3% of its performance. 我们不再使用带部分选主元的髙斯消元法，而是最终采用迭代法。我们有另一个基准测试来模拟这种特定问题，这个基准测试名为HPCG。它采用的是共轭梯度算法。共轭梯度算法是一个迭代过程：你先对解做一个猜测，然后进行迭代。这种迭代最终有望收敛到正确的解。三维偏微分方程通过 27 点模板进行离散化，这会产生一个矩阵，其结构正如右上角的矩阵所示。这是一个稀疏矩阵，矩阵中含有大量的零，并且具有对角条纹，这真实反映了矩阵中非零元素的结构。然后我们对其进行迭代。因此，其计算核心与之前的基准测试截然不同。前一个基准测试以矩阵乘法作为其计算核心；而这个测试以稀疏矩阵乘向量作为其计算核心。我们要移动更多的数据，却不会进行那么多的浮点运算。我们将对系统施加巨大的压力，而这种压力直接反映在性能表现上。这是该基准测试（HPCG）的前五名计算机榜单。同样，中国的“Line Shine”计算机名列榜单第一，它拥有 1300 万个核心。对于稠密矩阵问题，它的运行速度达到了 EFLOPS 级别（略高于 1 EFLOPS），位居第一；但对于这个稀疏矩阵计算的基准测试，它的运行速度是 22 PetaFLOPS。在求解时间和峰值性能利用率方面，存在着相当大的差距——我们仅达到了峰值性能的0.8%。相比理论峰值，实际发挥出的性能非常低。你可以顺着榜单往下看，看看其他机器的表现。因此，在解决这些机器上运行的典型问题时，它们的实际性能表现其实非常糟糕。再次强调，这主要是因为数据移动，因为在系统中推送数据是一项成本极高的操作。这里表现最好的机器之一是日本的超级计算机，但也仅能达到其理论峰值性能的 3%——仅仅 3% 的性能。 We have HPC in this mix working now with generative AI has really reorder the computing landscape in and it's done that across three axes. The first thing is the capital investment that goes on in AI is just incredible. One of the biggest machines for AI is a machine in Tennessee. It's called Colossus. It's being built by Elon Musk. This machine will have 1 million Nvidia GPUs in it 1 million GPU. And the cost of that machine looks like 30 to 45 billion USD dollars for the system. That's just a hardware cost of this machine. The energy that's going to be required for this system is measured not in terms of megawatts, but in terms of gigawatts of power, this machine will cost about $1.8 billion just to turn it on. For that, it's an incredible cost associated with the power consumption for this computer. For AI, we have this market capitalization of companies, so the market capitalization is what a company worth. We have a lot of companies involved in in this area today. If we take a look at the market capitalization, we see traditional computer companies like IBM, AMD, Intel, HP have a market capitalization which in aggregate is around $2 trillion. If we take a look at those companies that are investing in AI, we see this incredible situation. Companies like Facebook, Amazon, Microsoft, Google, Apple, and Nvidia have this market capitalization Nvidia's. Capitalization is over five trillion us dollars just an incredible situation with those companies. And compared to the traditional companies. 当前，高性能计算与生成式人工智能的结合，真正重新洗牌了计算格局，这种重塑体现在三个维度。首先，人工智能领域的资本投入简直令人难以置信。目前用于 AI 的最大机器之一位于田纳西州，名为 Colossus，由埃隆·马斯克建造。这台机器将拥有 100 万个 Nvidia GPU——足足 100 万个 GPU。该系统的硬件成本预计在 300 亿到 450 亿美元之间。而且，该系统所需的能源不再以兆瓦来衡量，而是以吉瓦（十亿瓦）来衡量。光是把这台机器开机，就要花费大约 18 亿美元的电费。因此，这台计算机伴随着令人难以置信的巨额功耗成本。在人工智能领域，我们看到了惊人的公司市值。市值代表了一家公司的价值。如今有很多公司涉足这一领域。如果我们观察市值，会发现传统的计算机公司——如 IBM、AMD、Intel、HP——它们的总市值加起来大约在 2 万亿美元左右。而如果我们看看那些正在大力投资 AI 的公司，就会发现一种不可思议的局面：像 Facebook、Amazon、Microsoft、Google、Apple 和Nvidia 这些公司，其市值呈爆炸式增长（其中仅 Nvidia 一家公司的市值就超过了 5 万亿美元）。与传统公司相比，这简直是一个不可思议的格局。 So AI is not simply another HPC workload. It's changing the hardware that the scientific computers have access to and there's three consequences of that. The first one is there's an investment in dense matrix computations. I mentioned that most of the large scale scientific computing is not doing dense matrix computations, but in AI, that's exactly what we're doing. Dense matrix computations. The matrices are small, but the matrices don't require high precision, doesn't require high precision. It requires low precision. So we can get away with 16 bits or 8 bit or even 4 bit floating point computations rather than 64 bit or 32 bit, has much greater pressure on memory management and energy for these systems. And software ecosystem that we have is today is being increasingly optimized for AI there's this tension that we see. The between HPC and AI systems, AI systems need low precision. HPC in general, needs high precision. We look at how the manufacturers are building hardware to satisfy both of those demands. Matrix multiply is central to AI if we take a look at the neural layers that are involved in the algorithms, we end up needing to do something like this matrix multiply here. We have X, which is the input data. We have W, which are the learned weights. We need to multiply them together and come up with the output, which is then used to cycle through again. So these models have a very important characteristics associated with them. And that's the matrix multiply that we have. 因此，AI并不单纯是又一个HPC工作负载，它正在改变科学计算机所能接触到的硬件生态，这带来了三个后果：第一，大量资金涌入并投资于稠密矩阵计算。我刚才提到，绝大多数大规模科学计算并不做稠密矩阵计算，但在 AI 领域，这正是我们所做的事情——稠密矩阵计算。尽管矩阵的尺寸较小，但它们不需要高精度，而是需要低精度。因此，我们可以使用 16 位、8 位甚至 4 位浮点运算，而不必使用64位或32位。这给这些系统的内存管理和能耗带来了巨大压力。此外，我们现有的软件生态系统也正日益围绕AI进行优化。我们看到了这样一种张力：HPC与AI 系统之间存在需求冲突。AI 系统需要低精度，而 HPC 通常需要高精度。我们正在观察制造商们如何构建硬件来同时满足这两种需求。矩阵乘法是AI的核心。如果我们看一下算法中涉及的神经网络层，最终我们需要执行类似这里的矩阵乘法：我们有X（输入数据），我们有W（通过学习得到的权重）。我们需要将它们相乘，得出输出结果，然后再将结果循环送入下一层。因此，这些模型具有非常重要的特征，那就是我们所面对的大量矩阵乘法。 So AI does not require GPUs. However, what it really requires is a high-throughput matrix operations. Those matrix multiplies very high bandwidth memory, efficient data motion, and the software ecosystem that can make that work. So GPUs happen to provide that combination today, but it's not really necessary. in fact with the line shine computer in China that machine has no GPUs in it. It uses an ARM architecture and that ARM architecture was modified by the people designing the eye Line Shine to do a matrix, multiply instruction in the ARM processor. They have the capability of doing that matrix multiply and they have a lot of high-speed memory very close to that processor. So they can move data very quickly between those two components. So they've designed a system which uses no GPUs and they've implemented it. that is able to accommodate the AI capabilities as well as what's happening in a high-performance computing systems. The hardware design for AI is focused on reduced precision to reduce a number of things. It reduces the energy and the data movement costs associated with that. Rather than using 64 bit, they can get away with 16 bit 8 bit, even 4-bit floating point operations. There's a lot of research or interest in doing integer emulation of floating. for this. There's a number of tools that we have that can be used to do high performance computing computations. Those tools come from the mathematics, they come from iterative refinement, stochastic rounding, randomization sketching, and hierarchical preconditioners that can be used. So the attempt here is to use low precision as much as possible in the high-performance computing scenario and use these techniques to help us get to a point where we have an acceptable solution that has the required accuracy that's needed and hopefully avoiding communication along the way and getting the benefits of the reduced energy requirements. 所以，AI 并不一定非要使用 GPU。然而，它真正需要的是高吞吐量的矩阵运算——这些矩阵乘法需要极高的内存带宽、高效的数据移动以及能够使其运转的软件生态系统。碰巧的是，GPU 在今天同时满足了这些组合需求，但它并不是绝对必需的。事实上，中国的那台“Line Shine”计算机就没有使用任何 GPU。它采用的是 ARM架构，并且由设计团队对该 ARM 处理器进行了修改，使其能够直接在处理器内部执行矩阵乘法指令。他们具备执行这种矩阵乘法的功能，并且在处理器附近配备了大量的高速内存，因此他们可以在这两个组件之间非常快速地移动数据。由此，他们设计并实现了一个不使用 GPU 的系统，该系统既能满足 AI 的计算能力，也能适应高性能计算系统的需求。面向 AI 的硬件设计专注于降低精度，以减少多项开销。它降低了能耗以及与之相关的数据移动成本。他们没有使用 64 位，而是完全可以采用 16 位、8位甚至 4 位的浮点运算。目前有大量的研究和兴趣正投入到利用整数来仿真浮点运算中。我们拥有一系列可以用于高性能计算的数学工具。这些工具来源于数学领域，包括：迭代精化、随机舍入、随机化草图以及分层预处理。这里的核心尝试是：在 HPC 场景中尽可能多地使用低精度，并借助这些技术，让我们最终能够得到满足所需精度要求的、可接受的解，同时在过程中尽量避免数据通信，并收获降低能耗带来的好处。 I want to point out what's happening with Nvidia. Nvidia has a processor called Hopper back in 2022. That processor had the following characteristics in terms of 64-bit operations 32-bit operations, and 16-bit floating point operations. 2 years later, they had a new architecture. And that architecture was called Blackwell and take a look at what's happened to 64-bit operations. They went from 66 TeraFLOPS to peak performance down to 40 TeraFLOPS. The newer processor actually got less performance for doing 64-bit operations. 32-bit operations increased going from roughly 500 to over 1,000 TeraFLOPS. 16-bit operations increased again about a factor of two and they've introduced 4-bit floating point operations and you can see the performance for 4-bit operations there. 2 years later, the new architecture came out from Nvidia called Rubin. Again, 64 bit operations are being reduced in terms of their performance characteristics, down to 33 teraflops where we get an increase for 32 bit 16 bit and 4 bit floating point operations going to 25 teraflops so incredible increase and that's being that's being driven by the AI computations Nvidia has pivoted from scientific computations pivoted to addressing the needs of AI because of the market is so great and they're not investing as much real estate on the silicon for doing 64-bit operations, the effects of that we're not getting the kind of performance for HPC as we had hoped to get with newer generations of the GPU of Nvidia GPU. That's a rather striking accommodation. 我想指出 Nvidia 正在发生的变化。Nvidia 在 2022 年推出了一款名为 Hopper的处理器。该处理器在 64 位运算、32位运算和 16 位浮点运算方面具备各自的性能特征。两年后，他们推出了新架构，名为Blackwell。看看 64 位运算发生了什么：其峰值性能从 66 TeraFLOPS 下降到了 40 TeraFLOPS。这款更新的处理器在执行 64 位运算时的性能实际上反而变低了！而 32 位运算性能有所增加，从大约 500 TeraFLOPS 跃升至 1000 多 TeraFLOPS。16 位运算的性能再次翻了一番，并且他们还引入了 4 位浮点运算，你可以看到其 4 位运算的性能表现。又过了两年，Nvidia 推出了名为 Rubin的新架构。同样，64 位运算的性能特征进一步被削减，降至 33 TeraFLOPS；与此同时，32 位、16 位和 4 位浮点运算的性能则大幅跃升——这种令人难以置信的性能增长是由 AI 计算驱动的。Nvidia 已经将业务重心从科学计算转向了满足 AI 的需求，因为 AI 的市场实在是太庞大了。他们在硅片上分配给 64 位运算的硬件面积在减少。其直接后果是：我们在 HPC 领域无法从新一代 Nvidia GPU 中获得我们原本期望的那种 64 位性能提升。这是一个相当引人注目的妥协与转变。 From the scientific side, we'd like to look at algorithms which I call responsibly reckless. We want an algorithm which is a fast algorithm, and it may be unstable. You may get the wrong answer, but hopefully you don't get the wrong answer very often. And you have some way to detect that. In fact, you're getting the wrong answer. But these algorithms are avoiding data movement, you're avoiding synchronization of the parallel processing and they're using mixed precision to accomplish their task. So using the lower precision whenever possible and hopefully getting some effects of the higher precision computations and then they check for instability. And if the problem is not getting the right answer, they'll end up aborting the problem and going back and using the more traditional way, other more traditional being the more stable algorithm to do the computation we need to do that. Check to see, in fact, if we've gotten the wrong answer. So mixed precision is faster because it does low less communication. You have small amounts of data. Memory traffic, network traffic is reduced. You have a smaller footprint associated with the computation. The arithmetic is going to be faster because you're using that part of the machine which was designed to run faster and hopefully you can improve the algorithm performance without compromising the quality of the solutions that you have. Here's an example where that works. We're going to solve Ax=b. In this case here, the matrix is a dense matrix. We're going to factor the matrix. We're going to decompose that matrix an into two components, the LU factor and we're going do that using 16-bit arithmetic. We're going to give an approximation to the factorization. That approximation is going to be used to drive an iterative process whereby we enhance the accuracy of the solution that we get with that approximation. And hopefully come to the point where we get to 64-bit accuracy at the end of the iteration. This algorithm is an old algorithm. It's called iterative refinement and it's being used in this context to mix the precisions 16-bit and 64-bit arithmetic with checks to ensure that you get the right accuracy. So as you run this algorithm. And the problem is well conditioned, you're going to end up with a solution which is as accurate as if you had done the whole computation using 64-bit arithmetic. The graph here on the right is the performance characteristics of those two algorithms. The blue is giving us an indication of the performance for a dense matrix calculation using Gaussian elimination with partial pivoting. That algorithm takes a certain amount of time and consumes a certain amount of energy. If we use this mixed precision algorithm, we can solve the problem much faster and get the same accuracy as we did use 64-bit arithmetic. In this particular case, it's 8 times faster on it, uses 8 times less energy. In that way, we have a situation where it's a big win for the solution stressing the lower precision in arithmetic while still maintaining the higher precision accuracy. 从科学计算的角度来看，我们希望寻找那些我称之为“责任式激进”（responsibly reckless）的算法。我们希望有一种运行速度极快的算法，虽然它可能有些不稳定，有时可能会给出错误答案，但只要我们能保证它不会频繁出错，并且有办法检测出它何时得到了错误答案。这类算法的核心在于：它们避免了数据通信，避免了并行处理的同步开销，并利用混合精度来完成任务。也就是说，只要可能就使用较低的精度，同时希望兼顾较高精度计算的一些效果，然后检查其不稳定性。如果发现计算结果不对，就会选择中止该任务，转而退回到传统的方式——即采用更稳定的算法来重新进行计算。我们需要进行这种检查，以确信我们没有得到错误答案。因此，混合精度计算速度更快，因为它产生的通信量更少。数据量变小了，内存流量和网络流量也随之减少，计算所占用的内存空间也更小。由于你使用的是专为高速运行而设计的硬件部分，算术运算的速度会更快。希望我们能在不折损所得解之质量的前提下，提升算法的性能。这里有一个行之有效的例子。我们要求解 $Ax=b$。在这个例子中，矩阵是一个稠密矩阵。我们要对该矩阵进行因子分解。我们将矩阵$A$ 分解为两个分量——LU 分解，并且使用 16 位算术来完成这一步。我们将得到一个近似的分解结果。然后，利用这个近似结果去驱动一个迭代过程，在这个迭代过程中，我们不断增强由该近似解带来的精度，并希望在迭代结束时达到 64 位的精度。这种算法是一个历史悠久的算法，被称为迭代精化。在此语境下，它被用来将16 位和 64 位算术混合使用，并通过校验机制来确保获得正确的精度。因此，当你运行该算法时，只要问题条件良好，你最终得到的解其准确度将完全不逊色于全程使用 64 位算术所得到的解。右侧的图表展示了这两种算法的性能特征。蓝色线条展示了使用带部分选主元的髙斯消元法进行稠密矩阵计算的性能。该算法需要耗费一定的时间和能量。而如果我们使用这种混合精度算法，我们就能以快得多的速度解决问题，同时获得与使用 64 位算术完全相同的精度。在这种特定情况下，它的速度提升了 8 倍，能耗降低了 8 倍。通过这种方式，我们在压榨低精度算术潜力的同时，依然保持了高精度的准确性，取得了巨大的成功。 So the next era for HPC, this exascale point that we've managed to hit is not the destination that we're looking for. It's just a milestone along the way. The peak performance is no longer enough we need to measure the energy, the time to solution and the confidence in the answers that we have. AI hardware is really creating an opportunity, I'll say, for scientific computing, using low precision for speed and efficiency and using high precision selectively where we hopefully can validate and test for this correctness. The future is going to be driven by this idea of hardware algorithms and software that are co-designed together to give us the right kind of solutions. Our goal is not really simply to do more computations, but to do more trusted science per second and also per watt. We're going to mix energy and time to solution in this context. The peak performance is not the same as scientific performance AI hardware offers enormous low precision capability and scientific computing really has to control the errors associated with it. The object is not simply more flops, but fewer joules and less time to solution that we can trust. There's plenty of room at the Top for these high-performance machines. We've seen a lot of hardware that's constantly changing. We've gone from scalar to vector machines, machines that use distributed memory, machines that use accelerators as in the form of GPUs. And then now today we're looking at using mixed precision to augment the kind of things that we have at our disposal. The algorithms and software really are evolving with the hardware changes come take place on those hardware changes require other software on the algorithms to be restructured to accommodate those algorithms. There was a paper published in Science a few years ago by Charles Leiserson and company. And the title of the paper was There is plenty of room at the Top: What will drive computer performance after Moore's law? where they describe and look at a number of issues related to those software algorithms and hardware that I've talked about. And this is a riff on Feynman's lecture back in 1959 where he was talking about. There's plenty of room at the bottom where he was talking about quantum computing and how would great potential it might have in the future. The takeaway here is that exascale is an important milestone, but it's not the final destination. The next era of high-performance computing is not gonna be defined simply by how many operations the machine can execute, it will be defined by how intelligently we can use the precision, the communication, the energy, and the algorithms to get this trusted science as a result. So that's where there's still plenty of room at the Top. Thank you. 因此，高性能计算的下一个时代——我们刚刚达到的百亿亿次节点——并不是我们最终要寻找的终点站，它只是沿途的一个里程碑。单纯追求峰值性能已经不够了，我们需要衡量能耗、求解时间以及对所得答案的置信度。可以说，AI 硬件确实为科学计算创造了一个机遇：利用低精度来换取速度和效率，同时有选择地使用高精度来验证和测试正确性。未来将由硬件、算法和软件三者共同协同设计的理念来驱动，从而为我们提供正确的解决方案。我们的目标不只是单纯做更多的计算，而是在每一秒、每一瓦的功耗下，产出更多值得信赖的科学成果。在此背景下，我们要将能耗与求解时间结合起来考量。峰值性能并不等于科学计算的真实性能。AI 硬件提供了巨大的低精度算力，而科学计算必须严格控制与其相关的误差。我们的目标不仅仅是追求更多的 flops，而是追求更少的焦耳消耗、更短的求解时间，以及我们能够信赖的结果。这些高性能机器的“高处”依然大有可为。我们见证了硬件的不断演进：从标量机到向量机，再到使用分布式内存的机器，以及采用 GPU 形式加速器的机器；而现在，我们正在着眼于利用混合精度来扩充我们手头的工具箱。算法和软件确实随着硬件的变化而演进，硬件的变革反过来也要求对软件和算法进行重构，以适应这些新硬件。几年前，《科学》杂志上发表了一篇由查尔斯·莱瑟森等人撰写的论文，标题是：《高处大有可为：摩尔定律之后是什么在驱动计算机性能？》。他们在文中描述并探讨了许多与我刚才谈到的软件、算法和硬件相关的问题。这其实是对费曼 1959 年著名演讲《底部大有可为》的一次致敬与改编，当年费曼探讨的是量子计算及其在未来的巨大潜力。我们得到的启示是：Exascale是一个重要的里程碑，但它不是终点站。高性能计算的下一个时代将不再仅仅由机器能执行多少次运算来定义，而是将由我们能以多聪明的智巧去利用精度、通信、能量和算法，从而得出值得信赖的科学成果来定义。正因如此，“高处依然大有可为”。 谢谢大家。