电子科技大学：《机器学习 Machine Learning》课程教学资源（课件讲稿）第6章 深度神经网络

深度神经网络 电子科技大学 师君

电子科技大学 师 君

信息处理系统结构 图灵结构 运算与存储分离 CPU (intel) Core"2 Quad 。指令存储：冯诺依曼结构、哈弗结构 Memory 传输瓶颈 nVIDIA. 5品品 GPU Control Arithmetic C Unit Logic Unit U XILINX Input Output DSP XA3SD3400A

 图灵结构 ◦ 指令存储：冯诺依曼结构、哈弗结构 CPU GPU DSP 运算与存储分离 传输瓶颈

信息处理系统结构 Brain structure of the bird(goose) cerebrum cerebellum olfactory bulb ，神经网络结构 pituitary optic lobe 由“神经元”互联构成的复杂信息处理系统。 理解神经系统的工作原理、处理人工智能相关的问题。 运算与存储一 起构成网络基 本单元

 神经网络结构 运算与存储一 起构成网络基 本单元 由“神经元”互联构成的复杂信息处理系统。 理解神经系统的工作原理、处理人工智能相关的问题

信息处理系统结构 https://www.theregister.co.uk/2011/08/18/ibm_ darpa synapse_project/ ,图灵结构VS神经网络 Like most DARPA projects,Synapse has some impressive goals and ones that may not pan out.There is a lot of talk about "dawn of a new paradigm"and "dawn of a new age"as researchers try to create brain-like systems.The problem,according to DARPA,is that von Neumann machines,while great for playing Angry Birds and wasting time at work,are less efficient than biological computers-the ripply, fat-encrusted gray stuff between your ears-between a factor of 1 million to 1 billion.It takes an increasingly cbyomplex von Neumann machine to handle increasingly complex data streaming in from the environment

https://www.theregister.co.uk/2011/08/18/ibm_ darpa_synapse_project/  图灵结构 vs 神经网络 Like most DARPA projects, Synapse has some impressive goals and ones that may not pan out. There is a lot of talk about "dawn of a new paradigm" and "dawn of a new age" as researchers try to create brain-like systems. The problem, according to DARPA, is that von Neumann machines, while great for playing Angry Birds and wasting time at work, are less efficient than biological computers – the ripply, fat-encrusted gray stuff between your ears –between a factor of 1 million to 1 billion. It takes an increasingly cbyomplex von Neumann machine to handle increasingly complex data streaming in from the environment

信息处理系统结构 https://www.theregister.co.uk/2011/08/18/ibm_ darpa synapse_project/ [log] von Neumann Machines What is the Machine difference? Neuromorphic Complexity Machines e.g.Gates; Memory; Human level performance Neurons; Dawn of a new age Synapses Power; Dawn of a new Size paradigm Program Objective "simple" "complex" [log] Environmental Complexity e.g.Input Combinatorics

https://www.theregister.co.uk/2011/08/18/ibm_ darpa_synapse_project/  图灵结构

信息处理系统结构 https://en.wikipedia.org/wiki/SyNAPSE ,认知计算(Cognitive Computer) o SyNAPSE The program is being undertaken by HRL Laboratories(HRL), Hewlett-Packard,and IBM Research.In November 2008,IBM and its collaborators were awarded $4.9 million in funding from DARPA while HRL and its collaborators were awarded $5.9 million in funding from DARPA.For the next phase of the project,DARPA added $16.1 million more to the IBM effort while HRL received an additional $10.7 million.In 2011, DARPA added $21 million more to the IBM project.[1]and an additional $17.9 million to the HRL project.[2]

 认知计算(Cognitive Computer) ◦ SyNAPSE  The program is being undertaken by HRL Laboratories (HRL), Hewlett-Packard, and IBM Research. In November 2008, IBM and its collaborators were awarded $4.9 million in funding from DARPA while HRL and its collaborators were awarded $5.9 million in funding from DARPA. For the next phase of the project, DARPA added $16.1 million more to the IBM effort while HRL received an additional $10.7 million. In 2011, DARPA added $21 million more to the IBM project.[1] and an additional $17.9 million to the HRL project.[2] https://en.wikipedia.org/wiki/SyNAPSE

DARPA SyNAPSE Program Plan Phase 0 Phase 1 Phase 2 Phase 3 Phase 4 nA2。题VNEK6 Process and ~106 neuron single Component Synapse CMOS Process Component Circuit chip implementation ~108 neuron Development Integration multi-chip robot Development 2ec中onle Microcircuit Architecture System Level 106 Neuron Design 108 neuron design Architecture for Simulation and Development for simulation and Development Hardware Layout Comprehensive Design hardware layout Capability uonejnw!s 8 Simulate Large Neural ~106 neuron level -108 neuron level "Human"level Design Subsystem Dynamics Benchmark Benchmark (~1010 neuron) 亚 Build Expand Refine Expand Sustain Sustain 11

信息处理系统结构 https://en.wikipedia.org/wiki/Cognitive computer https://en.wikipedia.org/wiki/Electroencephalography https://baike.baidu.com/item/%E8%84%91%E7%94%B5%E6%B3%A2/1599805?fr=aladdin ,认知计算(Cognitive Computer) https://tieba.baidu.com/p/1443142593 ?red tag=3441879498 TrueNorth Neuromorphic CMOS integrated circuit produced by IBM in 2014.It is a manycore processor network on a chip design,with 4096 cores,each one having 256 programmable simulated neurons for a total of just over a million neurons.In turn,each neuron has 256 programmable "synapses"that convey the signals between them.Hence,the total number of programmable synapses is just over 268 million(228).Its basic transistor count is 5.4 billion. 人脑 IBM neural芯片 神经元 860~1000亿 256×4096 神经突触 1000/神经元 256/神经元 时钟周期 <100Hz 10Hz?

 认知计算(Cognitive Computer) ◦ TrueNorth  Neuromorphic CMOS integrated circuit produced by IBM in 2014. It is a manycore processor network on a chip design, with 4096 cores, each one having 256 programmable simulated neurons for a total of just over a million neurons. In turn, each neuron has 256 programmable "synapses" that convey the signals between them. Hence, the total number of programmable synapses is just over 268 million (228). Its basic transistor count is 5.4 billion. https://en.wikipedia.org/wiki/Cognitive_computer 人脑 IBM neural 芯片 神经元 860~1000亿 256×4096 神经突触 1000/神经元 256/神经元 时钟周期 < 100Hz 10Hz ？ https://en.wikipedia.org/wiki/Electroencephalography https://baike.baidu.com/item/%E8%84%91%E7%94%B5%E6%B3%A2/1599805?fr=aladdin https://tieba.baidu.com/p/1443142593 ?red_tag=3441879498

Band Frequency Location (Hz) Normally adult slow-wave sleep subcortical les frontally in adults,posteriorly in 。in babies 。diffuse lesions Delta 32 Somatosensory cortex sound and sight)72][73] decline,espec Also is shown during short-term memory matching of been proven f recognized objects,sounds,or tactile sensations ·Mu suppressio Mu 8-12 Sensorimotor cortex ·Shows rest-state motor neurons.7④ Deficits in Mu in autism.[75]

信息处理系统结构 ARTICLE https:/dol.0rg/10.1038/s41586-018-0180-5 https://www.jiqizhixin.com/arti Equivalent-accuracy accelerated neural- cles/2018-06-12-4 network training using analogue memory Stefano Ambrogio',Pritish Narayanan',Hsinyu Tsai,Robert M.Shelby Irem Boybat2.Carmelo di Nolfo,Severin Sidler Massimo Giordano',Martina Bodinil,Nathan C.P Farinha',Benjamin Killeen',Christina Cheng Yassine Jaoudi Geoffrey W.Burr ,IBM AI7芯片 o Synaptic cell for Analogue-Memory-Based DNN 突触单元：每个单元由一对相变存储器 (PCM)单元和三个晶体管和一个电容器的组 合构成，PCM将权重存储在电阻，便于长期 保存，电容权重存储电荷，便于快速更新。 训练时，只有电容权重更新，经过数千张图 片之后，权重传输到PCM以长期存储。 能源效率达到GPU280倍的， 同样面积上实现100倍的算力

 IBM AI芯片 ◦ Synaptic cell for Analogue-Memory-Based DNN https://www.jiqizhixin.com/arti cles/2018-06-12-4 突触单元：每个单元由一对相变存储器 ( PCM ) 单元和三个晶体管和一个电容器的组 合构成， PCM将权重存储在电阻，便于长期 保存，电容权重存储电荷，便于快速更新。 训练时，只有电容权重更新,经过数千张图 片之后，权重传输到 PCM 以长期存储。 能源效率达到 GPU 280 倍的， 同样面积上实现 100 倍的算力