中国科学院：CERN专题计算学校《T-CSC数据存储》课程教学资源（讲义）Writing Parallel software（booklet）

Writing Parallel software Writing Parallel software Sebastien Ponce sebastien.ponce@cern.ch CERN Thematic CERN School of Computing 2022 1/46 S.Ponce-CERN

Writing Parallel software 1 / 46 S. Ponce - CERN Intro Threading Safety Solutions Writing Parallel software S´ebastien Ponce sebastien.ponce@cern.ch CERN Thematic CERN School of Computing 2022

Writing Parallel software Outline Introduction and expectations Thread-safety issues oScope what is parallelism ●Data races o Finding parallelism o Thread safety Expectations Thread-safety solutions 2Threading,theory and practice Avoid the problem oProcesses and threads o Replicate o Python threading ●Atomics C++threading and async o Locking Credits to Danilo Piparo for the original talk in previous tCSCs ..and all the content I shamelessly stole from it 2/46 S.Ponce-CERN

Writing Parallel software 2 / 46 S. Ponce - CERN Intro Threading Safety Solutions Outline 1 Introduction and expectations Scope : what is parallelism ? Finding parallelism Expectations 2 Threading, theory and practice Processes and threads Python threading C ++threading and async 3 Thread-safety issues Data races Thread safety 4 Thread-safety solutions Avoid the problem Replicate Atomics Locking Credits to Danilo Piparo for the original talk in previous tCSCs ... and all the content I shamelessly stole from it

Writing Parallel software Introduction and expectations Threading.theory and practice Introduction and expectations Scope what is parallelism Thread-safety issues Finding parallelism Expectations Thread-safety solutions what find expect 3/46 S.Ponce-CERN

Writing Parallel software 3 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Introduction and expectations 1 Introduction and expectations Scope : what is parallelism ? Finding parallelism Expectations 2 Threading, theory and practice 3 Thread-safety issues 4 Thread-safety solutions

Writing Parallel software “ Parallelism Definition o the ability to make 2 or more things concurrently more parallel 2nd parallel flow main flow start end parallel flow concepts o concurrency running 2 things in parallel o asynchronicity disentangle launching a task from getting its result what find empest 4/46 S.Ponce-CERN

Writing Parallel software 4 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Parallelism Definition the ability to make 2 or more things concurrently start end main flow parallel flow 2 nd parallel flow more parallel concepts concurrency running 2 things in parallel asynchronicity disentangle launching a task from getting its result

Writing Parallel software Intro c" Different flavors along the history Multiple levels of parallelism o machines(forever)/processors(for 40 years)/cores(for 25 years) o so not really a new topic Matching different programming techniques o RPC,MPI,map/reduce,.../multi-processing multi-threading Why is it trendy now o on many core processors you cannot avoid it individual cores are too slow o multi-core/multi-threading is tricky o and a major source of nasty bugs what find empect 5/46 S.Ponce-CERN

Writing Parallel software 5 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Different flavors along the history Multiple levels of parallelism machines (forever) / processors (for 40 years) / cores (for 25 years) so not really a new topic ! Matching different programming techniques RPC, MPI, map/reduce, .../ multi-processing / multi-threading Why is it trendy now ? on many core processors you cannot avoid it individual cores are too slow multi-core/multi-threading is tricky and a major source of nasty bugs

Writing Parallel software Intro Why would you use parallelism From the software point of view 2 main targets o multiplexing different tasks on the same machine o making one heavy task faster o by running different subparts of it in parallel From the hardware point of view o do not waste available resources o make best value for money Net consequence:we need to find things to be done in parallel what find expect 6/46 S.Ponce-CERN

Writing Parallel software 6 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Why would you use parallelism ? From the software point of view 2 main targets multiplexing different tasks on the same machine making one heavy task faster by running different subparts of it in parallel From the hardware point of view do not waste available resources make best value for money Net consequence : we need to find things to be done in parallel

Writing Parallel software 花5 Finding parallelism Task parallelism Sum act on the work flow Measures o break the work in tasks Sum squares o run several concurrently Data parallelism datal Computex2 osplit data in pieces data2 Computex2 o run on several concurrently data3 Computex2 what find empect 7/46 S.Ponce-CERN

Writing Parallel software 7 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Finding parallelism Task parallelism act on the work flow break the work in tasks run several concurrently Measures Sum Sum squares σ Data parallelism split data in pieces run on several concurrently data1 data2 data3 Compute χ 2 Compute χ 2 Compute χ 2

Writing Parallel software Task parallelism Concept parallelism achieved through the partition of load into "baskets of work" consumed by a pool of resources. Implications o requires splitting the processing into blocks o and dealing with dependencies between them o well adapted to data processing frameworks Thread/process pools usually useful a pool a workers is created at start o and reused during all processing tasks are mapped to available workers what find expect 8/46 S.Ponce-CERN

Writing Parallel software 8 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Task parallelism Concept parallelism achieved through the partition of load into “baskets of work” consumed by a pool of resources. Implications requires splitting the processing into blocks and dealing with dependencies between them well adapted to data processing frameworks Thread/process pools usually useful a pool a workers is created at start and reused during all processing tasks are mapped to available workers

Writing Parallel software Task parallelism example Event reconstruction case o processing consists in running a bunch of algorithms o originally as a sequence ○I■0■○ onow as a dependency graph o using a thread pool and a scheduler mapping what can be done to available threads 010 what find expect 9/46 S.Ponce-CERN

Writing Parallel software 9 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Task parallelism example Event reconstruction case processing consists in running a bunch of algorithms originally as a sequence now as a dependency graph using a thread pool and a scheduler mapping what can be done to available threads

Writing Parallel software Data parallelism Concept parallelism achieved through the application of the same transformation to multiple pieces of data Implications o requires independent pieces of data o may have an impact on data structures o and potentially on memory Practically for our event processing example 时Hp 10/46 S.Ponce-CERN

Writing Parallel software 10 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Data parallelism Concept parallelism achieved through the application of the same transformation to multiple pieces of data Implications requires independent pieces of data may have an impact on data structures and potentially on memory Practically for our event processing example