北京大学：《大规模数据处理——云计算 Mass Data Processing Cloud Computing》课程教学资源（PPT课件）分布式文件系统 Distributed File systems

DFS@wikipedia(1/2) In computing,a distributed file system or network file system is any file system that allows access to files from multiple hosts sharing via a computer network.[1 This makes it possible for multiple users on multiple machines to share files and storage resources. The client nodes do not have direct access to the underlying block storage but interact over the network using a protocol.This makes it possible to restrict access to the file system depending on access lists or capabilities on both the servers and the clients, depending on how the protocol is designed

DFS@wikipedia (1/2) • In computing, a distributed file system or network file system is any file system that allows access to files from multiple hosts sharing via a computer network. [1] This makes it possible for multiple users on multiple machines to share files and storage resources. • The client nodes do not have direct access to the underlying block storage but interact over the network using a protocol. This makes it possible to restrict access to the file system depending on access lists or capabilities on both the servers and the clients, depending on how the protocol is designed

DFS@wikipedia(2/2) In contrast,in a shared disk file system all nodes have equal access to the block storage where the file system is located.On these systems the access control must reside on the client. Distributed file systems may include facilities for transparent replication and fault tolerance.That is,when a limited number of nodes in a file system go offline,the system continues to work without any data loss. The difference between a distributed file system and a distributed data store can be vague,but DFSes are generally geared towards use on local area networks

DFS@wikipedia (2/2) • In contrast, in a shared disk file system all nodes have equal access to the block storage where the file system is located. On these systems the access control must reside on the client. • Distributed file systems may include facilities for transparent replication and fault tolerance. That is, when a limited number of nodes in a file system go offline, the system continues to work without any data loss. • The difference between a distributed file system and a distributed data store can be vague, but DFSes are generally geared towards use on local area networks

Outline File systems overview NFS AFS (Andrew File System) ·Google File System

Outline • File systems overview • NFS & AFS (Andrew File System) • Google File System

File Systems Overview System that permanently stores data Usually layered on top of a lower-level physical storage medium ·Divided into logical units called“files” -Addressable by a filename ("foo.txt") -Usually supports hierarchical nesting (directories)

File Systems Overview • System that permanently stores data • Usually layered on top of a lower-level physical storage medium • Divided into logical units called “files” – Addressable by a filename (“foo.txt”) – Usually supports hierarchical nesting (directories)

File Paths A file path joins file directory names into a relative or absolute address to identify a file -Absolute:/home/aaron/foo.txt -Relative:docs/someFile.doc The shortest absolute path to a file is called its canonical path The set of all canonical paths establishes the namespace for the file system

File Paths • A file path joins file & directory names into a relative or absolute address to identify a file – Absolute: /home/aaron/foo.txt – Relative: docs/someFile.doc • The shortest absolute path to a file is called its canonical path • The set of all canonical paths establishes the namespace for the file system

What Gets Stored User data itself is the bulk of the file system's contents Also includes meta-data on a drive-wide and per-file basis: Drive-wide: Per-file: available space name formatting info owner character set modification date physical layout

What Gets Stored • User data itself is the bulk of the file system's contents • Also includes meta-data on a drive-wide and per-file basis: Drive-wide: available space formatting info character set ... Per-file: name owner modification date physical layout

High-Level Organization ·Files are organized in a"tree”structure made of nested directories ·One directory acts as the“root'" ·links”(symlinks,shortcuts,etc)provide simple means of providing multiple access paths to one file ·Other file systems can be“mounted"and dropped in as sub-hierarchies (other drives, network shares)

High-Level Organization • Files are organized in a “tree” structure made of nested directories • One directory acts as the “root” • “links” (symlinks, shortcuts, etc) provide simple means of providing multiple access paths to one file • Other file systems can be “mounted” and dropped in as sub-hierarchies (other drives, network shares)

Low-Level Organization (1/2) File data and meta-data stored separately File descriptors meta-data stored in inodes Large tree or table at designated location on disk Tells how to look up file contents Meta-data may be replicated to increase system reliability

Low-Level Organization (1/2) • File data and meta-data stored separately • File descriptors + meta-data stored in inodes – Large tree or table at designated location on disk – Tells how to look up file contents • Meta-data may be replicated to increase system reliability

Low-Level Organization(2/2) ·“Standard"”read-write medium is a hard drive (other media:CDROM,tape,... Viewed as a sequential array of blocks Must address ~1 KB chunk at a time ·Tree structure is“flattened"into blocks Overlapping reads/writes/deletes can cause fragmentation:files are often not stored with a linear layout -inodes store all block ids related to file

Low-Level Organization (2/2) • “Standard” read-write medium is a hard drive (other media: CDROM, tape, ...) • Viewed as a sequential array of blocks • Must address ~1 KB chunk at a time • Tree structure is “flattened” into blocks • Overlapping reads/writes/deletes can cause fragmentation: files are often not stored with a linear layout – inodes store all block ids related to file

Fragmentation A B C (free space) A B C A (free space) A (free space) C A (free space) A D C A D (free)

Fragmentation A B C (free space) A B C A (free space) A (free space) C A (free space) A D C A D (free)