Distributed Memory

Shared Memory

 Shared memory parallel computers vary widely, but generally have in common the ability for all processors to access all memory as global address space

• Multiple processors can operate independently but share the same memory resources.

• Changes in a memory location effected by one processor are visible to all other processors.

• Shared memory machines can be divided into two main classes based upon memory access times: UMA and NUMA

Uniform Memory Access (UMA):

• Most commonly represented today by Symmetric Multiprocessor (SMP) machines
• Identical processors
• Equal access and access times to memory
• Sometimes called CC-UMA - Cache Coherent UMA. Cache coherent means if one processor updates a location in shared memory, all the other processors know about the update. Cache coherency is accomplished at the hardware level.

 Non-Uniform Memory Access (NUMA):

• Often made by physically linking two or more SMPs
• One SMP can directly access memory of another SMP
• Not all processors have equal access time to all memories
• Memory access across link is slower
• If cache coherency is maintained, then may also be called CC-NUMA - Cache Coherent NUMA

Distributed Memory

 Like shared memory systems, distributed memory systems vary widely but share a common characteristic. Distributed memory systems require a communication network to connect inter-processor memory.

• Shared memory
– Means a single address space, so equal to shared address space
– Opposite: multiple address space memory
• Distributed memory
– Physical memory is divided into modules with some placed near each processor
– Opposite: centralized memory

and another statement from the internet regarding the programming model:

Parallel Programming Models 

There are several parallel programming models in common use:

• Shared Memory
• Threads
• Message Passing
• Data Parallel
• Hybrid

• Parallel programming models exist as an abstraction above hardware and memory architectures.

• Although it might not seem apparent, these models are NOT specific to a particular type of machine or memory architecture. In fact, any of these models can (theoretically) be implemented on any underlying hardware. Two examples:

  1. Shared memory model on a distributed memory machine: Kendall Square Research (KSR) ALLCACHE approach.

     Machine memory was physically distributed, but appeared to the user as a single shared memory (global address space). Generically, this approach is referred to as "virtual shared memory". Note: although KSR is no longer in business, there is no reason to suggest that a similar implementation will not be made available by another vendor in the future.

  2. Message passing model on a shared memory machine: MPI on SGI Origin.

     The SGI Origin employed the CC-NUMA type of shared memory architecture, where every task has direct access to global memory. However, the ability to send and receive messages with MPI, as is commonly done over a network of distributed memory machines, is not only implemented but is very commonly used.

• Which model to use is often a combination of what is available and personal choice. There is no "best" model, although there certainly are better implementations of some models over others.

 My undertanding till now:

From the perspective of memory parallelism, we have shared memory and distributed memory, here shared memory is equivalent to shared address space, while distributed memory is the same as distributed address space. we can catorize share memory into UMA and NUMA. Moreover, shared memory and disitributed memory are not mutually opposite: shared memory's opposite is mutiple address space memory, while the opposite of distributed memory is centralized memory. (A furture work about the concepts of these 4 "memory" needed.) 

On the other hand, ther is also a term of "shared memory " from the perspective of programming model (distributed memory programming model has the name of "message passing"). shared memory programming model is for the hardware architecture with "shared memory (shared addresss space)", while mesage passing is for "distributed memory (distributed address space)", but it's always true: we can adpot shared memory model to distributed memory machinse, or apply message passing model to shared memory machines.

reference:

1. http://www.llnl.gov/computing/tutorials/parallel_comp/

2. Slides of CS546 (Dr. Lan)

3. Wiki pidia