Side note: As far as I know, this is a poorly documented topic given its importance as a security issue.
Too Long; Don't Read: A process's virtual address space is fully isolated from another. The Linux kernel has access to the whole memory as it runs in kernel mode. It provides system calls that allow a process, under certain circumstances (see Ptrace access mode checking below), to access the memory of another.
There are system calls in the Linux kernel that allow reading/writing memory of other process:
process_vm_readv() and process_vm_writev() (same manual page)
These system calls transfer data between the address space of the calling process ("the local process") and the process identified by pid ("the remote process"). The data moves directly between the address spaces of the two processes, without passing through kernel space.
The last sentence refers to what happens in kernel mode (the kernel actually copies between two physical addresses). The user mode cannot access other virtual address space. For technical details, take a look at the implementation patch.
Regarding the permissions needed:
Permission to read from or write to another process is governed by a ptrace access mode PTRACE_MODE_ATTACH_REALCREDS check; see ptrace().
ptrace()
The ptrace() system call provides a means by which one process (the "tracer") may observe and control the execution of another process (the "tracee"), and examine and change the tracee's memory and registers.
Regarding the permissions needed, according to ptrace() manual page:
Ptrace access mode checking
Various parts of the kernel-user-space API (not just ptrace() operations), require so-called "ptrace access mode" checks, whose outcome determines whether an operation is permitted (or, in a few cases, causes a "read" operation to return sanitized data). These checks are performed in cases where one process can inspect sensitive information about, or in some cases modify the state of, another process. The checks are based on factors such as the credentials and capabilities of the two processes, whether or not the "target" process is dumpable, and the results of checks performed by any enabled Linux Security Module (LSM)—for example, SELinux, Yama, or Smack—and by the commoncap LSM (which is always invoked).
Related stuff: