Doc: MPI Sync?

[ax3l]

Open brain storming if we want to document which calls are collective and synchronizing.

Wording loosely after the MPI standard, section 2.4:
https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf

[franzpoeschel]

On collective operations:

A procedure is collective if all processes in a process group need to invoke the
procedure. A collective call may or may not be synchronizing. Collective calls over
the same communicator must be executed in the same order by all members of the
process group.

On the distinction between collective and synchronizing:

Collective operations can (but are not required to) complete as soon as the caller’s
participation in the collective communication is finished. A blocking operation is complete
as soon as the call returns. A nonblocking (immediate) call requires a separate completion
call (cf. Section 3.7). The completion of a collective operation indicates that the caller is free
to modify locations in the communication buffer. It does not indicate that other processes
in the group have completed or even started the operation (unless otherwise implied by the
description of the operation). Thus, a collective communication operation may, or may not,
have the effect of synchronizing all calling processes. This statement excludes, of course,
the barrier operation.

Our usage diverges from both terms:

• collective: Those operations which we declare as collective, but not synchronizing, may be executed in divergent orders on different MPI ranks. The above definition requires that the orders should be the same.
  (This is somewhat fine, we just document stricter requirements than needed)
• synchronizing is a guarantee that the operation implements MPI_Barrier() semantics. However, when series.flush() returns, we do not actually guarantee to our users that all ranks have reached the flush operation. We should avoid this term.

Suggestion:

• Use the term collective for operations that must be executed by all MPI ranks in the same order. This is in agreement with the MPI terminology, and with it we do not make any guarantee that we cannot hold.
  This PR currently calls these operations synchronizing.
• MPI does not have a notion for "operation that is enqueued now and flushed later". These are similar to nonblocking operations, but not the same: Nonblocking operations are communication operations. What we want to document for operations such as setAttribute() are requirements about coherent data declaration.
  I suggest to use a term outside MPI terminology for that: coherent operations.

[franzpoeschel]

Suggested change

	A **synchronizing** operation will synchronize the MPI ranks that participate in it.
	A **synchronizing** operation will synchronize the MPI ranks that participate in it. All synchronizing operations are collective.

EDIT: I think we should not use this term at all, see here.

[ax3l]

Wouldn't "non-blocking collective" be the right description for our .flush()? (like MPI_IBarrier)

I think we just do not have as MPI has a "corresponding completion operation" (test/wait). (We kind of would need one generally, e.g., to ensure a checkpoint is done. So far I have closed, in doubt, to be sure. Of course, there is always the additional challenge that a consistent MPI view does not correspond to a consistent PFS view.)

[ax3l]

@franzpoeschel can you double check the details on availableChunks and potentially fix this doc line in a separate PR?

[ax3l]

coherent data declaration

Or maybe even simpler: consistent data declaration?