aio_reap(2)

NAME

aio_reap() — wait for multiple asynchronous I/O requests

SYNOPSIS

#include <sys/lock.h>

typedef struct aio_completion_data {
struct aiocb *aio_aiocb;
int aio_return;
int aio_error;
} aio_completion_t;

int aio_reap(aio_completion_t list[], int nent, const struct timespec *timeout, int waitfor, int *completed_count);

DESCRIPTION

The aio_reap() function is an extension of the aio_suspend() library function. Refer to the aio_suspend() manpage for a general description of the interface, the aiocb data structure and other structures relevant for asynchronous I/O (AIO). The timeout function parameter has the same meaning as for the aio_suspend() function.

This function suspends the calling thread until at least waitfor AIO operations scheduled earlier by the process have completed, until a signal interrupts the function, or, if timeout is not NULL, until the time interval specified by timeout has passed.

In addition, this function supports a light-weight polling mode. (See below.)

To use aio_reap(), the reap mechanism must be initiated by making a light-weight polling call to aio_reap(). This call must be made before calling any other AIO functions. (See Usage Notes below) After this call, the use of aio_error(), aio_return(), and aio_suspend() becomes undefined. In future releases, aio_cancel() and aio_fsync() may also become undefined. The aio_reap() mechanism is meant to be as efficient as possible, and is not meant for mixing with these other more costly features.

Each aiocb request structure to be returned by this function must have been scheduled earlier by aio_write(), aio_read(), or lio_listio() functions.

None of the AIO requests to be reaped may use the lio_listio() asynchronous notification mechanism, or the LIO_WAIT flag. If any of the aiocb structures in the list array were submitted with asynchronous notification or LIO_WAIT, the result is undefined.

The waitfor parameter specifies the number of AIO requests that must be completed before the function returns to the calling process. The value must be greater than 0 and less than or equal to nent and MAX_AIO_REAP, which is defined in aio.h. MAX_AIO_REAP defines the maximum number of AIO requests that the system is capable of reaping in one invocation of aio_reap().

The list argument is an array of aio_completion_t structures, and each element of the array must be zeroed out before calling aio_reap(). There must be at least nent number of elements in this array. For each IO collected by aio_reap(), the fields of an aio_completion_t in list will be filled in as follows:

aio_aiocb: pointer to the aiocb structure that issued this I/O
aio_return: analogous to value obtained from aio_return()
aio_error: analogous to value obtained from aio_error()

(Note that the position of items in this list does not necessarily reflect the order of the members in the structure.)

The number of AIO requests reaped (and, analogously, the number of aio_completion_t structures filled in) is returned into the address specified by completed_count.

Use of the aio_reap() interface results in complete processing of each IO, eliminating the need to call aio_return() or aio_error() for each IO.

The use of aio_reap() in an application should not be mixed with the older aio_return() or aio_error() functions. Mixing of the two methods of IO collection will result in undefined behavior and possible data loss.

The timeout parameter must be either NULL or point to a timespec structure that has a timeout time specified. If a NULL pointer is specified, then no timeout handling is performed and aio_reap() blocks until waitfor AIO requests are available. If timeout points to a zero-value timespec structure, aio_reap() returns immediately after checking for any completed AIO requests (behaves similar to poll()).

The aio_reap() function call returns to the calling process in any of the following conditions:

1.: the specified number of completed requests are available, or
2.: the timeout time has elapsed, or
3.: a signal has interrupted the function, or
4.: an error situation is detected.

In all cases, aio_reap() will attempt to collect as many IOs as possible at the time of its return, even if less than waitfor IOs have completed when a timeout, signal, or error interrupts aio_reap(). If more than waitfor IOs have completed, aio_reap() will attempt to return up to nent IOs. As a result, the value of completed_count should be checked after all calls to aio_reap() (regardless of its return value or errno status).

Light-Weight Polling Mode

When called with all NULL arguments (except for completed_count), aio_reap will simply return in completed_count the number of IOs that have completed and are ready for collection.

In light-weight polling mode, aio_reap() is implemented to execute quickly, without requiring the cost of a regular system call. This allows applications to poll for completed IOs without incurring significant costs.

(Note: all releases will provide this polling functionality, but the actual mechanism by which lightweight polling is implemented is release specific. However, it should in all cases be faster than a regular system call.)

Multithreading Notes

The aio_reap() interface is completely thread safe, but there are three issues that multithreaded applications should be aware of:

The first is starvation. When multiple threads concurrently perform aio_reap() calls, there is no guarantee that completed I/Os will be distributed fairly across the calls. For example, if 50 I/Os are issued and two threads call aio_reap() with waitfor=25 and nent=50, there is no guarantee that both threads will collect any particular number of I/Os. In fact, one possible outcome is that the first thread's call to aio_reap() will collect all 50 I/Os, and the second thread will have none to collect. Multithreaded applications should take this possibility into consideration, and make appropriate use of timeout values to prevent unbounded waits in starvation situations.

Another related issue is that when threads perform aio_reap() calls, they can collect I/Os issued by any thread in the process (not just the calling thread). As an example, if thread A and thread B each issue 10 I/Os, and then each calls aio_reap() with waitfor=10 and nent=10, no assumptions can be made about which I/Os each call to aio_reap() will collect. Thread A's call to aio_reap() might collect all 10 of thread A's I/Os, all 10 of thread B's I/Os, or any mixture of the two. The only guaranteed behavior is that the first call to aio_reap() will collect the first 10 I/Os to complete.

The third issue is that it is possible for multiple threads to disrupt execution of aio_reap() if they corrupt the arguments to aio_reap() during its execution. For example, problems can arise if one thread of a multithreaded application invalidates the completed_count pointer used by another thread concurrently executing a call to aio_reap() (e.g. by freeing or mlocking the corresponding memory). If this happens, it is possible for aio_reap() to complete I/Os and fill in aio_completion_t structures but then be unable to update completed_count. In this case aio_reap() will return with errno set to EFAULT, but will have no way to let the application know how many IOs it has completed. The completed I/Os will be stored in list, but the application will not be able to know how many I/Os have completed, so it is possible to lose I/Os in this case. Multithreaded applications must avoid this situation.

Usage Notes

To use aio_reap(), link in the realtime library by specifying -lrt on the compiler or linker command line. Then, before making any AIO function calls, initialize the reap mechanism by calling aio_reap() in lightweight mode, as shown in this example:

     int ignore; 

     (void)aio_reap(NULL, 0, NULL, 0, &ignore); 

For this single call to initialize aio_reap(), all output (including return value) should be ignored.

RETURN VALUE

aio_reap() returns 0 if waitfor or more AIO requests have completed. The reaped requests (the aiocb addresses of the completed AIO requests) are copied to the array list, and the number of AIO requests reaped is returned into the address specified by completed_count.

If the function was interrupted by a signal, the function returns -1 and errno is set to EINTR. If the function was interrupted because the timeout time has elapsed, the function returns -1 and errno is set to EAGAIN. In either case, actual number of requests completed at the time of the interrupt is returned in completed_count, and these requests are copied to list -- even if the number of AIO requests reaped is less than waitfor.

If the total number of outstanding AIO requests for that process is less than waitfor, aio_reap() will return -1 and set errno to E2BIG after all remaining requests have been reaped. The only exception is if timeout time has elapsed, in which case the preceding rule applies.

If an error is discovered, the return value is -1. errno contains additional information.

In light-weight polling mode, aio_reap returns 0 and sets completed_count to the number of IOs that are ready for collection. If an irrecoverable error occurs, aio_reap will return -1 and leave completed_count unchanged.

ERRORS

If aio_reap() returns -1, errno contains one of the following errors:

EINVAL: waitfor is less than or equal to 0, greater than nent, or greater than the configured maximum possible number of AIO requests the system can reap in one call MAX_AIO_REAP).
EINVAL: timeout time is not a valid timespec.
EINVAL: nent is less than waitfor or greater than the configured maximum possible number of AIO requests the system can reap in one call MAX_AIO_REAP).
ENOMEM: No free memory available.
EFAULT: Error copying data from or to the user address space.
EAGAIN: The timeout time has elapsed and less than waitfor AIO requests are available.
EINTR: A signal interrupted the aio_reap() function and less than waitfor AIO requests are available.
E2BIG: The system had fewer than waitfor outstanding AIO requests available.
ENOTSUP: aio_reap() has not been properly initialized. That is, the first AIO function called by this process was not aio_reap(), so aio_reap() is not available for use.

APPLICATION USAGE

Note that if aio_reap() returns -1, the caller should check the value of errno and completed_count to determine whether any AIO requests have been reaped.

aio_reap() is designed for high performance handling of a large number of outstanding AIO requests. It increases the throughput of AIO by reducing the number of calls to reap AIO requests, and by not hanging up on individual AIO requests that may be taking a long time to complete.

AUTHOR

aio_reap() was developed by HP.