Scalable I/O library for parallel access to task-local files
SIONlib is a scalable I/O library for the parallel access to task-local files. The library not only supports writing and reading binary data to or from from several thousands of processors into a single or a small number of physical files but also provides for global open and close functions to access SIONlib file in parallel. SIONlib provides different interfaces: parallel access using MPI, OpenMp, or their combination and sequential access for post-processing utilities.
Besides collective open and close functions, a distinctive feature of SIONlib is that all writing and reading is done asynchronously using, for instance, standard C file I/O operations (POSIX). Due to this simplified file handling, I/O can be done with the standard C-I/O functionality (fwrite, fread) without changing the existing program. As an additional file management advantage, only one large file containing all the data needs to be maintained - instead of thousands of separate files.
In order function effectively, SIONlib requires information on the BLOCKSIZE of the underlying filesystem, i.e., the BLOCKSIZE is the smallest part of a file which can be locked. For instance, IBM GPFS has blocksizes of one ore two Megabytes. Given that each processor writes its data to same file, SIONlib assigns different regions (e.g., data blocks) of the file to a processor. Hence, for each processor, SIONlib needs the estimated data size which would be written to the file (chunk size). SIONlib can also handle extensions (additional chunks), if the end of a data block is reached. Both, the estimated chunk size and filesystem's blocksize are used to calculate the individual start position in the file for each processor. While a certain block of the file is locked only by one processor, efficient parallel writing and reading is feasible.
If you are interested in upcoming SIONlib versions, we recommend to add your mail address to the SIONlib-news mailing list.