Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


SIONlib: Scalable I/O library for parallel access to task-local files

Version (1.5.3) available (see also Release notes)

Welcome to SIONlib

SIONlib is a scalable I/O library for the parallel access to task-local files. The library not only supports writing and reading binary data to or from from several thousands of processors into a single or a small number of physical files but also provides for global open and close functions to access SIONlib file in parallel. SIONlib provides different interfaces: parallel access using MPI, OpenMp, or their combination and sequential access for post-processing utilities.

SIONlib is a joint project of:
Forschungszentrum Jülich, Jülich Supercomputing Centre
German Research School for Simulation Sciences
Laboratory for Parallel Programming

Besides collective open and close functions, a distinctive feature of SIONlib is that all writing and reading is done asynchronously using, for instance, standard C file I/O operations (POSIX). Due to this simplified file handling, I/O can be done with the standard C-I/O functionality (fwrite, fread) without changing the existing program. As an additional file management advantage, only one large file containing all the data needs to be maintained - instead of thousands of separate files.

In order function effectively, SIONlib requires information on the BLOCKSIZE of the underlying filesystem, i.e., the BLOCKSIZE is the smallest part of a file which can be locked. For instance, IBM GPFS has blocksizes of one ore two Megabytes. Given that each processor writes its data to same file, SIONlib assigns different regions (e.g., data blocks) of the file to a processor. Hence, for each processor, SIONlib needs the estimated data size which would be written to the file (chunk size). SIONlib can also handle extensions (additional chunks), if the end of a data block is reached. Both, the estimated chunk size and filesystem's blocksize are used to calculate the individual start position in the file for each processor. While a certain block of the file is locked only by one processor, efficient parallel writing and reading is feasible.

To send us comments, questions, or bug reports, please send us an email.

Further informations:
Sion API
Sion File Format
Examples
Utilities
Further Documentation and References
Download
Installation, Debugging, Error Messages
Release Notes & Copyright

Contact: sionlib_jsc@fz-juelich.de
WWW: http://www.fz-juelich.de/jsc/sionlib


Servicemeu