Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


SIONlib: Scalable I/O library for parallel access to task-local files

Templates for parallel I/O with SIONlib
Sample parallel MPI program for I/O with SIONlib
Serial program for SIONlib file access
Other examples

Templates for parallel I/O with sionlib

All templates can also be found in the distribution package under the PATH:
sionlib/examples

Parallel write:
sid=sion_paropen_mpi( ... ,chunksize, comm, &fileptr, ...)  # collective

loop: {
sion_ensure_free_space(sid,nbytes); # non-collective
fwrite(data,1,nbytes,fileptr)
}
sion_parclose_mpi(sid); # collective

- the call to sion_ensure_free_space can be omitted, if it is guaranteed that not more bytes as specified in chunksize are written to the file.
- chunksize can be different on different tasks
- if the data does not fit in the current chunk, sion_ensure_free_space would assign a new chunk in the file to the task and also advance the file pointer to the new position. This operation is non-collective; all information about the locally used chunks and the number of bytes written to the chunks are buffered in memory and stored in the sion file in the collective sion_parclose_mpi.
- one parameter of the open function is an MPI communicator, which allows opening a sion file from a subset of all MPI tasks


Parallel read:
sid=sion_paropen_mpi( ... ,&chunksize, comm, &fileptr, ...) # collective

while((!sion_feof(sid))) { # non-collective
btoread=sion_bytes_avail_in_block(sid); # non-collective
bread=fread(localbuffer,1,btoread,fileptr);
}
sion_parclose_mpi(sid); # collective

- it must be ensured that the number of bytes read from a chunk is not larger as the number of bytes written to it. For this the function sion_bytes_avail_in_block provides this number for the current chunk.

- if all bytes of a chunk are already read and there are more chunks available for this task, sion_feof will advance the filepointer to the start position of the next chunk in the sion file

- sion_paropen_mpi is collective. The meta data will read only by one task and broadcasted to all other tasks. It is also possible to open/close the sion file on each with serial sion functions without collective operations, see function sion_open_rank.


Parallel read without collective operation:
sid=sion_open_rank( ... ,&chunksize, rank, &fileptr, ...)   # non-collective

while((!sion_feof(sid))) { # non-collective
btoread=sion_bytes_avail_in_block(sid); # non-collective
bread=fread(localbuffer,1,btoread,fileptr);
}
sion_close(sid); # non-collective

- the meta-information of the sion file will be read from each task. This is a parallel access to same filesystem blocks, however this are only read accesses which should not be lock the filesystem block.

top of page

Sample parallel MPI program for I/O with SIONlib

This simple parallel program can be found under the PATH:
sionlib/examples/simple
This directory contain also Makefiles to build the program.
...
#include "sion.h"
#define FNAMELEN 255
#define BUFSIZE (1024*1024)

int main(int argc, char **argv)
{

int rank, size, globalrank, sid, i, numFiles;
char fname[FNAMELEN], *newfname=NULL;
int ;
MPI_Comm gComm, lComm;
sion_int64 chunksize,left;
sion_int32 fsblksize;
size_t btoread, bread, bwrote;
char *localbuffer;
FILE *fileptr;

MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

/* allocate and initalize a buffer */
localbuffer = (char *) malloc(BUFSIZE);
srand(time(NULL));
for (i = 0; i < BUFSIZE; i++) localbuffer[i] = (char) rand() % 256;

/* inital parameters */
strcpy(fname, "parfile.sion");
numFiles = 1;
gComm=lComm= MPI_COMM_WORLD;
chunksize = 10*1024*1024;
fsblksize = 1*1024*1024;
globalrank = rank;

/* write */
sid = sion_paropen_mpi(fname, "bw", &numFiles, gComm, &lComm,
&chunksize, &fsblksize, &globalrank, &fileptr, &newfname)
;
left=BUFSIZE;
while (left > 0) {
sion_ensure_free_space(sid, left);
bwrote = fwrite(localbuffer, 1, left, fileptr);
left -= bwrote;
}
sion_parclose_mpi(sid);

printf("Task %02d: wrote sionfile -> %s\n",rank,newfname);


/* read */
sid=sion_paropen_mpi(fname,"br",&numFiles,MPI_COMM_WORLD,&lComm,
&chunksize,&fsblksize, &globalrank, &fileptr, &newfname);

while((!sion_feof(sid))) {
btoread=sion_bytes_avail_in_block(sid);
bread=fread(localbuffer,1,btoread,fileptr);
}
sion_parclose_mpi(sid);

printf("Task %02d: read sionfile -> %s\n",rank,newfname);

MPI_Finalize();

return (0);
}
top of page

Serial program for SIONlib file access

Templates for accessing SION-files from a serial program are shown below. The templates can also be found in the distribution package under the PATH: sionlib/examples


Serial write:
sid=sion_open( ...,chunksize, &fileptr)

rank_loop: {
sion_seek(sid,rank,SION_CURRENT_BLK,SION_CURRENT_POS);

sion_ensure_free_space(sid,nbytes);
fwrite(...,fileptr)
}
sion_close(id);

Serial read:
sid=sion_open( ...,chunksize, &fileptr)

sion_get_locations(sid,&size,&blocks,&globalskip,&start_of_varheader,
&sion_localsizes,&sion_globalranks,
&sion_chunkcount,&sion_chunksizes);

loop: {
sion_seek(sid,rank,blknr,pos);

fread(...,fileptr)
}
sion_close(id);

- access to all ranks and chunks is possible (sion_seek)
- sion_get_locations returns pointers to internal fields, containing the number of chunks written by each task (sion_chunkcount) and their sizes (sion_chunksizes)

top of page

Other examples

The example directory also contains in sub-directory
./examples/pepc
an example application dependant converter program for the parallel simulation application PEPC (Multi-Purpose Parallel Tree-Code,
PEPC). This serial program converts data files between the native PEPC ASCII format and a binary format using sionfiles and can be used to build converter programs for own applications.

top of page


Servicemeu

Homepage