Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


Data Conversion

Exchange of unformatted (binary) files between JUGENE and JUROPA/HPC-FF

JUGENE is using big endian data encoding while Intel/AMD-based computers are working with little endian encoding for their unformatted I/O. That means that normally files, created on the first type of machine, are not usable without problems on the 2nd type of machine.

The Intel Fortran compiler offers several possibilities to change the endianess:

  1. The compiler option -convert big_endian, which uses big endian encoding for unformatted files. No modifications of the source code are necessary but all I/O is affected.
  2. Several environment variables allow big endian encoding. Also here, no modifications of the source code are necessary. The conversion can be switched on for certain defined Fortran unit numbers or file types with a certain suffix.
  3. The non-standard CONVERT specifier in the OPEN statement (CONVERT='LITTLE_ENDIAN' or 'BIG_ENDIAN') The source needs to be modified, but one can select the OPEN statements which should be affected.

All possibilities are described below and example routines are given. Please, download the routines, unpack the archive with

gunzip Litte_Big_Endian_tar.gz
tar -xvf Litte_Big_Endian_tar
cd Litte_Big_Endian/

and follow the instructions below in order to test the conversion.

  1. Compiler option -convert big_endian:

    All availables integer values are encoded using write or decoded using read statements. That means, integer length 1-, 2-, 4- and 8-byte are usable for unformatted I/O.

    When using REAL or COMPLEX values, only SINGLE and DOUBLE values (4 or 8 bytes long, respectively) are handled in the right way. Values of 16 bytes length are encoded differently on both maschine types, e.g. the maximum exponent for a 16-byte a REAL variable is E+4932 on Intel and E+309 on IBM, respectively.

    So it is not possible to use option -convert big_endian for the exchange of unformatted files containing 16 byte REAL or 32 byte COMPLEX values.

    To test the file exchange please change into the directory

    Converting_via_compiler_options/

    There you find some routines which create a file named Little_Big_Endian_binary.file on the first machine type and read this file on the 2nd type.


    File transfer from JUGENE to Intel/AMD computers (e.g. JUROPA)

    Compile routines on JUGENE:


    xlf90 -o write_to_binary_file.exe declaration_module.f90 write_to_binary_file.f90

    Execute the program:


    ./write_to_binary_file.exe

    A file Little_Big_Endian_binary.file is created. Transfer this file to the INTEL/AMD machine.

    Compile on this system, using the Intel-compiler (ifort):

    ifort -o read_from_binary_file.exe -convert big_endian declaration_module.f90 read_from_binary_file.f90

    Execute the program:

    read_from_binary_file.exe

    The following output should be created:

    integer kind = 1, number expected = 1, number read = 1 O.K.
    integer kind = 2, number expected = 2, number read = 2 O.K.
    integer kind = 4, number expected = 4, number read = 4 O.K.
    integer kind = 8, number expected = 8, number read = 8 O.K.

    real kind = 4, number expected = 4.00, number read = 4.00 O.K.
    real kind = 8, number expected = 8.00, number read = 8.00 O.K.
    real kind = 16, number expected = 16.00, number read =****** ===> Failure <===

    complex kind = 4, number expected = ( 104.00 204.00), number read = ( 104.00 204.00) O.K.
    complex kind = 8, number expected = ( 108.00 208.00), number read = ( 108.00 208.00) O.K.
    complex kind = 16, number expected = ( 116.00 216.00), number read = (**************) ===> Failure <===

    All available types of numbers are checked and you can see that the test for quadruple precision has failed.

    File transfer from Intel/AMD computers (e.g. JUROPA) to JUGENE

    Compile routines on JUROPA:

    ifort -o write_to_binary_file.exe -convert big_endian declaration_module.f90 write_to_binary_file.f90

    Execute the program:

    ./write_to_binary_file.exe

    A file Little_Big_Endian_binary.file is created.
    Transfer this file to JUGENE.

    Compile there, using the IBM compiler (xlf):

    xlf90 -o read_from_binary_file.exe declaration_module.f90 read_from_binary_file.f90

    Execute the program:

    read_from_binary_file.exe

    The following output should be created:

    integer kind = 1, number expected = 1, number read = 1 O.K.
    integer kind = 2, number expected = 2, number read = 2 O.K.
    integer kind = 4, number expected = 4, number read = 4 O.K.
    integer kind = 8, number expected = 8, number read = 8 O.K.

    real kind = 4, number expected = 4.00, number read = 4.00 O.K.
    real kind = 8, number expected = 8.00, number read = 8.00 O.K.
    real kind = 16, number expected = 16.00, number read = 2.38 ===>Failure<===

    complex kind = 4, number expected = ( 104.00 204.00), number read = (104.00 204.00) O.K.
    complex kind = 8, number expected = ( 108.00 208.00), number read = (108.00 208.00) O.K.
    complex kind = 16, number expected = ( 116.00 216.00), number read = (2.73 2.84) ===> Failure <===

    All available types of numbers are checked and you can see that the test for quadruple precision has failed.

  2. Conversion by specification of environment variables

    All available integer values are encoded using write or decoded using read statements. That means, integer length 1-, 2-, 4- and 8-byte are usable for unformatted I/O.

    When using REAL or COMPLEX values only SINGLE and DOUBLE values (4 or 8 bytes long, respectively) are handled in the right way. Values of 16 bytes length are encoded differently on both maschine types, e.g. the maximum exponent for a 16-byte REAL variable is E+4932 on Intel and E+309 on IBM, respectively.

    So it is not possible to use the environment variables described in the following section for the exchange of unformatted files containing 16-byte REAL or 32-byte COMPLEX values.

    Compile the program as usual, using ifort without any conversion options.
    Before execution enter:

    export FORT_CONVERTxx=BIG_ENDIAN
    xx is the number of the Fortran unit, where the data has to be converted.
    You also can enter:

    export FORT_CONVERT_suff=BIG_ENDIAN
    suff is a suffix. All files having the suffix and being connected to the program are encoded/decoded in BIG_ENDIAN.

    Similar can be accomplished using the environment variable F_UFMTENDIAN. More informations are available under Intel Development Tools.

    To test the file exchange please change into the directory

    Converting_via_compiler_options/

    There you will find some routines which create a file named Little_Big_Endian_binary.file on the first machine type and read this file on the 2nd type.


    File transfer from JUGENE to Intel/AMD computers (e.g. JUROPA)

    Compile the routines on JUGENE:

    xlf90 -o write_to_binary_file.exe declaration_module.f90 write_to_binary_file.f90

    Execute the program:

    ./write_to_binary_file.exe

    A file Little_Big_Endian_binary.file is created. Transfer this file to the INTEL/AMD machine

    Compile there, using the Intel compiler (ifort):

    ifort -o read_from_binary_file.exe -convert big_endian declaration_module.f90 read_from_binary_file.f90

    Before execution set the environment variable

    export FORT_CONVERT_file=BIG_ENDIAN

    You can also set

    export FORT_CONVERT1 BIG_ENDIAN

    which means that all binary data written to FORTRAN unit 1 is encoded to BIG_ENDIAN, and data read from FORTRAN unit 1 is decoded from BIG_ENDIAN, respectively.

    Excute the program:

    read_from_binary_file.exe

    The following output should be created:

    integer kind = 1, number expected = 1, number read = 1 O.K.
    integer kind = 2, number expected = 2, number read = 2 O.K.
    integer kind = 4, number expected = 4, number read = 4 O.K.
    integer kind = 8, number expected = 8, number read = 8 O.K.

    real kind = 4, number expected = 4.00, number read = 4.00 O.K.
    real kind = 8, number expected = 8.00, number read = 8.00 O.K.
    real kind = 16, number expected = 16.00, number read =****** ===> Failure <===

    complex kind = 4, number expected = ( 104.00 204.00), number read = ( 104.00 204.00) O.K.
    complex kind = 8, number expected = ( 108.00 208.00), number read = ( 108.00 208.00) O.K.
    complex kind = 16, number expected = ( 116.00 216.00), number read = (**************) ===> Failure <===

    All available types of numbers are checked and you can see that the test for quadruple precision has failed.

    File transfer from Intel/AMD computers (e.g. JUROPA) to JUGENE

    Compile the routines on JUROPA:

    ifort -o write_to_binary_file.exe declaration_module.f90 write_to_binary_file.f90

    Before execution set the environment variable:

    export FORT_CONVERT_file=BIG_ENDIAN

    You also can set

    export FORT_CONVERT1 BIG_ENDIAN

    which means that all binary data written to FORTRAN unit 1 is encoded to BIG_ENDIAN, and data read from FORTRAN unit 1 is decoded from BIG_ENDIAN, respectively.

    Execute the program:

    ./write_to_binary_file.exe

    A file Little_Big_Endian_binary.file is created.

    Transfer this file to JUGENE. Compile there, using the IBM compiler (xlf):

    xlf90 -o read_from_binary_file.exe declaration_module.f90 read_from_binary_file.f90

    Execute the program:

    read_from_binary_file.exe

    The following output should be created:

    integer kind = 1, number expected = 1, number read = 1 O.K.
    integer kind = 2, number expected = 2, number read = 2 O.K.
    integer kind = 4, number expected = 4, number read = 4 O.K.
    integer kind = 8, number expected = 8, number read = 8 O.K.

    real kind = 4, number expected = 4.00, number read = 4.00 O.K.
    real kind = 8, number expected = 8.00, number read = 8.00 O.K.
    real kind = 16, number expected = 16.00, number read = 2.38 ===> Failure <===

    complex kind = 4, number expected = ( 104.00 204.00), number read = ( 104.00 204.00) O.K.
    complex kind = 8, number expected = ( 108.00 208.00), number read = ( 108.00 208.00) O.K.
    complex kind = 16, number expected = ( 116.00 216.00), number read = ( 2.73 2.84) ===> Failure <===

    All available types of numbers are checked and you can see that the test for quadruple precision has failed.

  3. Specifier CONVERT for the OPEN statement

    All available integer values are encoded using write or decoded using read statements according to the specifier of the corresponding OPEN statements. That means, integer length 1-, 2-, 4- and 8-byte are usable for unformatted I/O.

    When using REAL or COMPLEX values only SINGLE and DOUBLE values (4 or 8 bytes long, respectively) are handled in the right way. Values of 16 bytes length are encoded differently on both maschine types, e.g. the maximum exponent for a 16-byte REAL variable is E+4932 on Intel and E+309 on IBM, respectively.

    Like in cases 1) and 2) it is not possible to use the CONVERT specifier for the exchange of unformatted files containing 16-byte REAL or 32-byte COMPLEX values.

    To test the file-exchange please change into the directory

    Converting_via_open_statement/

    Here, you will find some routines which create a file named Little_Big_Endian_binary.file on the first machine type and read this file on the 2nd type.

    Since the CONVERT specifier is non-standard and therefore not known to the xlf compiler, a warning is issued when compiling the routines. However, the specifier is ignored by default and hence, this warning can be ignored. (It can be avoided by using corresponding preprocessor directives.)

    File transfer from JUGENE to Intel/AMD computers (e.g. JUROPA)

    Compile the routines on JUGENE:

    xlf90 -o write_to_binary_file.exe declaration_module.f90 write_to_binary_file.f90

    Execute the program:

    ./write_to_binary_file.exe

    A file Little_Big_Endian_binary.file is created. Transfer this file to the INTEL/AMD machine.

    Compile there, using the Intel compiler (ifort):

    ifort -o read_from_binary_file.exe declaration_module.f90 read_from_binary_file.f90

    Execute the program:

    read_from_binary_file.exe

    The following output should be created:

    integer kind = 1, number expected = 1, number read = 1 O.K.
    integer kind = 2, number expected = 2, number read = 2 O.K.
    integer kind = 4, number expected = 4, number read = 4 O.K.
    integer kind = 8, number expected = 8, number read = 8 O.K.

    real kind = 4, number expected = 4.00, number read = 4.00 O.K.
    real kind = 8, number expected = 8.00, number read = 8.00 O.K.
    real kind = 16, number expected = 16.00, number read =****** ===> Failure <===

    complex kind = 4, number expected = ( 104.00 204.00), number read = ( 104.00 204.00) O.K.
    complex kind = 8, number expected = ( 108.00 208.00), number read = ( 108.00 208.00) O.K.
    complex kind = 16, number expected = ( 116.00 216.00), number read = (**************) ===> Failure <===

    All available types of numbers are checked and you can see that the test for quadruple precision has failed.

    File transfer from Intel/AMD computers (e.g. JUROPA) to JUGENE

    Compile the routines on JUROPA:

    ifort -o write_to_binary_file.exe declaration_module.f90 write_to_binary_file.f90

    Execute the program:

    ./write_to_binary_file.exe

    A file Little_Big_Endian_binary.file is created. Transfer this file to JUGENE. Compile there, using the IBM compiler (xlf):

    xlf90 -o read_from_binary_file.exe declaration_module.f90 read_from_binary_file.f90

    Execute the program:

    read_from_binary_file.exe

    The following output should be created:

    integer kind = 1, number expected = 1, number read = 1 O.K.
    integer kind = 2, number expected = 2, number read = 2 O.K.
    integer kind = 4, number expected = 4, number read = 4 O.K.
    integer kind = 8, number expected = 8, number read = 8 O.K.

    real kind = 4, number expected = 4.00, number read = 4.00 O.K.
    real kind = 8, number expected = 8.00, number read = 8.00 O.K.
    real kind = 16, number expected = 16.00, number read = 2.38 ===> Failure <===

    complex kind = 4, number expected = ( 104.00 204.00), number read = ( 104.00 204.00) O.K.
    complex kind = 8, number expected = ( 108.00 208.00), number read = ( 108.00 208.00) O.K.
    complex kind = 16, number expected = ( 116.00 216.00), number read = ( 2.73 2.84) ===> Failure <===

    All available types of numbers are checked and you can see that the test for quadruple precision has failed.


Servicemeu

Homepage