TITech Data Compression Library for Large Scale Data

Abstract

Scale of scientific and engineering computing is becoming larger year by year. In these years, transient solution calculations came to be performed frequently in many science and engineering fields, and resultant data storage came to be a serious issue. For example, in the case of storing data from 100 steps of a caliculation which fully uses 1 GB main memory, outputting the memory image as is to the storage would be the most efficient way from the performance point of view. However, the output will occupy 100 GB of space on the storage. Performing ordinary data compression/decompression to the data on the storage afterwards would require vast amount of time while saving not so much storage space. Thus, a library which enables in-process compression of data preserving precision of data required by post processing applications (such as visualization) and retaining a high performance is highly required.
Development of the Data Compression Library for Large Scale Data is an attempt to satisfy such requirements.

The "TITech Data Compression Library for Large Scale Data" has the following characteristics:

• Its architecture handles data transformation and data compression independently. The data transformation supports various types of transformation (including lossy transformation) for various natures of data. The data compression portion can build-in various compression algorithms as required (development of compression algorithms is not in the scope of the "TITech Data Compression Library for Large Scale Data" development).

• It can handle array data of up to three dimentions, and also time series data (data with another dimention for time) of such arrays, for transformation and compression.

• The transformed/compressed archives have a data structure suitable for porting among heterogeneous computer platforms and for efficient decompression processing.

The "TITech Data Compression Library for Large Scale Data" has been being developed for C and Fortran (F90) languages by the workgroup, using SGI (TM) Origin(TM) 2000 / Onyx2(TM) server (OS: IRIX (TM) 6.5) as the target environment. Although portability of the library to other computer platforms is considered in the development, reports from those who performed actual porting should be referred to for porting/testing status.