Opened 8 years ago

#44 new enhancement

Analysis should consider automatically compressing SI VIFF files

Reported by: flip Owned by:
Priority: minor Milestone:
Component: general/unknown Version:
Keywords: Cc: bsoher

Description

Vespa's export (save to VIFF) supports zlib compression by simply passing a flag to the export function. In RFPulse and Simulation, users can turn this flag on via the export dialog. (It's off by default.)

Analysis doesn't have an export dialog, so users never get a chance to change that setting.

When I saved 4007_smh_spat.rsd as a VIFF file, it ballooned to ~39.9Mb (compared to the 16Mb .rsd file).

The source of VIFF's obesity is that it stores the data multiple times whereas the .rsd file stores only the raw data. For this 64x64x512 dataset, storing it multiple times really adds up. In the XML, the elements block_raw, block_spatial, and block_spectral each contain a 64x64x512 numpy array.

Since the .rsd file requires 16Mb to represent its array, IMO it's a triumph that VIFF stores three copies of it in 2.5x the space.

However, I doubt our users will be singing hosannas about using "only" 2.5x the disk space, and I can't blame them.

GZipping the file (more or less equivalent to applying Vespa's glib compression) reduces the file to 30.2Mb -- still big, but not nearly as bad.

Here's those file sizes summarized:
.rsd file - 16Mb
Compressed VIFF - 30.2Mb (1.88x bigger than .rsd)
Uncompressed VIFF - 39.9Mb (2.5x bigger than .rsd)

Change History (0)

Note: See TracTickets for help on using tickets.