Thursday, April 30, 2015

Sparse Files

I am still a great fan and proponent of the Syncback SE or Pro backup program sold by 2BrightSparks.

I commented on and recommended Syncback here

I noticed in their latest email that it supported Sparse Files. More advanced OS allows via an API call to create a seemingly large file (reported as large by directory listing commands) but with a bytes used footprint equal to only the data stored.

Example,   I create a 1GB sparse file, it shows up as 1GB in size but the 1MB of data stored inside it means the file is still actually only taking up 1MB of actual space on the disk.

I wondered, hmm .... which OS that I use actually have this facility built in.

Wikipedia Sparse files

Wikipedia File Systems

The basic requirements are that 

a) The installed filesystem that your files are stored on must support Sparse

b) The OS too needs to support sparse

So I did some digging as to Sparse support on the OS that I use daily:

Windows NTFS

Well yes of course.

GetVolumeInformation() - Determine if fs supports sparse
Code example sparse in C#
Command line create sparse with fsutil
NTFS Sparse files notes programmers


At a file level I ssh'd command line into my Android 5.1.1 phone and created a sparse file without issue.

also: Sparse file image format Android


On MacOS, the default filesystem is HFS+ which does not support sparse files. You can optionally format a volume with UFS which does support sparse files. HFS+ is the default filesystem on MacOS.

If you use UFS  (Unix File System) under OSX i.e. the one that nobody uses (!) then there is Sparse File support.

Slightly less flexibily but more usefully to end users is the Sparse image format for disk images  (.dmg)  This enables a single file to contain an image of a hard disk with greater efficiency and is used by various backup software including Apple Time machine.


My production Linux systems run Centos 64bit Linux and use the default XFS filesystem which supports sparse files.  The above image shows that whilst the filesystem thinks the file is 4G big the space used is zero.

Copy regular file to a sparse file
Various UNIX Sparse notes


I'm not hands on daily here just daily reading  (today it was 
IBM z13 Configuration Setup) so over to Google:
I did not find any mention of /sparse files/ under native zOS for IBM mainframes.   I do find mentions in the UNIX utilities such as System Z Personal Development Tool but these just relate to Linux Sparse files.

To summarise: Windows and Linux (including Android) support Sparse files as a matter of course on their normal/usual filesystems.  Whilst Apple's OSX supports sparse files at an OS level the default Apple HFS filesystem used unless specifically overidden by a user does not.


NTFS sparse files
Windows Sparse files by 2brightsparks
Apple HFS+ File System