disk compression?

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

andrea catto' - 30 Apr 2004 21:17 GMT

guys, I want to understand how the transparent disk compression works,
I already know it works as a compression layer between the OS and the
cluster/sector access.

but I am confused,

even if a cluster or chunk or block can be shrunk into less than its
original size, there will be unused parts and the end in such a unit.

the compression would be effective only if those unused units are reutilized
somehow, but how ?
I don't get it.

please help me out

Reply to this Message

philo - 30 Apr 2004 21:59 GMT

> guys, I want to understand how the transparent disk compression works,
> I already know it works as a compression layer between the OS and the
[quoted text clipped - 10 lines]
>
> please help me out

see this

http://storagereview.com/guide2000/ref/hdd/file/comprWhy.html

Reply to this Message

andrea catto' - 30 Apr 2004 22:35 GMT

thank you philo, but after reading that it still leaves me without answers
it talk about the high level and sectors and stuff, but my question
is.......
even if 512bytes sectors shrinks to 128.... how's the rest of the sector
going to be utilized ???
and if it is not, what's the benefit ?

> > guys, I want to understand how the transparent disk compression works,
> > I already know it works as a compression layer between the OS and the
[quoted text clipped - 15 lines]
>
> http://storagereview.com/guide2000/ref/hdd/file/comprWhy.html

Reply to this Message

philo - 30 Apr 2004 23:07 GMT

> thank you philo, but after reading that it still leaves me without answers
> it talk about the high level and sectors and stuff, but my question
> is.......
> even if 512bytes sectors shrinks to 128.... how's the rest of the sector
> going to be utilized ???
> and if it is not, what's the benefit ?

well i am hardly an expert ...but it's my understanding that compression
does not change the cluster size...it merely allows for more data to be
stored
within each cluster.

also note: with dos , win3x/win9x drive compression is not really a good
idea
becasue with a disk error, caused by a bad shutdown etc, you could end up
loosing all the data on your drive.
with a compressed volume, all your files are really compressed into
one large file.

btw: NT based operating systems on NTFS partitions have a safer form of
compression, which does not rely on a compressed volume

go look around on google...there's alot of info out there

Reply to this Message

andrea catto' - 01 May 2004 00:06 GMT

thank you for the answer
the fact is that ntfs is safer because it's journaled, so regardless of the
compression, there is a journal of transactions that either get validated or
not.
so it's almost impossible to risk anything.

I also have tried to look on google..
all I found is an example of how ext2 e2compr functions, which has some sort
of lousy explanation at all.

I should try and download the source code since it's linux and understand it

> > thank you philo, but after reading that it still leaves me without answers
> > it talk about the high level and sectors and stuff, but my question
[quoted text clipped - 19 lines]
>
> go look around on google...there's alot of info out there

Reply to this Message

Steve Baron - KB3MM - 01 May 2004 00:21 GMT

HMMM! Isn't the size of a cluster fixed i.e. the number of bytes in the
cluster is invariant and purely a funtion of the number of sectos in the
cluster and the ize of the sector?

Reply to this Message

AlmostBob - 01 May 2004 00:46 GMT

Yes disk compression compresses in much the same way as .zip or .rar
compresses files so can fit more, it doesnt change the cluster size or sector
size it acts as a service layer between the OS and disk controller

Signature

Adaware http://www.lavasoft.de
spybot http://security.kolla.de
AVG http://www.grisoft.com
Panda online scan http://www.pandasoftware.com/ActiveScan/
Catalog of removal tools http://www.pandasoftware.com/download/utilities/
Blocking Unwanted Parasites with a Hosts file
http://mvps.org/winhelp2002/hosts.htm
links provided as a courtesy,
Grateful thanks to the authors/webmasters

| HMMM! Isn't the size of a cluster fixed i.e. the number of bytes in the
| cluster is invariant and purely a funtion of the number of sectos in the
[quoted text clipped - 24 lines]
| >
| > go look around on google...there's alot of info out there

Reply to this Message

andrea catto' - 01 May 2004 01:06 GMT

but the thing I don't get is.......

if a cluster is designed to be 4 sectors = 2048 bytes, and you can get those
2048 bytes compressed into for example only 1 sector....

the 1st will be written only, ok so far, but what's happening to the other 3
????
they will stay unutilized ???
if so there is no benefit.
how do those 3 unused sectors get used ?

> Yes disk compression compresses in much the same way as .zip or .rar
> compresses files so can fit more, it doesnt change the cluster size or sector
[quoted text clipped - 28 lines]
> | >
> | > go look around on google...there's alot of info out there

Reply to this Message

AlmostBob - 01 May 2004 01:37 GMT

At the disk, only the compressed file is written so only the number of
clusters required to write the compressed data is used, like winzip or winrar
do with files eg a 5Kb text file may be compressed by "your favorite
compression disk program" to 2kb so occupies a single 2kb cluster on the disk.
IIRC the entire compressed drive is stored as a single cabinet file on the
Host drive with a little overhead to maintain pointers to the beginning of
each file within the cabinet not as a larger number of small single compressed
files, allowing files compressed to smaller than 1 cluster to be placed
consecutively without occupying a full physical drive cluster.
info is here http://www.pcguide.com/ref/hdd/file/compr.htm

Signature

| but the thing I don't get is.......
|
[quoted text clipped - 46 lines]
| > | >
| > | > go look around on google...there's alot of info out there

Reply to this Message

Jeff Richards - 01 May 2004 01:53 GMT

If you are talking about Drivespace or similar, there is only one file at
the file system level, so cluster size becomes almost irrelevant (maximum
slack is one cluster). The Drivespace manager organises storage within the
compressed volume according to its own rules, which does not use the disk
cluster size. It appears that file storage within the compressed volume
operates at the disc sector level, so maximum slack is 512b.
http://www.pcguide.com/ref/hdd/file/comprSlack-c.html

Signature

Jeff Richards
MS MVP W95/W98

> but the thing I don't get is.......
>
[quoted text clipped - 6 lines]
> if so there is no benefit.
> how do those 3 unused sectors get used ?

Reply to this Message

andrea catto' - 02 May 2004 18:57 GMT

thank you, I got it now, partially figured it out by myself after a lot of
speculation, and partially confirmed with your article.

since the msdos days I no longer followed low level disk management so I was
forgetting how clusters are managed as independent units that are
individually taken care of by the OS.

the OS, at least in the fat, has a representation of each cluster in form of
pointers within the FAT, so the number of clusters in a disk equals the
number of pointers (multiplied by 4/8 bytes each) in the fat.

baiscally, since 99% of the apps deal with files, not low level, they don't
know nor they care of how the files are distributed across the disk.
the OS normally already may jump from place A to B any time it needs to deal
with the next data chunk of a file.
to the file this is totally transparent, a file only deals with a (virtual)
progressive file pointer/position.
basically because of this abstraction layer the compression is a piece of
cake to implement.
instead of writing the data to a cluster as its demanded, the OS takes an
extra step and 'tries' compressing it, if it's reasonably smaller, it'll be
able to fit more per cluster-unit therefore reducing the odds of requiring a
new cluster-allocation from the OS.
that's way clear now.

I am hoping this explains the idea to others too since I like sharing.

1st highest level layer) the seamless and transparent file management, which
allows dealing with virtually sequential file pointers/access.
2nd layer) OS file management which decides when it's time to 'skip' to
another cluster-unit to allow the progressive file pointers seamlessely.
3rd layer) (IF COMPRESSION IS SET ONLY) try to compress as much as possible
before writing to fit more in a cluster-unit.
4th layer) (as suggested) cacheing.
5th layer) almost lowest level disk coordination regardless of the geometry
(which I recall it was int 25h,int 26h in dos)
6th layer) lowest level disk geometry coordination considering
head/disk/clusters etc... (which I recall it was int 13h in dos).

does this make sense to you all ?

Reply to this Message

Jeff Richards - 02 May 2004 22:54 GMT

Not quite. If compression is active the compression manager allocates space
for the compressed data within the compressed volume file, using its own
file system. If the CVF needs to expand as a result of including the new
data, the compression driver asks the OS to allocate space for the CVF, and
the FAT is updated accordingly. As there will often be unused space in the
CVF then many changes that occur to compressed files involve nothing more
than writing new data within the CVF, and no changes to any FAT entries
occur.

Signature

Jeff Richards
MS MVP W95/W98

> thank you, I got it now, partially figured it out by myself after a lot of
> speculation, and partially confirmed with your article.
[quoted text clipped - 52 lines]
> >
> > please help me out

Reply to this Message

andrea catto' - 04 May 2004 01:02 GMT

sorry I noticed now this is not an NT/NTFS file system thing, I was
referring to that instead.
which compresses the files individually without having the create a virtual
drive.
but thanks.

> Not quite. If compression is active the compression manager allocates space
> for the compressed data within the compressed volume file, using its own
[quoted text clipped - 70 lines]
> > >
> > > please help me out

Reply to this Message