Considerable size drop

[DevHackr]

I have been going through these archives in an effort to preserve them. One thing I noticed was that the files are considerably bloated. This a mix of things. File formats that are outdated, older versions of photoshop, duplicates of the same files, and a healthy amount of junk.

I began with 41.9gb worth of materials.

#1 first step was combining all these torrents onto one folder called "Acclaim Archives"
#2 Running a scan for exact duplicates of the same file. Same name/size I removed any duplicates
#3 removal of the garbage 0kb files and empty directories.
#4 converting the bloated and outdated TIFF format images to comparable PNG files of the same quality.
#5 Updating the decade old photoshop files to newer, better, more compressed photoshop formats of today.
#6 updating the outdated quark QXD files to the newer quark QXP format. No loss, better compression.
#7 updated the outdated video files to current day standards.

The work isn't done as of yet, but I have managed to compress 41.9GB down to 26.7GB. I still have more stuff to update, it's not nearly finished.

I still need to:
#1 finish the PSD file updates
#2 finish updating the QXDs to QXPs
#3 check the text and document files to see what can be shrunk
#4 7z the entire thing and see how small I can get it. I'll keep you updated as I get further or finish.

[bobzee]

There's something strangely addictive about compressing stuff. Good work.

http://i272.photobucket.com/albums/jj191/bobzee666/forums/compress.gif

[APE]

Not bad. Not bad at all.

IMO the originals should be preserved for posterity, but for general consumption this should be more than adequate.

[DevHackr]

Yes, I agree. Sadly my biggest failure is the fact that the dates of the newer files does not correctly represent the original documents. (except the tiff2png which the program matched the dates)

I was a bit disappointed with the contents of these torrents. It was almost entirely strategy guide documents

[chalmo]

http://i272.photobucket.com/albums/jj191/bobzee666/forums/compress.gif

When my brother first introduced me to PKZip back in the early 90s, I thought I was a genius by thinking of rezipping files over and over again until they were a couple of bytes. Yeah, I'm not the brightest.

[n64coder]

Could you really compress files over and over with pkzip until they're a few bytes? I didn't think that would be possible. I would think subsequent compressions would yield roughly the same or larger size.

I guess there was a bug with pkzip or it used some sort of lossy compression algorithm.

[APE]

Could you really compress files over and over with pkzip until they're a few bytes? I didn't think that would be possible. I would think subsequent compressions would yield roughly the same or larger size.

I guess there was a bug with pkzip or it used some sort of lossy compression algorithm.

No you really can't. I can't claim to understand how the compression algorithm works but its my understanding that there is a small amount of overhead necessary to be able to take the compressed data and decompress it. If you compress something to its maximum potential compressing it again will just add overhead to it thus increasing the size of the resulting file.

Not by much mind you, but if you take a tarball, 7zip it, RAR it, then ZIP it again you'll have a file larger than the original tarball.

[7Force]

And if you zip/7zip/rar/whatever something that's already been compressed with another method, like an Xvid video or something, the file will remain roughly the same size.

[syntax error]

Do the have an ancient version of the Photoshop plugin Genuine Fractals on tape?

[DevHackr]

Outcome after a few weeks of puttering 7.89gb in 81 97.5mb rar pieces

[DevHackr]

7.49gb after a few weeks, and after rarin' them.

So I shaved off 34.41 GB off the total archives. A little easier to distribute now

[Druid II]

Does your conversion preserve all the data in the old formats? I'm thinking of minor crap like metadata and such, which conversion apps often do not check.


#7 updated the outdated video files to current day standards.

... Does that mean you transcoded them, thus lowering their quality?

[DevHackr]

It would have been excessively difficult to make the video fies WORSE. They were 320/240 grainy pieces of crap. I did take care to make sure they had the same colors, framerates, audio, etc. However they're not bloated late 90's quicktime anymore

[Melchior]

Be careful with the TIFF files!

A file can contain more than one image, many utilities DO NOT COMPREHEND THIS!

So you may be losing extra 'pages' of images...

Blindly compressing and converting this is a good way to loose data. (If anyone cares)

[r34per]

whats the status of this? :)

[Master13]

I also am wondering when this version will be released? That is a major drop and could save a bunch of time Thanks for your work Devhackr

[ASSEMbler]

I would like to replace the torrents with the streamlined version.

[Master13]

Status? I'm still looking forward to a release.

[CodeAsm]

I stick with my originals, and believe they should stay acceseble.

[Master13]

I know its just I don't have enough room on my hard drive right now for all of this

[ASSEMbler]

Anyone guide me to redo this I will launch a new torrent.

[APE]

I stick with my originals, and believe they should stay accessible.

If the content remains 100% identical for all intents and purposes with the file size dropping I'm all for it.

On the other hand the originals shouldn't be wiped off the face of the earth due to inconvenience. It'd almost be akin to digitizing the moon landing in HD and burning the original magnetic tapes because they're bulky and cost money to store properly.

I'd be worried that by taking an oldler formatted file and updating it to its modern equivalent you'd either change the contained data from slightly/imperceptively to a lot. For preservation the former is acceptable if no other options are available with the latter being wholesale unacceptable. I'd bet good money that the way Photoshop stores data has changed quite a bit, likely tossing away legacy support/features in the process.

[BLUamnEsiac]

As long as lossless formats are used, there shouldn't be any drop in quality. I do have several versions of Photoshop (5.0, CS2, CS3, Elements 8.0) and backward compatibility seems to be great. I'm fairly certain that features within the past 15 years should still be supported in current versions.


If the content remains 100% identical for all intents and purposes with the file size dropping I'm all for it.

On the other hand the originals shouldn't be wiped off the face of the earth due to inconvenience. It'd almost be akin to digitizing the moon landing in HD and burning the original magnetic tapes because they're bulky and cost money to store properly.

I'd be worried that by taking an oldler formatted file and updating it to its modern equivalent you'd either change the contained data from slightly/imperceptively to a lot. For preservation the former is acceptable if no other options are available with the latter being wholesale unacceptable. I'd bet good money that the way Photoshop stores data has changed quite a bit, likely tossing away legacy support/features in the process.

[DevHackr]

Sorry for being so out of touch! I do still have the files, smooshed down a good amount. I attempted to break it up and upload it to megaupload, but my connection is shitty at best and it kept dropping the upload. If someone can setup an ftp I'd be happy to drop it on there

[ASSEMbler]

Can you perhaps private torrent it to me and then I will torrent it on the server?

[DevHackr]

If you're up to it, sure. I will need a bit of direction on what to do. But once it's up I can leave it on for a bit maybe pop a few seeds

[weej]

Hi All,
Did this ever happen? ('this' being that the streamlined torrent replaced the original 45 gb one)

[Master13]

No its never been released. I've been waiting for it but distribution seems to be the issue

[xclaw]

I can assist with a private FTP if needed. Then a seed on some public tracker.

[Zetrox2k]

Hey guys, just a quick update on this;

Devhackr has done some serious cleaning up, and deserves a thumbs up on this :encouragement:

Currently, DevHackr is in the process of uploading the data into my personal FTP. Here's the plan;

1. Receive archives (der...)
2. Extract/Test archives
3. Repack, with 10% recovery data
4. Upload and seed from 1gbit Seedbox.

Unfortunately my FTP wont be able to cope with an army of people downloading, so I think investing in a 1gbit Seedbox is the next best thing.

Cheers guys!

-Z