very big files

[zak]

i read about this before years, but remembered it when i have compressed the 2MB pi.txt in this thread (http://www.thinbasic.com/community/showthread.php?11120-regular-expressions-usage&p=83098), it is compressed to about half, because pi digits somewhat random and can't be represented in a more concise form.
so i have tried the attached code to make a 51 MB file with just zero's and newlines, when we compress it its new size will be about 55 KB, and you can reduce the compressed file size more by removing the newlines.
do not try to open the resulted text files since it will hang your editor.

if we can describe something big then it can be compressed efficiently


Uses "file","console"
Dim st,s As String
Dim i As Long
PrintL "wait........."
s = "0000000000000000000000000000000000000000000000000000" + Chr$(13) + Chr$(10)

For i=1 To 10000
st=st + s
Next
'FILE_Save(APP_SourcePath +"big.txt",st)
For i=1 To 100
FILE_Append(APP_SourcePath +"bigFile.txt",st)
Next
PrintL "finish"
WaitKey

[ErosOlmi]

Concatenation is quite memory intensive and very time consuming.
Better to create your string in one go and save all at once ;)

On my computer a 54Mb file is save into 0.8 secs and less that 8 secs to save a 527Mb file

Ciao
Eros


Uses "file","console"

Dim sBuffer As String
Dim sFileOut As String = APP_SourcePath + "bigFile.txt"
Dim t1, t2 As Double
Dim lRepeat As Long = 1000000 '---< Increment to have bigger string

sBuffer = "0000000000000000000000000000000000000000000000000000" & $CRLF

PrintL "Wait: writing a file" & sFileOut
PrintL "File will be " & Format$(Len(sBuffer) * lRepeat) & " bytes"

t1 = Timer
FILE_Save(APP_SourcePath +"bigFile.txt", Repeat$(lRepeat, sBuffer))
t2 = Timer

PrintL
PrintL "---------------------------------------------------------"
PrintL "Finished in " & Format$(t2 - t1, "#0.000") & " seconds"
PrintL "---Press a key to continue---"
WaitKey