Load from password protected zip files : Model Handling

[lassad]

Hello

In game programming it is very useful to load 3D models and stuff from zipped archives , and for security reasons it is better to protect them by password

In other game engines it is possible to load from these archives by specific functions.

Is it possible to add a module to thinbasic that handles this type of archives and create somes other functions to load from memory ....


Thanks

[ErosOlmi]

Ciao,

I'm not export in games creation so maybe other will reply on specific issues.

thinBasic has functions working with compressed zipped files. More info at http://www.thinbasic.com/public/products/thinBasic/help/html/index.html?zlib.htm
Maybe ZLib_ExtractToZip (http://www.thinbasic.com/public/products/thinBasic/help/html/index.html?zlib_extracttostring.htm)can be used to extract any binary file and access them as memory areas.

Anyway we can think to develop more specific functionalities if needed.

Eros

[lassad]

Thanks for reply,

I have seen that there is a module in thinbasic wich is called Zlib. but it does not extract files to memory , and it does not handle password protected zip files.

In other languages as Purebasic for example : there is a module called PUREZIP, it s a very useful module in this case.

Can someone make a wrapper for thinbasic?

[ReneMiner]

there's Zlib_ExtractToString from tB-Versions above 1.9.16.x that allows to extract single files from a zip-archive to memory. Poke that string anywhere you want. Even to HEAP...

[ErosOlmi]

Thanks for reply,

I have seen that there is a module in thinbasic wich is called Zlib. but it does not extract files to memory , and it does not handle password protected zip files.

In other languages as Purebasic for example : there is a module called PUREZIP, it s a very useful module in this case.

Can someone make a wrapper for thinbasic?


Hi Lassad,

a string in thinBasic is a powerful dynamic memory area. You can get a pointer to string data using STRPTR function.
Otherwise it is very easy to get a heap memory allocation using thinBasic Heap functions.

An example. You can use test data you can find into \SampleScripts\zLib\
Here we extract from a known zip file a known file name stored into a subdirectory inside the ZIP file

'----------------------------------------------------------------------------- '-----------------------------------------------------------------------------
uses "ZLib"

Dim sBuffer As String
Dim ZIPFileName As String


Double T1 = Timer
ZIPFileName = APP_ScriptPath & "TestFolder\ZipFiles.tBasic.ZIP"


sBuffer = ZLib_ExtractToString(ZIPFileName, "Data\ZipFiles.tBasic.UZIP.TXT")


Double T2 = Timer

If sBuffer <> "" Then
MsgBox 0, "File Found. Buffer is " & Len(sBuffer) & " bytes." & $CRLF &
"Time taken To Extract: " & Format$(T2-T1, "#0.000")
Else
MsgBox 0, "File not found in ZIP"
End If


The above example extract into a string buffer 2.8 MB of data in 0.012 seconds on my PC so it is quite fast.


Once you have a string buffer, you can get a pointer to it using STRPTR

Long pData = StrPtr(sBuffer)


or you can store the buffer into a Heap returning Heap memory pointer

Long pData = HEAP_AllocByStr(sBuffer)

If you give more info on how you think to use compressed data, I can see what I can do more.

Regarding password protected files, you are right: there is no any function working on password protected zip files.
I will see what I can do to add something, maybe this can be useful to others.

Ciao
Eros

[ErosOlmi]

I'm a PureBasic customer, just installed PureBasic 5.42 LTS but I cannot find an official PUREZIP module inside help.

Can you please point me where I can find documentation of the mentioned module?

Thanks a lot
Eros

[lassad]

I'm a PureBasic customer, just installed PureBasic 5.42 LTS but I cannot find an official PUREZIP module inside help.

Can you please point me where I can find documentation of the mentioned module?

Thanks a lot
Eros

It is working only with purebasic 5.0

9578

Im looking for loading M15 model and its texture from protected zip . hope it can be done some how (without extracting into drive)

[ErosOlmi]

If you attach here an example ZIP file with one or two password protected files inside I can make some test.
Just an example with a known dummy password you can post here.

I think I've already done something but I need some tests.

Thanks
Eros

[primo]

purezip does not support password as reported in july 2014 http://purebasic.fr/english/viewtopic.php?f=3&t=59871

[ErosOlmi]

Added password parameter to ZLib_ExtractToString.
Will be present in next thinBasic beta version.

sBuffer = ZLib_ExtractToString(ZIPFileName, sFileNameToExtract, sPassword)

I'm trying to make more tests on different zip files created with different compressing applications.
If you have some ZIP example with protected password file inside, just attach in this posts giving the password of the file and I will test.

If all will work fine I will also work on something else:

ZLib_ExtractToHeap that will do the same as ZLib_ExtractToString but will allocate an Heap memory area returning it's handle.

[lassad]

Added password parameter to ZLib_ExtractToString.
Will be present in next thinBasic beta version.

sBuffer = ZLib_ExtractToString(ZIPFileName, sFileNameToExtract, sPassword)

I'm trying to make more tests on different zip files created with different compressing applications.
If you have some ZIP example with protected password file inside, just attach in this posts giving the password of the file and I will test.

If all will work fine I will also work on something else:

ZLib_ExtractToHeap that will do the same as ZLib_ExtractToString but will allocate an Heap memory area returning it's handle.


Here
9579
password = "passeport"

the Zlib_ExtractoSting that can manage password protected zip files will be a important advance toward a greater project : the Filesystem module (like Add3Darchive procedure in purebasic)

But dont forget , the final goal is that we can load resources like M15 files from memory (after extracting them from zip files to string)
I looked in the TB modules and did not find an instruction that can do that (except for MakeTexture from string)

Thank you for support

[primo]

the Filesystem module (like Add3Darchive procedure in purebasic): it is also does not support password, the zipped archives in purebasic internal folders "C:\PureBasic\Examples\3D\Data\Packs" like skybox.zip we can open them easily. it is a long time requested feature but does not fullfilled yet.

[ReneMiner]

...

But dont forget , the final goal is that we can load resources like M15 files from memory (after extracting them from zip files to string)
I looked in the TB modules and did not find an instruction that can do that (except for MakeTexture from string)

Thank you for support


yes, I guess Petr will have to add some function to create models from memory or string-buffer

[ErosOlmi]

Here attached something to test

Install new module
Attached thinBasic_zLib.dll module, new zLib module. I had undergo from zlib_1.2.5 to zLib_1.2.3 because password protected
Please Extract it under your \thinBasic\Lib\ directory substituting your current one.

How to use
New syntax of ZLib_ExtractToZip is:

StringBuffer = ZLib_ExtractToZip(sZipFile, FileNameToExtract [, sPassword])
If StringBuffer is not empty than you will have the uncompressed file and you can do whatever you want.


Example
Get Media.zip file from lassad previous post.
Place it in the same directory of your script.
Than execute the following script.
If all will went fine, you will have 2 new file extracted on disk.


uses "ZLib"

Dim sBuffer As String
Dim ZIPFileName As String

ZIPFileName = APP_ScriptPath & "media.ZIP"

sBuffer = ZLib_ExtractToString(ZIPFileName, "bones.jpg", "passeport")
Save_File(APP_ScriptPath & "bones.jpg", sBuffer)

sBuffer = ZLib_ExtractToString(ZIPFileName, "bones_all.x", "passeport")
Save_File(APP_ScriptPath & "bones_all.x", sBuffer)


If all is working fine, I will see what other functions I can develop.
For sure something like ZLib_ExtractToHeap, exactly the same of ZLib_ExtractToString but will return an Heap memory pointer.

Let me know.
Eros

[Petr Schreiber]

I hear you guys!

M15 from memory should not be a problem, let me look into it!


Petr

[lassad]

Here attached something to test

Install new module
Attached thinBasic_zLib.dll module, new zLib module. I had undergo from zlib_1.2.5 to zLib_1.2.3 because password protected
Please Extract it under your \thinBasic\Lib\ directory substituting your current one.

How to use
New syntax of ZLib_ExtractToZip is:

StringBuffer = ZLib_ExtractToZip(sZipFile, FileNameToExtract [, sPassword])
If StringBuffer is not empty than you will have the uncompressed file and you can do whatever you want.


Example
Get Media.zip file from lassad previous post.
Place it in the same directory of your script.
Than execute the following script.
If all will went fine, you will have 2 new file extracted on disk.


uses "ZLib"

Dim sBuffer As String
Dim ZIPFileName As String

ZIPFileName = APP_ScriptPath & "media.ZIP"

sBuffer = ZLib_ExtractToString(ZIPFileName, "bones.jpg", "passeport")
Save_File(APP_ScriptPath & "bones.jpg", sBuffer)

sBuffer = ZLib_ExtractToString(ZIPFileName, "bones_all.x", "passeport")
Save_File(APP_ScriptPath & "bones_all.x", sBuffer)


If all is working fine, I will see what other functions I can develop.
For sure something like ZLib_ExtractToHeap, exactly the same of ZLib_ExtractToString but will return an Heap memory pointer.

Let me know.
Eros

works great

good job eros

[Billbo]

Lassad,

Why do you show media.zipmedia.zip as the download file, yet it downloads as media.zip? And, you use media.zip in you script.

Bill

[lassad]

Lassad,

Why do you show media.zipmedia.zip as the download file, yet it downloads as media.zip? And, you use media.zip in you script.

Bill

Sorry , a problem in the name typing, i will correct

[lassad]

I hear you guys!

M15 from memory should not be a problem, let me look into it!


Petr

I saw in other libs (for purebasic) some functions like : CatchMesh (MemoryBlock), CatchAnimMesh (MemoryBlock) and CatchTexture(MemoryBlock)

Wish it could give you an idea how to start

Thanks for support

[Petr Schreiber]

I checked the code now, loading M15 from memory is okay, but I wonder - how would you expect it to seek for textures?
Because by default, it looks for the on HDD. Maybe passing array with texture IDs? Let me know!


Petr

[lassad]

I checked the code now, loading M15 from memory is okay, but I wonder - how would you expect it to seek for textures?
Because by default, it looks for the on HDD. Maybe passing array with texture IDs? Let me know!


Petr

I think the best way is to make a big function called LoadM15FromZip(Zipfile,M15file,Pass) witch :
1) extracts M15 to memory
2) Extracts Texture names from M15
3) Looks for names as files in the same zipfile
4) Extracts Textures to memory from zip file
5) Catches the textures from memory and Mesh from memory



Procedure.i LoadMeshFromZip(nomfichier.s,zipfile.s=#CurrentZipArchive,pass.s=#ZipPassword)
Static myFileinfo.PureZIP_FileInfo

If PureZIP_Archive_Read(zipfile)
PureZIP_SetArchivePassword(pass)
NB=PureZIP_FindFile(zipfile, nomfichier)
PureZIP_GetFileInfo(zipfile, NB, @myFileinfo)
Debug "Found " + myFileinfo\FileName
Debug "unCompressed Size: " + Str(myFileinfo\unCompressedSize)
*MemoireTampon1 = AllocateMemory(myFileinfo\unCompressedSize)
Debug PureZIP_ExtractMemory(zipfile,NB,*MemoireTampon1, MemorySize(*MemoireTampon1))

Texte$ = PeekS(*MemoireTampon1,myFileinfo\unCompressedSize)

position=1
posf = 1
Repeat

Position = FindString(Texte$, "TextureFilename",posf)

If Position >0

posi=FindString(Texte$, Chr(34) , position+1 )
posf=FindString(Texte$, Chr(34) , posi+1 )
;Debug posi
;Debug posf
If posf>posi

TextureName$ = Mid(Texte$, posi+1, posf-posi-1)
Debug "Found Texture : "+ TextureName$
NB.i=PureZIP_FindFile(zipfile, TextureName$)
PureZIP_GetFileInfo(zipfile, NB, @myFileinfo)
*MemoireTampon = AllocateMemory(myFileinfo\unCompressedSize)
PureZIP_ExtractMemory(zipfile,NB,*MemoireTampon,myFileinfo\unCompressedSize)
Debug "Extracting texture.."
textur = MP_catchtexture(*MemoireTampon,myFileinfo\unCompressedSize,1,1)
MP_TextureSetName (Textur ,TextureName$)

EndIf
EndIf

Until position=0

mesh = MP_catchMesh(*MemoireTampon1,MemorySize(*MemoireTampon1))
Debug "Model loaded"
FreeMemory(*MemoireTampon1)
PureZIP_Archive_Close()
ProcedureReturn mesh
EndIf


EndProcedure



This is how i do in purebasic
the procedure loads a .X text mesh where the texture file names are indicated by "TextureFilename" string
the MP_catchMesh function looks for a variable called textur by defaut, and applies it on the mesh, if the textur variable is empty so the mesh will appear bold
so in the program we must preload the texture by textur=catchtextur function before calling it

Hope it may help

[lassad]

No replies since 2 weeks ?

seems that the project is aborted.:(

[Petr Schreiber]

Don't give up so easily, and read forum more carefuly before making such a statement please ;)

You can see I started my work on redesigning the m15 in order to integrate your request more easily:
http://www.thinbasic.com/community/showthread.php?12666-M15-reimplementation-via-GBuffers&p=92841#post92841

Long story short - I am working on making M15 better already for some time... It will take some initial amount of work, but from the result all should benefit.


Petr

[lassad]

happy to hear that