Storing massive amounts of data in INI files

[marcuslee]

It would seem to me that one can store massive amounts of data in an ini file. Plus, if the data is similar, it could be stored in the same field, which can easily be parsed or split into an array.

The following code doesn't even begin to show what an INI file could do (methinks), but it is a start.



Uses "INI"

Dim x, n As String
Dim z(10), d(10) As String
Dim y,v,b As Byte
Dim iFile As String = APP_ScriptPath + "file.ini"

INI_SetKey(iFile, "1", "1", "23|26|65|76|87|43|21|35|43|99")
x = INI_GetKey(iFile, "1", "1")

y = Parse(x,z,"|")
v = Split(x,"|",d)

n = d(3) + z(2)

b = MsgBox (0,n)


Now, I see a possible problem ... hopefully there isn't one. Is there a limit to the amount of data that an INI file can or should hold?

I have an idea for a non-graphical game that might require a lot of variables/game data to be stored in a file. When I say a lot, I mean hundreds if not thousands. Much of the data would be related to each other (earlier versions of the same piece of data, for example), so they could be stored together as I have shown in the above example.

Is an INI file a good way to do this? Are there other ways? Pros and cons?




Mark

[Michael Clease]

Marcus an ini file is just a text file so it's no worse or better that any other method it does have one advantage that you can edit it in notepad,wordpad,word... and doesn't require a program specially written.

If you keep the format of the file in some sort of order you could overlay an array of UDT'S which makes it more like a record system.

[marcuslee]

If you keep the format of the file in some sort of order you could overlay an array of UDT'S which makes it more like a record system.

How would an overlay work? A simple example?



Mark

[Michael Clease]

Uses "File"

Type tRecord
one As Byte
two As Byte
three As Byte
four As Byte
Txt As String*10
End Type

Dim FileBuffer As String
Dim FileName As String
Dim sMsg As String
Dim FirstValue As Byte

FileBuffer = FILE_Load(FileName)

' This must be called after filling the buffer or you can use REDIM
Dim MyRecord(10) As tRecord At StrPtr(FileBuffer)

FirstValue = MyRecord(1).One



Forgot the array :oops:

[marcuslee]

Extending your example, would you use something like the following line?

INI_SetKey(FileName, "1", "1", MyRecord)


Of course, the name of the section and the record could be different in the file. Those are just labels that don't mean anything to the program ... just organizing for the programmer, correct?



Mark

[Michael Clease]

Sorry I forgot the INI thing and just thought of the way I would do it :)

If you use INI you will be bashing the HDD and performance for lots of entries would be quite slow.



Uses "File","INI"

Type tRecord
one As Byte
two As Byte
three As Byte
four As Byte
Txt As String*10
End Type

Dim FileBuffer As String
Dim FileName As String = APP_ScriptPath + "file.ini"
Dim sMsg As String
Dim FirstValue As Byte

' FileBuffer = FILE_Load(FileName)
FileBuffer = String$(SizeOf(tRecord)*10, $SPC)

' This must be called after filling the buffer or you can use REDIM
Dim MyRecord(10) As tRecord At StrPtr(FileBuffer)

MyRecord(1).One = 54
MyRecord(1).two = 55
MyRecord(1).three = 56
MyRecord(1).four = 57
MyRecord(1).txt = "Record 1"
MyRecord(2).txt = "Record 2"


INI_SetKey(FileName, "1", "1", MyRecord(1).One)
INI_SetKey(FileName, "1", "2", MyRecord(1).two)
INI_SetKey(FileName, "1", "3", MyRecord(1).three)
INI_SetKey(FileName, "1", "4", MyRecord(1).four)
INI_SetKey(FileName, "1", "Txt", MyRecord(1).txt)

INI_SetKey(FileName, "FullString", "String", FileBuffer)



As you can see its a lot of work for not much gain, as I said I would just do it without ini its already formatted by the udt you just need to watch the filebuffer is large enough for the array.

[marcuslee]

If you use INI you will be bashing the HDD and performance for lots of entries would be quite slow.

...

As you can see its a lot of work for not much gain, as I said I would just do it without ini its already formatted by the udt you just need to watch the filebuffer is large enough for the array.

I thought that might be the case ... slow performance for lots of data. What would be a good way to store a massive amount of data in a file for later retrieval, the next time the game is played?


Mark

[Michael Clease]

What will the data be? What sort of format is the data?

If its a repeating format then my first example would work quite happily and you could save the FileBuffer at the end of the program which reduces bashing.

[marcuslee]

What will the data be? What sort of format is the data?


The data and format will vary across the board: numbers and strings, long and short, precise (floating point) and integer.



If its a repeating format then my first example would work quite happily and you could save the FileBuffer at the end of the program which reduces bashing.

A little confused here.

Your first example didn't include INI, right? If so, what type of file would it be?

How does knowing the size of the file speed up the program? I'm sorry if I misunderstand.




Mark

[Michael Clease]

The file type would be custom it would be your format :o

Where did I say the size of the file would speed it up ?

you say "The data and format will vary across the board: numbers and strings, long and short, precise (floating point) and integer."

but will this data be same for each record?

Because I load the all of the records from one file in one lump and then edit that information in memory I only access the hdd ONCE to read and then when I finish with it at the end of the program I could SAVE it then.

[marcuslee]

Where did I say the size of the file would speed it up ?


I guess your comment about reducing bashing made me think speed.



you say "The data and format will vary across the board: numbers and strings, long and short, precise (floating point) and integer."

but will this data be same for each record?


The way I picture it now (which could change) is that there might be a certain number of variables that need to be saved ... say anywhere from 25 to 1000 or more depending on the complexity of the game.

That will be for just one iteration of the game. I want to be able to keep track of the decisions you make in the game so each set of variables will be needed in an archival mode.

Opening a file for "random" could work, I suppose. The player wouldn't need to save the entire file to memory. If you play the game long enough, you may not have enough ram to do that (for a really complex version of the game). In archival mode, the player could simply reopen the file and get the needed data.

Does this make sense?



Mark

[Michael Clease]

Not really.

I just dont have enough information about what your are trying to do.

Its like saying "I want to build a house" then not telling me what materials to build the house from, I can keep guessing but no surprises I wont.

Mike

[marcuslee]

I just dont have enough information about what your are trying to do.

Well, I could give you more info on the game, but I think that might just confuse you.

So, how about a small code example of what I just described to get started.



Type tRecord
a As Byte
b As Byte
c As Byte
d As Integer
e As Integer
f As Integer
g As Integer
h As Integer
i As Integer
j As Integer
k As Integer
l As DWord
m As DWord
n As DWord
o As DWord
p As DWord
q As DWord
r As DWord
s As DWord
t As DWord
u As DWord
v As DWord
w As DWord
x As DWord
y As DWord
z As DWord
aa As Extended
bb As Extended
cc As Extended
dd As String
ee As String
ff As String
gg As String
hh As String
ii As Currency
jj As Currency
kk As Currency
ll As Currency
mm As Double
nn As Double
oo As Double
pp As Double
qq As Double
rr As Double
ss As Double
tt As Double
End Type


Okay, now ... a little bit more about my game idea.

Now, there would be a limitless amount of these records. It would just depend on how long you play the game. Each turn of the game would be one record, and if you liked the game, and it was challenging and engaging, you could easily advance the game a thousand times or more.

It is a simulation city builder, sort of like sim city without the graphics. The closest that comes to it now is a game on Facebook called Metropolis. Of course, as I always dream big, I would want it to be bigger than that. Who knows if I will get there! Probably not, but trying is half the fun.




Mark

[Michael Clease]

I do think your over estimating the size of the data, try this.



Type tRecord
a As Byte
b As Byte
c As Byte
d As Integer
e As Integer
f As Integer
g As Integer
h As Integer
i As Integer
j As Integer
k As Integer
l As DWord
m As DWord
n As DWord
o As DWord
p As DWord
q As DWord
r As DWord
s As DWord
t As DWord
u As DWord
v As DWord
w As DWord
x As DWord
y As DWord
z As DWord
aa As Extended
bb As Extended
cc As Extended
dd As String
ee As String
ff As String
gg As String
hh As String
ii As Currency
jj As Currency
kk As Currency
ll As Currency
mm As Double
nn As Double
oo As Double
pp As Double
qq As Double
rr As Double
ss As Double
tt As Double
End Type


Dim FileBuffer As String
Dim FileName As String = APP_ScriptPath + "file.ini"
Dim sMsg As String
Dim FirstValue As Byte

FileBuffer = String$(SizeOf(tRecord)*1, 64)

' This must be called after filling the buffer or you can use REDIM
Dim MyRecord(1) As tRecord At StrPtr(FileBuffer)

sMsg = "Record size *1 = " + Str$( SizeOf(tRecord)*1 )+" bytes"+$CRLF
sMsg += "Record size *100 = " + Str$( SizeOf(tRecord)*100/1024 )+" kbytes"+$CRLF
sMsg += "Record size *1000 = " + Str$( SizeOf(tRecord)*1000/1024 )+" kbytes"+$CRLF
sMsg += "Record size *10000 = " + Str$( SizeOf(tRecord)*10000/1024 )+" kbytes"+$CRLF

MsgBox 0, sMsg


Dont forget you are in control of the data so if you keep a thousand records in memory and then dump them all to a file and start a new set you can.

This is one of the challenges in writing that sort of game, I would search for examples how other people have organised the internal usage of memory.

[marcuslee]

I do think your over estimating the size of the data, try this.


Perhaps. The problem I foresee is that the data could be large if you play the game long enough. Of course, I could program a limit so that nothing gets larger than what I dim it for.



Dont forget you are in control of the data so if you keep a thousand records in memory and then dump them all to a file and start a new set you can.


I don't see why I would need to keep everything in memory at one time, so worrying about the size and speed of the game may be moot in the end anyway. More experimentation is needed.


This is one of the challenges in writing that sort of game, I would search for examples how other people have organised the internal usage of memory.

Could you point me in the right direction here? I'm not familiar with this type of thing and/or resource.



Mark