Trouble in (my) Paradise

[LCSims]

Greets,

After taking way longer than I first envisioned, I've got my most important program functioning and spitting out the results that I had hoped for. But and there's always a but, I'm running into random crashes of thinBasic and the thinCore.dll. So I come looking for any guidance on the issue, before I fully consider rewriting a bunch of code.

This thread gives some insight into what I work with, as far as data and files; http://www.thinbasic.com/community/s...ad.php?t=11464

Some of the files are rather large, such as for the U.S. of A., where the data in the files is a grid of 10801x10801. I subsequently changed these files into quarters and their make up is a grid of 5401x5401. Using some sample code given by Eros in the linked thread, I load the file into a string using FILE_Load and then parse it into two separate strings. One string is the first six lines which give geographic location and then the second string reads the rest of the file. The second string then gets parsed into MyMatrix(), giving me my 5401 rows and 5401 columns. After that, I get into the data manipulation and then write out the data to a file.

Before the manipulation loop ends, I go and Redim MyMatrix() and then the program starts all over reading the next data file, parsing into two string and parsing the data string into MyMatrix(). This is when I get random crashes from thinBasic, listing thinCore.dll as the faulting module.

Memory usage is not so great to blow through the Win32 limit. At a certain point parsing the data string thinBasic may get up to around 1.2gb of memory use before settling back down to around 1gb of memory use until MyMatrix() gets Redim'd. I have inserted a Sleep(10000) to allow a little time for the memory drain to occur before the loop starts all over again. I tried taking out the Redim MyMatrix() command and instead leaving MyMatrix at (5401,5401) and using nested loops to reset the elements back to a null string of "". That crashed also.

The crashes can come at any time, but they appear after the first process and always back at the top of the code, where I am trying to Parse another file. The error messages are strange, as in one from yesterday where it said the variable SourceFileName should be of range 1 to 1, but was 2372. What is strange is SourceFileName is a string variable.

So is there any thoughts as to what I can do to make the processing of the large matrix more secure? Could I programmatically generate a new name for MyMatrix(), so that subsequent loops might create MyMatrix1(), MyMatrix2() and so on, in the hopes that any repetitive use of the matrix might be a contributing factor to the crashes? Otherwise I fear that I am going to have to consider stepping through the source file line by line and incurring, what I believe would be, a much longer processing time.

Thanks for any assistance,

Lance

P.S. for Eros: I'm more than happy to share thinBasic source and my data files. I can upload to my web site if you wanted to take a look. But you MUST promise not to laugh at my spaghetti code!

[ErosOlmi]

P.S. for Eros: I'm more than happy to share thinBasic source and my data files. I can upload to my web site if you wanted to take a look. But you MUST promise not to laugh at my spaghetti code!

As John said, I will be happy too to have a look at data and to some code enough to show the crash.
If I can help solving the problem of fixing any bug inside thinBasic I will be happy.

Because you mentioned about more than 1 GB, my mind start to think it can be an internal thinBasic limit.
For example moving 1 GB of string buffer into another string buffer in some internal function can require 3GB of memory, creating a crash.


PS: it is impossible you created spaghetti code: thinBasic has no GOTO and no GOSUB

[ErosOlmi]

Is already written everywhere in thinBasic documentation:

"... thinBasic is a Basic like language interpreter ..."

[LCSims]

Greets to all,

I've just about finished uploading the file to my site, but Eros' first comment set off a bell inside my head and I want to try something first.

The largest file that thinBasic has to load is 220mb, with 99.9% of that size being read into MyMatrix(). But one thing I haven't done and am going to try out, is I read all the data into a string and then Parse the string into MyMatrix(). That initial string still is populated from the file read and I'm going to null it out once MyMatrix() is filled and see if the random crashes still appear. That should reduce any memory load by a considerable amount. I hope?

And Eros, I taught myself how to program back on Quick Basic. Me and GOTO / GOSUB are old friends! I'm positive a "modern day programmer" will look at my code and say "Huh?", as Object Orientation and inheritance and the rest of those weird things were just be developed.

Lance

[LCSims]

Greets to all,

The edits weren't successful, as the program crashed after 5 of 30 files. But the memory usage did go down by a couple hundred MB, down into the 800-850mb range.

And I forgot to mention that I'm on v1.9.0.0, dated December 13, 2011.

For John: You wrote "Are these single index element arrays you're working with? Would a combination of element and associative based arrays that can be redefined at will with any type of any value be helpful. (system memory is the only limitation. number of dimensions are unlimited)". And best I can understand I'd respond with No, they are space delimited files after the first six lines. The first six lines are for positioning of the data and rows after that are for the data. The amount of data that this type of GIS file can hold can range from very small to very large and they don't have to be square as mine are. It's just how I deal with the material, which is in one degree increment of latitude and longitude. Bottom line I guess would be, if there's a better way of working with the data it's beyond my capabilities.

I'm hoping to find the "limit" of thinBasic, so to speak, as while these files appear large, I have other that are of greater resolution and will need to understand how to slice and dice them to a workable size for thinBasic to handle.

Thanks to all for any help along the way,

Lance

[ErosOlmi]

I Lance,

I'm checking your code and script execution.
I have to admit it is not easy: I do not know the matter, you code is quite complex, there are too many global variables and this increase complexity when debugging.

In any case I can see many point where things can be improved but first I need to understand what the program is doing.
I will add some console output data in order to understand where the program is crashing. So far no crash.

I will give you back as soon as I will discover something.

Ciao
Eros

[LCSims]

Greets Eros,

Too many global variables? I've tried making variables local to the sub-routine, but then it tends to make me even more confused as I work through the code.

At the top of the folder structure are a few screen grabs of when thinBasic crashes. You will note it's saying a variable is out of range, but references the MyMatrix() fill process.

I appreciate your taking the time and I've figured that if I have to, I can decrease the size of the files being read. But then I'd need some guidance to as what an acceptable size for the MyMatrix() data might be. As I mentioned at the top, the original file size is 10801x10801 and I've quartered that. So the next step is to go and make 16 files at 2701x2701. My Canada source files are 4801x4801, so would those need to be quartered? Other areas of the world might be 1201x1201, so where is the "sweet spot" on file size?

I think I mentioned that I've run 27 files through and have thinBasic crash on #28. Then when I restart I might get to #5 before it crashes.

I've earned all the gray hair on my head, thank you!

Best regards,

Lance

[ErosOlmi]

At the moment I'm trying to optimize Work1 function.
Looping though the 29 millions cells of MyMatrix just to rewrite them into a new file takes too long (almost 2 minutes here)
Maybe I will develop new dedicated thinBasic native functions to optimize such a loops.

[ErosOlmi]

Hi Lance,

so far I was just able to identify where the problem occurs: WORK2W function
I've modified your source in order to add some debug info to each function in such a way I could understand where the problem was occurring.

I will send by mail modified source code (with some optimization in append) plus error log.
I hope you can understand some thing more.
I think one file is double copied or erased just before it is loaded and parsed.

I will go on testing tomorrow.

Ciao
Eros

PS: please download and install current thinBasic 1.9.6
It is much better than 1.9.0

[LCSims]

Greets Eros,

I will get 1.9.6.0 installed once I finish this post and will check out the file with your email.

I appreciate the time and effort. And the fact that you didn't say my code was horrible. Cryptic? Yes, it confuses me if I don't work on it for a short time! Now to get it to work all the time and make it's way through about 15,000 files. That's the goal! Hit the Run button and let it churn through a whole lot of files.

Best regards,

Lance