timer / doevents / inet problem

[martin]

In my script I have actived a Timer with DIALOG SET TIMER. The interval value is 250.
When the timer is fired it runs the DOEVENTS ON command.
CASE %WM_TIMER
DOEVENTS ON

Also I use some INET_URLdownload commands. But I noticed that when INET_URLdownload is running the timer is not active anymore. But when I put also "DOEVENTS ON" after INET_URLdownload the timer will be hit again. I don't understand how this exactly works and how i can improve my code.

I hope my problem is clear, otherwise I will show some code.

thanks in advance for any help

[ErosOlmi]

Martin,

first never call a DOEVENTS inside a timer because it will create unpredictable results. Timer events are already releasing cpu to other process.

The problem is INET_URLdownload: it is a synchronous function, it means it will not return until it has finished to download the file. So if you call that function inside a timer event again you will get unpredictable results again.

What you need is an asynchronous function, that is a function you fire and than you can check its going on till finished.
In this case you could fire the function when needed and than install a timer to check download status.

Problem is that I need to create such a function.
Something like \SampleScripts\TcpUdp\DownloadPage.tBasic

I will see what I can do.
Eros

[ErosOlmi]

Martin,

I think I've already developed something interesting.
Be patient for 1 or 2 days and I will release something to test.

Eros

[martin]

Wonderful!! :D

[ErosOlmi]

Ok,

I've already developed something but I'm not 100% satisfied.
Application is working fine, you have full control of what to download and control of download stepping but I would like to have more: download callback in order to be able for thinBasic INet module to automatically call a script function indicated by the programmer as soon as there is an event in download stepping.

Anyhow, for the moment is not bad. Below an example of the code and a bundled exe you can check. I think when I will release you will be able to use also in a timer callback: you will need to add some checking in the timer in order to see if current data chunk has already been downloaded or you have to wait till next timer event.

New features will be present in next release (out very soon).
I'm not sure if I will be able to already include a callback version.

Ciao
Eros



'---Declare need modules
uses "console"
uses "INET"
uses "file"

'---Some needed variables
LOCAL bResult AS LONG
LOCAL hSession AS DWORD
LOCAL hInternet AS DWORD

LOCAL sUrl AS string

LOCAL strBuffer AS STRING
LOCAL pstrBuffer AS long
LOCAL strBufferLen AS long
LOCAL strData AS STRING

LOCAL cbBytesRead AS DWORD
local TotalBytesRead as dword
local FileSize as dword

local nSecs as long
local T0, T1 as double

local lBufferLen as long = 1024

'---Initializes an application's use of the WinINet Internet functions.
hSession = INET_Internet_Open("Test download from thinBasic script", %INTERNET_OPEN_TYPE_DIRECT, "", "", 0)
IF hSession = %NULL THEN
printl "INET_Internet_Open error"
stop
END IF

'---Opens a resource specified by a complete HTTP URL.
'---More or less like a standard local OPEN file operation
sUrl = "http://www.thinbasic.biz/projects/thinbasic/thinBasic_1.7.8.0.zip"
printl "Downloading " & sUrl
hInternet = INET_Internet_OpenUrl(hSession, sUrl, "", 0, %INTERNET_FLAG_NO_CACHE_WRITE, 0)
IF hInternet = %NULL THEN
printl "INET_Internet_OpenUrl error"
INET_Internet_CloseHandle hSession
stop
END IF

'---Setup a %BufferLen bytes buffer
strBuffer = STRING$(lBufferLen, $SPC)
pstrBuffer = STRPTR(strBuffer)

'---Query remote server about the size of the file we are going to download
INET_Http_QueryInfo(hInternet, %HTTP_QUERY_CONTENT_LENGTH, pstrBuffer, lBufferLen)
FileSize = val(left$(strBuffer, lBufferLen))
printl "Real file size in bytes: " & FileSize

'---Ok we can start a loop for file download. This can be automated in a loop or in a timer function
'---Reads the data in %BufferLen bytes chunks
T0 = timer
DO
bResult = INET_Internet_ReadFile(hInternet, pstrBuffer, len(strBuffer), cbBytesRead)
IF bResult = 0 OR cbBytesRead = 0 THEN EXIT DO
IF cbBytesRead = LEN(strBuffer) THEN
strData &= strBuffer
ELSE
strData &= LEFT$(strBuffer, cbBytesRead)
END IF
TotalBytesRead += cbBytesRead
T1 = timer
printat "Elapsed seconds : " & format$(T1-T0, "#0") , 40, 3
printat "Bytes read so far : " & TotalBytesRead , 40, 4
printat "KBytes per second : " & format$((TotalBytesRead / max(T1-T0, 1))/1024, "#0.00"), 40, 5
printat "Estimated seconds : " & format$((FileSize * (T1-T0))/TotalBytesRead, "#0") , 40, 6
LOOP

'---Closes all used handles
INET_Internet_CloseHandle hInternet
INET_Internet_CloseHandle hSession

'---Output buffer to a file
printl
printl
printl
printl
printl
printl "---Saving file: " & app_sourcepath & "thinBasic_1.7.8.0.zip"
file_save(app_sourcepath & "thinBasic_1.7.8.0.zip", strData)
printl
printl "---All done! Press any key to finish---"
WAITKEY

[martin]

Hello Eros, thanks for your explanation. I didn't realize that a DoEvent inside a timerfunction is unwise. The example and code looks very promising and I am looking forward (i think we all do) to the new official release. I want to thank you for always answering my/our questions and for your hard work to realize our wishes and requests. It's unbelievable what you're doing for us and sometimes I wonder if you have time to sleep.

Martin

[ErosOlmi]

It's unbelievable what you're doing for us and sometimes I wonder if you have time to sleep.


Yes, be sure I have time to sleep, work in real job, family and even other (little) things.

One of the big advantage of thinBasic (those developing thinBasic modules know what I mean) is its internal structure. Adding new keywords is very straightforward.
The most difficult part is to understand what is the right direction for solving a problem but when you have got it, adding in thinBasic is easy.

[ErosOlmi]

I think I have my first callback monitored Internet download version done.
I'm still discovering this functionality by myself because I didn't find much help on the web apart people having problems with it :D

API function I'm using is InternetSetStatusCallback (http://msdn.microsoft.com/en-us/library/aa385120(VS.85).aspx) and the relative CallBack template function (http://msdn.microsoft.com/en-us/library/aa385121(VS.85).aspx)

Hope to succeed and have something to release very soon.

Eros

[martin]

That's really great news Eros. I'm looking forward to try this new function :yahoo:

[ErosOlmi]

Attached a preview on how it will look like. I'm working on adding more internal error checking and help material.

Mainly after having opened connection with

INET_Internet_OpenUrl

and you have a valid Internet handle, you setup a callback for the that handle using

INET_Internet_SetStatusCallBack

From that moment at every status change, INet will automatically call your designated callback.

Considering you are developing a User Interface application I suppose you will work in a way like the following:

setup some data to be able to control your download (a UDT is perfect)
setup a callback function (maybe common to all downloads)
somewhere, open your url and get its Internet handle
setup a timer (maybe common to all active downloads) active only if you have one or more downloads active
in you timer event use INET_Internet_ReadFile to get data from active connections and this will automatically call your designated callback. In there your will monitor your download status


Ciao
Eros

[Petr Schreiber]

I like a lot how it is designed, but I must report that I never got full 10 159 378 bytes, but usually few bytes less ( each time differentely ).

I have no problems downloading the file using Firefox3/IE7.

[ErosOlmi]

I like a lot how it is designed, but I must report that I never got full 10 159 378 bytes, but usually few bytes less ( each time differentely ).

It doesn't matter. Size is just computed for stat purposes.
Important is INET_Internet_ReadFile buffer size. That will determine the finish of the download.
Callback status function is just used to monitor going on when not inside a loop but inside a timer message. It will allow to have some sort of multi-tasking behave. I will create an example showing a possible way on how to use it, a kind of download manager.



I have no problems downloading the file using Firefox3/IE7.

Used API are OS integrated and not dependant from the installed browser so there should not be problems unless you are on very old OS.

Ciao
Eros

[Petr Schreiber]

Hi Eros,

well, I have Windows XP SP2 + all updates before SP3.
When the script reports its finished, I try to unzip downloaded file and it complains "end of signature not found" or something similar - and as I said, 2 executions, 2 different resultant file sizes.

I will check if it is not corrupted by antivirus.


Petr

[Petr Schreiber]

My suspection became reality - when having NOD32 (http://www.eset.com) online, it finished with bad size, with AV disabled no problem.
I think NOD32 scans any download streams coming to PC, so maybe it interfered in some way.

[ErosOlmi]

All developed functionalities are now available on current thinBasic beta 1.7.8.0
Some functions has a little different syntax so please refer to example in \thinBasic\SampleScripts\INet\Internet_FileDownload.tbasic

I will create a User Interface example very soon showing a sort of Download Manager.
I'm not sure if all needed functions are already present, maybe I will develop something more on this area

Let me know.
Eros

[martin]

Works great!

But again I have a problem with reading international characters.

In your example I replaced sURL with http://nl.wikipedia.org/wiki/Knäckebröd and then I save strData with file_save("C:\test.txt", strData)

If I run the script and then open test.txt in NotePad, I can see the word Knäckebröd is written correctly. But if I insert this line in the script MSGBOX 0,grab$(strData,"<title>","</title>") it outputs Knäckebröd as Knäckebröd.

I guess I must change the font, but how?

Martin

[ErosOlmi]

Thanks martin.

I have to investigate further on this, sorry not to have an immediate solution.

Eros

[ErosOlmi]

I think I've got the problem and possibly found a solution thanks to a post from José Roca forum (as usual his forum is a mine of info and source code): http://www.jose.it-berater.org/smfforum/index.php?topic=74.0

The page you pointed for testing is UTF-8 encoded while thinBasic works on ANSI codes.
I will release soon a new update of thinBasic beta version in which I've developed few new functions able to convert from UFT-8 to Wide to Ansi.

Ciao
Eros

[martin]

Super! I really need correct output for analyzing webpages with searchresults. Thanks for your research!

[martin]

Hi Eros, I don't understand how to use the new utf8-commands. In the following script I wrote "function=UTF8ToAnsi$(strdata)" but that doesn't help to display correct characters. How should I use these new functions?


uses "inet"

dim html as string
dim title as string

html=download("http://nl.wikipedia.org/wiki/Knäckebröd")
title=grab$(html,"<title>","</title>")
msgbox 0,title

function download(sUrl as string) as string
local hInternet,MyContext,hFile,cbBytesRead,TotalBytesRead,FileSize as dword
local bResult,pstrBuffer,lBufferLen as long
local strBuffer,strData AS STRING

lBufferLen=1024

do
hInternet = INET_Internet_Open("test", %INTERNET_OPEN_TYPE_DIRECT)
IF hInternet <> %NULL THEN EXIT DO
loop

do
hfile=INET_Internet_OpenUrl(hInternet, sUrl, "", %INTERNET_FLAG_NO_CACHE_WRITE, MyContext)
if hfile<>%NULL then exit do
loop

do
filesize=INET_Http_QueryInfo(hFile, %HTTP_QUERY_CONTENT_LENGTH)
if filesize<>%NULL then exit do
loop

strBuffer = STRING$(lBufferLen, $SPC)
pstrBuffer = STRPTR(strBuffer)

DO
bResult = INET_Internet_ReadFile(hFile, pstrBuffer, len(strBuffer), cbBytesRead)
IF bResult = 0 OR cbBytesRead = 0 THEN EXIT DO

IF cbBytesRead = LEN(strBuffer) THEN
strData &= strBuffer
ELSE
strData &= LEFT$(strBuffer, cbBytesRead)
END IF

TotalBytesRead += cbBytesRead

doevents

LOOP

INET_Internet_CloseHandle hInternet
INET_Internet_CloseHandle hFile

function=UTF8ToAnsi$(strdata)
end function

[ErosOlmi]

Marting,

try the following, change from:

function = UTF8ToAnsi$(strdata)
to

function = acode$(UTF8ToWideChar$(strdata))
in function "download"

Mainly you need to pass from UTF8 to Unicode to Ansi.

I'm still studying how those conversions take place, to be honest I'm not an expert of this field.

Ciao
Eros

[ErosOlmi]

Martin,

I just saw your example. Why are you making that DO/LOOP like:


do
hInternet = INET_Internet_Open("test", %INTERNET_OPEN_TYPE_DIRECT)
IF hInternet <> %NULL THEN EXIT DO
loop


Problem is that if function fails and hInternet will be zero, ode will enter into an infinite loop.

[martin]

whoohoo that works perfect! :eusadance:

I know that this loop can be endless. What my script does is testing a lot of internetpages and then it put the results in a listview. So this download-function is called from a timer routine. So the main script is still running and my thought was that the loop will end if the internet connection is restored. But I agree I must think about a better way.

[ErosOlmi]

whoohoo that works perfect! :eusadance:


Seems a good start ;)