asynchronous file operations

[DirectuX]

Hi,

thinBasic\SampleScripts\UI\Power\Power_Messages.tBasic is a sample that can detect power status changes via callback management.

I'm in search of such behaviour for complete filesystem interaction.

As a start I studied the file copy feature:

Both FILE_Copy (https://www.thinbasic.com/public/products/thinBasic/help/html/file_copy.htm) and FILE_ShellCopy (https://www.thinbasic.com/public/products/thinBasic/help/html/file_shellcopy.htm) are synchronous functions. None can expose the copy progression to the script.

Instead, MS documentation shows two relevant entries: IFileOperation interface (https://docs.microsoft.com/fr-fr/windows/win32/api/shobjidl_core/nn-shobjidl_core-ifileoperation) and CopyFileExW function (https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexw).

:arrow: Before moving in any direction, I would like to ask :

advices for which of these two last way-suggestions would be the fittest and convenient to implement ?
in your opinion, which shape the implementation should take ?
if you think of another way ?
any other thought about this ?

[DirectuX]

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
:anyone:

[ErosOlmi]

This is an example on how to use shell file operations.
Hope can be of some help.

References: https://docs.microsoft.com/en-us/windows/win32/api/shellapi/nf-shellapi-shfileoperationa

Following example just copy script itself into a copy with extension .TXT



'--------------------------------------------------------------------
' https://docs.microsoft.com/en-us/windows/win32/api/shellapi/nf-shellapi-shfileoperationa
'--------------------------------------------------------------------




'---Shell File Operations
%FO_MOVE = &H0001
%FO_COPY = &H0002
%FO_DELETE = &H0003
%FO_RENAME = &H0004


'---Shell File Options
%FOF_MULTIDESTFILES = &H0001
%FOF_CONFIRMMOUSE = &H0002
%FOF_SILENT = &H0004 ' don't display progress UI (confirm prompts may be displayed still)
%FOF_RENAMEONCOLLISION = &H0008 ' automatically rename the source files to avoid the collisions
%FOF_NOCONFIRMATION = &H0010 ' don't display confirmation UI, assume "yes" for cases that can be bypassed, "no" for those that can not
%FOF_WANTMAPPINGHANDLE = &H0020 ' Fill in SHFILEOPSTRUCT.hNameMappings
' Must be freed using SHFreeNameMappings
%FOF_ALLOWUNDO = &H0040 ' enable undo including Recycle behavior for IFileOperation::Delete()
%FOF_FILESONLY = &H0080 ' only operate on the files (non folders), both files and folders are assumed without this
%FOF_SIMPLEPROGRESS = &H0100 ' means don't show names of files
%FOF_NOCONFIRMMKDIR = &H0200 ' don't dispplay confirmatino UI before making any needed directories, assume "Yes" in these cases
%FOF_NOERRORUI = &H0400 ' don't put up error UI, other UI may be displayed, progress, confirmations
%FOF_NOCOPYSECURITYATTRIBS = &H0800 ' dont copy file security attributes (ACLs)
%FOF_NORECURSION = &H1000 ' don't recurse into directories for operations that would recurse
%FOF_NO_CONNECTED_ELEMENTS = &H2000 ' don't operate on connected elements ("xxx_files" folders that go with .htm files)
%FOF_WANTNUKEWARNING = &H4000 ' during delete operation, warn if nuking instead of recycling (partially overrides FOF_NOCONFIRMATION)
%FOF_NORECURSEREPARSE = &H8000?? ' deprecated; the operations engine always does the right thing on FolderLink objects (symlinks, reparse points, folder shortcuts)
%FOF_NO_UI = (%FOF_SILENT OR %FOF_NOCONFIRMATION OR %FOF_NOERRORUI OR %FOF_NOCONFIRMMKDIR) ' don't display any UI at all




TYPE SHFILEOPSTRUCTA
hwnd AS DWORD ' HWND
wFunc AS DWORD
pFrom AS ASCIIZ PTR
pTo AS ASCIIZ PTR
fFlags AS Word
fAnyOperationsAborted AS LONG ' BOOL
hNameMappings AS DWORD ' LPVOID
lpszProgressTitle AS ASCIIZ PTR ' only used if FOF_SIMPLEPROGRESS
END TYPE


DECLARE FUNCTION SHFileOperationA LIB "Shell32.dll" ALIAS "SHFileOperationA" (byref lpFileOp AS SHFILEOPSTRUCTA) AS LONG


'--------------------------------------------------------------------
' Copy file routine, using shell function (with dialog)
'--------------------------------------------------------------------
FUNCTION ShellFileCopy(BYVAL Source AS STRING, BYVAL Destination AS STRING) AS LONG
Local lRes As Long
Local shfos As SHFILEOPSTRUCTA


Source = Source + Chr$(0) + Chr$(0)
Destination = Destination + Chr$(0) + Chr$(0)


shfos.wFunc = %FO_COPY
shfos.pFrom = StrPtr(Source)
shfos.pTo = StrPtr(Destination)
shfos.fFlags = %FOF_NOCONFIRMMKDIR Or %FOF_MULTIDESTFILES


'Note: if to use FOF_MULTIDESTFILES, source and destination strings can
'contain multiple files, separated with CHR$(0), and end with CHR$(0,0)


lRes = SHFileOperationA(shfos)


If lRes <> %Null Or shfos.fAnyOperationsAborted <> 0 Then
'user aborted, do whatever is needed..
END IF


FUNCTION = lRes
END FUNCTION


uses "Console"


string sFileSource = APP_ScriptFullName
string sFileDestination = APP_ScriptFullName & ".copy.txt"
long lRet


printl "Copying: " & sFileSource
printl "To.....: " & sFileDestination


lRet = ShellFileCopy(sFileSource, sFileDestination)


printl "Result.: " & lret


printl "All done. Press a key to end"
WaitKey

[DirectuX]

Thanks you Eros for the sample.
I tried it, but this does the same as thinbasic's FILE_ShellCopy function (synchronous).

That is why I proposed two other functions that (to my knowledge) would allow the same behaviour than for thinBasic\SampleScripts\UI\Power\Power_Messages.tBasic (callbacks).

What do you think about it ?

[ErosOlmi]

I think the API to use is CopyFileExA
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa

Using a CODEPTR to a lpProgressRoutine
https://docs.microsoft.com/it-it/windows/win32/api/winbase/nc-winbase-lpprogress_routine
Example using PowerBasic language: https://forum.powerbasic.com/forum/user-to-user-discussions/powerbasic-for-windows/782794-copyfileex-problems-with-copy-progress-routine#post782798

Problem is that thinBasic allow CODEPTR only to functions with a maximum of 4 parameters while in this case you need 9 parameters.
I think the only way is to develop a compiled FILE module function that wrap CopyFileEx allowing passing a script function that will be called when needed.

Will see what I can do but not just round the corner.

[DirectuX]

I think the API to use is CopyFileExA
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa

The W is for unicode. Some filename are foreign.



Using a CODEPTR to a lpProgressRoutine
https://docs.microsoft.com/it-it/windows/win32/api/winbase/nc-winbase-lpprogress_routine
Example using PowerBasic language: https://forum.powerbasic.com/forum/user-to-user-discussions/powerbasic-for-windows/782794-copyfileex-problems-with-copy-progress-routine#post782798

Problem is that thinBasic allow CODEPTR only to functions with a maximum of 4 parameters while in this case you need 9 parameters.
Thought it would be easier :eek:
I see LPPROGRESS_ROUTINE callback function
Can it be received by example in a string or an udt that I cut after ? (aka 1 parameter ?) <- or my logic is wrong ? Edit1: From tb help: script function can have from zero to 4 BYVAL LONG or DWORD parameters


I think the only way is to develop a compiled FILE module function that wrap CopyFileEx allowing passing a script function that will be called when needed.

Will see what I can do but not just round the corner.
Would a freeBasic #COMPILED section do the work ? I haven't listed if it's about CopyFileEx only.

Edit2: https://www.freebasic.net/wiki/wikka.php?wakka=KeyPgOpProcptr

[DirectuX]

At least , I can think of 'move' function that can be long too. If I'm not mistaken , others operations (rename, delete...) can be managed (synchronously = script blocking) with standard thinBasic functions.

[ErosOlmi]

I've made some tests and maybe I will have something for tomorrow.

[DirectuX]

how quick ! :wizard:

[ErosOlmi]

Updated thinBasic 1.11.3 at https://www.thinbasic.biz/projects/thinbasic/thinBasic_1.11.3.0.zip

Added FILE_CopyEX function (not yet documented) that partially wrap CopyFileExA https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa

How to use?
Check example in \thinBasic\SampleScripts\File\File_CopyEX.tbasic reported here below.
Copy one file into destination and automatically call a "CopyProgressRoutine" callback function during copy execution in order to have the option to progress some info or cancel operation.
IMPORTANT "CopyProgressRoutine" callback function can have any name but MUST EXACTLY be defined like in the example (otherwise script will GPF)

I've tested with files larger up to 140MB as seems very fast.
Data chunk size cannot be controlled.

If I'm on the right track I will wrap CopyFileExW and possibly add more options.
Let me know if it what you needed.

Ciao
Eros




'---References
' https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa
' https://docs.microsoft.com/it-it/windows/win32/api/winbase/nc-winbase-lpprogress_routine
'---------------------------------------------------------------------------------------------


uses "Console"
uses "File"


callback Function CopyProgressRoutine(
ByVal TotalFileSize As Quad, ' // total file size, in bytes
ByVal TotalBytesTransferred As Quad, ' // total number of bytes transferred
ByVal StreamSize As Quad, ' // total number of bytes for this stream
ByVal StreamBytesTransferred As Quad, ' // total number of bytes transferred for this stream
ByVal dwStreamNumber As Long, ' // the current stream
ByVal dwCallbackReason As Long, ' // reason for callback
ByVal hSourceFile As Long, ' // handle to the source file
ByVal hDestinationFile As Long, ' // handle to the destination file
ByVal lpData As Long ) As Long


'Function should send back one of the following:
' %FILE_PROGRESS_CONTINUE Continue the copy operation.
' %FILE_PROGRESS_CANCEL Cancel the copy operation and delete the destination file.
' %FILE_PROGRESS_STOP Stop the copy operation. It can be restarted at a later time.
' %FILE_PROGRESS_QUIET Continue the copy operation, but stop invoking CopyProgressRoutine to report progress.


printl TotalFileSize, TotalBytesTransferred At 10, 10

function = %FILE_PROGRESS_CONTINUE


end Function


string sFileSource = APP_ScriptFullName
string sFileDestination = sFileSource & ".File_CopyEX.txt"
long lRet


printl "Copying: " & sFileSource
printl "To.....: " & sFileDestination


lRet = file_copyex(sFileSource, sFileDestination, CopyProgressRoutine)


printl "Result.: " & lret


printl "All done. Press a key to end"
WaitKey

[DirectuX]

Updated thinBasic 1.11.3 at https://www.thinbasic.biz/projects/thinbasic/thinBasic_1.11.3.0.zip

Added FILE_CopyEX function (not yet documented) that partially wrap CopyFileExA https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa

How to use?
Check example in \thinBasic\SampleScripts\File\File_CopyEX.tbasic reported here below.
Copy one file into destination and automatically call a "CopyProgressRoutine" callback function during copy execution in order to have the option to progress some info or cancel operation.
IMPORTANT "CopyProgressRoutine" callback function can have any name but MUST EXACTLY be defined like in the example (otherwise script will GPF)

I've tested with files larger up to 140MB as seems very fast.

I tested it. Yes, fast for regular files.
I tested it for GB files too. Works.

From documentation (https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa)
dwCopyFlags
Flags that specify how the file is to be copied.


Value
Meaning



COPY_FILE_NO_BUFFERING
0x00001000

The copy operation is performed using unbuffered I/O, bypassing system I/O cache resources. Recommended for very large file transfers.






Data chunk size cannot be controlled.

That would be for optimization against sector size. (Former I've seen noticeable speed improvement when adjusting this for RAID disks; today I've no RAID)



If I'm on the right track I will wrap CopyFileExW and possibly add more options.
Let me know if it what you needed.


Ah ! I've missed something. I thought that with callbacks coding style, script would be able to continue while the filesystem operations are processed, and be notified regularly on process progress. In fact, the script waits for the copy to end.



thinBasic\SampleScripts\UI\Power\Power_Messages.tBasic is a sample that can detect power status changes via callback management.
I'm in search of such behaviour for complete filesystem interaction.
As a start I studied the file copy feature:
Both FILE_Copy (https://www.thinbasic.com/public/products/thinBasic/help/html/file_copy.htm) and FILE_ShellCopy (https://www.thinbasic.com/public/products/thinBasic/help/html/file_shellcopy.htm) are synchronous functions. None can expose the copy progression to the script.


So... having a progression feedback, we are at half way.


Before moving in any direction, I would like to ask (...)

I think I mislead myself with ideas like DIALOG SHOW MODELESS instruction and callbacks. That is one reason I started questioning :blush:
Now, I foresee even more the difficulty to have asynchronous file operation.



Let me know if it what you needed.


Except the non script-blocking disk-operations that is essential.



For information. My project is not alpha ready but on right track.
Roughly : I have:
1 filesystem(FS) unit
1 DB unit
1 UI unit (no started yet)

user can work offline to queue operations (disks and shares can be offline), by interacting on DB through UI.
script tries to execute FS operations as soon as possible ( rename, copie, move, delete) while not blocking user work
script maintain db up to date. (not yet explored (https://docs.microsoft.com/fr-fr/windows/win32/wmisdk/monitoring-events))

thinking...

maybe I should split the project in two scripts ? (I'd rather not)

I want to start simple (manual db update) and add feature on progress.

Features for alpha :
manual db update
display (2 panes) disks folder tree + files + duplicates
basic file operations



PS : not going into multi-threading like xLeaves's LzBot , is this concept (https://docs.microsoft.com/en-us/dotnet/visual-basic/programming-guide/concepts/async/) possible in thinBasic ? (this is single thread)

[ErosOlmi]

Maybe you need a multy thread programming language so you can run one thread for each file copy operation and have one main thread controlling the process of child threads.

At the moment thinBasic is not able to execute part of the script (for example a function) in a separate thread.
One way could be to use FreeBasic compiled code, have thinBasic script as orchestrator and FreeBasic run copy threads.

Script \thinBasic\SampleScripts\FreeBASIC\FB_Sample_04_PrintAndThread.tBasic can be a start to look at.
Main thinBasic script waits and show results while compiled FreeBasic code execute a thread changing a value.

[DirectuX]

Thanks for your guidance Eros,

I'll keep you informed. :)

[DirectuX]

I think the API to use is CopyFileExA
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa
My mistake, it is nowhere written that CopyFileEx returns immediately. It is just said that it has progress report.
Still studying... win32/fileio/synchronous-and-asynchronous-i-o (https://docs.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o),
win32/ipc/synchronous-and-overlapped-input-and-output (https://docs.microsoft.com/en-us/windows/win32/ipc/synchronous-and-overlapped-input-and-output)



Maybe you need a multy thread programming language
The more I read, the more I understand that it's not the sole solution. -> multithreading -> multitasking



One way could be to use FreeBasic compiled code, have thinBasic script as orchestrator and FreeBasic run copy threads.
Script \thinBasic\SampleScripts\FreeBASIC\FB_Sample_04_PrintAndThread.tBasic can be a start to look at.
Main thinBasic script waits and show results while compiled FreeBasic code execute a thread changing a value.

Looked at. Share your thought... for last resort.

[DirectuX]

As a start.

I'm unsure if the declarations in this draft are correct.
The buffer seems empty.



uses "Console"
uses "File"

#Region "Declarations"


%INVALID_HANDLE_VALUE = &HFFFFFFFF as dword

'[] [REF] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-samr/262970b7-cd4a-41f4-8c4d-5a27f0092aaa
%GENERIC_READ = 0x80000000 as dword ' Specifies access control suitable for reading the object.
%GENERIC_WRITE = 0x40000000 as dword ' Specifies access control suitable for updating attributes on the object.

'[] [REF] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew
%ERROR_FILE_NOT_FOUND = 2
%FILE_SHARE_DELETE = 0x00000004 as dword ' Enables subsequent open operations on a file or device to request delete access. Otherwise, other processes cannot open the file or device if they request delete access.
%FILE_SHARE_READ = 0x00000001 as dword ' Enables subsequent open operations on a file or device to request read access. Otherwise, other processes cannot open the file or device if they request read access.
%FILE_SHARE_WRITE = 0x00000002 as dword ' Enables subsequent open operations on a file or device to request write access. Otherwise, other processes cannot open the file or device if they request write access.
%OPEN_EXISTING = 3 as dword ' Opens a file or device, only if it exists.
%FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000 as dword ' Access is intended to be sequential from beginning to end.
%FILE_ATTRIBUTE_SYSTEM = 4 as dword ' The file is part of or used exclusively by an operating system.
%FILE_FLAG_OVERLAPPED = 0x40000000 as DWord ' The file or device is being opened or created for asynchronous I/O. When subsequent I/O operations are completed on this handle, the event specified in the OVERLAPPED structure will be set to the signaled state.
%FILE_FLAG_NO_BUFFERING = 0x20000000 as DWord 'The file or device is being opened with no system caching for data reads and writes. This flag does not affect hard disk caching or memory mapped files.

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/minwinbase/ns-minwinbase-overlapped
Type overlapped
Internal as quad ' The status code for the I/O request. When the request is issued, the system sets this member to STATUS_PENDING to indicate that the operation has not yet started. When the request is completed, the system sets this member to the status code for the completed request.
InternalHigh as quad ' The number of bytes transferred for the I/O request. The system sets this member if the request is completed without errors.
Offset as DWord ' The low-order portion of the file position at which to start the I/O request, as specified by the user. This member is nonzero only when performing I/O requests on a seeking device that supports the concept of an offset (also referred to as a file pointer mechanism), such as a file. Otherwise, this member must be zero.
OffsetHigh as DWord ' The high-order portion of the file position at which to start the I/O request, as specified by the user. This member is nonzero only when performing I/O requests on a seeking device that supports the concept of an offset (also referred to as a file pointer mechanism), such as a file. Otherwise, this member must be zero.
Pointer as Long' Reserved for system use; do not use after initialization to zero.
hEvent as long' A handle to the event that will be set to a signaled state by the system when the operation has completed.
end type


'[] [REF] https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/aa379560(v%3Dvs.85)
TYPE SECURITY_ATTRIBUTES DWORD
nLength AS DWORD
lpSecurityDescriptor AS DWORD
bInheritHandle AS LONG
END TYPE

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/fileapi/nf-fileapi-readfileex
Declare Function ReadFileEx lib "KERNEL32.DLL" Alias "ReadFileEx" (
ByVal hFile As Long, ' A handle to the file or I/O device
byref lpBuffer As Asciiz , ' A pointer to a buffer that receives the data read from the file or device.
ByVal nNumberOfBytesToRead As Long, ' The number of bytes to be read.
byref lpOverlapped As overlapped, ' A pointer to an OVERLAPPED data structure that supplies data to be used during the asynchronous (overlapped) file read operation.
ByVal lpCompletionRoutine As long ' A pointer to the completion routine to be called when the read operation is complete and the calling thread is in an alertable wait state.
) As Long

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/fileapi/nf-fileapi-createfilea
DECLARE FUNCTION CreateFileA lib "KERNEL32.DLL" ALIAS "CreateFileA" (
byref lpFileName AS ASCIIZ, ' The name of the file or device to be created or opened.
byval dwDesiredAccess AS DWORD, ' The requested access to the file or device, which can be summarized as read, write, both or neither zero).
byval dwShareMode AS DWORD, ' The requested sharing mode of the file or device, which can be read, write, both, delete, all of these, or none
byref lpSecurityAttributes AS SECURITY_ATTRIBUTES, ' A pointer to a SECURITY_ATTRIBUTES structure. This parameter can be NULL.
byval dwCreationDisposition AS DWORD, ' An action to take on a file or device that exists or does not exist.
byval dwFlagsAndAttributes AS DWORD, ' The file or device attributes and flags, FILE_ATTRIBUTE_NORMAL being the most common default value for files.
byval hTemplateFile AS DWORD ' A valid handle to a template file with the GENERIC_READ access right. This parameter can be NULL.
) AS DWORD ' If the function succeeds, the return value is an open handle to the specified file, device, named pipe, or mail slot.

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/handleapi/nf-handleapi-closehandle
Declare Function CloseHandle lib "KERNEL32.DLL" Alias "CloseHandle" (byval hObject as DWORD) AS LONG

#EndRegion

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/minwinbase/nc-minwinbase-lpoverlapped_completion_routine
function LpoverlappedCompletionRoutine(
byval dwErrorCode as DWord, ' The I/O completion status. This parameter can be one of the system error codes.
Byval dwNumberOfBytesTransfered as DWORD, ' The number of bytes transferred. If an error occurs, this parameter is zero.
byval lpOverlapped as long ' A pointer to the OVERLAPPED structure specified by the asynchronous I/O function.
) as Long
Printl lpOverlapped.InternalHigh
printl dwErrorCode
printl dwNumberOfBytesTransfered

return 0
end Function


'
' DIALOG SHOW MODELESS hDlg Call DlgCallback
'
' Do
'
' DIALOG DOEVENTS 0 To Count
'
' if not EOF
' if not cancelled
' doReadNextBlock
'
' Loop While Count
'


Function TBMain() as Long

Local sFileSource as asciiz ' Full path to file to be read
local voverlapped as overlapped
Local hFile as Long ' handle to file to be read
Local lpSecurityAttributes as SECURITY_ATTRIBUTES
Local success as long
Local lpBuffer as asciiz * 512 ' cluster size
Local result as long

Global fileSize as DWord

sFileSource = "t:\test.txt" ' File to read
fileSize = FILE_Size(sFileSource)

Printl "File : " & sFileSource
Printl "Size (Bytes) : " & fileSize

hFile = CreateFileA(sFileSource, %GENERIC_READ, %FILE_SHARE_READ, lpSecurityAttributes, %OPEN_EXISTING, %FILE_FLAG_NO_BUFFERING or %FILE_FLAG_OVERLAPPED, Null)

if hFile <> %INVALID_HANDLE_VALUE then

Printl "File handle = " & hFile

result = ReadFileEx(hFile, lpBuffer, 512, voverlapped, CodePtr(LpoverlappedCompletionRoutine))

printl "ReadFileEx result (<> 0 means no error) : " & result
Printl "Buffer dump : " & lpBuffer

Else
Printl "Failed to get a file handle." in %CONSOLE_FOREGROUND_RED or %CONSOLE_FOREGROUND_INTENSITY
success = %FALSE
endif

closeHandle(hFile)

printl "Press a key to end."

WaitKey

if success = %FALSE then
Return %FALSE
Else
Return %TRUE
endif

end function

[DirectuX]

Progress... :D




uses "Console"
uses "File"

#Region "Declarations"


%INVALID_HANDLE_VALUE = &HFFFFFFFF as dword

'[] [REF] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-samr/262970b7-cd4a-41f4-8c4d-5a27f0092aaa
%GENERIC_READ = 0x80000000 as dword ' Specifies access control suitable for reading the object.
%GENERIC_WRITE = 0x40000000 as dword ' Specifies access control suitable for updating attributes on the object.

'[] [REF] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew
%ERROR_FILE_NOT_FOUND = 2
%FILE_SHARE_DELETE = 0x00000004 as dword ' Enables subsequent open operations on a file or device to request delete access. Otherwise, other processes cannot open the file or device if they request delete access.
%FILE_SHARE_READ = 0x00000001 as dword ' Enables subsequent open operations on a file or device to request read access. Otherwise, other processes cannot open the file or device if they request read access.
%FILE_SHARE_WRITE = 0x00000002 as dword ' Enables subsequent open operations on a file or device to request write access. Otherwise, other processes cannot open the file or device if they request write access.
%OPEN_EXISTING = 3 as dword ' Opens a file or device, only if it exists.
%FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000 as dword ' Access is intended to be sequential from beginning to end.
%FILE_ATTRIBUTE_SYSTEM = 4 as dword ' The file is part of or used exclusively by an operating system.
%FILE_FLAG_OVERLAPPED = 0x40000000 as DWord ' The file or device is being opened or created for asynchronous I/O. When subsequent I/O operations are completed on this handle, the event specified in the OVERLAPPED structure will be set to the signaled state.
%FILE_FLAG_NO_BUFFERING = 0x20000000 as DWord 'The file or device is being opened with no system caching for data reads and writes. This flag does not affect hard disk caching or memory mapped files.

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/minwinbase/ns-minwinbase-overlapped
Type overlapped
Internal as quad ' The status code for the I/O request. When the request is issued, the system sets this member to STATUS_PENDING to indicate that the operation has not yet started. When the request is completed, the system sets this member to the status code for the completed request.
InternalHigh as quad ' The number of bytes transferred for the I/O request. The system sets this member if the request is completed without errors.
Offset as DWord ' The low-order portion of the file position at which to start the I/O request, as specified by the user. This member is nonzero only when performing I/O requests on a seeking device that supports the concept of an offset (also referred to as a file pointer mechanism), such as a file. Otherwise, this member must be zero.
OffsetHigh as DWord ' The high-order portion of the file position at which to start the I/O request, as specified by the user. This member is nonzero only when performing I/O requests on a seeking device that supports the concept of an offset (also referred to as a file pointer mechanism), such as a file. Otherwise, this member must be zero.
Pointer as Long' Reserved for system use; do not use after initialization to zero.
hEvent as long' A handle to the event that will be set to a signaled state by the system when the operation has completed.
end type


'[] [REF] https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/aa379560(v%3Dvs.85)
TYPE SECURITY_ATTRIBUTES DWORD
nLength AS DWORD
lpSecurityDescriptor AS DWORD
bInheritHandle AS LONG
END TYPE

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/fileapi/nf-fileapi-readfileex
Declare Function ReadFileEx lib "KERNEL32.DLL" Alias "ReadFileEx" (
ByVal hFile As Long, ' A handle to the file or I/O device
byref lpBuffer As Asciiz , ' A pointer to a buffer that receives the data read from the file or device.
ByVal nNumberOfBytesToRead As Long, ' The number of bytes to be read.
byref lpOverlapped As overlapped, ' A pointer to an OVERLAPPED data structure that supplies data to be used during the asynchronous (overlapped) file read operation.
ByVal lpCompletionRoutine As long ' A pointer to the completion routine to be called when the read operation is complete and the calling thread is in an alertable wait state.
) As Long

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/fileapi/nf-fileapi-createfilea
DECLARE FUNCTION CreateFileA lib "KERNEL32.DLL" ALIAS "CreateFileA" (
byref lpFileName AS ASCIIZ, ' The name of the file or device to be created or opened.
byval dwDesiredAccess AS DWORD, ' The requested access to the file or device, which can be summarized as read, write, both or neither zero).
byval dwShareMode AS DWORD, ' The requested sharing mode of the file or device, which can be read, write, both, delete, all of these, or none
byref lpSecurityAttributes AS SECURITY_ATTRIBUTES, ' A pointer to a SECURITY_ATTRIBUTES structure. This parameter can be NULL.
byval dwCreationDisposition AS DWORD, ' An action to take on a file or device that exists or does not exist.
byval dwFlagsAndAttributes AS DWORD, ' The file or device attributes and flags, FILE_ATTRIBUTE_NORMAL being the most common default value for files.
byval hTemplateFile AS DWORD ' A valid handle to a template file with the GENERIC_READ access right. This parameter can be NULL.
) AS DWORD ' If the function succeeds, the return value is an open handle to the specified file, device, named pipe, or mail slot.

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/handleapi/nf-handleapi-closehandle
Declare Function CloseHandle lib "KERNEL32.DLL" Alias "CloseHandle" (byval hObject as DWORD) AS LONG

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/synchapi/nf-synchapi-sleepex
declare function SleepEx lib "KERNEL32.DLL" Alias "SleepEx" (
byval dwMilliseconds as dword,
byval bAlertable as long
)as Long
#EndRegion

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/minwinbase/nc-minwinbase-lpoverlapped_completion_routine
function LpoverlappedCompletionRoutine(
byval dwErrorCode as DWord, ' The I/O completion status. This parameter can be one of the system error codes.
Byval dwNumberOfBytesTransfered as DWORD, ' The number of bytes transferred. If an error occurs, this parameter is zero.
byval lpOverlapped as long ' A pointer to the OVERLAPPED structure specified by the asynchronous I/O function.
) as Long

printl "inside LpoverlappedCompletionRoutine"

printl "errCode = " & dwErrorCode
printl "Bytes read = " & dwNumberOfBytesTransfered

return 0
end Function


'
' DIALOG SHOW MODELESS hDlg Call DlgCallback
'
' Do
'
' DIALOG DOEVENTS 0 To Count
'
' if not EOF
' if not cancelled
' doReadNextBlock
'
' Loop While Count
'


Function TBMain() as Long

Global sFileSource as asciiz ' Full path to file to be read
Global voverlapped as overlapped
Global hFile as Long ' handle to file to be read
Global lpSecurityAttributes as SECURITY_ATTRIBUTES
Global success as long
Global lpBuffer as asciiz * 512 ' cluster size
Global result as long

Global fileSize as DWord

sFileSource = "t:\test.txt" ' File to read
fileSize = FILE_Size(sFileSource)

Printl "File : " & sFileSource
Printl "Size (Bytes) : " & fileSize

hFile = CreateFileA(sFileSource, %GENERIC_READ, %FILE_SHARE_READ, lpSecurityAttributes, %OPEN_EXISTING, %FILE_FLAG_NO_BUFFERING or %FILE_FLAG_OVERLAPPED, Null)

if hFile <> %INVALID_HANDLE_VALUE then

Printl "File handle = " & hFile

result = ReadFileEx(hFile, lpBuffer, 512, voverlapped, CodePtr(LpoverlappedCompletionRoutine))

printl "ReadFileEx result (<> 0 means no error) : " & result

SleepEx(100,%TRUE)
Printl "Buffer dump : " & lpBuffer

Else
Printl "Failed to get a file handle." in %CONSOLE_FOREGROUND_RED or %CONSOLE_FOREGROUND_INTENSITY
success = %FALSE
endif



printl "Press a key to end."
DoEvents
WaitKey

closeHandle(hFile)

if success = %FALSE then
Return %FALSE
Else
Return %TRUE
endif

end function

[DirectuX]

More progress ! :dance1:


uses "Console"
uses "File"
uses "UI"

dim myTime as DWord = Timer

#Region "Declarations"

%INVALID_HANDLE_VALUE = &HFFFFFFFF as dword

'[] [REF] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-samr/262970b7-cd4a-41f4-8c4d-5a27f0092aaa
%GENERIC_READ = 0x80000000 as dword ' Specifies access control suitable for reading the object.
%GENERIC_WRITE = 0x40000000 as dword ' Specifies access control suitable for updating attributes on the object.

'[] [REF] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew
%ERROR_FILE_NOT_FOUND = 2
%FILE_SHARE_DELETE = 0x00000004 as dword ' Enables subsequent open operations on a file or device to request delete access. Otherwise, other processes cannot open the file or device if they request delete access.
%FILE_SHARE_READ = 0x00000001 as dword ' Enables subsequent open operations on a file or device to request read access. Otherwise, other processes cannot open the file or device if they request read access.
%FILE_SHARE_WRITE = 0x00000002 as dword ' Enables subsequent open operations on a file or device to request write access. Otherwise, other processes cannot open the file or device if they request write access.
%OPEN_EXISTING = 3 as dword ' Opens a file or device, only if it exists.
%FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000 as dword ' Access is intended to be sequential from beginning to end.
%FILE_ATTRIBUTE_SYSTEM = 4 as dword ' The file is part of or used exclusively by an operating system.
%FILE_FLAG_OVERLAPPED = 0x40000000 as DWord ' The file or device is being opened or created for asynchronous I/O. When subsequent I/O operations are completed on this handle, the event specified in the OVERLAPPED structure will be set to the signaled state.
%FILE_FLAG_NO_BUFFERING = 0x20000000 as DWord 'The file or device is being opened with no system caching for data reads and writes. This flag does not affect hard disk caching or memory mapped files.

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/minwinbase/ns-minwinbase-overlapped
Type overlapped
Internal as quad ' The status code for the I/O request. When the request is issued, the system sets this member to STATUS_PENDING to indicate that the operation has not yet started. When the request is completed, the system sets this member to the status code for the completed request.
InternalHigh as quad ' The number of bytes transferred for the I/O request. The system sets this member if the request is completed without errors.
Offset as DWord ' The low-order portion of the file position at which to start the I/O request, as specified by the user. This member is nonzero only when performing I/O requests on a seeking device that supports the concept of an offset (also referred to as a file pointer mechanism), such as a file. Otherwise, this member must be zero.
OffsetHigh as DWord ' The high-order portion of the file position at which to start the I/O request, as specified by the user. This member is nonzero only when performing I/O requests on a seeking device that supports the concept of an offset (also referred to as a file pointer mechanism), such as a file. Otherwise, this member must be zero.
Pointer as Long' Reserved for system use; do not use after initialization to zero.
hEvent as long' A handle to the event that will be set to a signaled state by the system when the operation has completed.
end type

'[] [REF] https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/aa379560(v%3Dvs.85)
TYPE SECURITY_ATTRIBUTES DWORD
nLength AS DWORD
lpSecurityDescriptor AS DWORD
bInheritHandle AS LONG
END TYPE

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/fileapi/nf-fileapi-readfileex
Declare Function ReadFileEx lib "KERNEL32.DLL" Alias "ReadFileEx" (
ByVal hFile As Long, ' A handle to the file or I/O device
byref lpBuffer As string , ' A pointer to a buffer that receives the data read from the file or device.
ByVal nNumberOfBytesToRead As Long, ' The number of bytes to be read.
byref lpOverlapped As overlapped, ' A pointer to an OVERLAPPED data structure that supplies data to be used during the asynchronous (overlapped) file read operation.
ByVal lpCompletionRoutine As long ' A pointer to the completion routine to be called when the read operation is complete and the calling thread is in an alertable wait state.
) As Long

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/fileapi/nf-fileapi-createfilea
DECLARE FUNCTION CreateFileA lib "KERNEL32.DLL" ALIAS "CreateFileA" (
byref lpFileName AS ASCIIZ, ' The name of the file or device to be created or opened.
byval dwDesiredAccess AS DWORD, ' The requested access to the file or device, which can be summarized as read, write, both or neither zero).
byval dwShareMode AS DWORD, ' The requested sharing mode of the file or device, which can be read, write, both, delete, all of these, or none
byref lpSecurityAttributes AS SECURITY_ATTRIBUTES, ' A pointer to a SECURITY_ATTRIBUTES structure. This parameter can be NULL.
byval dwCreationDisposition AS DWORD, ' An action to take on a file or device that exists or does not exist.
byval dwFlagsAndAttributes AS DWORD, ' The file or device attributes and flags, FILE_ATTRIBUTE_NORMAL being the most common default value for files.
byval hTemplateFile AS DWORD ' A valid handle to a template file with the GENERIC_READ access right. This parameter can be NULL.
) AS DWORD ' If the function succeeds, the return value is an open handle to the specified file, device, named pipe, or mail slot.

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/handleapi/nf-handleapi-closehandle
Declare Function CloseHandle lib "KERNEL32.DLL" Alias "CloseHandle" (byval hObject as DWORD) AS LONG

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/synchapi/nf-synchapi-sleepex
declare function SleepEx lib "KERNEL32.DLL" Alias "SleepEx" (
byval dwMilliseconds as dword,
byval bAlertable as long
)as Long
#EndRegion

'[] [REF] https://docs.microsoft.com/fr-fr/windows/win32/api/minwinbase/nc-minwinbase-lpoverlapped_completion_routine
function LpoverlappedCompletionRoutine(
byval dwErrorCode as DWord, ' The I/O completion status. This parameter can be one of the system error codes.
Byval dwNumberOfBytesTransfered as DWORD, ' The number of bytes transferred. If an error occurs, this parameter is zero.
byval lpOverlapped as long ' A pointer to the OVERLAPPED structure specified by the asynchronous I/O function.
) as Long

printl "inside LpoverlappedCompletionRoutine"

printl "errCode = " & dwErrorCode
printl "Bytes read = " & dwNumberOfBytesTransfered

CONTROL SET TEXT hDlg, %label2, lpBuffer

return 0
end Function

Begin ControlID
%Label1
%Label2
%Button1
End ControlID


'
' DIALOG SHOW MODELESS hDlg Call DlgCallback
'
' Do
'
' DIALOG DOEVENTS 0 To Count
'
' if not EOF
' if not cancelled
' doReadNextBlock
'
' Loop While Count
'
CALLBACK Function DlgCallback() As Long
Select Case CBMSG
Case %WM_Command
If CBCTLMSG = %BN_CLICKED Then
SELECT CASE cbctl
case %Button1
call start_read
end Select
endif
Case %WM_DESTROY
MSGBOX 0, "Window is to be destroyed."
End Select
End Function

function start_read()

Global voverlapped as overlapped
Global hFile as Long ' handle to file to be read
Global lpBuffer as string * 512 ' cluster size

Local lpSecurityAttributes as SECURITY_ATTRIBUTES
Local result as long
Local success as long

hFile = CreateFileA(sFileSource, %GENERIC_READ, %FILE_SHARE_READ, lpSecurityAttributes, %OPEN_EXISTING, %FILE_FLAG_NO_BUFFERING or %FILE_FLAG_OVERLAPPED, Null)

if hFile <> %INVALID_HANDLE_VALUE then

Printl "File handle = " & hFile
result = ReadFileEx(hFile, lpBuffer, 512, voverlapped, CodePtr(LpoverlappedCompletionRoutine))
readInProgress = %TRUE
printl "ReadFileEx result (<> 0 means no error) : " & result

Else
Printl "Failed to get a file handle." in %CONSOLE_FOREGROUND_RED or %CONSOLE_FOREGROUND_INTENSITY
success = %FALSE
endif

if success = %FALSE then
Return %FALSE
Else
Return %TRUE
endif


end Function

Function TBMain() as Long

Local count as Integer
Global hDlg as DWord ' Dialog holder

Global sFileSource as asciiz ' Full path to file to be read
Global fileSize as DWord
Global readInProgress as Boolean

sFileSource = "t:\pg200.txt" ' File to read
fileSize = FILE_Size(sFileSource)

Printl "File : " & sFileSource
Printl "Size (Bytes) : " & fileSize

call MakeUi

Do
DIALOG DOEVENTS 0 To Count
SleepEx(1,%TRUE)
updateLabel1
Loop While Count

closeHandle(hFile)

end function

Function updateLabel1()

Local sLabel1 as String
Local nTime as DWord

nTime = Timer - myTime
sLabel1 = "Script is running since : " & nTime & " seconds"
CONTROL SET TEXT hDlg, %label1, sLabel1

end function

Function MakeUi()

DIALOG NEW 0, "form1", -1, -1, 195, 180, %WS_DLGFRAME OR %DS_CENTER OR %WS_CAPTION OR %WS_SYSMENU, 0 TO hDlg

CONTROL ADD Label , hDlg, %label1, "...", 5, 15, 185, 15, %WS_BORDER OR %ES_RIGHT, %WS_EX_CLIENTEDGE
CONTROL ADD Label , hDlg, %label2, "...", 5, 35, 185, 15, %WS_BORDER OR %ES_RIGHT, %WS_EX_CLIENTEDGE
CONTROL ADD BUTTON , hDlg, %button1,"Start READ", 65, 80, 45, 20

DIALOG SHOW MODELESS hDlg Call DlgCallback

end function

Have still to find how to prog read_next_chunk function.

Note: have to explore (show dialog modal + dialog set timer) as an alternative

Eros, side question : how to show dialog modeless with #resource ? , I didn't find.

[DirectuX]

Finally,

this sample shows that thinbasic is capable of doing non-blocking file operation.
but : in this state this is slow,
I can't increase the chunk size, I don't know why (20480 bytes is maximum).
Pipe is not implemented in this example.

Again, I ask opinion from TB community.
Anyone interested in this challenge ?

10131

[DirectuX]

Updated thinBasic 1.11.3 at https://www.thinbasic.biz/projects/thinbasic/thinBasic_1.11.3.0.zip

Added FILE_CopyEX function (not yet documented) that partially wrap CopyFileExA https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa

How to use?
Check example in \thinBasic\SampleScripts\File\File_CopyEX.tbasic reported here below.
Copy one file into destination and automatically call a "CopyProgressRoutine" callback function during copy execution in order to have the option to progress some info or cancel operation.
IMPORTANT "CopyProgressRoutine" callback function can have any name but MUST EXACTLY be defined like in the example (otherwise script will GPF)

I've tested with files larger up to 140MB as seems very fast.
Data chunk size cannot be controlled.

If I'm on the right track I will wrap CopyFileExW and possibly add more options.
Let me know if it what you needed.

Ciao
Eros




'---References
' https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa
' https://docs.microsoft.com/it-it/windows/win32/api/winbase/nc-winbase-lpprogress_routine
'---------------------------------------------------------------------------------------------


uses "Console"
uses "File"


callback Function CopyProgressRoutine( ...





I made a sample based upon this script and this thinBasic very version. One can have a GUI and at the same time make non blocking copies and follow progress.

10136

Note about the pause feature: Eros, if possible, can we have a discussion on the Pause feature ?

'%FILE_PROGRESS_STOP Stop the copy operation. It can be restarted at a later time.
It is not working as I expected, because if %FILE_PROGRESS_STOP is returned, the file_copyex ends.
How do you think it is intended to implement the copy continue ?

[DirectuX]

No, didn't change anything in that area.
I'm working mostly in thinAir

Can you send me a complete code example giving error so I can test it?

Eros, you can use the complete script in this above #19 post (https://www.thinbasic.com/community/showthread.php?13026-asynchronous-file-operations&p=95457&viewfull=1#post95457) (it's the modal version but it has the same thindebug error as the modeless one (https://www.thinbasic.com/community/showthread.php?12949-thinBasic-1-11-x-x&p=95531&viewfull=1#post95531))

Edit: here is the modeless version 10148, but I think I will continue dev with the modal version.