Network byte order

[Michael Clease]

I was wondering what was the fastest way to convert data that is stored in network byte order.

so if i had a file that had the number 13 as a DWORD stored in a sequence

MSB LSB
&h00 00 00 0D

rather than
LSB MSB
&h0d 00 00 00

this is my first Idea

mid$(FileBuffer,IHDRpos-4) = strreverse$(mid$(FileBuffer,IHDRpos-4,4))

Its for PNG file format



All integers that require more than one byte must be in network byte order: the most significant byte comes
first, then the less significant bytes in descending order of significance (MSB LSB for two-byte integers, B3
B2 B1 B0 for four-byte integers). The highest bit (value 128) of a byte is numbered bit 7; the lowest bit
(value 1) is numbered bit 0. Values are unsigned unless otherwise noted. Values explicitly noted as signed
are represented in two’s complement notation.
Unless otherwise stated, four-byte unsigned integers are limited to the range 0 to 231 − 1 to accommodate
languages that have difficulty with unsigned four-byte values. Similarly, four-byte signed integers are limited
to the range −(231 − 1) to 231 − 1 to accommodate languages that have difficulty with the value −231.


Anyone got any different ideas

[ErosOlmi]

Abraxas,

I cannot help on this because I've not experteese on images but I can quite easily make a native function to revert bytes into a memory area giving start memory position, len.

Let me know.
Eros

[Michael Clease]

I think its got less to do with images and more to do with networking



12.5 Byte order
It has been asked why PNG uses network byte order. We have selected one byte ordering and used it consistently.
Which order in particular is of little relevance, but network byte order has the advantage that routines
to convert to and from it are already available on any platform that supports TCP/IP networking, including
all PC platforms. The functions are trivial and will be included in the reference implementation.


If your willing to make a new function I am willing to try it.

thanks

Mike

[ErosOlmi]

Abraxas,

Microsoft has helped us developing needed functions. See reference here: http://msdn2.microsoft.com/en-us/library/ms741394.aspx
Add the following in your code and you should be able to make relevant conversions:


'The htons function converts a u_short from host to TCP/IP network byte order (which is big-endian).
DECLARE FUNCTION htons LIB "wsock32.dll" ALIAS "htons" (BYVAL hostshort AS WORD) AS WORD
'The ntohs function converts a u_short from TCP/IP network byte order to host byte order (which is little-endian on Intel processors).
DECLARE FUNCTION ntohs LIB "wsock32.dll" ALIAS "ntohs" (BYVAL netshort AS WORD) AS WORD

'The htonl function converts a u_long from host to TCP/IP network byte order (which is big endian).
DECLARE FUNCTION htonl LIB "wsock32.dll" ALIAS "htonl" (BYVAL hostlong AS DWORD) AS DWORD
'The ntohl function converts a u_long from TCP/IP network order to host byte order (which is little-endian on Intel processors).
DECLARE FUNCTION ntohl LIB "wsock32.dll" ALIAS "ntohl" (BYVAL netlong AS DWORD) AS DWORD


First 2 functions work on 16 bit numbers.
Second 2 functions work on 32bit numbers.

Let me know if all is working fine.

Ciao
Eros

[Michael Clease]

thanks will try when I go home for lunch.

how about adding them to TcpUDP if you think they would fit.

[ErosOlmi]

thanks will try when I go home for lunch.

how about adding them to TcpUDP if you think they would fit.


Yes, maybe worth to add.

Here an example using above defined API functions.


'The htons function converts a u_short from host to TCP/IP network byte order (which is big-endian).
DECLARE FUNCTION htons LIB "wsock32.dll" ALIAS "htons" (BYVAL hostshort AS WORD) AS WORD
'The ntohs function converts a u_short from TCP/IP network byte order to host byte order (which is little-endian on Intel processors).
DECLARE FUNCTION ntohs LIB "wsock32.dll" ALIAS "ntohs" (BYVAL netshort AS WORD) AS WORD

'The htonl function converts a u_long from host to TCP/IP network byte order (which is big endian).
DECLARE FUNCTION htonl LIB "wsock32.dll" ALIAS "htonl" (BYVAL hostlong AS DWORD) AS DWORD
'The ntohl function converts a u_long from TCP/IP network order to host byte order (which is little-endian on Intel processors).
DECLARE FUNCTION ntohl LIB "wsock32.dll" ALIAS "ntohl" (BYVAL netlong AS DWORD) AS DWORD

'---
' This example fill a memory buffer with a sequence of DWORDs
' Every DWORD number is followed by its relative BIG ENDIAN notation
'---

uses "console"

'---Define a string
dim sBuffer as string
'--Define a pointer to the string buffer
dim psBuffer as dword value strptr(sBuffer)

'---String len. For this example it MUST BE a multiple of 8
dim sLen as long value 256

'---Just for testing ...
'---Fill string buffer with null
sBuffer = string$(sLen, 0)

'---Define a counter for the for/next
dim Counter as long

'---Now define 2 pointers to a DWORD with absolute position!!
dim p1Dwd, p2Dwd as dword at 0

'---Now loop for the len
for Counter = 0 to 256 step 8
'---Set the first pointer to a DWORD to some position in the buffer and assign a value
setat(p1Dwd, psBuffer + Counter)
p1Dwd = Counter^2

'---Set the second pointer to point 4 bytes after first pointer and assign BIG ENDIAN of the first DWORD
setat(p2Dwd, GetAt(p1Dwd) + 4)
p2Dwd = htonl(p1dWd)

'---Output some data
console_writeline format$(Counter, "000") & " " & format$(p1Dwd, "#0") & " " & format$(p2Dwd, "#0")
next

console_waitkey

[Michael Clease]

On testing "ntohl" seems to be almost twice as fast as "mid$(FileBuffer,IHDRpos-4) = strreverse$(mid$(FileBuffer,IHDRPOS,4))"

so I think it would be a good addition.

thanks again for the good find.

Mike

[ErosOlmi]

Perfect!

Just to give some more info, moving memory areas is always very slow compared to direct byte pointers.
What strreverse$(mid$(FileBuffer,IHDRPOS,4)) does is a lot of memory allocation, bytes copy, bytes assignments. It is not efficient at all especially when few bytes are involved.
First MID$ get a copy of FileBuffer (and it can be a big string), than it extract a piece of the original string allocating another string passing it to STRREVERSE$ that will return another allocated string. A lot of work.

Another way to do the job is using absolute variables, that is variables dimensioned with AT clause. Absolute variables are logical structures that use memory allocated by something else.

For example, to manage 4 bytes inside a string we can create an absolute array of 4 bytes in this way:



DIM FileBuffer AS STRING
'---Here load some data inside the string and fill with NULLs
FileBuffer = STRING$(256, 0)

'---To consider the first 4 bytes of the allocated string as single bytes just do something like
DIM MyBytes(4) AS BYTE AT STRPTR(FileBuffer)

'---Now when using MyBytes in reality you are changing the first 4 bytes of the string
MyBytes(1) = 10
MyBytes(2) = 20
MyBytes(3) = 30
MyBytes(4) = 40
'---SWAP values
SWAP MyBytes(1), MyBytes(4)
SWAP MyBytes(2), MyBytes(3)

'---Show results
msgbox 0, "Swapped bytes:" & $crlf & _
MyBytes(1) & $crlf & _
MyBytes(2) & $crlf & _
MyBytes(3) & $crlf & _
MyBytes(4) & $crlf & _
""

'---Or we can consider the first 4 bytes like 2 INTEGERs
DIM MyINT(2) AS integer AT STRPTR(FileBuffer)

'---Now again, when using MyINT in reality you are changing the first 4 bytes of the string
MyINT(1) = 1000
MyINT(2) = 2000
'---SWAP values
SWAP MyINT(1), MyINT(2)

'---Show results
msgbox 0, "Swapped integers:" & $crlf & _
MyINT(1) & $crlf & _
MyINT(2) & $crlf & _
""

'---And if we move the pointer of the absolute variable we can change all the other bytes inside the original string
SETAT(MyINT, GETAT(MyINT) + 4)

'---Now MyINT will point to byte 5, 6, 7, 8 of the string considering it as 2 integers

'---... and so on ...


There is a lot of power in DIM ... AT because it is like using C pointers but simulating them with standard variables that are more easy to managed.

Ciao
Eros

[Michael Clease]

Nice example. Would make a good tutorial on addressing memory techniques.

thanks, will look tonight if I get time.

Mike