string to number problem fasm [solved]

[aliceinchains]

I used mul and the result of fffffh is 0000001a:2f00ffff edx:eax.
I add all the edx's values together and I mul a pow then add to result is what eax is why isn't it coming out properly?

edit: what I mean is what do I have to do to edx:eax to get the correct result 00000000:000fffff

edit: it was my code that was to blame not mul

[Nable]

It would be much better if you show the code of your function. Or you can use debugger and single-step through your code.

[aliceinchains]

the title of the topic is a string to number function written in fasm the code is irrelevant if you have personally taken ascii values converted them to 0-15 multiplied them by a power of 16 and added them to a result I'm wondering if you have to do something to edx:eax when it splits the result when MUL'ing that will move the finished result into EAX...

edit: or just say its my code...

[BrightLight]

MUL stores the result in EDX:EAX.

[Brendan]

Hi,

the title of the topic is a string to number function written in fasm the code is irrelevant if you have personally taken ascii values converted them to 0-15 multiplied them by a power of 16 and added them to a result I'm wondering if you have to do something to edx:eax when it splits the result when MUL'ing that will move the finished result into EAX...

Don't use MUL, use SHR.

For example (untested, NASM):

Code: Select all

;Convert hexadecimal value in ASCIIZ string into unsigned 64-bit integer
;
;Input
; esi      Address of ASCIIZ string
;
;Output
; carry    Set if error (overflow or bad character)
; edx:eax  Resulting value (if carry is clear)

convertHexASCIIZtoU64:
    push ebx
    xor eax,eax
    xor edx,edx           ;edx:eax = temp = 0

.nextCharacter:
    movzx ebx,byte [esi]  ;ebx = next character
    test bl,bl            ;Is it the zero terminator?
    je .done              ; yes, end of string found
    sub bl,'0'            ;bl = character - '0'
    jc .badCharacter      ;Bad character if character < '0'
    cmp bl,9              ;Was the character '0' to '9'?
    jbe .gotDigit         ; yes
    sub bl,'A'-'0'        ;bl = character - 'A'
    jc .badCharacter      ;Bad character if (character > '9') && ( character < 'A')
    cmp bl,0x0F           ;Was the character > 'F'?
    ja .badCharacter      ; yes, bad character

.gotDigit:                ;ebx = digit
    test edx,0xF00000000  ;Will it overflow if we shift it?
    jne .overflow         ; yes, number too large for unsigned 64-bit integer
    shrd edx,eax,4
    shr eax,4             ;edx:eax = temp * 16

    add eax,ebx
    adc edx,0             ;edx:eax = temp * 16 + digit
    add esi,1             ;esi = address of next character
    jmp .nextCharacter

.done:
    pop ebx
    clc
    ret

.overflow:
.badCharacter:
    pop ebx
    stc
    ret

Cheers,

Brendan

[aliceinchains]

I found out it was my code I hadn't used any big numbers with mul before so I was unsure if my output was correct so going to mark as solved.

@Brendan ty for code but 0x40h isn't a digit but I'm using a 16bit terminal to input variable hex or base 10 numbers so I cant just shr sry

edit: this is what I use for ascii

Code: Select all

nibbletonumber:
        cmp al, 0x2f
        jle .bad
        cmp al, 0x67
        jge .bad
        cmp al, 0x39
        jle .num
        cmp al, 0x40
        jle .bad
        cmp al, 0x61
        jge .hexlow
        cmp al, 0x47
        jge .bad
        jmp .hex
.bad:
        std
        ret
.num:
        cld
        clc
        sub al, 0x30
        ret
.hexlow:
        cld
        stc
        sub al, 0x57
        ret
.hex:
        cld
        stc
        sub al, 0x37
        ret