visual studio – How to compile 64-bit binaries for latest versions of OpenSSL (3.5.x) on the Windows 10

I am trying to compile latest version of the OpenSSL on Windows 10 machine (version 3.5.4).

Prerequisites are:

1. I ran x64 Native Tools Command Prompt for Visual Studio 2022 Community Edition

2. I downloaded latest version of zlib library (version 1.3.1) and compiled it successfully

3. I downloaded latest version of OpenSSL (version 3.5.4) and run the folowing command to configure build environment:

   perl Configure no-rc5 no-idea enable-mdc2 enable-zlib VC-WIN32 -I..\zlib /LIBPATH:..\zlib

   It completes without errors

4. Run nmake. Compilation completes successfully. But at the end of the process I receive following error when build process tries to link compiled .obj files into final libcrypto-3.dll:

...
cmd /C ""link" /nologo /debug /machine:x64 /LIBPATH:..\zlib /dll /nologo /debug  \
/machine:x64 /LIBPATH:..\zlib @C:\Users\XXXX\AppData\Local\Temp\nmDAB3.tmp \
/implib:libcrypto.lib || (DEL /Q libcrypto-3.* libcrypto.lib & EXIT 1)"

crypto\aes\libcrypto-shlib-aes-586.obj : fatal error LNK1112: module machine type 'x86' conflicts with target machine type 'x64'

Analysis

It looks to me that issue is that linker is trying to mix 32-bit and 64-bit binaries which obviously fails. So I used dumpbin /headers command and checked gzlib.obj and bitness of this object file is 64-bit.Then I checked other object files from the OpenSSL compilation land and see that their bitness is also 64-bit. Specifically, C files compiled by CC compiler (CL.EXE pointing to proper MS compiler capable to produce 64 bit binary) are compiled correctly into 64-bit bitness. The problem is caused by object files produced by NASM, reported libcrypto-shlib-aes-586.obj is indeed 32-bit.

Attempt

I tried to force NASM to produce 64-bit binary by passing -f win64 switch instead but this fails misserably with a lot of errors like:

        "nasm"  -f win64  -o crypto\aes\libcrypto-shlib-aes-586.obj crypto\aes\libcrypto-shlib-aes-586.obj.asm
crypto\aes\libcrypto-shlib-aes-586.obj.asm:972: error: instruction not supported in 64-bit mode
crypto\aes\libcrypto-shlib-aes-586.obj.asm:973: error: instruction not supported in 64-bit mode
crypto\aes\libcrypto-shlib-aes-586.obj.asm:974: error: instruction not supported in 64-bit mode
...

Looking into aes-586.S it seems to me, that this is really 32-bit code, using instructions like:

    push    ebp
    push    ebx
    push    esi
    push    edi

and setting bitness on section code use32 class=code align=64. I dig a bit deeper and I can see that all those assembly files are really generated by set of PERL scripts. There is PERLASM_SCHEME flag in which control dialect of the assembler but not really bitness, PERL scripts that generate assembly files are 32-bit. Which makes sense also based on their names, referring to 586 processor family.

but there is PERL file named /crypto/eas/asm/aesni-x86_64.pl suggesting there is 64-bit implementation, I just have not found any logic on Configure and generated makefile scripts which would drive its selection instead of picking up /crypto/eas/asm/aes-586.pl.

Which leads me to surprising conclusion – at least on Windows platform, it is not “possible” to build OpenSSL on 64-bit.

Well, I know, that in software everything is possible and I feel I am close so if someone can please point me to the correct direction how to resolve it, it would be great.