Why is my assembler subroutine for Shellsort not sorting the array when integrated with a C program
In a group work where we are trying to integrate an assembler subroutine for sorting numbers
Shellsort with a C program, the sorting algorithm has been tested as a standalone program and works correctly, but we are encountering issues when linking it with the C code. Specifically, the output array contains the original numbers in the same order instead of being sorted.
We are passing the pointer to the array through the RCX register, which is the correct method for passing function parameters in Intel 64. Since the C source code was provided by our lead, we suspect there might be an issue in our assembly code.
We are using this function signature in the C code to call the assembly routine:
We read the input from a file into the array:
We call the sorting routine like this:
So after running the program using NASM 2.15 assembler and GCC 11.2.0 compiler on the x86-64 architecture with Ubuntu 22.04 OS, the output array contains the numbers in the original order instead of being sorted.
I expected the assembler routine to modify the array in place and return the sorted numbers.
The program runs without any errors, but the output file contains the same numbers in their original order, so the sorting routine didn’t work as expected.
Could someone help identify what’s wrong with the assembler code, especially how we’re passing the array and length through registers using RCX and RDX? Are there any issues with how we’re modifying the array in the Shellsort routine?
1 Reply