I did some reading and found this interesting discussion regarding why there is such a difference
Yes. It is known that some workloads, mostly involving malloc, and heavy
amounts of C string operations benchmark poorly verses glibc.This is largely because the security hardening features of musl and
Alpine are not zero cost, but also because musl does not contain
micro-architecture specific optimizations, meaning that on glibc you
might get strlen/strcpy type functions that are hand-tuned for the exact
CPU you are using.In practice though, performance is adequate for most workloads.