Let's consider what modern Neanderthal genetics is actually telling us that everyone is afraid to come out and say including the evidence for sexually asymmetric mating.
We now know that Neanderthals were a race of white-skinned blonds living in Europe that were invaded by hoards of black Homo sapiens coming out of Africa. It's well known that black guys like to have sex with blond white girls! :-) Seriously most men do tend to prefer blond white girls, which is why millions of brunette girls dye their hair blond. Result over time is massive black on blond mating that essentially diluted Neanderthals out of existence and resulted in modern Europeans with 2-3% of Neanderthal genes including those for white skin and to some extent lighter hair.
The obvious conclusion here is that H. sapiens men preferentially had sex with Neanderthal females but Neanderthal men were less likely to have sex with H. sapiens females
This raises a number of interesting questions. Did Neanderthal men find H. sapiens females unattractive, or was it just a case of H. sapiens being the dominant species when it came to mating? Sort of reminds me of mating between black slaves and their white masters in early America in reverse. Many black females had children by their white masters but almost no white females had children by their black male slaves.
So Neanderthals never died out, they were simply assimilated into the H. sapiens genome by preferential mating because those Neanderthal girls were so sexually attractive! This is the correct explanation in my opinion. So Neanderthals didn't die out, they weren’t killed off or out competed by modern humans or their diseases; they were just absorbed into modern Europeans who now have 2-3% Neanderthal genes as a result.
How many times must I say this before it sinks in? The Neanderthals didn't go anywhere. They were simply absorbed into our own gene pool by interbreeding. We are 2-3% Neanderthal. Assume there were 33x as many H. sapiens than Neanderthal and that they all interbred over a few thousand years. You get exactly what we see now, a single species of H. sapiens with 2-3% Neanderthal ancestry.