You get this grey color inside legend keys because you use
stat_smooth() that as default makes also confidence interval
around the line with some fill (grey if
fill= isn't used
One solution is to set
stat_smooth() if you don't need the confidence intervals.
+stat_smooth(method = "loess", formula = y ~ x, level=0, size
aes(group = gender, colour=gender),se=FALSE)
Another solution is to use the function
override.aes= to remove fill from the legend but keep
confidence intervals around lines.