I live very close to the Stony Point line and have done for over 11 years.
The trains can be heard more or less clearly at any time depending largely on prevailing wind direction and ambient noise.
All trains whistle for the various level crossings and the passenger service usually whistles to warn of moving away from station stops as well.
The Long Island steelies are big and quite heavy trains and take a fair amount of grunt to lift over the summit at Langwarrin. There is an up run at around 12.45am which I can often hear climbing all the way from Baxter and until it cuts off to take the curve approaching Frankston. That may be for around 10 minutes though it can seem like more.
Sometimes that's reassuring and at other times it wakes me annoyingly from sleep. Some nights I don't hear it at all. That isn't necessarily related to weather conditions though it can sound as though it's coming into the house on a quiet night with a light southerly breeze.
I wouldn't be unduly concerned about living close to the line. There's little activity after 8pm except for the Friday night late trains so the steelies does stand out when everything else is quiet but I don't find it a nuisance.