A brief field report in the Feb 2014 Light Railways has this to say:
A passing visit to this 240m-high mount (above sea level), near Derrinallum, found the gate closed – it transpires that it is only open on the first Sunday, monthly, from 1pm until 4pm. In 1912/13 volcanic scoria was quarried for railway ballast by the Ways & Works Department of the Victoria Railways (VR) during the construction of the Gheringhap (near Geelong) to Maroona (near Ararat) railway – a distance of one hundred miles, and the last major broad gauge railway to be built in Victoria.
A branch railway, about two kilometres in length, was built south from near Derrinallum station to the northern base of Mt Elephant, where a quarry was opened. The quarry is easily visible on Google aerial photos, which seems to show a road on the railway formation between the main road and the quarry. However, all is not as it seems as, for the first 200m after entering the property, the access road is paralleled by the railway formation on a 2m-high embankment.