Hi Jamie
This does look as though this is where the format is set but when I tested the date was able to be input in a variety of date formats and the spreadsheet would convert this to the equivalent five digit number for the date which was able to be imported into a custom date field.
I think the sever date format needs to match the spreadsheet regional date format so if using a server in US you may have to change the spreadsheet to US date too.