Hi, Ben
CSV files can be funny and I had a lot of problems with Totara Sync when I first started using it at our company.
Couple of things I would check would be the Element and Source settings for users.
If 'Source contains all records' is set to 'No', I believe you need to apply the delete flag in your CSV file. I always add a zero (0) to the deleted and timemodified columns.
Check the Source settings and make sure the character encoding is set to 'UTF-8' and and delimiter is set to 'comma'. I think this is set by default though.
I would try removing your double-quotes, too. I don't use any double-quotes in my CSV files and it uploads fine.
If you're using Excel to generate your CSV, it can throw in extra commas sometimes from what I've noticed, so watch out for that one.
In some cases, double-qoutes are needed though and it's worth doing some reading up on CSV file formatting. It certainly helped me understand it better.
Cheers
Tom