Article Title: Using exFAT as a common filesystem Article Year: 2019 Article Status: Static Author: Robert Walliczek Author E-mail Contact: galaxyverge+how2@pm.me ------- Summary ------- The exFAT filesystem lets you store data that can be read and written using Linux, Windows 7 and newer, and Mac OS X Snow Leopard and newer. Using diskpart in Windows, erase and format the hard drive as GPT with its only partition exFAT of sector size 4096 bytes. Check and repair unmounted exFAT partitions using fsck_exfat in Snow Leopard or Linux. ------------ Instructions ------------ ) Turn off the computer. ) Connect the drive to the computer hardware. ) Turn on the computer and boot into Windows. - If you get a BSOD stop error as a result of a zero sector size, turn off the computer and boot into a Linux or Mac operating system, run fsck_exfat to repair the drive, then copy any files from it and erase the drive. Alternatively, boot into a Windows installation disc and erase the drive. - If desired, before erasing the drive, copy files from the drive. ) Run cmd in administrator mode. ) Issue: diskpart list disk ) Visually identify the new disk and its number. In these instructions, it is disk 1. ) Issue: select disk 1 list disk ) Visually confirm that the selection * is at the left of the intended disk entry row. If it is not, then go back and select the correct disk. ) Issue: list partition - If there are partitions listed, select and delete them, preferably in reverse order. Issue "list partition" to visually identify the selected partition. In these instructions, there are two partitions already. Partition 1 is an EFI system partition and partition 2 is a primary data partition. ) Issue: select partition 2 list partition delete partition select partition 1 list partition delete partition override ) Issue: list disk - If the selected disk does not have * next to it within the GPT column, it is a Master Boot Record disk and should be converted to GPT if it is larger than 128 MB. ) Issue: convert gpt ) Issue: create partition primary assign list volume ) Visually identify the new volume's drive letter. In these instructions, it is F. It should already have * at its left to indicate that it is selected. You did not create the volume using "create volume". ) Devise a label for the drive. In these instructions, it is the Greek alphabet letter name "Phi". ) Issue: format fs=exFAT label="Phi" unit=4096 quick override exit ----- Notes ----- - Format a drive into exFAT for compatibility with common operating systems. This will erase all its files. - exFAT is suitable for data storage, not operating system file installation or any other situation where user file permissions should be recorded. For example, install the operating systems onto partitions of an SSD using suitable filesystems, then use a separate HDD with a single exFAT partition for cross-platform common data storage. - exFAT is intended for solid state drives and works with hard disk drives also, but seems fragile because it is not perfectly handled by the operating systems and lacks error checking features to mitigate data corruption. - exFAT is owned by Microsoft. - exFAT compatibility: - On Linux CentOS, install the exfat-utils and fuse-exfat EPEL packages. - On Mac, use updated Snow Leopard or newer. - On Windows, use Windows 7 or newer. Windows XP x64 and Windows Server 2003 have KB955704 to add exFAT support. Microsoft stopped hosting that update for Windows XP x86. - These instructions have you use Windows to set up the drive. - On a Mac, specify the startup disk at boot by turning on or restarting the computer and holding the Option key after the keyboard initializes -- after the startup chime sounds -- then selecting the desired disk on the EFI screen using the arrow and enter keys. - List connected disks: - On Linux, issue: fdisk -l - On Mac, issue: diskutil list - On Windows, issue: diskpart list disk exit - Do not use chkdsk on exFAT volumes within Windows 7 and older. Microsoft KB2843376 for Windows 7 is no longer listed and does not work. That hotfix is still available for Windows 8 and Windows Server 2012. - Do not repair exFAT volumes using Disk Utility on Mac; diskutil does not properly use fsck_exfat. - Check and repair exFAT volumes by un-mounting the partition and running fsck_exfat on Linux or Mac OS X Snow Leopard operating systems. If fsck_exfat is run on a mounted partition, it will open the partition in read-only mode and fail with write errors while checking the active bitmap. If fsck_exfat asks, do not mark the volume as clean. An error in fsck_exfat is not necessarily a problem with the volume. Example exFAT partition checking using Terminal on Mac OS X Snow Leopard: $ diskutil list /dev/disk0 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *1.0 TB disk0 1: EFI 209.7 MB disk0s1 2: Apple_HFS Iota 50.0 GB disk0s2 3: Apple_HFS Eta 150.0 GB disk0s3 4: Microsoft Basic Data BOOTCAMP 724.2 GB disk0s4 /dev/disk1 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *2.0 TB disk1 1: EFI 209.7 MB disk1s1 2: Apple_HFS Beta 2.0 TB disk1s2 /dev/disk2 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *2.0 TB disk2 1: EFI 209.7 MB disk2s1 2: Apple_HFS Theta 2.0 TB disk2s2 /dev/disk3 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *4.0 TB disk3 1: Microsoft Basic Data Phi 4.0 TB disk3s1 $ diskutil unmount /dev/disk3s1 Volume Phi on disk3s1 unmounted - If it says that the resource is busy, open Disk Utility and verify and repair the partition's disk /dev/disk3. $ sudo fsck_exfat -y /dev/disk3s1 ** Checking volume. ** Checking main boot region. ** Checking system files. ** Volume name is Phi. ** Checking upper case translation table. ** Checking file system hierarchy. ** Checking active bitmap. ** Rechecking main boot region. ** Rechecking alternate boot region. ** The volume Phi appears to be OK. - The more files a volume has, the longer it takes to check its file system hierarchy. - The larger a volume is, the longer it takes to check its active bitmap. $ diskutil mount /dev/disk3s1 Volume Phi on /dev/disk3s1 mounted - exFAT does not support file ownership permissions such as with chmod and chown. FAT file modification times have a precision of 2 seconds. Transfer files in bulk to a mounted exFAT volume using rsync. - Example dry run using option -n: $ sudo rsync -rulthWP --inplace --compress-level=0 --no-p --no-o --no-g --modify-window=1 -n "/Volumes/Eta/Users/silent" "/Volumes/Phi/" building file list ... 299513 files to consider silent/ [long list of files] sent 9.43M bytes received 1.80M bytes 641.83K bytes/sec total size is 85.33G speedup is 7596.81 - If --compress-level=0 is unrecognized, try --no-compress. - Remove the -n option to actually transfer files. - Insert option -q for quiet mode if automating as with cron so that only errors are displayed. - You could issue this through nohup (no hang-up) so it runs in the background. Simply prepend "nohup" to the command. The standard output would be appended to a nohup.out file in the current directory, and errors would be shown in the current terminal while the terminal pipe is still active. Control-C will interrupt the pipe and terminate the nohup command. Closing the terminal window will interrupt the terminal pipe without interrupting nohup. $ nohup sudo rsync - Change the current directory to relocate nohup.out. You can isolate a change of directory to its statement by parenthesizing the statement: $ (cd /Volumes/Phi/;nohup sudo rsync) ----- Links ----- Unix Commands: https://bhami.com/rosetta.html#disks Microsoft KB2843376, chkdsk exFAT compatibility for Windows 7: https://support.microsoft.com/en-us/kb/2843376 Microsoft Update Catalog entry for KB2843376, chkdsk exFAT compatibility now only available for Windows 8 and Windows Server 2012: https://support.microsoft.com/en-us/kb/955704 Microsoft KB955704, exFAT compatibility for Windows XP, now only x64, and Windows Server 2003: https://support.microsoft.com/en-us/kb/955704 Brief definitions of a Macintosh file's data fork, resource fork, finder info, and extended attributes, related to rsync -E: https://support.grouplogic.com/?p=1496 exFAT (DOS) timestamp format, "Seconds are stored in two-second increments": https://shullich.blogspot.com/2009/12/time-stamp-format.html ------------------- Discussion Segments ------------------- July 23, 2019 Nathan: In troubleshooting a problem with one of my hard drives, namely repairing a 5TB exFAT volume following damage during a power outage, especially after foolishly running chkdsk on said volume in Win7x64, I happened upon an article/document recently which you wrote concerning the exFAT filesystem which led me to seek your advice. I had previously read about Microsoft KB2843376 for Windows 7, but in looking into it further, came to the same realization you noted in the article: "Do not use chkdsk on exFAT volumes within Windows 7 and older. Microsoft KB2843376 for Windows 7 is no longer listed and does not work." Unfortunately, I have already run the tool, and my drive is now read only, and is showing up with 13GB free (should be ~2TB free) and many of the files are no longer showing up. In your article, you advise: "Check and repair exFAT volumes by un-mounting the partition and running fsck_exfat on Linux or Mac OS X Snow Leopard operating systems." My question is: Given that I have already run chkdsk in Win7x64, should I still run fsck_exfat next? Or would something else (such as running chkdsk on a Windows 8 machine with the KB2843376 hotfix) be more appropriate? Thank you! Author: Hi! I'm glad you got in touch with me and I'm happy to help however I can. I only use up to Windows 7 and have no access to Windows 8, so I can't speak for that, but if you have access to an updated Snow Leopard, I'd say use that. Otherwise, since exFAT really is Microsoft's proprietary format, I'd say go with whatever new or updated version of Windows you have. As long as the files on the drive are not overwritten, even an incorrect run of the diagnostics software like chkdsk or fsck_exfat or diskutil should be OK. Sorry for not having more concrete solutions, but what I wrote in the article is what should work -- if you have access to that particular operating system. A more involved approach would be to use a free linux-based OS in a virtual machine, but again, it's Microsoft's file system format, so it's probably best to stick with Microsoft first and see if that works. I'd be happy to discuss this further, and you can let me know how it goes if you like. Author: I am using exFAT with at least one 4TB internal HDD. I kind of don't like it, and I added another paragraph to the notes section in the article: - exFAT is intended for solid state drives and works with hard disk drives also, but seems fragile because it is not perfectly handled by the operating systems and lacks error checking features to mitigate data corruption. The only issues I've had so far are those you've had and that the drive gets marked dirty and placed into read-only or unrecognized state, where I must then boot into Snow Leopard and run fsck_exfat on it. So don't be dismayed or afraid of losing the data on the drive with diagnostic tool failures. Try Windows 8's chkdsk, and if that doesn't work, I'd say go ahead and use fsck_exfat on a Snow Leopard retail disk installation that is updated to 10.6.8, which should work. Otherwise, you could do a CentOS 7 minimum install on a small partition on your main hard drive and boot into that from the EFI or BIOS screen and install the extra fuse and other linux packages via yum. For more on that linux setup, check online -- I think stack overflow has an article on exfat and linux. Nathan: Yes, I am a bit of a novice and chose exFAT on a bit of a whim, thinking it might come in handy if/when I decided to transition to some version of Linux as my primary OS as Windows drops Windows 7 support, but I was not really up to speed on the problems it could introduce... I will try the Windows 8 chkdsk route and let you know how it goes sometime in the next few days or over the weekend as I don't have it immediately available. What are your current thoughts on a filesystem for such a drive? Would you just use NTFS? Author: I have not seen documentation of any filesystem other than exFAT that is supported by Mac, Windows, and some Linux variants. For common file storage, exFAT is useful; I am stuck with it because I really do use it for multi-platform common storage: I download files from The Internet to the hard drive and process those files using software on Windows and Mac [mostly the command-line youtube-dl program from youtube-dl.org to download YouTube videos, and to back up my other hard drives that contain Windows and Mac applications and files... where I don't care much about details like modification date precision and file permissions], and I store encrypted password databases and other such things that are used by both OS boot partitions. Certainly, there are alternatives, including porting superior filesystems, but that either requires dangerously or prohibitively tinkering with the OS kernel, or separate software... which means (much perpetual) work, money, or both, and possibly inferior performance, a high risk of data loss, and possible hardware damage. There are horror stories and I have lost entire hard drives of data myself even from trusted brands like Parallels. The best alternative I can suggest to a current cross-platform filesystem at this point is to use the free VMWare Workstation Player on Windows and Linux, and the commercial VMWare Fusion on Mac, to make a small Linux VM on whatever hard drive and use its VMWare Tools to share folders such that you can have something like an XFS filesystem in the VM used as common file storage. This would mean running the VMWare program when you log into the host Windows or Mac operating system. This can also get you started on the Linux situation since you mentioned abandoning Windows 7 when Microsoft does. I had the same idea, but perhaps latently. Nathan: That seems like potentially a clever solution... I am not sure, however, if I fully get how it would work, although it could just be due to my own ignorance. I apologize if this is a stupid question, but if you did that, wouldn't the VMWare image just be resting on top of the underlying filesystem of the physical drive? In other words, if (as just an example) I created the virtual machine on a 4TB exFAT drive, but set up the Linux VM to use XFS, wouldn't it be just as likely (or perhaps more likely?) to become corrupted? Or am I misunderstanding what you are suggesting? Also, would it make sense to do what you suggested except to use an OpenSUSE VM with Btrfs rather than XFS? Author: Look here: https://en.wikipedia.org/wiki/Comparison_of_file_systems If you were using VMWare Fusion for Mac, you would do a bootcamp partition on the new hard drive. From what you've said, I gather you are using Windows 7, Windows 8, and the Linux variant you mentioned. Whatever you find is most suitable with your desired installed OS that will be running on the VM is fine. See the comparison of file systems link. With Linux, you would want to write the Linux OS installer as a bootable CD or flash drive, back up whatever files you have on the new HD to somewhere else, then erase that new hard drive and install the linux OS, which would typically install the OS in one partition and have a separate "/home" directory mount point as the other partition for the rest of the data, which can be of whatever filesystem is most compatible with that Linux OS. Then you would have a situation where you use VMWare Fusion for Mac or VMWare Workstation Player on Windows to make a Virtual Machine file to identify the contents of that drive as a linux OS, and you would set up the virtual machine settings to share the whole partition -- you could maybe share it both ways as a network drive, or however else is available, but you might only be able to use the VM to transfer files onto the disk. You'd therefore want to use a lot of integration options such as sharing the clipboard and sharing at least your Windows User folder with the VM guest OS. Alternatively, you could create the VM with a virtual disk file that increases in size or is preallocated with the maximum available disk space left on the new hard drive, then go ahead and transfer over the files from the drive to the virtual disk on the same drive using the installed OS. That way you don't have to erase the new hard drive, but that puts all your files into the virtual disk file(s) so that even if Windows could have read access or write access to the desired filesystem on the drive, you would be depending upon being able to access the files on the drive using a VMWare product. Let me know if the chkdsk works on Windows 8. Author: Actually, with the first option I mentioned, you would run VMWare Workstation Player in Windows, install the Linux OS into a virtual machine, then use the new HD as a shared folder using whatever filesystem is compatible with Linux and Windows (or maybe only the host or guest OS?). You could keep using exFAT by installing the linux OS's Fuse drivers or whatever makes exFAT compatible with linux, or you could experiment with another filesystem -- but I'd back up the files that are on the hard drive anyway if you are planning to mess around with partitions. Author: Btrfs has WinBtrfs: https://github.com/maharmstone/btrfs