Working with ntfs

[Forester]

Hi utu,

I will get to your question but first I have to respond to Harry Kuhman's post (# 6) under
http://www.knoppix.net/forum/threads...h-installation.

Harry is a Sys Admin and I take what Sys Admin's say very seriously. I didn't mean to imply that (I think) accessing Windows partitions from Linux is perfectly safe.

Now ... this is relevant though at first it may not appear so.

My Knoppix 6.4.3 USB installation is less that 4 weeks old. The second bit of software I installed was a plug-in for Eclipse. The plug-in didn't work so I installed it again. This time the plug-in worked fine. Eclipse is like Windows (heck it will even run under Windows): you've a GUI, all you do is point and click and press return when the OK button appears. What could possibly go wrong ?

When I came to remaster I discovered that /UNIONFS/lost+found was not empty. It contained, I believe, files missing from the first installation attempt. So, the file system screwed up.

This was a timely reminder that even accessing Linux file systems under Linux is not perfectly safe.

Anything we do carries some risk. Risk has two halves: the probability of shit happening and how deep the shit is when it does happen.

I am happy using Linux file systems because my experience (going back over many years) is that the probability is low and the consequences not serious. All those years I used Windows 98 I was not happy because experience taught me the probability the file system was damaged after a power cut was quite high and I could loose important files.

I don't know ntfs all that well. When someone whose authority I tend to respect says what Harry said about serious consequences, I think "I'd better not do that again." I don't know and I don't know how Harry knows but I do know that if I start repeating that kind of thing I'm effectively spreading rumour. Just because a rumour gets repeated doesn't make it any more or less true.

Harry wrote that often corruption is not noticed until long after the damage is done. That makes proving cause and effect very difficult. My very first PC gave me lots of file system problems. Until I installed a UPS. I had circumstantial evidence, but no proof, that power brown-outs were the root cause of my problems.

Someone might try statistical analysis to correlate using Linux to access ntfs with later file system problems under Windows but I suspect all they would infer is that inexperienced people tend to make more mistakes than experience people and that reckless people are more likely to wreck things than cautious people.

I can and do approve of Harry reminding people to back up their file systems.

I also approve of this thread. Many people new to Knoppix may see ntfs as their first option when they need some extra disk space. They need to understand a little the risk they may be taking.

[utu]

@Forester

Thanks for this excellent response.
It is a good beginning of a longer discussion I hope will continue.

@KL522, Harry Kuhman, and Capricorny

When I stepped on KL522's thread, posts #'s 5.Forester, 6.Kuhman,
7.KL522, 8.Capricorny, and 9.KL522 seemed to be appropriate to comment on.
Each (seemed to me) was concerned with some ntfs-related fear or uncertainty.

I hope KL522, Kuhman and Capricorny will also participate here.

@krishna

Groucho Marx once remarked, "I've been thrown out of better places than this."
I've got that feeling. Keep up the great work of the Thread Police.

[krishna.murphy]

There are several problems with NTFS, and IMHO it should be avoided where possible. That said, M$oft being as ubiquitous as it is, it is frequently necessary to deal with it even if one chooses not to operate on/under/within their OS-s on a daily basis. Thus I am grateful that ntfs-3g is available, and included in Knoppix. My usage/reasoning is detailed below, but I am not a member of the "don't ever write to NTFS in Linux" camp - it's not SO bad, these days, and sometimes necessary (or at least convenient.)

I am using Knoppix version 6.2.0-DVD from the FAT32-formatted first partition of my hard drive, and there are no doubts that this imposes limitations. In addition to the 4GB of KNOPPIX itself, I have a persistent store of 2 GB, wherein all my system additons (e.g. Google Chrome) and some files kept in home, etc. (all under ext3) reside, leaving about 2GB of FAT32 storage for certain downloads, Windows-readable. Since this is the system my wife and I use to watch downloaded TV and movies (mostly), it follows that more storage for media is required. I fill this need by using slack space on TWO other NTFS partitions with large amounts of relatively dead storage. I often (daily) have several torrents running simultaneously (sometimes more), which can definitely ratchet up the processor load noticeably, especially as those partitions fill up and the files get more fragmented.

This brings us to the most serious drawback of the use of NTFS - it requires defragging, in some form or the other, relatively often. The Linux ext2/3/4 filesystems are relatively immune to fragmentation, and what does exist therein (always some, usually not much) is not a serious handicap and NEVER requires user (or admin) intervention, being automatic.

I have also used Knoppix copied to an NTFS (Windows) partition, and thereunder running, causing auto-mounting when booted. While this is an anecdotal report, and certainly NOT an endorsement of such a practice, I can report that the most serious trouble I had using that system on a daily basis was also processor load, perhaps due to allowing the automatic use of the Windows swap file. Experientially, I found that ntfs3g was quite capable, and robustly handled the access Knoppix required of it to fulfill the role of providing underlying filesystem support, and shares files with Windows with no trouble (unless I tried to hibernate Windows and then access files it thought were locked while it was asleep.)

So, on balance, while I am not thrilled with being forced, as it were, to use a substandard filesystem, and in the near future I expect to eliminate my daily NTFS-access needs, I am content for the time being with using ntfs3g. I don't lose ANY data, and if I keep an eye on the processor load (via the cute little graphic available on the taskbar called "CPU usage monitor") to avoid making the system unresponsive by overloading it, it works fine (sort of, anyway.)

Cheers!
Krishna

[Forester]

@ Kishna

Thanks for telling us a little of how you use Knoppix. I see posts by Junior Members that don't say how they are trying to use Knoppix with replies from Senior Members saying you can do this and that and the other. I read these replies and I think "that's all very well but ...".

I am using a USB install of the Knoppix 6.4.3 EN DVD at work because I've been tasked to write some scripts for Linux servers and I don't agree that 'cross-development' from Windows is the better option. I dare not repartition the hard drive or anything: it isn't mine. I cannot boot Knoppix: my boss requires I use corporate e-mail and messaging services (so he can yell at me without embarrassment). I want a portable solution because next week I may be using a different PC and next month I may be working for a different client. My compromise was to install VirtualBox on Windows 7 and run Knoppix in a virtual machine. There are advantages as well as disadvantages.

At work I don't use ntfs-3g even though I use the ntfs file system. I have virtual disks, which are large ntfs files on the outside but Linux file systems on the inside. When Knoppix accesses these file systems, the (VirtualBox) driver calls Windows and Windows accesses the nfts file system. That's about as safe as it gets.

One nice thing about this arrangement is that I can keep all the work stuff that probably should not leave the building at work but still take Knoppix and any personal stuff home. I do that because I don't have the time at work to learn as much as I want to. I certainly can't be messing around remastering Knoppix on company time.

At home, you may have guessed, I run Linux, not Windows and I have no real use for ntfs-3g. It was not always that way. At one time I had a home network with several PC and a laptop running Windows 2000. I used Windows shares to make one large network file system. When I first installed Linux at home I naturally used a dual-boot arrangement. In order to 'share' files between Linux and Windows I learnt how to set up a samba network. When a PC was running Linux, I used ntfs-3g to mount the old Windows shares so that other PCs could access data over the samba network just as if the PC were still running Windows. Since I didn't really expect to use this all that often, I used the auto mounter to mount the ntfs file systems only when they were actually in use.

The solution is 'technically' sound but, I suppose, how 'safe' it is depends entirely on how safe ntfs-3g is. I was using this several years ago when it was first included in an Ubuntu distributed it (I don't use Ubuntu any more but that's a different story). I never heard of ntfs-3g being unreliable. I never had a problem but I never used it all that much. The auto mounter and the samba network are still in place but they only ever get used when I do a 'fire drill' to make sure they are still working.

I cannot say whether ntfs-3g is safe or not but I have messed around with different file systems and operating systems and machines and stuff to have a fair idea of what the options are and their pros and cons.

[Forester]

NTFS

So, on balance, while I am not thrilled with being forced, as it were, to use a substandard filesystem, ... I am content for the time being with using ntfs3g.

Hmm .... I fear that you may be being a tad libellous in branding (Windows) NTFS a sub-standard file system for I suspect much of what you complain about is a result of (the way you are) using (the Linux) ntfs-3g driver. Tut, tut. Shame on you.

This brings us to the most serious drawback of the use of NTFS - it requires defragging, in some form or the other, relatively often.

From the one time I read up on Windows NTFS, I remember something along the lines of "A whole generation grew up defragging FAT file systems under Windows 98 and continue religiously defragging NFTS file systems when there really is no need to." I've defragged NTFS file systems but only as a prelude to cuttings the b******ds down to size so I could use the disk space for something useful.

It is not that fragmentation does not occur on NTFS: it is more that fragmentation has no adverse affect on performance. If you do really have performance problems due to fragmentation, it may be an ntfs-3g issue.

I often (daily) have several torrents running simultaneously (sometimes more).

Think about what you're doing. You are down loading a large file in small chunks in random order and, you admit, sometimes you are down loading more than several at once. I think fragmentation is inevitable.

debian@aspen:~$ sudo filefrag /media/iso/KNOPPIX_V6.2.1DVD-2010-01-31-EN/KNOPPIX_V6.2.1DVD-2010-01-31-EN.iso
/media/iso/KNOPPIX_V6.2.1DVD-2010-01-31-EN/KNOPPIX_V6.2.1DVD-2010-01-31-EN.iso: 106113 extents found

That's a torrent download I did recently under Linux to an XFS file system. I quote Wikipedia: "XFS is particularly proficient at handling large files and at offering smooth data transfers." 106 133 extents: looks pretty fragmented to me. My bitTorrent client claims the download had less that 4 500 chunks !

I keep an eye on the processor load ... to avoid making the system unresponsive by overloading it, it works fine (sort of, anyway.)

The processor load surprised me at first.

For me, just one torrent download can max out my broad band. Not only does my browser become unresponsive, but so do the browsers anyone else might be using elsewhere in the house. So I don't: I limit the bandwidth of individual torrents but I can also limit the grand total.

If I had fast enough broadband, I'd expect the disk access to max out before the processor load (same effect - system goes treacle). Remember you're using ntfs-3g. That's a File System In User Space (aka FUSE) driver. No way is it going to be as efficient as a lean, mean Linux Kernel driver or the genuine Windows kernel driver from M$soft. Furthermore, since NTFS is a complex file system, I expect the design goals for ntfs-3g were correct operation rather than efficient use of processor or memory. Now I'm not so surprised.

... in the near future I expect to eliminate my daily NTFS-access needs ...

Sounds like you've a serious habit to kick.

If it were me, I'd free up space on the NTFS partitions, defrag 'em and shrink 'em and use the space free for a new partition with a file system better suited to the job. I'd do some research starting with jfs and xfs.

I suspect that even using the space freed for another NTFS partition dedicated to torrent downloads would give better performance. Also, if Harry's nightmare comes true, all you lose is a few TV recordings, not them and your OS, personal correspondence, bank records and the e-mail address of that cute little ....

Cheers!