The raw reading and writing speed depends mainly on:

• the performance of the used hard disk drives
• the network capacity  its topology

The better the performance of the hard drives used and the better throughput of the network, the higher performance of the whole system.
 
Our in-house commodity servers (which additionally are utilized by other applications ) with simple gigabyte Ethernet network have achieved  a petabyte-class installation on Linux (Debian) with goal=2.  We have write speeds of about 20-30 MiB/s and reads of 30-50MiB/s. For smaller blocks the write speed decreases, but reading is noteffected that much.

A similar FreeBSD based network has observed slightly better writes and worse reads, giving a slightly better performance overall. 

Generally speaking, it does not. The goal setting can influence the reading speed under certain conditions. For example, when a file has goal set to two or more, reading the same file at the same time by more than one client would be faster due to the ability for each client to utilize the load balancing effect of the file existing across more than one data server. But in practice, it is very rare in real world situations where several computers would read the same file at the same moment; therefore, the goal setting has little influence on the reading speeds.

Similarly, the writing speed is negligibly influenced by the goal setting. 

All read operations are parallel - there is no problem with concurrent reading of the same data by several clients at the same moment.

In our environment (ca. 500 TiB, 25 million files, 2 million folders distributed on 26 million chunks on 70 machines) the usage of chunkserver CPU (by constant file transfer) is about 15-20% and chunkserver RAM usually consumes about 100MiB (independent of amount of data). The master server consumes about 30% of CPU (ca. 1500 operations per second) and 8GiB RAM. CPU load depends on amount of operations and RAM on the total number of files and folders, not the total size of the files themselves. The RAM usage is proportional to the number of entries in the file system because the master server process keeps the entire metadata cached in memory for performance.

You can add / remove chunk servers on the fly. But keep in mind that it is not wise to disconnect a chunk server if this server contains the only copy of a chunk in the file system (the CGI monitor will mark these in orange). You can also disconnect (change) an individual hard drive. The scenario for this operation would be:

1. Mark the disk(s) for removal (see How to mark a disk for removal?)
2. Restart the chunkserver process
3. Wait for the replication (there should be no “undergoal” or “missing” chunks marked in yellow, orange or red in CGI monitor)
4. Stop the chunkserver process
5. Delete entry(ies) of the disconnected disk(s) in 'mfshdd.cfg'
6. Stop the chunkserver machine
7. Remove hard drive(s)
8. Start the machine
9. Start the chunkserver process

If you have hotswap disk(s) after step 5 you should follow these:

• 6. Unmount disk(s)
• 7. Remove hard drive(s)
• 8. Start the chunkserver process

If you follow the above steps work of client computers would be not interrupted and the whole operation would not be noticed by MooseFS users.

When you want to mark a disk for removal from a chunkserver you need to edit the chunkserver's mfshdd.cfg configuration file and put an asterisk '*' at the start of the line of the disk that is to be removed. For example, in this mfshdd.cfg we have marked "/mnt/hdd" for removal:
/mnt/hda
/mnt/hdb
/mnt/hdc
*/mnt/hdd
/mnt/hde

After changing the mfshdd.cfg you need to restart chunkserver (mfschunkserver restart).

Once the disk has been marked for removal and the chunkserver process has been restarted, the system will make an appropriate number of copies of the chunks stored on this disk, as to maintain the required "goal" number of copies.

Finally, before the disk can be disconnected you need to confirm there are no "undergoal" chunks on the other disks. This can be done using the CGI Monitor. On the "Info" tab select "Regular chunks state matrix" mode.

During our research and development we also observed the problem of slow metadata operations. We decided to aleviate some of the speed considerations by caching the file system structure in RAM on the metadata server. This is why metadata server has increased memory requirements. The cached metadata is frequently flushed out to logging files on the master server. Additionally, the metadata logger server(s) would also frequently receive updates to the metadata structure and log these to their file systems.

Every chunkserver sends its own disk usage increased by 256MB for each used partition/hdd, and a sum of these master sends to the client as total disk usage. If you have 3 chunkservers with 7 hdd each, your disk usage will be increased by 3*7*256MB (about 5GB). In practice, this is not usually a concern for example when you have 150TB of HDD space.  

There is one other thing. If you use disks exclusively for MooseFS on chunkservers df will show correct disk usage, but if you have other data on your MooseFS disks df will count your own files too.  

If you want to see usage of your MooseFS files use 'mfsdirinfo' command.

The system was initially designed for keeping large amounts (like several thousands) of very big files (of tens of gigabytes) and has a hard-coded chunk size of 64MiB and block size of 64KiB. Using a consistent block size helps improve the networking performance and efficiencies, as all nodes in the system are able to work with a single 'bucket' size. That’s why even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header. The whole transfer which takes place in the system is done in blocks of 64KiB. However it doesn’t have any impact on the performance. (A normal file system will typically also use some degree of block read-ahead, while sometimes will fetch some superfluous data).

The issue regarding the occupied space of a small file stored inside a MooseFS chunk is really more significant, but in our opinion it is still negligible. Let’s take 25 million files with a goal set to 2. Counting the storage overhead, this could create about 50 million 69 KiB chunks, that may not be completely utilized due to internal fragmentation (wherever the file size was less than the chunk size). So the overall wasted space for the 50 million chunks would be approximately 3.2TiB. By modern standards, this should not be a significant concern. A more typical, medium to large project with 100,000 small files would consume at most 13GiB of extra space due to this internal chunk fragmentation.

So it is quite reasonable to store source code files in a MooseFS system, either for active use during development or for long term reliable storage or archival purposes.

Perhaps the larger factor to consider is the comfort of developing the code taking into account the performance of a network file system. When using MooseFS (or any other network based file system such as NFS, CIFS for that matter) for a project under active development, the network filesystem performance may not able to perform file IO operations to the same speed that would be possible with a directly attached regular hard drive.

Some modern integrated development environments (IDE) such as Eclipse make frequent IO requests on several small workspace metadata files. Running Eclipse with the workspace folder on a MooseFS file system (and again, with any other networked file system) will experience slightly slower user interface performance, than running Eclipse with the workspace on a local hard drive.

This is more of a feature of the way these IDE products are designed, utilizing the file system as its active model and not employing memory based caching. So you may need to evaluate for yourself if using MooseFS for your working copy of active development within an IDE is right for you.

In a different example, using a typical text editor for source code editing and a version control system such as Subversion to check out project files into a MooseFS file system, does not typically observe any performance degradation. The IO overhead of the network file system nature of MooseFS is offset by the larger IO latency of interacting with the remote Subversion repository. And the individual file operations (open, save) do not have any observable latencies when using simple text editors (outside of complicated IDE products).

A more likely situation would be to have the Subversion repository files hosted within a MooseFS file system, where the svnserve or Apache + mod_svn would service requests to the subversion repository and users would check out working sandboxes onto their local hard drives (which possibly would not be using MooseFS).

Yes there is checksumming done by the system itself. We thought it would be CPU consuming but it is not really. Overhead is about 4B per a 64KiB block which is 4KiB per a 64MiB chunk (per goal).

The most important factor is RAM of mfsmaster machine, as the full file system structure is cached in RAM for speed. Besides RAM mfsmaster machine needs some space on HDD for main metadata file together with incremental logs.  
The size of the metadata file is dependent on the number of files (not on their sizes). The size of incremental logs depends on the number of operations per hour, but length (in hours) of this incremental log is configurable.  

1 million files takes approximately 300 MiB of RAM. Installation of 25 million files requires about 8GiB of RAM and 25GiB space on HDD.

MooseFS does not immediately erase files on deletion, to allow you revert the delete operation. Deleted files are kept in the trash bin for the configured amount of time before they are deleted.  

You can configure for how long files are kept in trash and empty the trash manually (to release the space). There are more details in Reference Guide in section "Operations specific for MooseFS".  

In short - the time of storing a deleted file can be verified by the mfsgettrashtime command and changed with mfssettrashtime.

Yes!

No. File data is divided into fragments (chunks) with a maximum of 64MiB each. The value of 64 MiB is hard coded into system so you cannot modify its size. We based the chunk size on real-world data and determined it was a very good compromise between number of chunks and speed of rebalancing / updating the filesystem. Of course if a file is smaller than 64 MiB it occupies less space.

In the systems we take care of, several file sizes well exceed 100GB with no noticable chunk size penalty.

Let's briefly discuss the process of writing to the file system and what programming consequences this bears.

In all contemporary filesystems, files are written through a buffer (write cache). As a result, execution of the write command itself only transfers the data to a buffer (cache), with no actual writing taking place. Hence, a confirmed execution of the write command does not mean that the data has been correctly written on a disc. It is only with the invocation and completion of the fsync (or close) command that causes all data kept within the buffers (cache) to get physically written out. If an error occurs while such buffer-kept data is being written, it could cause the fsync (or close) command to return an error response.   

The problem is that a vast majority of programmers do not test the close command status (which is generally a very common mistake). Consequently, a program writing data to a disc may "assume" that the data has been written correctly from a success response from the write command, while in actuality, it could have failed during the subsequent close command.

As far as MooseFS is concerned – first, it's write buffers are larger than classic file systems (an issue of efficiency); second, write errors may be more frequent than in case of a classic hard drive (the network nature of MooseFS introduces some additional error-inducing situations). As a consequence, the amount of data processed during execution of the close command is often significant and if an error occurs while the data is being written [from the close command], this will be returned as an error during the execution of the close command. Hence, before executing close, it is recommended (especially when using MooseFS) to perform an fsync operation after writing to a file and then checking the status of the result of the fsync operation. Then, for good measure, also check the return status of close as well.

NOTE! When stdio is used, the fflush function only executes the "write" command, so correct execution of fflush is not sufficient to be sure that all data has been written successfully – you should also check the status of fclose.

The above problem frequently occurs when redirecting a standard output of a program to a file in shell. Bash (and many other programs) do not check the status of the close execution. So the syntax of "application > outcome.txt" type may wrap up successfully in shell, while in fact there has been an error in writing out the "outcome.txt" file. You are strongly advised to avoid using the above shell output redirection syntax when writing to a MooseFS mount point. If necessary, you can create a simple program that reads the standard input and writes everything to a chosen file, where this simple program would correctly employ the appropriate check of the result status from the fsync command. For example,  "application | mysaver outcome.txt", where mysaver is the name of your writing program instead of application > outcome.txt.

Please note that the problem discussed above is in no way exceptional and does not stem directly from the characteristics of MooseFS itself. It may affect any system of files – only that network type systems are more prone to such difficulties. Technically speaking, the above recommendations should be followed at all times (also in cases where classic file systems are used).

Currently MooseFS imposes a maximum file size limit of 2 TiB (2,199,023,255,552 bytes). However we are considering removing this limitation in the near future, at which point the maximum file size will reach the limits of the operating system, which is currently 16EiB (18,446,744,073,709,551,616 bytes).

If you normally installed the system using "make install", you should be able to just to run "mfscgiserv" on the master machine:

/usr/local/sbin/mfscgiserv

You can also specify the HTTP port number to use:

/usr/local/sbin/mfscgiserv -P 9425

And then navigate your browser to the URL of the master server and this port, such as:

http://MASTER_IP:9425/

Can I run a mail server application on MooseFS? Mail server is a very busy application with a large number of small files - will I not lose any files?

   

Are there any suggestions for the network, MTU or bandwidth?

 

 

Does MooseFS support supplementary groups?