At the moment file locking works only "locally", where if a file is locked on a given client node (e.g. machine A) its kernel remembers it is locked for any other process on the same node. However another client node (e.g. machine B) doesn't know about it. A “global” locking feature should be introduced in the 1.7 branch.

For now one work around could be to use the mechanism of chunk locking for write. Which causes the entire chunk which contains the file to be locked while one node is writing to the file. Another client then not be able to write to this chunk and would attempt to try the write again every second. In certain edge cases this can in theory lead to starvation but practically it shouldn’t.

The only problem would be with simultaneous appending (writing at the end) to the same file by two clients. See also the thread “Append and seek while writing functionality” on the group archive:
https://sourceforge.net/mailarchive/message.php?msg_id=25488203

While chunk locking for write is a safe operation, it is not recommended. Instead it is better when different process on different machines write to different files and later on have some other system operation combine the data from these many files into one single target file (as is typically done in “map-reduce” processing).

Yes, at this moment it is possible to assign (any) IP address to chunk server via DHCP.

However, a planned enhancement for a future release is "chunkserver awareness". Where the master server remembers the chunkservers that have connected to it and in case any of them were disconnected, the master server would then be able to report the disconnection.

So for chunkserver awareness to work the master server would need to know IP addresses of the chunkservers and this IP addresses should not change between chunkserver restarts. Thus, if you do use DHCP, it is advisable to configure DHCP so that it gives the same IP address to the chunkserver machine based on its MAC address, i.e. to use “DHCP reservations”.

Our experiences from working in a production environment have shown that aggressive replication is not desirable as it can substantially slow down the whole system. The overall performance of the system is more important than equal utilization of hard drives over all of the chunk servers.. By default replication is configured to be a non-aggressive operation. At our environment normally it takes 3-4 weeks for a new chunkserver to get to a standard hdd utilization. Aggressive replication would make the whole system considerably slow for several days.

Replication speeds can be adjusted on master server startup by setting these two options:
CHUNKS_WRITE_REP_LIMIT
how many chunks may be saved in parallel on one chunkserver while replicating (by default 1).
CHUNKS_READ_REP_LIMIT
how many chunks may be read in parallel from one chunkserver while replicating (by default 5).

Tuning these in your environment will require experimentation. When adding a new chunkserver, try setting the first option to 5 and the second to 15 and restart the master server. After replication finishes you should restore these settings back to their default values and again restart the master server.

Yes, it is highly advisable to additionally backup the metadata file. This provides a worst case recovery option in the event for some reason the metalogger data is not useable for restoring the master server (for example the metalogger server is also destroyed).

The master server flushes metadata kept in RAM to the metadata.mfs.back binary file every hour on the hour (xx:00). So a good time to copy the metadata file is every hour on the half hour (30 minutes after the dump). This would limit the amount of data loss to about 1.5h of data. Backing up the file can be done using any conventional method of copying the metadata file – cp, scp, rsync, etc.

After restoring the system based on this backed up metadata file the most recently created files will have been lost. Additionally files that were appended to would have their previous size, at the time of the metadata backup. Files that were deleted would exist again. And files that were renamed or moved would be back to their previous names (and locations). But still you would have all of data for the files created in the X past years before the crashed occurred.

In the CGI monitor go to the “Disks” tab and choose “switch to hour” in “I/O stats” column and sort the results by “write” in “max time” column. Now look for disks which have a significantly larger write time. You can also sort by the “fsync” column and look at the results. Maximum times should not exceed 2 seconds (2 million microseconds). It is a good idea to find individual disks that are operating slower as they may be a bottleneck to the system.

It might be helpful to create a test operation that continuously copies some data to create enough load on the system for there to be observable statisics in the CGI monitor. On the “Disks” tab specify units of “minutes” instead of hours for the “I/O stats” column.

Once a disk has been discovered to replace it follow the usual operation of marking the chunkserver for replacement, and waiting until the color changes to indicate that all of the chunks stored on this chunkserver have been replicated to achieve the sufficient goal settings onto other chunkservers.

In the directory where all metadata and changelogs are located (by default /usr/local/var/mfs) there is also a .master.lock file. The PID of the process which keeps a lock on this file is the PID of the master server. So the PID can be obtained by using the fcntl function:

struct flock fl;
if (fcntl(fd,F_GETLK,&fl)<0) { // get lock owner
	return -1; // getting lock error 
}
if (fl.l_type!=F_UNLCK) {  // found lock
	return fl.l_pid; // return lock owner
}
return -1;

This is a way to mark chunks belonging to the non-existing (i.e. deleted) files. Deleting a file is done asynchronously in MooseFS. First a file is removed from metadata and its chunks are marked as unnecessary (goal=0). Later the chunks are removed during an "idle" time. This is much more efficient than erasing everything at the moment the file was deleted.

Unnecessary chunks may also appear after a recovery of the master server if they were created shortly before the failure and were not available in the restored metadata file.

Another case is when the master tries to delete a chunk and the operation returns an error. In this scenario if the master deletes information about it from metadata before the error occurs there is no guarantee that the chunk has actually been deleted, or if it still resides on the disk on the chunkserver.

Such chunks would be discovered by the master server and unless admin does something with them, they would eventually be deleted after a week.

No. Mfsmount writes every failure encountered during communication with chunkservers to the syslog. Transient communication problems with the network might cause IO errors to be displayed, but this does not mean data loss or that mfsmount will return an error code to the application. Each operation is retried by the client (mfsmount) several times and only after the number of failures (reported as try counter) reaches a certain limit (typically 30) is the error returned to the application that data was not read/saved.

Of course it is important to monitor these messages. When messages appear more often from one chunkserver than from the others it may mean there are issues with this chunkserver - maybe hard drive is broken, maybe network card has some problems - check its charts, hard disk operation times, etc. in the CGI monitor.

Note: Chunkserver IP is written in hexadecimal format.

This means that Zth try to write the chunk was not successful and writing of Y blocks sent to the chunkserver were not confirmed. After reconnecting these blocks would be sent again for saving. The limit of trials is set by default to 30.

This message is for informational purposes and doesn't mean data loss.

This means that Zth try to read the chunk was not successful and system will try to read the block again. If value of Z equals 1 it is a transitory problem and you should not worry about it. The limit of trials is set by default to 30.

XXXXXXXX is a chunkserver IP written in hexadecimal format. This can be helpful for determining if there is an issue with a chunkserver. When messages appear more often from one chunkserver than from the others it may mean there are issues with this chunkserver – check its charts, hard disk operation times, etc. in the CGI monitor.

When the master server goes down while mfsmount is already running, mfsmount doesn't disconnect the mounted resource and files awaiting to be saved would stay quite long in the queue while trying to reconnect to the master server. After a specified number of tries they eventually return EIO - "input/output error". On the other hand it is not possible to start mfsmount when the master server is offline.

There are several ways to assure that the master server is online, we present a few of these below.

1. Check if you can connect to the TCP port of the master server (e.g. socket connection test).
2. In order to assure that a MooseFS resource is mounted it is enough to check the inode number – MooseFS root will always have inode equal to 1. For example if we have MooseFS installation in /mnt/mfs then stat /mnt/mfs command (in Linux) will show:

   
   $ stat /mnt/mfs
     File: `/mnt/mfs'
     Size: xxxxxx    	Blocks: xxx        IO Block: 4096   directory
   Device: 13h/19d	Inode: 1           Links: xx
   (...)
   

3. Additionaly mfsmount creates a virtual hidden file .stats in the root mounted folder. For example, to get the statistics of mfsmount when MooseFS is mounted we can cat this .stats file, eg.:

   
   $ cat /mnt/mfs/.stats
   fuse_ops.statfs: 241
   fuse_ops.access: 0
   fuse_ops.lookup-cached: 707553
   fuse_ops.lookup: 603335
   fuse_ops.getattr-cached: 24927
   fuse_ops.getattr: 687750
   fuse_ops.setattr: 24018
   fuse_ops.mknod: 0
   fuse_ops.unlink: 23083
   fuse_ops.mkdir: 4
   fuse_ops.rmdir: 1
   fuse_ops.symlink: 3
   fuse_ops.readlink: 454
   fuse_ops.rename: 269
   (...)
   

4. If you want to be sure that master server properly responds you need to try to read the goal of any object, specifically of the root folder:

   
   $ mfsgetgoal /mnt/mfs
   /mnt/mfs: 2
   

   
   If you get a proper goal of the root folder you can be sure that the master server is up and running.