Merge pull request #33 from s3fs-fuse/for_v1.77

changed for version v1.77 and fixed man page

ChangeLog

@@ -1,6 +1,19 @@
 ChangeLog for S3FS
 ------------------
 
+Version 1.77 -- Apr 19, 2014
+issue 405(googlecode) - enable_content_md5 Input/output error
+issue #14 - s3fs -u should return 0 if there are no lost multiparts
+issue #16 - empty file is written to s3
+issue #18 - s3fs crashes with segfault
+issue #22 - Fix typos in docs for max_stat_cache_size
+issue #23 - curl ssl problems
+issue #28 - Address signedness warning in FdCache::Init
+
+Version 1.76 -- Jan 21, 2014
+issue #5 - du shows incorrect usage stats
+issue #8 - version in configure.ac is 1.74 for release 1.75
+
 Version 1.75 -- Jan 6, 2014
 issue #1 - Using %20 instead of the plus (+) sign for encoding spaces
 issue #3 - Fixed local timezone was incorrectly being applied to IAM and Last-Modified dates.

README

@@ -14,16 +14,18 @@ In order to compile s3fs, You'll need the following requirements:
 * FUSE (>= 2.8.4)
 * FUSE Kernel module installed and running (RHEL 4.x/CentOS 4.x users - read below)
 * OpenSSL-devel (0.9.8)
-* Subversion
+* Git
 
 If you're using YUM or APT to install those packages, then it might require additional packaging, allow it to be installed.
 
 Downloading & Compiling:
 ------------------------
-In order to download s3fs, user the following command:
-svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs-read-only
+In order to download s3fs, download from following url:
+https://github.com/s3fs-fuse/s3fs-fuse/archive/master.zip
+Or clone the following command:
+git clone git://github.com/s3fs-fuse/s3fs-fuse.git
 
-Go inside the directory that has been created (s3fs-read-only/s3fs) and run: ./autogen.sh
+Go inside the directory that has been created (s3fs-fuse) and run: ./autogen.sh
 This will generate a number of scripts in the project directory, including a configure script which you should run with: ./configure
 If configure succeeded, you can now run: make. If it didn't, make sure you meet the dependencies above.
 This should compile the code. If everything goes OK, you'll be greated with "ok!" at the end and you'll have a binary file called "s3fs"
@@ -42,21 +44,21 @@ Then run: s3fs mybucket[:path] /mnt/s3
 
 This will mount your bucket to /mnt/s3. You can do a simple "ls -l /mnt/s3" to see the content of your bucket.
 
-If you want to allow other people access the same bucket in the same machine, you can add "-o allow _other" to read/write/delete content of the bucket.
+If you want to allow other people access the same bucket in the same machine, you can add "-o allow_other" to read/write/delete content of the bucket.
 
 You can add a fixed mount point in /etc/fstab, here's an example:
 
 s3fs#mybucket /mnt/s3 fuse allow_other 0 0
 
 This will mount upon reboot (or by launching: mount -a) your bucket on your machine.
+If that does not work, probably you should specify with "_netdev" option in fstab.
 
-All other options can be read at: http://code.google.com/p/s3fs/wiki/FuseOverAmazon
+All other options can be read at: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
 
 Known Issues:
 -------------
 s3fs should be working fine with S3 storage. However, There are couple of limitations:
 
-* There is no full UID/GID support yet, everything looks as "root" and if you allow others to access the bucket, others can erase files. There is, however, permissions support built in.
 * Currently s3fs could hang the CPU if you have lots of time-outs. This is *NOT* a fault of s3fs but rather libcurl. This happends when you try to copy thousands of files in 1 session, it doesn't happend when you upload hundreds of files or less.
 * CentOS 4.x/RHEL 4.x users - if you use the kernel that shipped with your distribution and didn't upgrade to the latest kernel RedHat/CentOS gives, you might have a problem loading the "fuse" kernel. Please upgrade to the latest kernel (2.6.16 or above) and make sure "fuse" kernel module is compiled and loadable since FUSE requires this kernel module and s3fs requires it as well.
 * Moving/renaming/erasing files takes time since the whole file needs to be accessed first. A workaround could be to use s3fs's cache support with the use_cache option.

configure.ac

@@ -20,7 +20,7 @@
 dnl Process this file with autoconf to produce a configure script.
 
 AC_PREREQ(2.59)
-AC_INIT(s3fs, 1.76)
+AC_INIT(s3fs, 1.77)
 
 AC_CANONICAL_SYSTEM
 AM_INIT_AUTOMAKE()

doc/man/s3fs.1

@@ -8,6 +8,9 @@ S3FS \- FUSE-based file system backed by Amazon S3
 .SS unmounting
 .TP
 \fBumount mountpoint
+.SS utility mode ( remove interrupted multipart uploading objects )
+.TP
+\fBs3fs -u bucket
 .SH DESCRIPTION
 s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).
 .SH AUTHENTICATION
@@ -102,7 +105,7 @@ time to wait for connection before giving up.
 \fB\-o\fR readwrite_timeout (default="30" seconds)
 time to wait between read/write activity before giving up.
 .TP
-\fB\-o\fR max_stat_cache_size (default="10000" entries (about 4MB))
+\fB\-o\fR max_stat_cache_size (default="1000" entries (about 4MB))
 maximum number of entries in the stat cache
 .TP
 \fB\-o\fR stat_cache_expire (default is no expire)
@@ -125,14 +128,22 @@ maximum number of parallel request for listing objects.
 .TP
 \fB\-o\fR parallel_count (default="5")
 number of parallel request for uploading big objects.
-s3fs uploads large object(over 20MB) by multipart post request, and sends parallel requests.
+s3fs uploads large object(default:over 20MB) by multipart post request, and sends parallel requests.
 This option limits parallel request count which s3fs requests at once.
 It is necessary to set this value depending on a CPU and a network band.
+This option is lated to fd_page_size option and affects it.
 .TP
 \fB\-o\fR fd_page_size(default="52428800"(50MB))
 number of internal management page size for each file discriptor.
 For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
 This option should not be changed when you don't have a trouble with performance.
+This value is changed automatically by parallel_count and multipart_size values(fd_page_size value = parallel_count * multipart_size).
+.TP
+\fB\-o\fR multipart_size(default="10"(10MB))
+number of one part size in multipart uploading request.
+The default size is 10MB(10485760byte), this value is minimum size.
+Specify number of MB and over 10(MB).
+This option is lated to fd_page_size option and affects it.
 .TP
 \fB\-o\fR url (default="http://s3.amazonaws.com")
 sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com

src/curl.cpp

@@ -173,7 +173,8 @@ curltime_t       S3fsCurl::curl_times;
 curlprogress_t   S3fsCurl::curl_progress;
 string           S3fsCurl::curl_ca_bundle;
 mimes_t          S3fsCurl::mimeTypes;
-int              S3fsCurl::max_parallel_cnt    = 5;    // default
+int              S3fsCurl::max_parallel_cnt    = 5;              // default
+off_t            S3fsCurl::multipart_size      = MULTIPART_SIZE; // default
 
 //-------------------------------------------------------------------
 // Class methods for S3fsCurl
@@ -822,6 +823,16 @@ string S3fsCurl::SetIAMRole(const char* role)
   return old;
 }
 
+bool S3fsCurl::SetMultipartSize(off_t size)
+{
+  size = size * 1024 * 1024;
+  if(size < MULTIPART_SIZE){
+    return false;
+  }
+  S3fsCurl::multipart_size = size;
+  return true;
+}
+
 int S3fsCurl::SetMaxParallelCount(int value)
 {
   int old = S3fsCurl::max_parallel_cnt;
@@ -931,7 +942,7 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
     // Loop for setup parallel upload(multipart) request.
     for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_cnt && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
       // chunk size
-      chunk = remaining_bytes > MULTIPART_SIZE ?  MULTIPART_SIZE : remaining_bytes;
+      chunk = remaining_bytes > S3fsCurl::multipart_size ? S3fsCurl::multipart_size : remaining_bytes;
 
       // s3fscurl sub object
       S3fsCurl* s3fscurl_para            = new S3fsCurl(true);
@@ -1021,7 +1032,7 @@ int S3fsCurl::ParallelGetObjectRequest(const char* tpath, int fd, off_t start, s
     // Loop for setup parallel upload(multipart) request.
     for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_cnt && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
       // chunk size
-      chunk = remaining_bytes > MULTIPART_SIZE ?  MULTIPART_SIZE : remaining_bytes;
+      chunk = remaining_bytes > S3fsCurl::multipart_size ? S3fsCurl::multipart_size : remaining_bytes;
 
       // s3fscurl sub object
       S3fsCurl* s3fscurl_para = new S3fsCurl();
@@ -1561,6 +1572,11 @@ int S3fsCurl::RequestPerform(void)
         sleep(2);
         break;
 
+      case CURLE_SSL_CONNECT_ERROR:
+        DPRN("### CURLE_SSL_CONNECT_ERROR");
+        sleep(2);
+        break;
+
       case CURLE_SSL_CACERT:
         // try to locate cert, if successful, then set the
         // option and continue
@@ -2837,7 +2853,7 @@ int S3fsCurl::MultipartUploadRequest(const char* tpath, headers_t& meta, int fd,
   // cycle through open fd, pulling off 10MB chunks at a time
   for(remaining_bytes = st.st_size; 0 < remaining_bytes; remaining_bytes -= chunk){
     // chunk size
-    chunk = remaining_bytes > MULTIPART_SIZE ?  MULTIPART_SIZE : remaining_bytes;
+    chunk = remaining_bytes > S3fsCurl::multipart_size ? S3fsCurl::multipart_size : remaining_bytes;
 
     // set
     partdata.fd         = fd2;
@@ -3127,8 +3143,12 @@ int S3fsMultiCurl::MultiRead(void)
 
         // For retry
         if(RetryCallback){
-          retrycurl = RetryCallback(s3fscurl);
-          cMap_all[retrycurl->hCurl] = retrycurl;
+          if(NULL != (retrycurl = RetryCallback(s3fscurl))){
+            cMap_all[retrycurl->hCurl] = retrycurl;
+          }else{
+            // do not care, but set...
+            isRetry = false;
+          }
         }
         if(s3fscurl != retrycurl){
           s3fscurl->DestroyCurlHandle();
@@ -3426,6 +3446,14 @@ unsigned char* md5hexsum(int fd, off_t start, ssize_t size)
   ssize_t bytes;
   unsigned char* result;
 
+  if(-1 == size){
+    struct stat st;
+    if(-1 == fstat(fd, &st)){
+      return NULL;
+    }
+    size = static_cast<ssize_t>(st.st_size);
+  }
+
   // seek to top of file.
   if(-1 == lseek(fd, start, SEEK_SET)){
     return NULL;
@@ -3457,7 +3485,6 @@ unsigned char* md5hexsum(int fd, off_t start, ssize_t size)
     free(result);
     return NULL;
   }
-
   return result;
 }
 

src/curl.h

@@ -166,6 +166,7 @@ class S3fsCurl
     static std::string      curl_ca_bundle;
     static mimes_t          mimeTypes;
     static int              max_parallel_cnt;
+    static off_t            multipart_size;
 
     // variables
     CURL*                hCurl;
@@ -274,8 +275,11 @@ class S3fsCurl
     static long SetSslVerifyHostname(long value);
     static long GetSslVerifyHostname(void) { return S3fsCurl::ssl_verify_hostname; }
     static int SetMaxParallelCount(int value);
+    static int GetMaxParallelCount(void) { return S3fsCurl::max_parallel_cnt; }
     static std::string SetIAMRole(const char* role);
     static const char* GetIAMRole(void) { return S3fsCurl::IAM_role.c_str(); }
+    static bool SetMultipartSize(off_t size);
+    static off_t GetMultipartSize(void) { return S3fsCurl::multipart_size; }
 
     // methods
     bool CreateCurlHandle(bool force = false);

src/fdcache.cpp

@@ -51,8 +51,7 @@ using namespace std;
 //------------------------------------------------
 // Symbols
 //------------------------------------------------
-#define MAX_OBJECT_SIZE     68719476735LL           // 64GB - 1L
-#define MULTIPART_LOWLIMIT  (20 * 1024 * 1024)      // 20MB
+#define MAX_MULTIPART_CNT   10000                   // S3 multipart max count
 #define	FDPAGE_SIZE	    (50 * 1024 * 1024)      // 50MB(parallel uploading is 5 parallel(default) * 10 MB)
 
 //------------------------------------------------
@@ -266,7 +265,7 @@ int PageList::Init(off_t size, bool is_init)
 {
   Clear();
   for(off_t total = 0; total < size; total += FdManager::GetPageSize()){
-    size_t areasize = (total + FdManager::GetPageSize()) < size ? FdManager::GetPageSize() : static_cast<size_t>(size - total);
+    size_t areasize = (total + static_cast<off_t>(FdManager::GetPageSize())) < size ? FdManager::GetPageSize() : static_cast<size_t>(size - total);
     fdpage* page    = new fdpage(total, areasize, is_init);
     pages.push_back(page);
   }
@@ -798,7 +797,7 @@ int FdEntity::Load(off_t start, off_t size)
         break;
       }
       // download
-      if((*iter)->bytes >= MULTIPART_LOWLIMIT && !nomultipart){ // 20MB
+      if((*iter)->bytes >= (2 * S3fsCurl::GetMultipartSize()) && !nomultipart){ // default 20MB
         // parallel request
         // Additional time is needed for large files
         time_t backup = 0;
@@ -880,14 +879,14 @@ int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool
    *  - 1 to 10,000 parts are allowed
    *  - minimum size of parts is 5MB (expect for the last part)
    * 
-   * For our application, we will define part size to be 10MB (10 * 2^20 Bytes)
-   * maximum file size will be ~64 GB - 2 ** 36 
+   * For our application, we will define minimum part size to be 10MB (10 * 2^20 Bytes)
+   * minimum file size will be 64 GB - 2 ** 36 
    * 
    * Initially uploads will be done serially
    * 
    * If file is > 20MB, then multipart will kick in
    */
-  if(pagelist.Size() > MAX_OBJECT_SIZE){ // 64GB - 1
+  if(pagelist.Size() > (MAX_MULTIPART_CNT * S3fsCurl::GetMultipartSize())){
     // close f ?
     return -ENOTSUP;
   }
@@ -898,7 +897,7 @@ int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool
     return -errno;
   }
 
-  if(pagelist.Size() >= MULTIPART_LOWLIMIT && !nomultipart){ // 20MB
+  if(pagelist.Size() >= (2 * S3fsCurl::GetMultipartSize()) && !nomultipart){ // default 20MB
     // Additional time is needed for large files
     time_t backup = 0;
     if(120 > S3fsCurl::GetReadwriteTimeout()){

src/s3fs.cpp

@@ -2712,7 +2712,7 @@ static bool abort_uncomp_mp_list(uncomp_mp_list_t& list)
   char buff[1024];
 
   if(0 >= list.size()){
-    return false;
+    return true;
   }
   memset(buff, 0, sizeof(buff));
 
@@ -3574,11 +3574,15 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
         return -1;
       }
       S3fsCurl::SetMaxParallelCount(maxpara);
+
+      if(FdManager::GetPageSize() < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
+        FdManager::SetPageSize(static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount()));
+      }
       return 0;
     }
     if(0 == STR2NCMP(arg, "fd_page_size=")){
       size_t pagesize = static_cast<size_t>(s3fs_strtoofft(strchr(arg, '=') + sizeof(char)));
-      if((1024 * 1024) >= pagesize){
+      if(pagesize < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
         fprintf(stderr, "%s: argument should be over 1MB: fd_page_size\n", 
            program_name.c_str());
         return -1;
@@ -3586,6 +3590,17 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
       FdManager::SetPageSize(pagesize);
       return 0;
     }
+    if(0 == STR2NCMP(arg, "multipart_size=")){
+      off_t size = static_cast<off_t>(s3fs_strtoofft(strchr(arg, '=') + sizeof(char)));
+      if(!S3fsCurl::SetMultipartSize(size)){
+        fprintf(stderr, "%s: multipart_size option could not be specified over 10(MB)\n", program_name.c_str());
+        return -1;
+      }
+      if(FdManager::GetPageSize() < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
+        FdManager::SetPageSize(static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount()));
+      }
+      return 0;
+    }
     if(0 == STR2NCMP(arg, "ahbe_conf=")){
       string ahbe_conf = strchr(arg, '=') + sizeof(char);
       if(!AdditionalHeader::get()->Load(ahbe_conf.c_str())){

src/s3fs_util.cpp

@@ -889,7 +889,7 @@ void show_help (void)
     "   readwrite_timeout (default=\"30\" seconds)\n"
     "      - time to wait between read/write activity before giving up\n"
     "\n"
-    "   max_stat_cache_size (default=\"10000\" entries (about 4MB))\n"
+    "   max_stat_cache_size (default=\"1000\" entries (about 4MB))\n"
     "      - maximum number of entries in the stat cache\n"
     "\n"
     "   stat_cache_expire (default is no expire)\n"