56 Commits
v1.74 ... v1.78

Author SHA1 Message Date
38e6857824 Merge pull request #56 from s3fs-fuse/version1.78
version number increament.
2014-09-15 22:30:51 +09:00
ca72b9a6d0 version number increament. 2014-09-15 13:26:35 +00:00
741831344a Merge pull request #55 from s3fs-fuse/cleanupcodes
Cleaned up codes for next packaging.
2014-09-08 00:10:49 +09:00
7a7c7572ea Cleaned up codes for next packaging. 2014-09-07 15:08:27 +00:00
4c32bc0aa5 Merge pull request #54 from s3fs-fuse/bugfix#50
fixed a bug about configure.ac. issue #50
2014-09-07 22:58:07 +09:00
0e9cfeb808 fixed a bug about configure.ac. issue #50 2014-09-07 13:53:20 +00:00
ae4ae88b6d Merge pull request #53 from s3fs-fuse/sse-c
Support for SSE-C #39
2014-09-07 22:31:33 +09:00
f0c33f8ef2 clean codes 2014-08-27 00:59:49 +00:00
e3a33343b9 Merge pull request #51 from masahide/master
Removed BOM
2014-08-27 02:24:47 +09:00
20b1c207be fixed issue #39 2014-08-26 17:11:10 +00:00
f1ca5d0340 :set nobomb 2014-08-25 19:18:34 +09:00
cbec8da9a3 fixed a bug issue #49 2014-08-14 15:58:06 +00:00
7a55eab399 Support for SSE-C, issue #39 2014-07-19 19:02:55 +00:00
95f8cab139 Merge pull request #44 from s3fs-fuse/fixbug#40
Fixed a bug issue #40
2014-06-29 02:42:49 +09:00
c1a6d76fc3 Fixed a bug issue #40 2014-06-28 17:36:35 +00:00
08929696f7 Merge pull request #43 from s3fs-fuse/fixbug#41
Fixed a bug issue #41
2014-06-29 02:26:10 +09:00
ba34ba181a Fixed a bug issue #41 2014-06-28 17:24:25 +00:00
d2c887a371 Merge pull request #38 from s3fs-fuse/path-request-style
Added explanation in man page for support for path API request style.
2014-06-03 23:48:27 +09:00
d5113c0501 Added explanation in man page for support for path API request style. 2014-06-03 14:45:39 +00:00
29a37645dd Merge pull request #37 from Andrew-Dunn/path-request-style
Added support for path API request style.
2014-06-03 23:37:00 +09:00
601482eff5 Added support for path API request style.
Rather than using virtual host style requests, path style requests can be used
instead.

i.e. rather than bucketname.s3.amazon.com/... the s3fs will be able to request
from s3.amazon.com/bucketname/...

This is useful for S3 compatible APIs which don't support the virtual host style
request.

It is enabled with the new option, `use_path_style_request`.

Example:

    /usr/bin/s3fs data ~/netcdf -o url="https://swift.rc.nectar.org.au:8888/" -o use_path_request_style -o allow_other -o uid=500 -o gid=500
2014-06-04 00:03:49 +10:00
f141bbd4b4 Merge pull request #36 from s3fs-fuse/gc#417
Changed codes for CR code in passwd file(googlecode issue#417).
2014-06-03 01:16:03 +09:00
61020370d5 Changed codes for CR code in passwd file(googlecode issue#417). 2014-06-02 16:12:55 +00:00
f1f7e76be5 Merge pull request #35 from s3fs-fuse/cryptlibs
Supports more two SSL libraries for NSS and GnuTLS.
2014-06-01 23:15:59 +09:00
160196798b Changed initializing logic for nss lib/openssl lib/s3fs own. 2014-06-01 03:54:02 +00:00
edad91186f Changed configuration switch from 'enable' to 'with' for libs 2014-05-10 16:45:46 +00:00
cd27f0aa54 Supported another crypt libraries as GnuTLS and NSS, and added configure options 2014-05-06 14:23:05 +00:00
c7665f80ab Merge pull request #33 from s3fs-fuse/for_v1.77
changed for version v1.77 and fixed man page
2014-04-20 01:10:21 +09:00
1a4065b0fb changed for version v1.77 and fixed man page 2014-04-19 16:08:10 +00:00
a4465105f7 Merge pull request #32 from s3fs-fuse/sslproblem#23
Retry to send request at CURLE_SSL_CONNECT_ERROR
2014-04-20 00:53:09 +09:00
8bba566774 Retry to send request at CURLE_SSL_CONNECT_ERROR 2014-04-04 16:23:56 +00:00
157612e7e7 Merge pull request #28 from andrewgaul/fdcache-init-signedness-warning
Address signedness warning in FdCache::Init
2014-04-05 01:02:13 +09:00
6148415b4b Merge pull request #31 from s3fs-fuse/upperlimit#16
Added multipart_size option for #16
2014-04-05 00:53:04 +09:00
d475e22774 Address signedness warning in FdCache::Init
This commit allows GCC 4.8 to compile s3fs withing warnings.
2014-04-01 23:50:08 -07:00
4762e53b5d Added multipart_size option for #16 2014-03-30 07:53:41 +00:00
e23ea87953 Merge pull request #26 from s3fs-fuse/bugfix#18
Fixed a bug #18(losing check retry error)
2014-03-30 15:44:59 +09:00
d7563309a2 Fixed a bug #18(losing check retry error) 2014-03-30 06:40:49 +00:00
c003076053 Merge pull request #22 from theasci/max_stat_cache_size_typo
Fix typos in docs for max_stat_cache_size (the default is actually 1000)
2014-03-16 13:48:21 +09:00
74fb29d9fb Fix typos in docs for max_stat_cache_size (the default is actually 1000) 2014-03-06 12:44:10 -05:00
b35f8ded46 Merge pull request #20 from s3fs-fuse/fix_bugs
Fixed a bug: issue #14, s3fs -u should return 0 if there are no lost multiparts
2014-03-04 02:02:59 +09:00
16487c4e26 Fixed a bug: issue #14, s3fs -u should return 0 if there are no lost multiparts 2014-03-03 17:00:20 +00:00
39402696ce Merge pull request #19 from s3fs-fuse/test_md5
Fixed a bug(googlecode issue 405), enable_content_md5 Input/output error
2014-03-04 01:24:23 +09:00
52d56d15e4 Fixed a bug(googlecode issue 405), enable_content_md5 Input/output error 2014-03-03 16:19:08 +00:00
36509351f0 Merge branch 'master' of https://github.com/s3fs-fuse/s3fs-fuse 2014-01-21 15:49:24 +00:00
00792a6555 Update ChangeLog 2014-01-21 15:47:58 +00:00
fdabb7cbbe Merge pull request #9 from s3fs-fuse/issue/#8
version number up to v1.76 for new.
2014-01-21 07:37:38 -08:00
775d8758ef version number up to v1.76 for new. 2014-01-21 15:33:02 +00:00
31c979b290 Merge pull request #7 from s3fs-fuse/issue#5
Fixed a bug - issue#5
2014-01-16 08:09:50 -08:00
5fd33405af Fixed a bug - issue#5 2014-01-16 16:04:41 +00:00
1d1e8f3e7d Update ChangeLog 2014-01-06 17:12:41 +00:00
654c58c90a Merge pull request #4 from pdeschen/patch-1
Fix compilation error on MacOSX with missing const
2014-01-05 05:24:54 -08:00
db3bd7c366 Fix compilation error on MacOSX with missing const 2014-01-04 21:24:27 -05:00
26187b954e Merge pull request #3 from worpet/master
Fixed local timezone was incorrectly being applied to IAM and Last-Modified dates.
2014-01-01 05:15:51 -08:00
33ec3739e2 Update s3fs_util.cpp
Fixed local time was incorrectly being applied to IAM and Last-Modified dates.
2013-12-30 09:35:39 -06:00
d9f13dbdcb Merge pull request #1 from dejaeghd/master
Using %20 instead of the plus (+) sign for encoding spaces
2013-12-17 07:57:15 -08:00
d5626fe595 Changed url encoding of space character to use %20 instead of the plus (+) sign. 2013-12-17 15:12:03 +01:00
26 changed files with 2231 additions and 516 deletions

130
ChangeLog
View File

@ -1,10 +1,134 @@
ChangeLog for S3FS
ChangeLog for S3FS
------------------
Version 1.1 -- Mon Oct 18 2010
Version 1.78 -- Sep 15, 2014
issue #29 - Possible to create Debian/Ubuntu packages?(googlecode issue 109)
issue 417(googlecode) - Password file with DOS format is not handled properly
issue #41 - Failed making signature
issue #40 - Moving a directory containing more than 1000 files truncates the
directory
issue #49 - use_sse is ignored when creating new files
issue #39 - Support for SSE-C
issue #50 - Cannot find pkg-config when configured with any SSL backend except
openssl
Version 1.77 -- Apr 19, 2014
issue 405(googlecode) - enable_content_md5 Input/output error
issue #14 - s3fs -u should return 0 if there are no lost multiparts
issue #16 - empty file is written to s3
issue #18 - s3fs crashes with segfault
issue #22 - Fix typos in docs for max_stat_cache_size
issue #23 - curl ssl problems
issue #28 - Address signedness warning in FdCache::Init
Version 1.76 -- Jan 21, 2014
issue #5 - du shows incorrect usage stats
issue #8 - version in configure.ac is 1.74 for release 1.75
Version 1.75 -- Jan 6, 2014
issue #1 - Using %20 instead of the plus (+) sign for encoding spaces
issue #3 - Fixed local timezone was incorrectly being applied to IAM and Last-Modified dates.
issue #4 - Fix compilation error on MacOSX with missing const
Version 1.74 -- Nov 24, 2013
This version is initial version on Github, same as on GoogleCodes(s3fs).
https://github.com/s3fs-fuse/s3fs-fuse/releases/tag/v1.74
see more detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.74.tar.gz
Version 1.73 -- Aug 23, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.73.tar.gz
Version 1.72 -- Aug 10, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.72.tar.gz
Version 1.71 -- Jun 15, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.71.tar.gz
Version 1.70 -- Jun 01, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.70.tar.gz
Version 1.69 -- May 15, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.69.tar.gz
Version 1.68 -- Apr 30, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.68.tar.gz
Version 1.67 -- Apr 13, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.67.tar.gz
Version 1.66 -- Apr 06, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.66.tar.gz
Version 1.65 -- Mar 30, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.65.tar.gz
Version 1.64 -- Mar 23, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.64.tar.gz
Version 1.63 -- Feb 24, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.63.tar.gz
Version 1.62 -- Jan 27, 2013
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.62.tar.gz
Version 1.61 -- Aug 30, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.61.tar.gz
Version 1.60 -- Aug 29, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.60.tar.gz
Version 1.59 -- Jul 28, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.59.tar.gz
Version 1.58 -- Jul 19, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.58.tar.gz
Version 1.57 -- Jul 07, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.57.tar.gz
Version 1.56 -- Jul 07, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.56.tar.gz
Version 1.55 -- Jul 02, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.55.tar.gz
Version 1.54 -- Jun 25, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.54.tar.gz
Version 1.53 -- Jun 22, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.53.tar.gz
Version 1.40 -- Feb 11, 2011
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.40.tar.gz
Version 1.33 -- Dec 30, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.33.tar.gz
Version 1.25 -- Dec 16, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.25.tar.gz
Version 1.19 -- Dec 2, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.19.tar.gz
Version 1.16 -- Nov 22, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.16.tar.gz
Version 1.10 -- Nov 6, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.10.tar.gz
Version 1.02 -- Oct 29, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.02.tar.gz
Version 1.01 -- Oct 28, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.01.tar.gz
Version 1.0 -- Oct 24, 2010
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.0.tar.gz
------
Version 1.1 -- Mon Oct 18 2010
Dan Moore reopens the project and fixes various issues that had accumulated in the tracker. Adrian Petrescu converts the project to autotools and posts it to GitHub.
Version 1.0 -- 2008
Randy Rizun releases a basic version of S3FS on Google Code.

View File

@ -1,3 +1,22 @@
######################################################################
# s3fs - FUSE-based file system backed by Amazon S3
#
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
######################################################################
SUBDIRS=src test doc
EXTRA_DIST=doc
@ -8,3 +27,4 @@ dist-hook:
release : dist ../utils/release.sh
../utils/release.sh $(DIST_ARCHIVES)

18
README
View File

@ -14,16 +14,20 @@ In order to compile s3fs, You'll need the following requirements:
* FUSE (>= 2.8.4)
* FUSE Kernel module installed and running (RHEL 4.x/CentOS 4.x users - read below)
* OpenSSL-devel (0.9.8)
* Subversion
GnuTLS(gcrypt and nettle)
NSS
* Git
If you're using YUM or APT to install those packages, then it might require additional packaging, allow it to be installed.
Downloading & Compiling:
------------------------
In order to download s3fs, user the following command:
svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs-read-only
In order to download s3fs, download from following url:
https://github.com/s3fs-fuse/s3fs-fuse/archive/master.zip
Or clone the following command:
git clone git://github.com/s3fs-fuse/s3fs-fuse.git
Go inside the directory that has been created (s3fs-read-only/s3fs) and run: ./autogen.sh
Go inside the directory that has been created (s3fs-fuse) and run: ./autogen.sh
This will generate a number of scripts in the project directory, including a configure script which you should run with: ./configure
If configure succeeded, you can now run: make. If it didn't, make sure you meet the dependencies above.
This should compile the code. If everything goes OK, you'll be greated with "ok!" at the end and you'll have a binary file called "s3fs"
@ -42,21 +46,21 @@ Then run: s3fs mybucket[:path] /mnt/s3
This will mount your bucket to /mnt/s3. You can do a simple "ls -l /mnt/s3" to see the content of your bucket.
If you want to allow other people access the same bucket in the same machine, you can add "-o allow _other" to read/write/delete content of the bucket.
If you want to allow other people access the same bucket in the same machine, you can add "-o allow_other" to read/write/delete content of the bucket.
You can add a fixed mount point in /etc/fstab, here's an example:
s3fs#mybucket /mnt/s3 fuse allow_other 0 0
This will mount upon reboot (or by launching: mount -a) your bucket on your machine.
If that does not work, probably you should specify with "_netdev" option in fstab.
All other options can be read at: http://code.google.com/p/s3fs/wiki/FuseOverAmazon
All other options can be read at: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
Known Issues:
-------------
s3fs should be working fine with S3 storage. However, There are couple of limitations:
* There is no full UID/GID support yet, everything looks as "root" and if you allow others to access the bucket, others can erase files. There is, however, permissions support built in.
* Currently s3fs could hang the CPU if you have lots of time-outs. This is *NOT* a fault of s3fs but rather libcurl. This happends when you try to copy thousands of files in 1 session, it doesn't happend when you upload hundreds of files or less.
* CentOS 4.x/RHEL 4.x users - if you use the kernel that shipped with your distribution and didn't upgrade to the latest kernel RedHat/CentOS gives, you might have a problem loading the "fuse" kernel. Please upgrade to the latest kernel (2.6.16 or above) and make sure "fuse" kernel module is compiled and loadable since FUSE requires this kernel module and s3fs requires it as well.
* Moving/renaming/erasing files takes time since the whole file needs to be accessed first. A workaround could be to use s3fs's cache support with the use_cache option.

View File

@ -20,87 +20,198 @@
dnl Process this file with autoconf to produce a configure script.
AC_PREREQ(2.59)
AC_INIT(s3fs, 1.74)
AC_INIT(s3fs, 1.78)
AC_CANONICAL_SYSTEM
AM_INIT_AUTOMAKE()
AC_PROG_CXX
AC_PROG_CC
CXXFLAGS="$CXXFLAGS -Wall -D_FILE_OFFSET_BITS=64"
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 libcrypto >= 0.9])
dnl ----------------------------------------------
dnl Choice SSL library
dnl ----------------------------------------------
auth_lib=na
nettle_lib=no
dnl malloc_trim function
AC_CHECK_FUNCS(malloc_trim, , )
dnl Initializing NSS(temporally)
AC_MSG_CHECKING([Initializing libcurl build with NSS])
AC_ARG_ENABLE(
nss-init,
dnl
dnl nettle library
dnl
AC_MSG_CHECKING([s3fs build with nettle(GnuTLS)])
AC_ARG_WITH(
nettle,
[AS_HELP_STRING([--with-nettle], [s3fs build with nettle in GnuTLS(default no)])],
[
AS_HELP_STRING(
[--enable-nss-init],
[Inilializing libcurl with NSS (default is no)]
)
],
[
case "${enableval}" in
case "${withval}" in
yes)
AC_MSG_RESULT(yes)
nss_init_enabled=yes
nettle_lib=yes
;;
*)
AC_MSG_RESULT(no)
;;
esac
],
[
AC_MSG_RESULT(no)
])
dnl
dnl use openssl library for ssl
dnl
AC_MSG_CHECKING([s3fs build with OpenSSL])
AC_ARG_WITH(
openssl,
[AS_HELP_STRING([--with-openssl], [s3fs build with OpenSSL(default is no)])],
[
case "${withval}" in
yes)
AC_MSG_RESULT(yes)
AS_IF(
[test $nettle_lib = no],
[auth_lib=openssl],
[AC_MSG_ERROR([could not set openssl with nettle, nettle is only for gnutls library])])
;;
*)
AC_MSG_RESULT(no)
;;
esac
],
[
AC_MSG_RESULT(no)
])
dnl
dnl use GnuTLS library for ssl
dnl
AC_MSG_CHECKING([s3fs build with GnuTLS])
AC_ARG_WITH(
gnutls,
[AS_HELP_STRING([--with-gnutls], [s3fs build with GnuTLS(default is no)])],
[
case "${withval}" in
yes)
AC_MSG_RESULT(yes)
AS_IF(
[test $auth_lib = na],
[
AS_IF(
[test $nettle_lib = no],
[auth_lib=gnutls],
[auth_lib=nettle])
],
[AC_MSG_ERROR([could not set gnutls because already set another ssl library])])
;;
*)
AC_MSG_RESULT(no)
;;
esac
],
[
AC_MSG_RESULT(no)
])
dnl
dnl use nss library for ssl
dnl
AC_MSG_CHECKING([s3fs build with NSS])
AC_ARG_WITH(
nss,
[AS_HELP_STRING([--with-nss], [s3fs build with NSS(default is no)])],
[
case "${withval}" in
yes)
AC_MSG_RESULT(yes)
AS_IF(
[test $auth_lib = na],
[
AS_IF(
[test $nettle_lib = no],
[auth_lib=nss],
[AC_MSG_ERROR([could not set openssl with nettle, nettle is only for gnutls library])])
],
[AC_MSG_ERROR([could not set nss because already set another ssl library])])
;;
*)
AC_MSG_RESULT(no)
nss_init_enabled=no
;;
esac
],
[
AC_MSG_RESULT(no)
nss_init_enabled=no
])
AS_IF(
[test $nss_init_enabled = yes],
[
AC_DEFINE(NSS_INIT_ENABLED, 1)
AC_CHECK_LIB(nss3, NSS_NoDB_Init, , [AC_MSG_ERROR(not found NSS libraries)])
AC_CHECK_LIB(plds4, PL_ArenaFinish, , [AC_MSG_ERROR(not found PL_ArenaFinish)])
AC_CHECK_LIB(nspr4, PR_Cleanup, , [AC_MSG_ERROR(not found PR_Cleanup)])
AC_CHECK_HEADER(nss.h, , [AC_MSG_ERROR(not found nss.h)])
AC_CHECK_HEADER(nspr4/prinit.h, , [AC_MSG_ERROR(not found prinit.h)])
AC_PATH_PROG(NSSCONFIG, [nss-config], no)
AS_IF(
[test $NSSCONFIG = no],
[
DEPS_CFLAGS="$DEPS_CFLAGS -I/usr/include/nss3"
DEPS_LIBS="$DEPS_LIBS -lnss3"
],
[
addcflags=`nss-config --cflags`
DEPS_CFLAGS="$DEPS_CFLAGS $addcflags"
dnl addlib=`nss-config --libs`
dnl DEPS_LIBS="$DEPS_LIBS $addlib"
DEPS_LIBS="$DEPS_LIBS -lnss3"
])
AC_PATH_PROG(NSPRCONFIG, [nspr-config], no)
AS_IF(
[test $NSPRCONFIG = no],
[
DEPS_CFLAGS="$DEPS_CFLAGS -I/usr/include/nspr4"
DEPS_LIBS="$DEPS_LIBS -lnspr4 -lplds4"
],
[
addcflags=`nspr-config --cflags`
DEPS_CFLAGS="$DEPS_CFLAGS $addcflags"
dnl addlib=`nspr-config --libs`
dnl DEPS_LIBS="$DEPS_LIBS $addlib"
DEPS_LIBS="$DEPS_LIBS -lnspr4 -lplds4"
])
])
[test $auth_lib = na],
AS_IF(
[test $nettle_lib = no],
[auth_lib=openssl],
[AC_MSG_ERROR([could not set nettle without GnuTLS library])]
)
)
AS_UNSET(nss_enabled)
dnl
dnl For PKG_CONFIG before checking nss/gnutls.
dnl this is redundant checking, but we need checking before following.
dnl
PKG_CHECK_MODULES([common_lib_checking], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6])
AC_MSG_CHECKING([compile s3fs with])
case "${auth_lib}" in
openssl)
AC_MSG_RESULT(OpenSSL)
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 libcrypto >= 0.9])
;;
gnutls)
AC_MSG_RESULT(GnuTLS-gcrypt)
gnutls_nettle=""
AC_CHECK_LIB(gnutls, gcry_control, [gnutls_nettle=0])
AS_IF([test "$gnutls_nettle" = ""], [AC_CHECK_LIB(gcrypt, gcry_control, [gnutls_nettle=0])])
AS_IF([test $gnutls_nettle = 0],
[
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 gnutls >= 2.12.0 ])
LIBS="-lgnutls -lgcrypt $LIBS"
AC_MSG_CHECKING([gnutls is build with])
AC_MSG_RESULT(gcrypt)
],
[AC_MSG_ERROR([GnuTLS found, but gcrypt not found])])
;;
nettle)
AC_MSG_RESULT(GnuTLS-nettle)
gnutls_nettle=""
AC_CHECK_LIB(gnutls, nettle_MD5Init, [gnutls_nettle=1])
AS_IF([test "$gnutls_nettle" = ""], [AC_CHECK_LIB(nettle, nettle_MD5Init, [gnutls_nettle=1])])
AS_IF([test $gnutls_nettle = 1],
[
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 nettle >= 2.7.1 ])
LIBS="-lgnutls -lnettle $LIBS"
AC_MSG_CHECKING([gnutls is build with])
AC_MSG_RESULT(nettle)
],
[AC_MSG_ERROR([GnuTLS found, but nettle not found])])
;;
nss)
AC_MSG_RESULT(NSS)
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 nss >= 3.15.0 ])
;;
*)
AC_MSG_ERROR([unknown ssl library type.])
;;
esac
AM_CONDITIONAL([USE_SSL_OPENSSL], [test "$auth_lib" = openssl])
AM_CONDITIONAL([USE_SSL_GNUTLS], [test "$auth_lib" = gnutls -o "$auth_lib" = nettle])
AM_CONDITIONAL([USE_GNUTLS_NETTLE], [test "$auth_lib" = nettle])
AM_CONDITIONAL([USE_SSL_NSS], [test "$auth_lib" = nss])
dnl ----------------------------------------------
dnl end of ssl library
dnl ----------------------------------------------
dnl malloc_trim function
AC_CHECK_FUNCS(malloc_trim, , )
AC_CONFIG_FILES(Makefile src/Makefile test/Makefile doc/Makefile)
AC_OUTPUT

View File

@ -1 +1,21 @@
######################################################################
# s3fs - FUSE-based file system backed by Amazon S3
#
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
######################################################################
dist_man1_MANS = man/s3fs.1

View File

@ -8,6 +8,9 @@ S3FS \- FUSE-based file system backed by Amazon S3
.SS unmounting
.TP
\fBumount mountpoint
.SS utility mode ( remove interrupted multipart uploading objects )
.TP
\fBs3fs -u bucket
.SH DESCRIPTION
s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).
.SH AUTHENTICATION
@ -68,9 +71,13 @@ this option can not be specified with use_sse.
(can specify use_rrs=1 for old version)
.TP
\fB\-o\fR use_sse (default is disable)
use Amazon's Server Site Encryption.
this option can not be specified with use_rrs.
(can specify use_sse=1 for old version)
use Amazon<EFBFBD>fs Server-Site Encryption or Server-Side Encryption with Customer-Provided Encryption Keys.
this option can not be specified with use_rrs. specifying only "use_sse" or "use_sse=1" enables Server-Side Encryption.(use_sse=1 for old version)
specifying this option with file path which has some SSE-C secret key enables Server-Side Encryption with Customer-Provided Encryption Keys.(use_sse=file)
the file must be 600 permission. the file can have some lines, each line is one SSE-C key. the first line in file is used as Customer-Provided Encryption Keys for uploading and change headers etc.
if there are some keys after first line, those are used downloading object which are encripted by not first key.
so that, you can keep all SSE-C keys in file, that is SSE-C key history.
if AWSSSECKEYS environment is set, you can set SSE-C key instead of this option.
.TP
\fB\-o\fR passwd_file (default="")
specify the path to the password file, which which takes precedence over the password in $HOME/.passwd-s3fs and /etc/passwd-s3fs
@ -102,7 +109,7 @@ time to wait for connection before giving up.
\fB\-o\fR readwrite_timeout (default="30" seconds)
time to wait between read/write activity before giving up.
.TP
\fB\-o\fR max_stat_cache_size (default="10000" entries (about 4MB))
\fB\-o\fR max_stat_cache_size (default="1000" entries (about 4MB))
maximum number of entries in the stat cache
.TP
\fB\-o\fR stat_cache_expire (default is no expire)
@ -125,14 +132,22 @@ maximum number of parallel request for listing objects.
.TP
\fB\-o\fR parallel_count (default="5")
number of parallel request for uploading big objects.
s3fs uploads large object(over 20MB) by multipart post request, and sends parallel requests.
s3fs uploads large object(default:over 20MB) by multipart post request, and sends parallel requests.
This option limits parallel request count which s3fs requests at once.
It is necessary to set this value depending on a CPU and a network band.
This option is lated to fd_page_size option and affects it.
.TP
\fB\-o\fR fd_page_size(default="52428800"(50MB))
number of internal management page size for each file discriptor.
For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
This option should not be changed when you don't have a trouble with performance.
This value is changed automatically by parallel_count and multipart_size values(fd_page_size value = parallel_count * multipart_size).
.TP
\fB\-o\fR multipart_size(default="10"(10MB))
number of one part size in multipart uploading request.
The default size is 10MB(10485760byte), this value is minimum size.
Specify number of MB and over 10(MB).
This option is lated to fd_page_size option and affects it.
.TP
\fB\-o\fR url (default="http://s3.amazonaws.com")
sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com
@ -160,6 +175,9 @@ If you set this option, s3fs do not use PUT with "x-amz-copy-source"(copy api).
For a distributed object storage which is compatibility S3 API without PUT(copy api).
This option is a subset of nocopyapi option. The nocopyapi option does not use copy-api for all command(ex. chmod, chown, touch, mv, etc), but this option does not use copy-api for only rename command(ex. mv).
If this option is specified with nocopapi, the s3fs ignores it.
.TP
\fB\-o\fR use_path_request_style (use legacy API calling style)
Enble compatibility with S3-like APIs which do not support the virtual-host request style, by using the older path request style.
.SH FUSE/MOUNT OPTIONS
.TP
Most of the generic mount options described in 'man mount' are supported (ro, rw, suid, nosuid, dev, nodev, exec, noexec, atime, noatime, sync async, dirsync). Filesystems are mounted with '-onodev,nosuid' by default, which can only be overridden by a privileged user.

View File

@ -1,7 +1,39 @@
######################################################################
# s3fs - FUSE-based file system backed by Amazon S3
#
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
######################################################################
bin_PROGRAMS=s3fs
AM_CPPFLAGS = $(DEPS_CFLAGS)
if USE_GNUTLS_NETTLE
AM_CPPFLAGS += -DUSE_GNUTLS_NETTLE
endif
s3fs_SOURCES = s3fs.cpp s3fs.h curl.cpp curl.h cache.cpp cache.h string_util.cpp string_util.h s3fs_util.cpp s3fs_util.h fdcache.cpp fdcache.h common_auth.cpp s3fs_auth.h common.h
if USE_SSL_OPENSSL
s3fs_SOURCES += openssl_auth.cpp
endif
if USE_SSL_GNUTLS
s3fs_SOURCES += gnutls_auth.cpp
endif
if USE_SSL_NSS
s3fs_SOURCES += nss_auth.cpp
endif
s3fs_SOURCES = s3fs.cpp s3fs.h curl.cpp curl.h cache.cpp cache.h string_util.cpp string_util.h s3fs_util.cpp s3fs_util.h fdcache.cpp fdcache.h common.h
s3fs_LDADD = $(DEPS_LIBS)

View File

@ -425,6 +425,7 @@ bool convert_header_to_stat(const char* path, headers_t& meta, struct stat* pst,
if(S_ISREG(pst->st_mode)){
pst->st_blocks = get_blocks(pst->st_size);
}
pst->st_blksize = 4096;
// mtime
pst->st_mtime = get_mtime(meta);
@ -439,3 +440,11 @@ bool convert_header_to_stat(const char* path, headers_t& meta, struct stat* pst,
return true;
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_CACHE_H_
#define S3FS_CACHE_H_
@ -105,3 +124,12 @@ class StatCache
bool convert_header_to_stat(const char* path, headers_t& meta, struct stat* pst, bool forcedir = false);
#endif // S3FS_CACHE_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_COMMON_H_
#define S3FS_COMMON_H_
@ -69,6 +88,7 @@ extern bool debug;
extern bool foreground;
extern bool foreground2;
extern bool nomultipart;
extern bool pathrequeststyle;
extern std::string program_name;
extern std::string service_path;
extern std::string host;
@ -76,3 +96,12 @@ extern std::string bucket;
extern std::string mount_prefix;
#endif // S3FS_COMMON_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

112
src/common_auth.cpp Normal file
View File

@ -0,0 +1,112 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <string>
#include "s3fs_auth.h"
using namespace std;
//-------------------------------------------------------------------
// Utility Function
//-------------------------------------------------------------------
char* s3fs_base64(unsigned char* input, size_t length)
{
static const char* base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
char* result;
if(!input || 0 >= length){
return NULL;
}
if(NULL == (result = (char*)malloc((((length / 3) + 1) * 4 + 1) * sizeof(char)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < length; rpos += 3){
parts[0] = (input[rpos] & 0xfc) >> 2;
parts[1] = ((input[rpos] & 0x03) << 4) | ((((rpos + 1) < length ? input[rpos + 1] : 0x00) & 0xf0) >> 4);
parts[2] = (rpos + 1) < length ? (((input[rpos + 1] & 0x0f) << 2) | ((((rpos + 2) < length ? input[rpos + 2] : 0x00) & 0xc0) >> 6)) : 0x40;
parts[3] = (rpos + 2) < length ? (input[rpos + 2] & 0x3f) : 0x40;
result[wpos++] = base[parts[0]];
result[wpos++] = base[parts[1]];
result[wpos++] = base[parts[2]];
result[wpos++] = base[parts[3]];
}
result[wpos] = '\0';
return result;
}
string s3fs_get_content_md5(int fd)
{
unsigned char* md5hex;
char* base64;
string Signature;
if(NULL == (md5hex = s3fs_md5hexsum(fd, 0, -1))){
return string("");
}
if(NULL == (base64 = s3fs_base64(md5hex, get_md5_digest_length()))){
return string(""); // ENOMEM
}
free(md5hex);
Signature = base64;
free(base64);
return Signature;
}
string s3fs_md5sum(int fd, off_t start, ssize_t size)
{
size_t digestlen = get_md5_digest_length();
char md5[2 * digestlen + 1];
char hexbuf[3];
unsigned char* md5hex;
if(NULL == (md5hex = s3fs_md5hexsum(fd, start, size))){
return string("");
}
memset(md5, 0, 2 * digestlen + 1);
for(size_t pos = 0; pos < digestlen; pos++){
snprintf(hexbuf, 3, "%02x", md5hex[pos]);
strncat(md5, hexbuf, 2);
}
free(md5hex);
return string(md5);
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

File diff suppressed because it is too large Load Diff

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_CURL_H_
#define S3FS_CURL_H_
@ -85,7 +104,7 @@ struct filepart
// for progress
struct case_insensitive_compare_func
{
bool operator()(const std::string& a, const std::string& b){
bool operator()(const std::string& a, const std::string& b) const {
return strcasecmp(a.c_str(), b.c_str()) < 0;
}
};
@ -100,18 +119,14 @@ class S3fsMultiCurl;
// class S3fsCurl
//----------------------------------------------
typedef std::map<std::string, std::string> iamcredmap_t;
typedef std::map<std::string, std::string> sseckeymap_t;
typedef std::list<sseckeymap_t> sseckeylist_t;
// share
#define SHARE_MUTEX_DNS 0
#define SHARE_MUTEX_SSL_SESSION 1
#define SHARE_MUTEX_MAX 2
// internal use struct for openssl
struct CRYPTO_dynlock_value
{
pthread_mutex_t dyn_mutex;
};
// Class for lapping curl
//
class S3fsCurl
@ -140,7 +155,6 @@ class S3fsCurl
// class variables
static pthread_mutex_t curl_handles_lock;
static pthread_mutex_t curl_share_lock[SHARE_MUTEX_MAX];
static pthread_mutex_t* crypt_mutex;
static bool is_initglobal_done;
static CURLSH* hCurlShare;
static bool is_dns_cache;
@ -151,6 +165,7 @@ class S3fsCurl
static bool is_public_bucket;
static std::string default_acl; // TODO: to enum
static bool is_use_rrs;
static sseckeylist_t sseckeys;
static bool is_use_sse;
static bool is_content_md5;
static bool is_verbose;
@ -160,12 +175,12 @@ class S3fsCurl
static time_t AWSAccessTokenExpire;
static std::string IAM_role;
static long ssl_verify_hostname;
static const EVP_MD* evp_md;
static curltime_t curl_times;
static curlprogress_t curl_progress;
static std::string curl_ca_bundle;
static mimes_t mimeTypes;
static int max_parallel_cnt;
static off_t multipart_size;
// variables
CURL* hCurl;
@ -189,6 +204,8 @@ class S3fsCurl
int b_postdata_remaining; // backup for retrying
off_t b_partdata_startpos; // backup for retrying
ssize_t b_partdata_size; // backup for retrying
bool b_ssekey_pos; // backup for retrying
std::string b_ssekey_md5; // backup for retrying
public:
// constructor/destructor
@ -205,11 +222,6 @@ class S3fsCurl
static void UnlockCurlShare(CURL* handle, curl_lock_data nLockData, void* useptr);
static bool InitCryptMutex(void);
static bool DestroyCryptMutex(void);
static void CryptMutexLock(int mode, int pos, const char* file, int line);
static unsigned long CryptGetThreadid(void);
static struct CRYPTO_dynlock_value* CreateDynCryptMutex(const char* file, int line);
static void DynCryptMutexLock(int mode, struct CRYPTO_dynlock_value* dyndata, const char* file, int line);
static void DestoryDynCryptMutex(struct CRYPTO_dynlock_value* dyndata, const char* file, int line);
static int CurlProgress(void *clientp, double dltotal, double dlnow, double ultotal, double ulnow);
static bool InitMimeType(const char* MimeFile = NULL);
@ -226,6 +238,7 @@ class S3fsCurl
static bool ParseIAMCredentialResponse(const char* response, iamcredmap_t& keyval);
static bool SetIAMCredentials(const char* response);
static bool PushbackSseKeys(std::string& onekey);
// methods
bool ResetHandle(void);
@ -235,7 +248,7 @@ class S3fsCurl
bool GetUploadId(std::string& upload_id);
int GetIAMCredentials(void);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool ow_sse_flg);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
int UploadMultipartPostSetup(const char* tpath, int part_num, std::string& upload_id);
int UploadMultipartPostRequest(const char* tpath, int part_num, std::string& upload_id);
@ -245,7 +258,7 @@ class S3fsCurl
// class methods
static bool InitS3fsCurl(const char* MimeFile = NULL);
static bool DestroyS3fsCurl(void);
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd);
static int ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
static bool CheckIAMCredentialUpdate(void);
@ -262,6 +275,12 @@ class S3fsCurl
static std::string SetDefaultAcl(const char* acl);
static bool SetUseRrs(bool flag);
static bool GetUseRrs(void) { return S3fsCurl::is_use_rrs; }
static bool SetSseKeys(const char* filepath);
static bool LoadEnvSseKeys(void);
static bool GetSseKey(std::string& md5, std::string& ssekey);
static bool GetSseKeyMd5(int pos, std::string& md5);
static int GetSseKeyCount(void);
static bool IsSseCustomMode(void);
static bool SetUseSse(bool flag);
static bool GetUseSse(void) { return S3fsCurl::is_use_sse; }
static bool SetContentMd5(bool flag);
@ -274,31 +293,35 @@ class S3fsCurl
static long SetSslVerifyHostname(long value);
static long GetSslVerifyHostname(void) { return S3fsCurl::ssl_verify_hostname; }
static int SetMaxParallelCount(int value);
static int GetMaxParallelCount(void) { return S3fsCurl::max_parallel_cnt; }
static std::string SetIAMRole(const char* role);
static const char* GetIAMRole(void) { return S3fsCurl::IAM_role.c_str(); }
static bool SetMultipartSize(off_t size);
static off_t GetMultipartSize(void) { return S3fsCurl::multipart_size; }
// methods
bool CreateCurlHandle(bool force = false);
bool DestroyCurlHandle(void);
bool AddSseKeyRequestHead(std::string& md5, bool is_copy);
bool GetResponseCode(long& responseCode);
int RequestPerform(void);
int DeleteRequest(const char* tpath);
bool PreHeadRequest(const char* tpath, const char* bpath = NULL, const char* savedpath = NULL);
bool PreHeadRequest(std::string& tpath, std::string& bpath, std::string& savedpath) {
return PreHeadRequest(tpath.c_str(), bpath.c_str(), savedpath.c_str());
bool PreHeadRequest(const char* tpath, const char* bpath = NULL, const char* savedpath = NULL, int ssekey_pos = -1);
bool PreHeadRequest(std::string& tpath, std::string& bpath, std::string& savedpath, int ssekey_pos = -1) {
return PreHeadRequest(tpath.c_str(), bpath.c_str(), savedpath.c_str(), ssekey_pos);
}
int HeadRequest(const char* tpath, headers_t& meta);
int PutHeadRequest(const char* tpath, headers_t& meta, bool ow_sse_flg);
int PutRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
int PutHeadRequest(const char* tpath, headers_t& meta, bool is_copy);
int PutRequest(const char* tpath, headers_t& meta, int fd);
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, std::string& ssekeymd5);
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
int CheckBucket(void);
int ListBucketRequest(const char* tpath, const char* query);
int MultipartListRequest(std::string& body);
int AbortMultipartUpload(const char* tpath, std::string& upload_id);
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta);
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta, bool is_copy);
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool is_copy);
int MultipartRenameRequest(const char* from, const char* to, headers_t& meta, off_t size);
// methods(valiables)
@ -318,6 +341,7 @@ class S3fsCurl
int GetMultipartRetryCount(void) const { return retry_count; }
void SetMultipartRetryCount(int retrycnt) { retry_count = retrycnt; }
bool IsOverMultipartRetryCount(void) const { return (retry_count >= S3fsCurl::retries); }
int GetLastPreHeadSeecKeyPos(void) const { return b_ssekey_pos; }
};
//----------------------------------------------
@ -400,3 +424,12 @@ struct curl_slist* curl_slist_sort_insert(struct curl_slist* list, const char* d
bool MakeUrlResource(const char* realpath, std::string& resourcepath, std::string& url);
#endif // S3FS_CURL_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -31,7 +31,6 @@
#include <string.h>
#include <assert.h>
#include <curl/curl.h>
#include <openssl/crypto.h>
#include <string>
#include <iostream>
#include <sstream>
@ -51,8 +50,7 @@ using namespace std;
//------------------------------------------------
// Symbols
//------------------------------------------------
#define MAX_OBJECT_SIZE 68719476735LL // 64GB - 1L
#define MULTIPART_LOWLIMIT (20 * 1024 * 1024) // 20MB
#define MAX_MULTIPART_CNT 10000 // S3 multipart max count
#define FDPAGE_SIZE (50 * 1024 * 1024) // 50MB(parallel uploading is 5 parallel(default) * 10 MB)
//------------------------------------------------
@ -266,7 +264,7 @@ int PageList::Init(off_t size, bool is_init)
{
Clear();
for(off_t total = 0; total < size; total += FdManager::GetPageSize()){
size_t areasize = (total + FdManager::GetPageSize()) < size ? FdManager::GetPageSize() : static_cast<size_t>(size - total);
size_t areasize = (total + static_cast<off_t>(FdManager::GetPageSize())) < size ? FdManager::GetPageSize() : static_cast<size_t>(size - total);
fdpage* page = new fdpage(total, areasize, is_init);
pages.push_back(page);
}
@ -798,7 +796,7 @@ int FdEntity::Load(off_t start, off_t size)
break;
}
// download
if((*iter)->bytes >= MULTIPART_LOWLIMIT && !nomultipart){ // 20MB
if((*iter)->bytes >= static_cast<size_t>(2 * S3fsCurl::GetMultipartSize()) && !nomultipart){ // default 20MB
// parallel request
// Additional time is needed for large files
time_t backup = 0;
@ -857,7 +855,7 @@ bool FdEntity::LoadFull(off_t* size, bool force_load)
return true;
}
int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool force_sync)
int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool force_sync)
{
int result;
@ -880,14 +878,14 @@ int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool
* - 1 to 10,000 parts are allowed
* - minimum size of parts is 5MB (expect for the last part)
*
* For our application, we will define part size to be 10MB (10 * 2^20 Bytes)
* maximum file size will be ~64 GB - 2 ** 36
* For our application, we will define minimum part size to be 10MB (10 * 2^20 Bytes)
* minimum file size will be 64 GB - 2 ** 36
*
* Initially uploads will be done serially
*
* If file is > 20MB, then multipart will kick in
*/
if(pagelist.Size() > MAX_OBJECT_SIZE){ // 64GB - 1
if(pagelist.Size() > (MAX_MULTIPART_CNT * S3fsCurl::GetMultipartSize())){
// close f ?
return -ENOTSUP;
}
@ -898,19 +896,19 @@ int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool
return -errno;
}
if(pagelist.Size() >= MULTIPART_LOWLIMIT && !nomultipart){ // 20MB
if(pagelist.Size() >= (2 * S3fsCurl::GetMultipartSize()) && !nomultipart){ // default 20MB
// Additional time is needed for large files
time_t backup = 0;
if(120 > S3fsCurl::GetReadwriteTimeout()){
backup = S3fsCurl::SetReadwriteTimeout(120);
}
result = S3fsCurl::ParallelMultipartUploadRequest(tpath ? tpath : path.c_str(), meta, fd, ow_sse_flg);
result = S3fsCurl::ParallelMultipartUploadRequest(tpath ? tpath : path.c_str(), meta, fd);
if(0 != backup){
S3fsCurl::SetReadwriteTimeout(backup);
}
}else{
S3fsCurl s3fscurl(true);
result = s3fscurl.PutRequest(tpath ? tpath : path.c_str(), meta, fd, ow_sse_flg);
result = s3fscurl.PutRequest(tpath ? tpath : path.c_str(), meta, fd);
}
// seek to head of file.
@ -1198,3 +1196,11 @@ bool FdManager::Close(FdEntity* ent)
return false;
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef FD_CACHE_H_
#define FD_CACHE_H_
@ -110,8 +129,8 @@ class FdEntity
bool SetAllDisable(void) { return SetAllStatus(false); }
bool LoadFull(off_t* size = NULL, bool force_load = false);
int Load(off_t start, off_t size);
int RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool force_sync = false);
int Flush(headers_t& meta, bool ow_sse_flg, bool force_sync = false) { return RowFlush(NULL, meta, ow_sse_flg, force_sync); }
int RowFlush(const char* tpath, headers_t& meta, bool force_sync = false);
int Flush(headers_t& meta, bool force_sync = false) { return RowFlush(NULL, meta, force_sync); }
ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false);
ssize_t Write(const char* bytes, off_t start, size_t size);
};
@ -154,3 +173,12 @@ class FdManager
};
#endif // FD_CACHE_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

266
src/gnutls_auth.cpp Normal file
View File

@ -0,0 +1,266 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>
#include <unistd.h>
#include <syslog.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <string.h>
#include <gcrypt.h>
#include <gnutls/gnutls.h>
#include <gnutls/crypto.h>
#ifdef USE_GNUTLS_NETTLE
#include <nettle/md5.h>
#include <nettle/sha1.h>
#include <nettle/hmac.h>
#endif
#include <string>
#include <map>
#include "common.h"
#include "s3fs_auth.h"
using namespace std;
//-------------------------------------------------------------------
// Utility Function for version
//-------------------------------------------------------------------
#ifdef USE_GNUTLS_NETTLE
const char* s3fs_crypt_lib_name(void)
{
static const char version[] = "GnuTLS(nettle)";
return version;
}
#else // USE_GNUTLS_NETTLE
const char* s3fs_crypt_lib_name(void)
{
static const char version[] = "GnuTLS(gcrypt)";
return version;
}
#endif // USE_GNUTLS_NETTLE
//-------------------------------------------------------------------
// Utility Function for global init
//-------------------------------------------------------------------
bool s3fs_init_global_ssl(void)
{
if(GNUTLS_E_SUCCESS != gnutls_global_init()){
return false;
}
return true;
}
bool s3fs_destroy_global_ssl(void)
{
gnutls_global_deinit();
return true;
}
//-------------------------------------------------------------------
// Utility Function for crypt lock
//-------------------------------------------------------------------
bool s3fs_init_crypt_mutex(void)
{
return true;
}
bool s3fs_destroy_crypt_mutex(void)
{
return true;
}
//-------------------------------------------------------------------
// Utility Function for HMAC
//-------------------------------------------------------------------
#ifdef USE_GNUTLS_NETTLE
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
{
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
return false;
}
if(NULL == (*digest = (unsigned char*)malloc(SHA1_DIGEST_SIZE))){
return false;
}
struct hmac_sha1_ctx ctx_hmac;
hmac_sha1_set_key(&ctx_hmac, keylen, reinterpret_cast<const uint8_t*>(key));
hmac_sha1_update(&ctx_hmac, datalen, reinterpret_cast<const uint8_t*>(data));
hmac_sha1_digest(&ctx_hmac, SHA1_DIGEST_SIZE, reinterpret_cast<uint8_t*>(*digest));
*digestlen = SHA1_DIGEST_SIZE;
return true;
}
#else // USE_GNUTLS_NETTLE
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
{
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
return false;
}
if(0 >= (*digestlen = gnutls_hmac_get_len(GNUTLS_MAC_SHA1))){
return false;
}
if(NULL == (*digest = (unsigned char*)malloc(*digestlen + 1))){
return false;
}
if(0 > gnutls_hmac_fast(GNUTLS_MAC_SHA1, key, keylen, data, datalen, *digest)){
free(*digest);
*digest = NULL;
return false;
}
return true;
}
#endif // USE_GNUTLS_NETTLE
//-------------------------------------------------------------------
// Utility Function for MD5
//-------------------------------------------------------------------
#define MD5_DIGEST_LENGTH 16
size_t get_md5_digest_length(void)
{
return MD5_DIGEST_LENGTH;
}
#ifdef USE_GNUTLS_NETTLE
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
{
struct md5_ctx ctx_md5;
unsigned char buf[512];
ssize_t bytes;
unsigned char* result;
// seek to top of file.
if(-1 == lseek(fd, start, SEEK_SET)){
return NULL;
}
memset(buf, 0, 512);
md5_init(&ctx_md5);
for(ssize_t total = 0; total < size; total += bytes){
bytes = 512 < (size - total) ? 512 : (size - total);
bytes = read(fd, buf, bytes);
if(0 == bytes){
// end of file
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
return NULL;
}
md5_update(&ctx_md5, bytes, buf);
memset(buf, 0, 512);
}
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
return NULL;
}
md5_digest(&ctx_md5, get_md5_digest_length(), result);
if(-1 == lseek(fd, start, SEEK_SET)){
free(result);
return NULL;
}
return result;
}
#else // USE_GNUTLS_NETTLE
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
{
gcry_md_hd_t ctx_md5;
gcry_error_t err;
char buf[512];
ssize_t bytes;
unsigned char* result;
if(-1 == size){
struct stat st;
if(-1 == fstat(fd, &st)){
return NULL;
}
size = static_cast<ssize_t>(st.st_size);
}
// seek to top of file.
if(-1 == lseek(fd, start, SEEK_SET)){
return NULL;
}
memset(buf, 0, 512);
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_md5, GCRY_MD_MD5, 0))){
DPRNN("MD5 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
return NULL;
}
for(ssize_t total = 0; total < size; total += bytes){
bytes = 512 < (size - total) ? 512 : (size - total);
bytes = read(fd, buf, bytes);
if(0 == bytes){
// end of file
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
return NULL;
}
gcry_md_write(ctx_md5, buf, bytes);
memset(buf, 0, 512);
}
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
return NULL;
}
memcpy(result, gcry_md_read(ctx_md5, 0), get_md5_digest_length());
gcry_md_close(ctx_md5);
if(-1 == lseek(fd, start, SEEK_SET)){
free(result);
return NULL;
}
return result;
}
#endif // USE_GNUTLS_NETTLE
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

203
src/nss_auth.cpp Normal file
View File

@ -0,0 +1,203 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>
#include <unistd.h>
#include <syslog.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <string.h>
#include <nss.h>
#include <pk11pub.h>
#include <hasht.h>
#include <prinit.h>
#include <string>
#include <map>
#include "common.h"
#include "s3fs_auth.h"
using namespace std;
//-------------------------------------------------------------------
// Utility Function for version
//-------------------------------------------------------------------
const char* s3fs_crypt_lib_name(void)
{
static const char version[] = "NSS";
return version;
}
//-------------------------------------------------------------------
// Utility Function for global init
//-------------------------------------------------------------------
bool s3fs_init_global_ssl(void)
{
NSS_Init(NULL);
NSS_NoDB_Init(NULL);
return true;
}
bool s3fs_destroy_global_ssl(void)
{
NSS_Shutdown();
PL_ArenaFinish();
PR_Cleanup();
return true;
}
//-------------------------------------------------------------------
// Utility Function for crypt lock
//-------------------------------------------------------------------
bool s3fs_init_crypt_mutex(void)
{
return true;
}
bool s3fs_destroy_crypt_mutex(void)
{
return true;
}
//-------------------------------------------------------------------
// Utility Function for HMAC
//-------------------------------------------------------------------
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
{
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
return false;
}
PK11SlotInfo* Slot;
PK11SymKey* pKey;
PK11Context* Context;
SECStatus SecStatus;
unsigned char tmpdigest[64];
SECItem KeySecItem = {siBuffer, reinterpret_cast<unsigned char*>(const_cast<void*>(key)), keylen};
SECItem NullSecItem = {siBuffer, NULL, 0};
if(NULL == (Slot = PK11_GetInternalKeySlot())){
return false;
}
if(NULL == (pKey = PK11_ImportSymKey(Slot, CKM_SHA_1_HMAC, PK11_OriginUnwrap, CKA_SIGN, &KeySecItem, NULL))){
PK11_FreeSlot(Slot);
return false;
}
if(NULL == (Context = PK11_CreateContextBySymKey(CKM_SHA_1_HMAC, CKA_SIGN, pKey, &NullSecItem))){
PK11_FreeSymKey(pKey);
PK11_FreeSlot(Slot);
return false;
}
*digestlen = 0;
if(SECSuccess != (SecStatus = PK11_DigestBegin(Context)) ||
SECSuccess != (SecStatus = PK11_DigestOp(Context, data, datalen)) ||
SECSuccess != (SecStatus = PK11_DigestFinal(Context, tmpdigest, digestlen, sizeof(tmpdigest))) )
{
PK11_DestroyContext(Context, PR_TRUE);
PK11_FreeSymKey(pKey);
PK11_FreeSlot(Slot);
return false;
}
PK11_DestroyContext(Context, PR_TRUE);
PK11_FreeSymKey(pKey);
PK11_FreeSlot(Slot);
if(NULL == (*digest = (unsigned char*)malloc(*digestlen))){
return false;
}
memcpy(*digest, tmpdigest, *digestlen);
return true;
}
//-------------------------------------------------------------------
// Utility Function for MD5
//-------------------------------------------------------------------
size_t get_md5_digest_length(void)
{
return MD5_LENGTH;
}
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
{
PK11Context* md5ctx;
unsigned char buf[512];
ssize_t bytes;
unsigned char* result;
unsigned int md5outlen;
if(-1 == size){
struct stat st;
if(-1 == fstat(fd, &st)){
return NULL;
}
size = static_cast<ssize_t>(st.st_size);
}
// seek to top of file.
if(-1 == lseek(fd, start, SEEK_SET)){
return NULL;
}
memset(buf, 0, 512);
md5ctx = PK11_CreateDigestContext(SEC_OID_MD5);
for(ssize_t total = 0; total < size; total += bytes){
bytes = 512 < (size - total) ? 512 : (size - total);
bytes = read(fd, buf, bytes);
if(0 == bytes){
// end of file
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
return NULL;
}
PK11_DigestOp(md5ctx, buf, bytes);
memset(buf, 0, 512);
}
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
PK11_DestroyContext(md5ctx, PR_TRUE);
return NULL;
}
PK11_DigestFinal(md5ctx, result, &md5outlen, get_md5_digest_length());
PK11_DestroyContext(md5ctx, PR_TRUE);
if(-1 == lseek(fd, start, SEEK_SET)){
free(result);
return NULL;
}
return result;
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

263
src/openssl_auth.cpp Normal file
View File

@ -0,0 +1,263 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>
#include <unistd.h>
#include <syslog.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <string.h>
#include <openssl/bio.h>
#include <openssl/buffer.h>
#include <openssl/evp.h>
#include <openssl/hmac.h>
#include <openssl/md5.h>
#include <openssl/crypto.h>
#include <openssl/err.h>
#include <string>
#include <map>
#include "common.h"
#include "s3fs_auth.h"
using namespace std;
//-------------------------------------------------------------------
// Utility Function for version
//-------------------------------------------------------------------
const char* s3fs_crypt_lib_name(void)
{
static const char version[] = "OpenSSL";
return version;
}
//-------------------------------------------------------------------
// Utility Function for global init
//-------------------------------------------------------------------
bool s3fs_init_global_ssl(void)
{
ERR_load_crypto_strings();
ERR_load_BIO_strings();
OpenSSL_add_all_algorithms();
return true;
}
bool s3fs_destroy_global_ssl(void)
{
EVP_cleanup();
ERR_free_strings();
return true;
}
//-------------------------------------------------------------------
// Utility Function for crypt lock
//-------------------------------------------------------------------
// internal use struct for openssl
struct CRYPTO_dynlock_value
{
pthread_mutex_t dyn_mutex;
};
static pthread_mutex_t* s3fs_crypt_mutex = NULL;
static void s3fs_crypt_mutex_lock(int mode, int pos, const char* file, int line)
{
if(s3fs_crypt_mutex){
if(mode & CRYPTO_LOCK){
pthread_mutex_lock(&s3fs_crypt_mutex[pos]);
}else{
pthread_mutex_unlock(&s3fs_crypt_mutex[pos]);
}
}
}
static unsigned long s3fs_crypt_get_threadid(void)
{
return static_cast<unsigned long>(pthread_self());
}
static struct CRYPTO_dynlock_value* s3fs_dyn_crypt_mutex(const char* file, int line)
{
struct CRYPTO_dynlock_value* dyndata;
if(NULL == (dyndata = static_cast<struct CRYPTO_dynlock_value*>(malloc(sizeof(struct CRYPTO_dynlock_value))))){
DPRNCRIT("Could not allocate memory for CRYPTO_dynlock_value");
return NULL;
}
pthread_mutex_init(&(dyndata->dyn_mutex), NULL);
return dyndata;
}
static void s3fs_dyn_crypt_mutex_lock(int mode, struct CRYPTO_dynlock_value* dyndata, const char* file, int line)
{
if(dyndata){
if(mode & CRYPTO_LOCK){
pthread_mutex_lock(&(dyndata->dyn_mutex));
}else{
pthread_mutex_unlock(&(dyndata->dyn_mutex));
}
}
}
static void s3fs_destroy_dyn_crypt_mutex(struct CRYPTO_dynlock_value* dyndata, const char* file, int line)
{
if(dyndata){
pthread_mutex_destroy(&(dyndata->dyn_mutex));
free(dyndata);
}
}
bool s3fs_init_crypt_mutex(void)
{
if(s3fs_crypt_mutex){
FPRNNN("s3fs_crypt_mutex is not NULL, destory it.");
if(!s3fs_destroy_crypt_mutex()){
DPRN("Failed to s3fs_crypt_mutex");
return false;
}
}
if(NULL == (s3fs_crypt_mutex = static_cast<pthread_mutex_t*>(malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t))))){
DPRNCRIT("Could not allocate memory for s3fs_crypt_mutex");
return false;
}
for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){
pthread_mutex_init(&s3fs_crypt_mutex[cnt], NULL);
}
// static lock
CRYPTO_set_locking_callback(s3fs_crypt_mutex_lock);
CRYPTO_set_id_callback(s3fs_crypt_get_threadid);
// dynamic lock
CRYPTO_set_dynlock_create_callback(s3fs_dyn_crypt_mutex);
CRYPTO_set_dynlock_lock_callback(s3fs_dyn_crypt_mutex_lock);
CRYPTO_set_dynlock_destroy_callback(s3fs_destroy_dyn_crypt_mutex);
return true;
}
bool s3fs_destroy_crypt_mutex(void)
{
if(!s3fs_crypt_mutex){
return true;
}
CRYPTO_set_dynlock_destroy_callback(NULL);
CRYPTO_set_dynlock_lock_callback(NULL);
CRYPTO_set_dynlock_create_callback(NULL);
CRYPTO_set_id_callback(NULL);
CRYPTO_set_locking_callback(NULL);
for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){
pthread_mutex_destroy(&s3fs_crypt_mutex[cnt]);
}
CRYPTO_cleanup_all_ex_data();
free(s3fs_crypt_mutex);
s3fs_crypt_mutex = NULL;
return true;
}
//-------------------------------------------------------------------
// Utility Function for HMAC
//-------------------------------------------------------------------
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
{
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
return false;
}
(*digestlen) = EVP_MAX_MD_SIZE * sizeof(unsigned char);
if(NULL == ((*digest) = (unsigned char*)malloc(*digestlen))){
return false;
}
HMAC(EVP_sha1(), key, keylen, data, datalen, *digest, digestlen);
return true;
}
//-------------------------------------------------------------------
// Utility Function for MD5
//-------------------------------------------------------------------
size_t get_md5_digest_length(void)
{
return MD5_DIGEST_LENGTH;
}
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
{
MD5_CTX md5ctx;
char buf[512];
ssize_t bytes;
unsigned char* result;
if(-1 == size){
struct stat st;
if(-1 == fstat(fd, &st)){
return NULL;
}
size = static_cast<ssize_t>(st.st_size);
}
// seek to top of file.
if(-1 == lseek(fd, start, SEEK_SET)){
return NULL;
}
memset(buf, 0, 512);
MD5_Init(&md5ctx);
for(ssize_t total = 0; total < size; total += bytes){
bytes = 512 < (size - total) ? 512 : (size - total);
bytes = read(fd, buf, bytes);
if(0 == bytes){
// end of file
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
return NULL;
}
MD5_Update(&md5ctx, buf, bytes);
memset(buf, 0, 512);
}
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
return NULL;
}
MD5_Final(result, &md5ctx);
if(-1 == lseek(fd, start, SEEK_SET)){
free(result);
return NULL;
}
return result;
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -31,7 +31,6 @@
#include <libxml/xpathInternals.h>
#include <libxml/tree.h>
#include <curl/curl.h>
#include <openssl/crypto.h>
#include <pwd.h>
#include <grp.h>
#include <getopt.h>
@ -50,6 +49,7 @@
#include "string_util.h"
#include "s3fs_util.h"
#include "fdcache.h"
#include "s3fs_auth.h"
using namespace std;
@ -83,6 +83,7 @@ bool debug = false;
bool foreground = false;
bool foreground2 = false;
bool nomultipart = false;
bool pathrequeststyle = false;
std::string program_name;
std::string service_path = "/";
std::string host = "http://s3.amazonaws.com";
@ -134,7 +135,7 @@ static xmlChar* get_base_exp(xmlDocPtr doc, const char* exp);
static xmlChar* get_prefix(xmlDocPtr doc);
static xmlChar* get_next_marker(xmlDocPtr doc);
static char* get_object_name(xmlDocPtr doc, xmlNodePtr node, const char* path);
static int put_headers(const char* path, headers_t& meta, bool ow_sse_flg);
static int put_headers(const char* path, headers_t& meta, bool is_copy);
static int rename_large_object(const char* from, const char* to);
static int create_file_object(const char* path, mode_t mode, uid_t uid, gid_t gid);
static int create_directory_object(const char* path, mode_t mode, time_t time, uid_t uid, gid_t gid);
@ -609,6 +610,30 @@ static int check_parent_object_access(const char* path, int mask)
return 0;
}
//
// This function is global, is called fom curl class(GetObject).
//
char* get_object_sseckey_md5(const char* path)
{
if(!path){
return NULL;
}
headers_t meta;
if(0 != get_object_attribute(path, NULL, &meta)){
DPRNNN("Failed to get object(%s) headers", path);
return NULL;
}
for(headers_t::iterator iter = meta.begin(); iter != meta.end(); ++iter){
string key = (*iter).first;
if(0 == strcasecmp(key.c_str(), "x-amz-server-side-encryption-customer-key-md5")){
return strdup((*iter).second.c_str());
}
}
return NULL;
}
static FdEntity* get_local_fent(const char* path, bool is_load)
{
struct stat stobj;
@ -643,7 +668,7 @@ static FdEntity* get_local_fent(const char* path, bool is_load)
* ow_sse_flg is for over writing sse header by use_sse option.
* @return fuse return code
*/
static int put_headers(const char* path, headers_t& meta, bool ow_sse_flg)
static int put_headers(const char* path, headers_t& meta, bool is_copy)
{
int result;
S3fsCurl s3fscurl(true);
@ -658,11 +683,11 @@ static int put_headers(const char* path, headers_t& meta, bool ow_sse_flg)
if(buf.st_size >= FIVE_GB){
// multipart
if(0 != (result = s3fscurl.MultipartHeadRequest(path, buf.st_size, meta))){
if(0 != (result = s3fscurl.MultipartHeadRequest(path, buf.st_size, meta, is_copy))){
return result;
}
}else{
if(0 != (result = s3fscurl.PutHeadRequest(path, meta, ow_sse_flg))){
if(0 != (result = s3fscurl.PutHeadRequest(path, meta, is_copy))){
return result;
}
}
@ -708,6 +733,8 @@ static int s3fs_getattr(const char* path, struct stat* stbuf)
}
FdManager::get()->Close(ent);
}
stbuf->st_blksize = 4096;
stbuf->st_blocks = get_blocks(stbuf->st_size);
}
FPRNINFO("[path=%s] uid=%u, gid=%u, mode=%04o", path, (unsigned int)(stbuf->st_uid), (unsigned int)(stbuf->st_gid), stbuf->st_mode);
S3FS_MALLOCTRIM(0);
@ -764,7 +791,7 @@ static int create_file_object(const char* path, mode_t mode, uid_t uid, gid_t gi
meta["x-amz-meta-mtime"] = str(time(NULL));
S3fsCurl s3fscurl(true);
return s3fscurl.PutRequest(path, meta, -1, false); // fd=-1 means for creating zero byte object.
return s3fscurl.PutRequest(path, meta, -1); // fd=-1 means for creating zero byte object.
}
static int s3fs_mknod(const char *path, mode_t mode, dev_t rdev)
@ -849,7 +876,7 @@ static int create_directory_object(const char* path, mode_t mode, time_t time, u
meta["x-amz-meta-mtime"] = str(time);
S3fsCurl s3fscurl;
return s3fscurl.PutRequest(tpath.c_str(), meta, -1, false); // fd=-1 means for creating zero byte object.
return s3fscurl.PutRequest(tpath.c_str(), meta, -1); // fd=-1 means for creating zero byte object.
}
static int s3fs_mkdir(const char* path, mode_t mode)
@ -1011,7 +1038,7 @@ static int s3fs_symlink(const char* from, const char* to)
return -errno;
}
// upload
if(0 != (result = ent->Flush(headers, true, true))){
if(0 != (result = ent->Flush(headers, true))){
DPRN("could not upload tmpfile(result=%d)", result);
}
FdManager::get()->Close(ent);
@ -1047,7 +1074,7 @@ static int rename_object(const char* from, const char* to)
meta["Content-Type"] = S3fsCurl::LookupMimeType(string(to));
meta["x-amz-metadata-directive"] = "REPLACE";
if(0 != (result = put_headers(to, meta, false))){
if(0 != (result = put_headers(to, meta, true))){
return result;
}
result = s3fs_unlink(from);
@ -1088,7 +1115,7 @@ static int rename_object_nocopy(const char* from, const char* to)
}
// upload
if(0 != (result = ent->RowFlush(to, meta, false, true))){
if(0 != (result = ent->RowFlush(to, meta, true))){
DPRN("could not upload file(%s): result=%d", to, result);
FdManager::get()->Close(ent);
return result;
@ -1388,7 +1415,7 @@ static int s3fs_chmod(const char* path, mode_t mode)
meta["x-amz-copy-source"] = urlEncode(service_path + bucket + get_realpath(strpath.c_str()));
meta["x-amz-metadata-directive"] = "REPLACE";
if(put_headers(strpath.c_str(), meta, false) != 0){
if(put_headers(strpath.c_str(), meta, true) != 0){
return -EIO;
}
StatCache::getStatCacheData()->DelStat(nowcache);
@ -1464,7 +1491,7 @@ static int s3fs_chmod_nocopy(const char* path, mode_t mode)
}
// upload
if(0 != (result = ent->Flush(meta, false, true))){
if(0 != (result = ent->Flush(meta, true))){
DPRN("could not upload file(%s): result=%d", strpath.c_str(), result);
FdManager::get()->Close(ent);
return result;
@ -1550,7 +1577,7 @@ static int s3fs_chown(const char* path, uid_t uid, gid_t gid)
meta["x-amz-copy-source"] = urlEncode(service_path + bucket + get_realpath(strpath.c_str()));
meta["x-amz-metadata-directive"] = "REPLACE";
if(put_headers(strpath.c_str(), meta, false) != 0){
if(put_headers(strpath.c_str(), meta, true) != 0){
return -EIO;
}
StatCache::getStatCacheData()->DelStat(nowcache);
@ -1636,7 +1663,7 @@ static int s3fs_chown_nocopy(const char* path, uid_t uid, gid_t gid)
}
// upload
if(0 != (result = ent->Flush(meta, false, true))){
if(0 != (result = ent->Flush(meta, true))){
DPRN("could not upload file(%s): result=%d", strpath.c_str(), result);
FdManager::get()->Close(ent);
return result;
@ -1708,7 +1735,7 @@ static int s3fs_utimens(const char* path, const struct timespec ts[2])
meta["x-amz-copy-source"] = urlEncode(service_path + bucket + get_realpath(strpath.c_str()));
meta["x-amz-metadata-directive"] = "REPLACE";
if(put_headers(strpath.c_str(), meta, false) != 0){
if(put_headers(strpath.c_str(), meta, true) != 0){
return -EIO;
}
StatCache::getStatCacheData()->DelStat(nowcache);
@ -1793,7 +1820,7 @@ static int s3fs_utimens_nocopy(const char* path, const struct timespec ts[2])
}
// upload
if(0 != (result = ent->Flush(meta, false, true))){
if(0 != (result = ent->Flush(meta, true))){
DPRN("could not upload file(%s): result=%d", strpath.c_str(), result);
FdManager::get()->Close(ent);
return result;
@ -1844,7 +1871,7 @@ static int s3fs_truncate(const char* path, off_t size)
}
// upload
if(0 != (result = ent->Flush(meta, false, true))){
if(0 != (result = ent->Flush(meta, true))){
DPRN("could not upload file(%s): result=%d", path, result);
FdManager::get()->Close(ent);
return result;
@ -1999,7 +2026,7 @@ static int s3fs_flush(const char* path, struct fuse_file_info* fi)
meta["x-amz-meta-mtime"] = str(ent_mtime);
}
}
result = ent->Flush(meta, true, false);
result = ent->Flush(meta, false);
FdManager::get()->Close(ent);
}
S3FS_MALLOCTRIM(0);
@ -2075,9 +2102,23 @@ static S3fsCurl* multi_head_retry_callback(S3fsCurl* s3fscurl)
if(!s3fscurl){
return NULL;
}
int ssec_key_pos = s3fscurl->GetLastPreHeadSeecKeyPos();
int next_retry_count = s3fscurl->GetMultipartRetryCount() + 1;
if(s3fscurl->IsOverMultipartRetryCount()){
DPRN("Over retry count(%d) limit(%s).", s3fscurl->GetMultipartRetryCount(), s3fscurl->GetSpacialSavedPath().c_str());
return NULL;
if(S3fsCurl::IsSseCustomMode()){
// If sse-c mode, start check not sse-c(ssec_key_pos = -1).
// do increment ssec_key_pos for checking all sse-c key.
next_retry_count = 0;
ssec_key_pos++;
if(S3fsCurl::GetSseKeyCount() <= ssec_key_pos){
DPRN("Over retry count(%d) limit(%s).", s3fscurl->GetMultipartRetryCount(), s3fscurl->GetSpacialSavedPath().c_str());
return NULL;
}
}else{
DPRN("Over retry count(%d) limit(%s).", s3fscurl->GetMultipartRetryCount(), s3fscurl->GetSpacialSavedPath().c_str());
return NULL;
}
}
S3fsCurl* newcurl = new S3fsCurl(s3fscurl->IsUseAhbe());
@ -2085,12 +2126,12 @@ static S3fsCurl* multi_head_retry_callback(S3fsCurl* s3fscurl)
string base_path = s3fscurl->GetBasePath();
string saved_path = s3fscurl->GetSpacialSavedPath();
if(!newcurl->PreHeadRequest(path, base_path, saved_path)){
if(!newcurl->PreHeadRequest(path, base_path, saved_path, ssec_key_pos)){
DPRN("Could not duplicate curl object(%s).", saved_path.c_str());
delete newcurl;
return NULL;
}
newcurl->SetMultipartRetryCount(s3fscurl->GetMultipartRetryCount() + 1);
newcurl->SetMultipartRetryCount(next_retry_count);
return newcurl;
}
@ -2132,6 +2173,8 @@ static int readdir_multi_head(const char* path, S3ObjList& head, void* buf, fuse
continue;
}
// First check for directory, start checking "not sse-c".
// If checking failed, retry to check with "sse-c" by retry callback func when sse-c mode.
S3fsCurl* s3fscurl = new S3fsCurl();
if(!s3fscurl->PreHeadRequest(disppath, (*iter), disppath)){ // target path = cache key path.(ex "dir/")
DPRNNN("Could not make curl object for head request(%s).", disppath.c_str());
@ -2269,8 +2312,20 @@ static int list_bucket(const char* path, S3ObjList& head, const char* delimiter)
next_marker = (char*)tmpch;
xmlFree(tmpch);
}else{
DPRN("Could not find next marker, thus break loop.");
truncated = false;
// If did not specify "delimiter", s3 did not return "NextMarker".
// On this case, can use lastest name for next marker.
//
string lastname;
if(!head.GetLastName(lastname)){
DPRN("Could not find next marker, thus break loop.");
truncated = false;
}else{
next_marker = s3_realpath.substr(1);
if(0 == s3_realpath.length() || '/' != s3_realpath[s3_realpath.length() - 1]){
next_marker += "/";
}
next_marker += lastname;
}
}
}
S3FS_XMLFREEDOC(doc);
@ -2593,7 +2648,13 @@ static int remote_mountpath_exists(const char* path)
static void* s3fs_init(struct fuse_conn_info* conn)
{
FPRN("init");
LOWSYSLOGPRINT(LOG_ERR, "init $Rev$");
LOWSYSLOGPRINT(LOG_ERR, "init v%s (%s)", VERSION, s3fs_crypt_lib_name());
// ssl init
if(!s3fs_init_global_ssl()){
fprintf(stderr, "%s: could not initialize for ssl libraries.\n", program_name.c_str());
exit(EXIT_FAILURE);
}
// init curl
if(!S3fsCurl::InitS3fsCurl("/etc/mime.types")){
@ -2635,6 +2696,8 @@ static void s3fs_destroy(void*)
if(is_remove_cache && !FdManager::DeleteCacheDirectory()){
DPRN("Could not remove cache directory.");
}
// ssl
s3fs_destroy_global_ssl();
}
static int s3fs_access(const char* path, int mask)
@ -2710,7 +2773,7 @@ static bool abort_uncomp_mp_list(uncomp_mp_list_t& list)
char buff[1024];
if(0 >= list.size()){
return false;
return true;
}
memset(buff, 0, sizeof(buff));
@ -2838,10 +2901,17 @@ static int s3fs_utility_mode(void)
return EXIT_FAILURE;
}
// ssl init
if(!s3fs_init_global_ssl()){
fprintf(stderr, "%s: could not initialize for ssl libraries.\n", program_name.c_str());
return EXIT_FAILURE;
}
// init curl
if(!S3fsCurl::InitS3fsCurl("/etc/mime.types")){
fprintf(stderr, "%s: Could not initiate curl library.\n", program_name.c_str());
LOWSYSLOGPRINT(LOG_ERR, "Could not initiate curl library.");
s3fs_destroy_global_ssl();
return EXIT_FAILURE;
}
@ -2886,6 +2956,10 @@ static int s3fs_utility_mode(void)
if(!S3fsCurl::DestroyS3fsCurl()){
DPRN("Could not release curl library.");
}
// ssl
s3fs_destroy_global_ssl();
return result;
}
@ -2955,10 +3029,18 @@ static int check_for_aws_format(void)
ifstream PF(passwd_file.c_str());
if(PF.good()){
while (getline(PF, line)){
if(line[0]=='#')
if(line[0]=='#'){
continue;
if(line.size() == 0)
}
if(line.size() == 0){
continue;
}
if('\r' == line[line.size() - 1]){
line = line.substr(0, line.size() - 1);
if(line.size() == 0){
continue;
}
}
first_pos = line.find_first_of(" \t");
if(first_pos != string::npos){
@ -3108,6 +3190,12 @@ static int read_passwd_file(void)
if(line.size() == 0){
continue;
}
if('\r' == line[line.size() - 1]){
line = line.substr(0, line.size() - 1);
if(line.size() == 0){
continue;
}
}
first_pos = line.find_first_of(" \t");
if(first_pos != string::npos){
@ -3474,22 +3562,44 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
return 0;
}
if(0 == strcmp(arg, "use_sse") || 0 == STR2NCMP(arg, "use_sse=")){
off_t sse = 1;
// for an old format.
if(0 == STR2NCMP(arg, "use_sse=")){
sse = s3fs_strtoofft(strchr(arg, '=') + sizeof(char));
}
if(0 == sse){
S3fsCurl::SetUseSse(false);
}else if(1 == sse){
if(S3fsCurl::GetUseRrs()){
fprintf(stderr, "%s: use_sse option could not be specified with use_rrs.\n", program_name.c_str());
return -1;
}
S3fsCurl::SetUseSse(true);
const char* ssecfile = &arg[strlen("use_sse=")];
if(0 == strcmp(ssecfile, "1")){
if(S3fsCurl::IsSseCustomMode()){
fprintf (stderr, "%s: already set SSE-C key by environment, and confrict use_sse option.\n", program_name.c_str());
return -1;
}
S3fsCurl::SetUseSse(true);
}else{
// testing sse-c, try to load AES256 keys
struct stat st;
if(0 != stat(ssecfile, &st)){
fprintf (stderr, "%s: could not open use_sse keys file(%s)\n", program_name.c_str(), ssecfile);
return -1;
}
if(st.st_mode & (S_IXUSR | S_IRWXG | S_IRWXO)){
fprintf (stderr, "%s: use_sse keys file %s should be 0600 permissions\n", program_name.c_str(), ssecfile);
return -1;
}
if(!S3fsCurl::SetSseKeys(ssecfile)){
fprintf (stderr, "%s: failed to load use_sse keys file %s\n", program_name.c_str(), ssecfile);
return -1;
}
}
}else{
fprintf(stderr, "%s: poorly formed argument to option: use_sse\n", program_name.c_str());
return -1;
if(S3fsCurl::GetUseRrs()){
fprintf(stderr, "%s: use_sse option could not be specified with use_rrs.\n", program_name.c_str());
return -1;
}
if(S3fsCurl::IsSseCustomMode()){
fprintf (stderr, "%s: already set SSE-C key by environment, and confrict use_sse option.\n", program_name.c_str());
return -1;
}
S3fsCurl::SetUseSse(true);
}
return 0;
}
@ -3572,11 +3682,15 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
return -1;
}
S3fsCurl::SetMaxParallelCount(maxpara);
if(FdManager::GetPageSize() < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
FdManager::SetPageSize(static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount()));
}
return 0;
}
if(0 == STR2NCMP(arg, "fd_page_size=")){
size_t pagesize = static_cast<size_t>(s3fs_strtoofft(strchr(arg, '=') + sizeof(char)));
if((1024 * 1024) >= pagesize){
if(pagesize < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
fprintf(stderr, "%s: argument should be over 1MB: fd_page_size\n",
program_name.c_str());
return -1;
@ -3584,6 +3698,17 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
FdManager::SetPageSize(pagesize);
return 0;
}
if(0 == STR2NCMP(arg, "multipart_size=")){
off_t size = static_cast<off_t>(s3fs_strtoofft(strchr(arg, '=') + sizeof(char)));
if(!S3fsCurl::SetMultipartSize(size)){
fprintf(stderr, "%s: multipart_size option could not be specified over 10(MB)\n", program_name.c_str());
return -1;
}
if(FdManager::GetPageSize() < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
FdManager::SetPageSize(static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount()));
}
return 0;
}
if(0 == STR2NCMP(arg, "ahbe_conf=")){
string ahbe_conf = strchr(arg, '=') + sizeof(char);
if(!AdditionalHeader::get()->Load(ahbe_conf.c_str())){
@ -3624,6 +3749,10 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
}
return 0;
}
if(0 == strcmp(arg, "use_path_request_style")){
pathrequeststyle = true;
return 0;
}
// debug option
//
@ -3723,6 +3852,9 @@ int main(int argc, char* argv[])
}
}
// Load SSE-C Key from env
S3fsCurl::LoadEnvSseKeys();
// clear this structure
memset(&s3fs_oper, 0, sizeof(s3fs_oper));
@ -3855,15 +3987,17 @@ int main(int argc, char* argv[])
s3fs_oper.access = s3fs_access;
s3fs_oper.create = s3fs_create;
// init NSS
S3FS_INIT_NSS();
if(!s3fs_init_global_ssl()){
fprintf(stderr, "%s: could not initialize for ssl libraries.\n", program_name.c_str());
exit(EXIT_FAILURE);
}
// now passing things off to fuse, fuse will finish evaluating the command line args
fuse_res = fuse_main(custom_args.argc, custom_args.argv, &s3fs_oper, NULL);
fuse_opt_free_args(&custom_args);
// cleanup NSS
S3FS_CLEANUP_NSS();
s3fs_destroy_global_ssl();
// cleanup xml2
xmlCleanupParser();
S3FS_MALLOCTRIM(0);
@ -3871,3 +4005,11 @@ int main(int argc, char* argv[])
exit(fuse_res);
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_S3_H_
#define S3FS_S3_H_
@ -65,36 +84,15 @@
#endif // HAVE_MALLOC_TRIM
//
// For initializing libcurl with NSS
// Normally libcurl initializes the NSS library, but usually allows
// you to initialize s3fs forcibly. Because Memory leak is reported
// in valgrind(about curl_global_init() function), and this is for
// the cancellation. When "--enable-nss-init" option is specified
// at configurarion, it makes NSS_INIT_ENABLED flag into Makefile.
// NOTICE
// This defines and macros is temporary, and this should be deleted.
//
#ifdef NSS_INIT_ENABLED
#include <nss.h>
#include <prinit.h>
#define S3FS_INIT_NSS() \
{ \
NSS_NoDB_Init(NULL); \
}
#define S3FS_CLEANUP_NSS() \
{ \
NSS_Shutdown(); \
PL_ArenaFinish(); \
PR_Cleanup(); \
}
#else // NSS_INIT_ENABLED
#define S3FS_INIT_NSS()
#define S3FS_CLEANUP_NSS()
#endif // NSS_INIT_ENABLED
char* get_object_sseckey_md5(const char* path);
#endif // S3FS_S3_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

54
src/s3fs_auth.h Normal file
View File

@ -0,0 +1,54 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_AUTH_H_
#define S3FS_AUTH_H_
//-------------------------------------------------------------------
// Utility functions for Authentication
//-------------------------------------------------------------------
//
// in common_auth.cpp
//
char* s3fs_base64(unsigned char* input, size_t length);
std::string s3fs_get_content_md5(int fd);
std::string s3fs_md5sum(int fd, off_t start, ssize_t size);
//
// in xxxxxx_auth.cpp
//
const char* s3fs_crypt_lib_name(void);
bool s3fs_init_global_ssl(void);
bool s3fs_destroy_global_ssl(void);
bool s3fs_init_crypt_mutex(void);
bool s3fs_destroy_crypt_mutex(void);
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen);
size_t get_md5_digest_length(void);
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size);
#endif // S3FS_AUTH_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -41,6 +41,7 @@
#include "s3fs_util.h"
#include "string_util.h"
#include "s3fs.h"
#include "s3fs_auth.h"
using namespace std;
@ -228,6 +229,26 @@ bool S3ObjList::IsDir(const char* name) const
return ps3obj->is_dir;
}
bool S3ObjList::GetLastName(std::string& lastname) const
{
bool result = false;
lastname = "";
for(s3obj_t::const_iterator iter = objects.begin(); iter != objects.end(); iter++){
if((*iter).second.orgname.length()){
if(0 > strcmp(lastname.c_str(), (*iter).second.orgname.c_str())){
lastname = (*iter).second.orgname;
result = true;
}
}else{
if(0 > strcmp(lastname.c_str(), (*iter).second.normalname.c_str())){
lastname = (*iter).second.normalname;
result = true;
}
}
}
return result;
}
bool S3ObjList::GetNameList(s3obj_list_t& list, bool OnlyNormalized, bool CutSlash) const
{
s3obj_t::const_iterator iter;
@ -311,7 +332,7 @@ MVNODE *create_mvnode(const char *old_path, const char *new_path, bool is_dir, b
return NULL;
}
if(NULL == (p_new_path = strdup(new_path))){
if(NULL == (p_new_path = strdup(new_path))){
free(p);
free(p_old_path);
printf("create_mvnode: could not allocation memory for p_new_path\n");
@ -329,7 +350,7 @@ MVNODE *create_mvnode(const char *old_path, const char *new_path, bool is_dir, b
}
//
// Add sorted MVNODE data(Ascending order)
// Add sorted MVNODE data(Ascending order)
//
MVNODE *add_mvnode(MVNODE** head, MVNODE** tail, const char *old_path, const char *new_path, bool is_dir, bool normdir)
{
@ -670,9 +691,9 @@ mode_t get_mode(headers_t& meta, const char* path, bool checkdir, bool forcedir)
}
}
// Checking the bitmask, if the last 3 bits are all zero then process as a regular
// file type (S_IFDIR or S_IFREG), otherwise return mode unmodified so that S_IFIFO,
// file type (S_IFDIR or S_IFREG), otherwise return mode unmodified so that S_IFIFO,
// S_IFSOCK, S_IFCHR, S_IFLNK and S_IFBLK devices can be processed properly by fuse.
if(!(mode & S_IFMT)){
if(!(mode & S_IFMT)){
if(!isS3sync){
if(checkdir){
if(forcedir){
@ -752,7 +773,7 @@ time_t cvtIAMExpireStringToTime(const char* s)
}
memset(&tm, 0, sizeof(struct tm));
strptime(s, "%Y-%m-%dT%H:%M:%S", &tm);
return mktime(&tm); // GMT
return timegm(&tm); // GMT
}
time_t get_lastmodified(const char* s)
@ -763,7 +784,7 @@ time_t get_lastmodified(const char* s)
}
memset(&tm, 0, sizeof(struct tm));
strptime(s, "%a, %d %b %Y %H:%M:%S %Z", &tm);
return mktime(&tm); // GMT
return timegm(&tm); // GMT
}
time_t get_lastmodified(headers_t& meta)
@ -822,7 +843,7 @@ void show_usage (void)
void show_help (void)
{
show_usage();
printf(
printf(
"\n"
"Mount an Amazon S3 bucket as a file system.\n"
"\n"
@ -854,7 +875,24 @@ void show_help (void)
" - this option makes Amazon's Reduced Redundancy Storage enable.\n"
"\n"
" use_sse (default is disable)\n"
" - this option makes Amazon's Server Site Encryption enable.\n"
" - use Amazon<EFBFBD>fs Server-Site Encryption or Server-Side Encryption\n"
" with Customer-Provided Encryption Keys.\n"
" this option can not be specified with use_rrs. specifying only \n"
" \"use_sse\" or \"use_sse=1\" enables Server-Side Encryption.\n"
" (use_sse=1 for old version)\n"
" specifying this option with file path which has some SSE-C\n"
" secret key enables Server-Side Encryption with Customer-Provided\n"
" Encryption Keys.(use_sse=file)\n"
" the file must be 600 permission. the file can have some lines,\n"
" each line is one SSE-C key. the first line in file is used as\n"
" Customer-Provided Encryption Keys for uploading and changing\n"
" headers etc.\n"
" if there are some keys after first line, those are used\n"
" downloading object which are encripted by not first key.\n"
" so that, you can keep all SSE-C keys in file, that is SSE-C\n"
" key history.\n"
" if AWSSSECKEYS environment is set, you can set SSE-C key instead\n"
" of this option.\n"
"\n"
" public_bucket (default=\"\" which means disabled)\n"
" - anonymously mount a public bucket when set to 1\n"
@ -889,7 +927,7 @@ void show_help (void)
" readwrite_timeout (default=\"30\" seconds)\n"
" - time to wait between read/write activity before giving up\n"
"\n"
" max_stat_cache_size (default=\"10000\" entries (about 4MB))\n"
" max_stat_cache_size (default=\"1000\" entries (about 4MB))\n"
" - maximum number of entries in the stat cache\n"
"\n"
" stat_cache_expire (default is no expire)\n"
@ -966,6 +1004,11 @@ void show_help (void)
" only rename command(ex. mv). If this option is specified with\n"
" nocopapi, the s3fs ignores it.\n"
"\n"
" use_path_request_style (use legacy API calling style)\n"
" Enble compatibility with S3-like APIs which do not support\n"
" the virtual-host request style, by using the older path request\n"
" style.\n"
"\n"
"FUSE/mount Options:\n"
"\n"
" Most of the generic mount options described in 'man mount' are\n"
@ -997,11 +1040,20 @@ void show_help (void)
void show_version(void)
{
printf(
"Amazon Simple Storage Service File System %s\n"
"Amazon Simple Storage Service File System V%s with %s\n"
"Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>\n"
"License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>\n"
"This is free software: you are free to change and redistribute it.\n"
"There is NO WARRANTY, to the extent permitted by law.\n", VERSION );
"There is NO WARRANTY, to the extent permitted by law.\n",
VERSION, s3fs_crypt_lib_name());
return;
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_S3FS_UTIL_H_
#define S3FS_S3FS_UTIL_H_
@ -51,6 +70,7 @@ class S3ObjList
std::string GetETag(const char* name) const;
bool IsDir(const char* name) const;
bool GetNameList(s3obj_list_t& list, bool OnlyNormalized = true, bool CutSlash = true) const;
bool GetLastName(std::string& lastname) const;
static bool MakeHierarchizedList(s3obj_list_t& list, bool haveSlash);
};
@ -116,3 +136,12 @@ void show_help(void);
void show_version(void);
#endif // S3FS_S3FS_UTIL_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -130,7 +130,9 @@ string urlEncode(const string &s)
} else if (s[i] == '.' || s[i] == '-' || s[i] == '*' || s[i] == '_') {
result += s[i];
} else if (s[i] == ' ') {
result += '+';
result += '%';
result += '2';
result += '0';
} else {
result += "%";
result += hexAlphabet[static_cast<unsigned char>(s[i]) / 16];
@ -184,8 +186,18 @@ string prepare_url(const char* url)
uri_length = 8;
}
uri = url_str.substr(0, uri_length);
host = bucket + "." + url_str.substr(uri_length, bucket_pos - uri_length).c_str();
path = url_str.substr((bucket_pos + bucket_length));
if(!pathrequeststyle){
host = bucket + "." + url_str.substr(uri_length, bucket_pos - uri_length).c_str();
path = url_str.substr((bucket_pos + bucket_length));
}else{
host = url_str.substr(uri_length, bucket_pos - uri_length).c_str();
string part = url_str.substr((bucket_pos + bucket_length));
if('/' != part[0]){
part = "/" + part;
}
path = "/" + bucket + part;
}
url_str = uri + host + path;
@ -206,3 +218,11 @@ string get_date()
return buf;
}
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_STRING_UTIL_H_
#define S3FS_STRING_UTIL_H_
@ -32,3 +51,12 @@ std::string prepare_url(const char* url);
bool get_keyword_value(std::string& target, const char* keyword, std::string& value);
#endif // S3FS_STRING_UTIL_H_
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/

View File

@ -1,3 +1,22 @@
######################################################################
# s3fs - FUSE-based file system backed by Amazon S3
#
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
######################################################################
TESTS=small-integration-test.sh
EXTRA_DIST = \