Compare commits
56 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 38e6857824 | |||
| ca72b9a6d0 | |||
| 741831344a | |||
| 7a7c7572ea | |||
| 4c32bc0aa5 | |||
| 0e9cfeb808 | |||
| ae4ae88b6d | |||
| f0c33f8ef2 | |||
| e3a33343b9 | |||
| 20b1c207be | |||
| f1ca5d0340 | |||
| cbec8da9a3 | |||
| 7a55eab399 | |||
| 95f8cab139 | |||
| c1a6d76fc3 | |||
| 08929696f7 | |||
| ba34ba181a | |||
| d2c887a371 | |||
| d5113c0501 | |||
| 29a37645dd | |||
| 601482eff5 | |||
| f141bbd4b4 | |||
| 61020370d5 | |||
| f1f7e76be5 | |||
| 160196798b | |||
| edad91186f | |||
| cd27f0aa54 | |||
| c7665f80ab | |||
| 1a4065b0fb | |||
| a4465105f7 | |||
| 8bba566774 | |||
| 157612e7e7 | |||
| 6148415b4b | |||
| d475e22774 | |||
| 4762e53b5d | |||
| e23ea87953 | |||
| d7563309a2 | |||
| c003076053 | |||
| 74fb29d9fb | |||
| b35f8ded46 | |||
| 16487c4e26 | |||
| 39402696ce | |||
| 52d56d15e4 | |||
| 36509351f0 | |||
| 00792a6555 | |||
| fdabb7cbbe | |||
| 775d8758ef | |||
| 31c979b290 | |||
| 5fd33405af | |||
| 1d1e8f3e7d | |||
| 654c58c90a | |||
| db3bd7c366 | |||
| 26187b954e | |||
| 33ec3739e2 | |||
| d9f13dbdcb | |||
| d5626fe595 |
130
ChangeLog
130
ChangeLog
@ -1,10 +1,134 @@
|
||||
ChangeLog for S3FS
|
||||
ChangeLog for S3FS
|
||||
------------------
|
||||
|
||||
Version 1.1 -- Mon Oct 18 2010
|
||||
Version 1.78 -- Sep 15, 2014
|
||||
issue #29 - Possible to create Debian/Ubuntu packages?(googlecode issue 109)
|
||||
issue 417(googlecode) - Password file with DOS format is not handled properly
|
||||
issue #41 - Failed making signature
|
||||
issue #40 - Moving a directory containing more than 1000 files truncates the
|
||||
directory
|
||||
issue #49 - use_sse is ignored when creating new files
|
||||
issue #39 - Support for SSE-C
|
||||
issue #50 - Cannot find pkg-config when configured with any SSL backend except
|
||||
openssl
|
||||
|
||||
Version 1.77 -- Apr 19, 2014
|
||||
issue 405(googlecode) - enable_content_md5 Input/output error
|
||||
issue #14 - s3fs -u should return 0 if there are no lost multiparts
|
||||
issue #16 - empty file is written to s3
|
||||
issue #18 - s3fs crashes with segfault
|
||||
issue #22 - Fix typos in docs for max_stat_cache_size
|
||||
issue #23 - curl ssl problems
|
||||
issue #28 - Address signedness warning in FdCache::Init
|
||||
|
||||
Version 1.76 -- Jan 21, 2014
|
||||
issue #5 - du shows incorrect usage stats
|
||||
issue #8 - version in configure.ac is 1.74 for release 1.75
|
||||
|
||||
Version 1.75 -- Jan 6, 2014
|
||||
issue #1 - Using %20 instead of the plus (+) sign for encoding spaces
|
||||
issue #3 - Fixed local timezone was incorrectly being applied to IAM and Last-Modified dates.
|
||||
issue #4 - Fix compilation error on MacOSX with missing const
|
||||
|
||||
Version 1.74 -- Nov 24, 2013
|
||||
This version is initial version on Github, same as on GoogleCodes(s3fs).
|
||||
https://github.com/s3fs-fuse/s3fs-fuse/releases/tag/v1.74
|
||||
see more detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.74.tar.gz
|
||||
|
||||
Version 1.73 -- Aug 23, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.73.tar.gz
|
||||
|
||||
Version 1.72 -- Aug 10, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.72.tar.gz
|
||||
|
||||
Version 1.71 -- Jun 15, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.71.tar.gz
|
||||
|
||||
Version 1.70 -- Jun 01, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.70.tar.gz
|
||||
|
||||
Version 1.69 -- May 15, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.69.tar.gz
|
||||
|
||||
Version 1.68 -- Apr 30, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.68.tar.gz
|
||||
|
||||
Version 1.67 -- Apr 13, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.67.tar.gz
|
||||
|
||||
Version 1.66 -- Apr 06, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.66.tar.gz
|
||||
|
||||
Version 1.65 -- Mar 30, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.65.tar.gz
|
||||
|
||||
Version 1.64 -- Mar 23, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.64.tar.gz
|
||||
|
||||
Version 1.63 -- Feb 24, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.63.tar.gz
|
||||
|
||||
Version 1.62 -- Jan 27, 2013
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.62.tar.gz
|
||||
|
||||
Version 1.61 -- Aug 30, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.61.tar.gz
|
||||
|
||||
Version 1.60 -- Aug 29, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.60.tar.gz
|
||||
|
||||
Version 1.59 -- Jul 28, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.59.tar.gz
|
||||
|
||||
Version 1.58 -- Jul 19, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.58.tar.gz
|
||||
|
||||
Version 1.57 -- Jul 07, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.57.tar.gz
|
||||
|
||||
Version 1.56 -- Jul 07, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.56.tar.gz
|
||||
|
||||
Version 1.55 -- Jul 02, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.55.tar.gz
|
||||
|
||||
Version 1.54 -- Jun 25, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.54.tar.gz
|
||||
|
||||
Version 1.53 -- Jun 22, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.53.tar.gz
|
||||
|
||||
Version 1.40 -- Feb 11, 2011
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.40.tar.gz
|
||||
|
||||
Version 1.33 -- Dec 30, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.33.tar.gz
|
||||
|
||||
Version 1.25 -- Dec 16, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.25.tar.gz
|
||||
|
||||
Version 1.19 -- Dec 2, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.19.tar.gz
|
||||
|
||||
Version 1.16 -- Nov 22, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.16.tar.gz
|
||||
|
||||
Version 1.10 -- Nov 6, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.10.tar.gz
|
||||
|
||||
Version 1.02 -- Oct 29, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.02.tar.gz
|
||||
|
||||
Version 1.01 -- Oct 28, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.01.tar.gz
|
||||
|
||||
Version 1.0 -- Oct 24, 2010
|
||||
see detail on googlecodes: http://code.google.com/p/s3fs/downloads/detail?name=s3fs-1.0.tar.gz
|
||||
|
||||
------
|
||||
Version 1.1 -- Mon Oct 18 2010
|
||||
Dan Moore reopens the project and fixes various issues that had accumulated in the tracker. Adrian Petrescu converts the project to autotools and posts it to GitHub.
|
||||
|
||||
Version 1.0 -- 2008
|
||||
|
||||
Randy Rizun releases a basic version of S3FS on Google Code.
|
||||
|
||||
|
||||
20
Makefile.am
20
Makefile.am
@ -1,3 +1,22 @@
|
||||
######################################################################
|
||||
# s3fs - FUSE-based file system backed by Amazon S3
|
||||
#
|
||||
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
######################################################################
|
||||
SUBDIRS=src test doc
|
||||
|
||||
EXTRA_DIST=doc
|
||||
@ -8,3 +27,4 @@ dist-hook:
|
||||
|
||||
release : dist ../utils/release.sh
|
||||
../utils/release.sh $(DIST_ARCHIVES)
|
||||
|
||||
|
||||
18
README
18
README
@ -14,16 +14,20 @@ In order to compile s3fs, You'll need the following requirements:
|
||||
* FUSE (>= 2.8.4)
|
||||
* FUSE Kernel module installed and running (RHEL 4.x/CentOS 4.x users - read below)
|
||||
* OpenSSL-devel (0.9.8)
|
||||
* Subversion
|
||||
GnuTLS(gcrypt and nettle)
|
||||
NSS
|
||||
* Git
|
||||
|
||||
If you're using YUM or APT to install those packages, then it might require additional packaging, allow it to be installed.
|
||||
|
||||
Downloading & Compiling:
|
||||
------------------------
|
||||
In order to download s3fs, user the following command:
|
||||
svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs-read-only
|
||||
In order to download s3fs, download from following url:
|
||||
https://github.com/s3fs-fuse/s3fs-fuse/archive/master.zip
|
||||
Or clone the following command:
|
||||
git clone git://github.com/s3fs-fuse/s3fs-fuse.git
|
||||
|
||||
Go inside the directory that has been created (s3fs-read-only/s3fs) and run: ./autogen.sh
|
||||
Go inside the directory that has been created (s3fs-fuse) and run: ./autogen.sh
|
||||
This will generate a number of scripts in the project directory, including a configure script which you should run with: ./configure
|
||||
If configure succeeded, you can now run: make. If it didn't, make sure you meet the dependencies above.
|
||||
This should compile the code. If everything goes OK, you'll be greated with "ok!" at the end and you'll have a binary file called "s3fs"
|
||||
@ -42,21 +46,21 @@ Then run: s3fs mybucket[:path] /mnt/s3
|
||||
|
||||
This will mount your bucket to /mnt/s3. You can do a simple "ls -l /mnt/s3" to see the content of your bucket.
|
||||
|
||||
If you want to allow other people access the same bucket in the same machine, you can add "-o allow _other" to read/write/delete content of the bucket.
|
||||
If you want to allow other people access the same bucket in the same machine, you can add "-o allow_other" to read/write/delete content of the bucket.
|
||||
|
||||
You can add a fixed mount point in /etc/fstab, here's an example:
|
||||
|
||||
s3fs#mybucket /mnt/s3 fuse allow_other 0 0
|
||||
|
||||
This will mount upon reboot (or by launching: mount -a) your bucket on your machine.
|
||||
If that does not work, probably you should specify with "_netdev" option in fstab.
|
||||
|
||||
All other options can be read at: http://code.google.com/p/s3fs/wiki/FuseOverAmazon
|
||||
All other options can be read at: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
|
||||
|
||||
Known Issues:
|
||||
-------------
|
||||
s3fs should be working fine with S3 storage. However, There are couple of limitations:
|
||||
|
||||
* There is no full UID/GID support yet, everything looks as "root" and if you allow others to access the bucket, others can erase files. There is, however, permissions support built in.
|
||||
* Currently s3fs could hang the CPU if you have lots of time-outs. This is *NOT* a fault of s3fs but rather libcurl. This happends when you try to copy thousands of files in 1 session, it doesn't happend when you upload hundreds of files or less.
|
||||
* CentOS 4.x/RHEL 4.x users - if you use the kernel that shipped with your distribution and didn't upgrade to the latest kernel RedHat/CentOS gives, you might have a problem loading the "fuse" kernel. Please upgrade to the latest kernel (2.6.16 or above) and make sure "fuse" kernel module is compiled and loadable since FUSE requires this kernel module and s3fs requires it as well.
|
||||
* Moving/renaming/erasing files takes time since the whole file needs to be accessed first. A workaround could be to use s3fs's cache support with the use_cache option.
|
||||
|
||||
225
configure.ac
225
configure.ac
@ -20,87 +20,198 @@
|
||||
dnl Process this file with autoconf to produce a configure script.
|
||||
|
||||
AC_PREREQ(2.59)
|
||||
AC_INIT(s3fs, 1.74)
|
||||
AC_INIT(s3fs, 1.78)
|
||||
|
||||
AC_CANONICAL_SYSTEM
|
||||
AM_INIT_AUTOMAKE()
|
||||
|
||||
AC_PROG_CXX
|
||||
AC_PROG_CC
|
||||
|
||||
CXXFLAGS="$CXXFLAGS -Wall -D_FILE_OFFSET_BITS=64"
|
||||
|
||||
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 libcrypto >= 0.9])
|
||||
dnl ----------------------------------------------
|
||||
dnl Choice SSL library
|
||||
dnl ----------------------------------------------
|
||||
auth_lib=na
|
||||
nettle_lib=no
|
||||
|
||||
dnl malloc_trim function
|
||||
AC_CHECK_FUNCS(malloc_trim, , )
|
||||
|
||||
dnl Initializing NSS(temporally)
|
||||
AC_MSG_CHECKING([Initializing libcurl build with NSS])
|
||||
AC_ARG_ENABLE(
|
||||
nss-init,
|
||||
dnl
|
||||
dnl nettle library
|
||||
dnl
|
||||
AC_MSG_CHECKING([s3fs build with nettle(GnuTLS)])
|
||||
AC_ARG_WITH(
|
||||
nettle,
|
||||
[AS_HELP_STRING([--with-nettle], [s3fs build with nettle in GnuTLS(default no)])],
|
||||
[
|
||||
AS_HELP_STRING(
|
||||
[--enable-nss-init],
|
||||
[Inilializing libcurl with NSS (default is no)]
|
||||
)
|
||||
],
|
||||
[
|
||||
case "${enableval}" in
|
||||
case "${withval}" in
|
||||
yes)
|
||||
AC_MSG_RESULT(yes)
|
||||
nss_init_enabled=yes
|
||||
nettle_lib=yes
|
||||
;;
|
||||
*)
|
||||
AC_MSG_RESULT(no)
|
||||
;;
|
||||
esac
|
||||
],
|
||||
[
|
||||
AC_MSG_RESULT(no)
|
||||
])
|
||||
|
||||
dnl
|
||||
dnl use openssl library for ssl
|
||||
dnl
|
||||
AC_MSG_CHECKING([s3fs build with OpenSSL])
|
||||
AC_ARG_WITH(
|
||||
openssl,
|
||||
[AS_HELP_STRING([--with-openssl], [s3fs build with OpenSSL(default is no)])],
|
||||
[
|
||||
case "${withval}" in
|
||||
yes)
|
||||
AC_MSG_RESULT(yes)
|
||||
AS_IF(
|
||||
[test $nettle_lib = no],
|
||||
[auth_lib=openssl],
|
||||
[AC_MSG_ERROR([could not set openssl with nettle, nettle is only for gnutls library])])
|
||||
;;
|
||||
*)
|
||||
AC_MSG_RESULT(no)
|
||||
;;
|
||||
esac
|
||||
],
|
||||
[
|
||||
AC_MSG_RESULT(no)
|
||||
])
|
||||
|
||||
dnl
|
||||
dnl use GnuTLS library for ssl
|
||||
dnl
|
||||
AC_MSG_CHECKING([s3fs build with GnuTLS])
|
||||
AC_ARG_WITH(
|
||||
gnutls,
|
||||
[AS_HELP_STRING([--with-gnutls], [s3fs build with GnuTLS(default is no)])],
|
||||
[
|
||||
case "${withval}" in
|
||||
yes)
|
||||
AC_MSG_RESULT(yes)
|
||||
AS_IF(
|
||||
[test $auth_lib = na],
|
||||
[
|
||||
AS_IF(
|
||||
[test $nettle_lib = no],
|
||||
[auth_lib=gnutls],
|
||||
[auth_lib=nettle])
|
||||
],
|
||||
[AC_MSG_ERROR([could not set gnutls because already set another ssl library])])
|
||||
;;
|
||||
*)
|
||||
AC_MSG_RESULT(no)
|
||||
;;
|
||||
esac
|
||||
],
|
||||
[
|
||||
AC_MSG_RESULT(no)
|
||||
])
|
||||
|
||||
dnl
|
||||
dnl use nss library for ssl
|
||||
dnl
|
||||
AC_MSG_CHECKING([s3fs build with NSS])
|
||||
AC_ARG_WITH(
|
||||
nss,
|
||||
[AS_HELP_STRING([--with-nss], [s3fs build with NSS(default is no)])],
|
||||
[
|
||||
case "${withval}" in
|
||||
yes)
|
||||
AC_MSG_RESULT(yes)
|
||||
AS_IF(
|
||||
[test $auth_lib = na],
|
||||
[
|
||||
AS_IF(
|
||||
[test $nettle_lib = no],
|
||||
[auth_lib=nss],
|
||||
[AC_MSG_ERROR([could not set openssl with nettle, nettle is only for gnutls library])])
|
||||
],
|
||||
[AC_MSG_ERROR([could not set nss because already set another ssl library])])
|
||||
;;
|
||||
*)
|
||||
AC_MSG_RESULT(no)
|
||||
nss_init_enabled=no
|
||||
;;
|
||||
esac
|
||||
],
|
||||
[
|
||||
AC_MSG_RESULT(no)
|
||||
nss_init_enabled=no
|
||||
])
|
||||
|
||||
AS_IF(
|
||||
[test $nss_init_enabled = yes],
|
||||
[
|
||||
AC_DEFINE(NSS_INIT_ENABLED, 1)
|
||||
AC_CHECK_LIB(nss3, NSS_NoDB_Init, , [AC_MSG_ERROR(not found NSS libraries)])
|
||||
AC_CHECK_LIB(plds4, PL_ArenaFinish, , [AC_MSG_ERROR(not found PL_ArenaFinish)])
|
||||
AC_CHECK_LIB(nspr4, PR_Cleanup, , [AC_MSG_ERROR(not found PR_Cleanup)])
|
||||
AC_CHECK_HEADER(nss.h, , [AC_MSG_ERROR(not found nss.h)])
|
||||
AC_CHECK_HEADER(nspr4/prinit.h, , [AC_MSG_ERROR(not found prinit.h)])
|
||||
AC_PATH_PROG(NSSCONFIG, [nss-config], no)
|
||||
AS_IF(
|
||||
[test $NSSCONFIG = no],
|
||||
[
|
||||
DEPS_CFLAGS="$DEPS_CFLAGS -I/usr/include/nss3"
|
||||
DEPS_LIBS="$DEPS_LIBS -lnss3"
|
||||
],
|
||||
[
|
||||
addcflags=`nss-config --cflags`
|
||||
DEPS_CFLAGS="$DEPS_CFLAGS $addcflags"
|
||||
dnl addlib=`nss-config --libs`
|
||||
dnl DEPS_LIBS="$DEPS_LIBS $addlib"
|
||||
DEPS_LIBS="$DEPS_LIBS -lnss3"
|
||||
])
|
||||
AC_PATH_PROG(NSPRCONFIG, [nspr-config], no)
|
||||
AS_IF(
|
||||
[test $NSPRCONFIG = no],
|
||||
[
|
||||
DEPS_CFLAGS="$DEPS_CFLAGS -I/usr/include/nspr4"
|
||||
DEPS_LIBS="$DEPS_LIBS -lnspr4 -lplds4"
|
||||
],
|
||||
[
|
||||
addcflags=`nspr-config --cflags`
|
||||
DEPS_CFLAGS="$DEPS_CFLAGS $addcflags"
|
||||
dnl addlib=`nspr-config --libs`
|
||||
dnl DEPS_LIBS="$DEPS_LIBS $addlib"
|
||||
DEPS_LIBS="$DEPS_LIBS -lnspr4 -lplds4"
|
||||
])
|
||||
])
|
||||
[test $auth_lib = na],
|
||||
AS_IF(
|
||||
[test $nettle_lib = no],
|
||||
[auth_lib=openssl],
|
||||
[AC_MSG_ERROR([could not set nettle without GnuTLS library])]
|
||||
)
|
||||
)
|
||||
|
||||
AS_UNSET(nss_enabled)
|
||||
dnl
|
||||
dnl For PKG_CONFIG before checking nss/gnutls.
|
||||
dnl this is redundant checking, but we need checking before following.
|
||||
dnl
|
||||
PKG_CHECK_MODULES([common_lib_checking], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6])
|
||||
|
||||
AC_MSG_CHECKING([compile s3fs with])
|
||||
case "${auth_lib}" in
|
||||
openssl)
|
||||
AC_MSG_RESULT(OpenSSL)
|
||||
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 libcrypto >= 0.9])
|
||||
;;
|
||||
gnutls)
|
||||
AC_MSG_RESULT(GnuTLS-gcrypt)
|
||||
gnutls_nettle=""
|
||||
AC_CHECK_LIB(gnutls, gcry_control, [gnutls_nettle=0])
|
||||
AS_IF([test "$gnutls_nettle" = ""], [AC_CHECK_LIB(gcrypt, gcry_control, [gnutls_nettle=0])])
|
||||
AS_IF([test $gnutls_nettle = 0],
|
||||
[
|
||||
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 gnutls >= 2.12.0 ])
|
||||
LIBS="-lgnutls -lgcrypt $LIBS"
|
||||
AC_MSG_CHECKING([gnutls is build with])
|
||||
AC_MSG_RESULT(gcrypt)
|
||||
],
|
||||
[AC_MSG_ERROR([GnuTLS found, but gcrypt not found])])
|
||||
;;
|
||||
nettle)
|
||||
AC_MSG_RESULT(GnuTLS-nettle)
|
||||
gnutls_nettle=""
|
||||
AC_CHECK_LIB(gnutls, nettle_MD5Init, [gnutls_nettle=1])
|
||||
AS_IF([test "$gnutls_nettle" = ""], [AC_CHECK_LIB(nettle, nettle_MD5Init, [gnutls_nettle=1])])
|
||||
AS_IF([test $gnutls_nettle = 1],
|
||||
[
|
||||
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 nettle >= 2.7.1 ])
|
||||
LIBS="-lgnutls -lnettle $LIBS"
|
||||
AC_MSG_CHECKING([gnutls is build with])
|
||||
AC_MSG_RESULT(nettle)
|
||||
],
|
||||
[AC_MSG_ERROR([GnuTLS found, but nettle not found])])
|
||||
;;
|
||||
nss)
|
||||
AC_MSG_RESULT(NSS)
|
||||
PKG_CHECK_MODULES([DEPS], [fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6 nss >= 3.15.0 ])
|
||||
;;
|
||||
*)
|
||||
AC_MSG_ERROR([unknown ssl library type.])
|
||||
;;
|
||||
esac
|
||||
|
||||
AM_CONDITIONAL([USE_SSL_OPENSSL], [test "$auth_lib" = openssl])
|
||||
AM_CONDITIONAL([USE_SSL_GNUTLS], [test "$auth_lib" = gnutls -o "$auth_lib" = nettle])
|
||||
AM_CONDITIONAL([USE_GNUTLS_NETTLE], [test "$auth_lib" = nettle])
|
||||
AM_CONDITIONAL([USE_SSL_NSS], [test "$auth_lib" = nss])
|
||||
|
||||
dnl ----------------------------------------------
|
||||
dnl end of ssl library
|
||||
dnl ----------------------------------------------
|
||||
|
||||
dnl malloc_trim function
|
||||
AC_CHECK_FUNCS(malloc_trim, , )
|
||||
|
||||
AC_CONFIG_FILES(Makefile src/Makefile test/Makefile doc/Makefile)
|
||||
AC_OUTPUT
|
||||
|
||||
@ -1 +1,21 @@
|
||||
######################################################################
|
||||
# s3fs - FUSE-based file system backed by Amazon S3
|
||||
#
|
||||
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
######################################################################
|
||||
dist_man1_MANS = man/s3fs.1
|
||||
|
||||
|
||||
@ -8,6 +8,9 @@ S3FS \- FUSE-based file system backed by Amazon S3
|
||||
.SS unmounting
|
||||
.TP
|
||||
\fBumount mountpoint
|
||||
.SS utility mode ( remove interrupted multipart uploading objects )
|
||||
.TP
|
||||
\fBs3fs -u bucket
|
||||
.SH DESCRIPTION
|
||||
s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).
|
||||
.SH AUTHENTICATION
|
||||
@ -68,9 +71,13 @@ this option can not be specified with use_sse.
|
||||
(can specify use_rrs=1 for old version)
|
||||
.TP
|
||||
\fB\-o\fR use_sse (default is disable)
|
||||
use Amazon's Server Site Encryption.
|
||||
this option can not be specified with use_rrs.
|
||||
(can specify use_sse=1 for old version)
|
||||
use Amazon<EFBFBD>fs Server-Site Encryption or Server-Side Encryption with Customer-Provided Encryption Keys.
|
||||
this option can not be specified with use_rrs. specifying only "use_sse" or "use_sse=1" enables Server-Side Encryption.(use_sse=1 for old version)
|
||||
specifying this option with file path which has some SSE-C secret key enables Server-Side Encryption with Customer-Provided Encryption Keys.(use_sse=file)
|
||||
the file must be 600 permission. the file can have some lines, each line is one SSE-C key. the first line in file is used as Customer-Provided Encryption Keys for uploading and change headers etc.
|
||||
if there are some keys after first line, those are used downloading object which are encripted by not first key.
|
||||
so that, you can keep all SSE-C keys in file, that is SSE-C key history.
|
||||
if AWSSSECKEYS environment is set, you can set SSE-C key instead of this option.
|
||||
.TP
|
||||
\fB\-o\fR passwd_file (default="")
|
||||
specify the path to the password file, which which takes precedence over the password in $HOME/.passwd-s3fs and /etc/passwd-s3fs
|
||||
@ -102,7 +109,7 @@ time to wait for connection before giving up.
|
||||
\fB\-o\fR readwrite_timeout (default="30" seconds)
|
||||
time to wait between read/write activity before giving up.
|
||||
.TP
|
||||
\fB\-o\fR max_stat_cache_size (default="10000" entries (about 4MB))
|
||||
\fB\-o\fR max_stat_cache_size (default="1000" entries (about 4MB))
|
||||
maximum number of entries in the stat cache
|
||||
.TP
|
||||
\fB\-o\fR stat_cache_expire (default is no expire)
|
||||
@ -125,14 +132,22 @@ maximum number of parallel request for listing objects.
|
||||
.TP
|
||||
\fB\-o\fR parallel_count (default="5")
|
||||
number of parallel request for uploading big objects.
|
||||
s3fs uploads large object(over 20MB) by multipart post request, and sends parallel requests.
|
||||
s3fs uploads large object(default:over 20MB) by multipart post request, and sends parallel requests.
|
||||
This option limits parallel request count which s3fs requests at once.
|
||||
It is necessary to set this value depending on a CPU and a network band.
|
||||
This option is lated to fd_page_size option and affects it.
|
||||
.TP
|
||||
\fB\-o\fR fd_page_size(default="52428800"(50MB))
|
||||
number of internal management page size for each file discriptor.
|
||||
For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
|
||||
This option should not be changed when you don't have a trouble with performance.
|
||||
This value is changed automatically by parallel_count and multipart_size values(fd_page_size value = parallel_count * multipart_size).
|
||||
.TP
|
||||
\fB\-o\fR multipart_size(default="10"(10MB))
|
||||
number of one part size in multipart uploading request.
|
||||
The default size is 10MB(10485760byte), this value is minimum size.
|
||||
Specify number of MB and over 10(MB).
|
||||
This option is lated to fd_page_size option and affects it.
|
||||
.TP
|
||||
\fB\-o\fR url (default="http://s3.amazonaws.com")
|
||||
sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com
|
||||
@ -160,6 +175,9 @@ If you set this option, s3fs do not use PUT with "x-amz-copy-source"(copy api).
|
||||
For a distributed object storage which is compatibility S3 API without PUT(copy api).
|
||||
This option is a subset of nocopyapi option. The nocopyapi option does not use copy-api for all command(ex. chmod, chown, touch, mv, etc), but this option does not use copy-api for only rename command(ex. mv).
|
||||
If this option is specified with nocopapi, the s3fs ignores it.
|
||||
.TP
|
||||
\fB\-o\fR use_path_request_style (use legacy API calling style)
|
||||
Enble compatibility with S3-like APIs which do not support the virtual-host request style, by using the older path request style.
|
||||
.SH FUSE/MOUNT OPTIONS
|
||||
.TP
|
||||
Most of the generic mount options described in 'man mount' are supported (ro, rw, suid, nosuid, dev, nodev, exec, noexec, atime, noatime, sync async, dirsync). Filesystems are mounted with '-onodev,nosuid' by default, which can only be overridden by a privileged user.
|
||||
|
||||
@ -1,7 +1,39 @@
|
||||
######################################################################
|
||||
# s3fs - FUSE-based file system backed by Amazon S3
|
||||
#
|
||||
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
######################################################################
|
||||
bin_PROGRAMS=s3fs
|
||||
|
||||
AM_CPPFLAGS = $(DEPS_CFLAGS)
|
||||
if USE_GNUTLS_NETTLE
|
||||
AM_CPPFLAGS += -DUSE_GNUTLS_NETTLE
|
||||
endif
|
||||
|
||||
s3fs_SOURCES = s3fs.cpp s3fs.h curl.cpp curl.h cache.cpp cache.h string_util.cpp string_util.h s3fs_util.cpp s3fs_util.h fdcache.cpp fdcache.h common_auth.cpp s3fs_auth.h common.h
|
||||
if USE_SSL_OPENSSL
|
||||
s3fs_SOURCES += openssl_auth.cpp
|
||||
endif
|
||||
if USE_SSL_GNUTLS
|
||||
s3fs_SOURCES += gnutls_auth.cpp
|
||||
endif
|
||||
if USE_SSL_NSS
|
||||
s3fs_SOURCES += nss_auth.cpp
|
||||
endif
|
||||
|
||||
s3fs_SOURCES = s3fs.cpp s3fs.h curl.cpp curl.h cache.cpp cache.h string_util.cpp string_util.h s3fs_util.cpp s3fs_util.h fdcache.cpp fdcache.h common.h
|
||||
s3fs_LDADD = $(DEPS_LIBS)
|
||||
|
||||
|
||||
@ -425,6 +425,7 @@ bool convert_header_to_stat(const char* path, headers_t& meta, struct stat* pst,
|
||||
if(S_ISREG(pst->st_mode)){
|
||||
pst->st_blocks = get_blocks(pst->st_size);
|
||||
}
|
||||
pst->st_blksize = 4096;
|
||||
|
||||
// mtime
|
||||
pst->st_mtime = get_mtime(meta);
|
||||
@ -439,3 +440,11 @@ bool convert_header_to_stat(const char* path, headers_t& meta, struct stat* pst,
|
||||
return true;
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
28
src/cache.h
28
src/cache.h
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_CACHE_H_
|
||||
#define S3FS_CACHE_H_
|
||||
|
||||
@ -105,3 +124,12 @@ class StatCache
|
||||
bool convert_header_to_stat(const char* path, headers_t& meta, struct stat* pst, bool forcedir = false);
|
||||
|
||||
#endif // S3FS_CACHE_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
29
src/common.h
29
src/common.h
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_COMMON_H_
|
||||
#define S3FS_COMMON_H_
|
||||
|
||||
@ -69,6 +88,7 @@ extern bool debug;
|
||||
extern bool foreground;
|
||||
extern bool foreground2;
|
||||
extern bool nomultipart;
|
||||
extern bool pathrequeststyle;
|
||||
extern std::string program_name;
|
||||
extern std::string service_path;
|
||||
extern std::string host;
|
||||
@ -76,3 +96,12 @@ extern std::string bucket;
|
||||
extern std::string mount_prefix;
|
||||
|
||||
#endif // S3FS_COMMON_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
112
src/common_auth.cpp
Normal file
112
src/common_auth.cpp
Normal file
@ -0,0 +1,112 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <string>
|
||||
|
||||
#include "s3fs_auth.h"
|
||||
|
||||
using namespace std;
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function
|
||||
//-------------------------------------------------------------------
|
||||
char* s3fs_base64(unsigned char* input, size_t length)
|
||||
{
|
||||
static const char* base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
|
||||
char* result;
|
||||
|
||||
if(!input || 0 >= length){
|
||||
return NULL;
|
||||
}
|
||||
if(NULL == (result = (char*)malloc((((length / 3) + 1) * 4 + 1) * sizeof(char)))){
|
||||
return NULL; // ENOMEM
|
||||
}
|
||||
|
||||
unsigned char parts[4];
|
||||
size_t rpos;
|
||||
size_t wpos;
|
||||
for(rpos = 0, wpos = 0; rpos < length; rpos += 3){
|
||||
parts[0] = (input[rpos] & 0xfc) >> 2;
|
||||
parts[1] = ((input[rpos] & 0x03) << 4) | ((((rpos + 1) < length ? input[rpos + 1] : 0x00) & 0xf0) >> 4);
|
||||
parts[2] = (rpos + 1) < length ? (((input[rpos + 1] & 0x0f) << 2) | ((((rpos + 2) < length ? input[rpos + 2] : 0x00) & 0xc0) >> 6)) : 0x40;
|
||||
parts[3] = (rpos + 2) < length ? (input[rpos + 2] & 0x3f) : 0x40;
|
||||
|
||||
result[wpos++] = base[parts[0]];
|
||||
result[wpos++] = base[parts[1]];
|
||||
result[wpos++] = base[parts[2]];
|
||||
result[wpos++] = base[parts[3]];
|
||||
}
|
||||
result[wpos] = '\0';
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
string s3fs_get_content_md5(int fd)
|
||||
{
|
||||
unsigned char* md5hex;
|
||||
char* base64;
|
||||
string Signature;
|
||||
|
||||
if(NULL == (md5hex = s3fs_md5hexsum(fd, 0, -1))){
|
||||
return string("");
|
||||
}
|
||||
if(NULL == (base64 = s3fs_base64(md5hex, get_md5_digest_length()))){
|
||||
return string(""); // ENOMEM
|
||||
}
|
||||
free(md5hex);
|
||||
|
||||
Signature = base64;
|
||||
free(base64);
|
||||
|
||||
return Signature;
|
||||
}
|
||||
|
||||
string s3fs_md5sum(int fd, off_t start, ssize_t size)
|
||||
{
|
||||
size_t digestlen = get_md5_digest_length();
|
||||
char md5[2 * digestlen + 1];
|
||||
char hexbuf[3];
|
||||
unsigned char* md5hex;
|
||||
|
||||
if(NULL == (md5hex = s3fs_md5hexsum(fd, start, size))){
|
||||
return string("");
|
||||
}
|
||||
|
||||
memset(md5, 0, 2 * digestlen + 1);
|
||||
for(size_t pos = 0; pos < digestlen; pos++){
|
||||
snprintf(hexbuf, 3, "%02x", md5hex[pos]);
|
||||
strncat(md5, hexbuf, 2);
|
||||
}
|
||||
free(md5hex);
|
||||
|
||||
return string(md5);
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
701
src/curl.cpp
701
src/curl.cpp
File diff suppressed because it is too large
Load Diff
81
src/curl.h
81
src/curl.h
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_CURL_H_
|
||||
#define S3FS_CURL_H_
|
||||
|
||||
@ -85,7 +104,7 @@ struct filepart
|
||||
// for progress
|
||||
struct case_insensitive_compare_func
|
||||
{
|
||||
bool operator()(const std::string& a, const std::string& b){
|
||||
bool operator()(const std::string& a, const std::string& b) const {
|
||||
return strcasecmp(a.c_str(), b.c_str()) < 0;
|
||||
}
|
||||
};
|
||||
@ -100,18 +119,14 @@ class S3fsMultiCurl;
|
||||
// class S3fsCurl
|
||||
//----------------------------------------------
|
||||
typedef std::map<std::string, std::string> iamcredmap_t;
|
||||
typedef std::map<std::string, std::string> sseckeymap_t;
|
||||
typedef std::list<sseckeymap_t> sseckeylist_t;
|
||||
|
||||
// share
|
||||
#define SHARE_MUTEX_DNS 0
|
||||
#define SHARE_MUTEX_SSL_SESSION 1
|
||||
#define SHARE_MUTEX_MAX 2
|
||||
|
||||
// internal use struct for openssl
|
||||
struct CRYPTO_dynlock_value
|
||||
{
|
||||
pthread_mutex_t dyn_mutex;
|
||||
};
|
||||
|
||||
// Class for lapping curl
|
||||
//
|
||||
class S3fsCurl
|
||||
@ -140,7 +155,6 @@ class S3fsCurl
|
||||
// class variables
|
||||
static pthread_mutex_t curl_handles_lock;
|
||||
static pthread_mutex_t curl_share_lock[SHARE_MUTEX_MAX];
|
||||
static pthread_mutex_t* crypt_mutex;
|
||||
static bool is_initglobal_done;
|
||||
static CURLSH* hCurlShare;
|
||||
static bool is_dns_cache;
|
||||
@ -151,6 +165,7 @@ class S3fsCurl
|
||||
static bool is_public_bucket;
|
||||
static std::string default_acl; // TODO: to enum
|
||||
static bool is_use_rrs;
|
||||
static sseckeylist_t sseckeys;
|
||||
static bool is_use_sse;
|
||||
static bool is_content_md5;
|
||||
static bool is_verbose;
|
||||
@ -160,12 +175,12 @@ class S3fsCurl
|
||||
static time_t AWSAccessTokenExpire;
|
||||
static std::string IAM_role;
|
||||
static long ssl_verify_hostname;
|
||||
static const EVP_MD* evp_md;
|
||||
static curltime_t curl_times;
|
||||
static curlprogress_t curl_progress;
|
||||
static std::string curl_ca_bundle;
|
||||
static mimes_t mimeTypes;
|
||||
static int max_parallel_cnt;
|
||||
static off_t multipart_size;
|
||||
|
||||
// variables
|
||||
CURL* hCurl;
|
||||
@ -189,6 +204,8 @@ class S3fsCurl
|
||||
int b_postdata_remaining; // backup for retrying
|
||||
off_t b_partdata_startpos; // backup for retrying
|
||||
ssize_t b_partdata_size; // backup for retrying
|
||||
bool b_ssekey_pos; // backup for retrying
|
||||
std::string b_ssekey_md5; // backup for retrying
|
||||
|
||||
public:
|
||||
// constructor/destructor
|
||||
@ -205,11 +222,6 @@ class S3fsCurl
|
||||
static void UnlockCurlShare(CURL* handle, curl_lock_data nLockData, void* useptr);
|
||||
static bool InitCryptMutex(void);
|
||||
static bool DestroyCryptMutex(void);
|
||||
static void CryptMutexLock(int mode, int pos, const char* file, int line);
|
||||
static unsigned long CryptGetThreadid(void);
|
||||
static struct CRYPTO_dynlock_value* CreateDynCryptMutex(const char* file, int line);
|
||||
static void DynCryptMutexLock(int mode, struct CRYPTO_dynlock_value* dyndata, const char* file, int line);
|
||||
static void DestoryDynCryptMutex(struct CRYPTO_dynlock_value* dyndata, const char* file, int line);
|
||||
static int CurlProgress(void *clientp, double dltotal, double dlnow, double ultotal, double ulnow);
|
||||
|
||||
static bool InitMimeType(const char* MimeFile = NULL);
|
||||
@ -226,6 +238,7 @@ class S3fsCurl
|
||||
|
||||
static bool ParseIAMCredentialResponse(const char* response, iamcredmap_t& keyval);
|
||||
static bool SetIAMCredentials(const char* response);
|
||||
static bool PushbackSseKeys(std::string& onekey);
|
||||
|
||||
// methods
|
||||
bool ResetHandle(void);
|
||||
@ -235,7 +248,7 @@ class S3fsCurl
|
||||
bool GetUploadId(std::string& upload_id);
|
||||
int GetIAMCredentials(void);
|
||||
|
||||
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool ow_sse_flg);
|
||||
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
|
||||
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
|
||||
int UploadMultipartPostSetup(const char* tpath, int part_num, std::string& upload_id);
|
||||
int UploadMultipartPostRequest(const char* tpath, int part_num, std::string& upload_id);
|
||||
@ -245,7 +258,7 @@ class S3fsCurl
|
||||
// class methods
|
||||
static bool InitS3fsCurl(const char* MimeFile = NULL);
|
||||
static bool DestroyS3fsCurl(void);
|
||||
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
|
||||
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd);
|
||||
static int ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
|
||||
static bool CheckIAMCredentialUpdate(void);
|
||||
|
||||
@ -262,6 +275,12 @@ class S3fsCurl
|
||||
static std::string SetDefaultAcl(const char* acl);
|
||||
static bool SetUseRrs(bool flag);
|
||||
static bool GetUseRrs(void) { return S3fsCurl::is_use_rrs; }
|
||||
static bool SetSseKeys(const char* filepath);
|
||||
static bool LoadEnvSseKeys(void);
|
||||
static bool GetSseKey(std::string& md5, std::string& ssekey);
|
||||
static bool GetSseKeyMd5(int pos, std::string& md5);
|
||||
static int GetSseKeyCount(void);
|
||||
static bool IsSseCustomMode(void);
|
||||
static bool SetUseSse(bool flag);
|
||||
static bool GetUseSse(void) { return S3fsCurl::is_use_sse; }
|
||||
static bool SetContentMd5(bool flag);
|
||||
@ -274,31 +293,35 @@ class S3fsCurl
|
||||
static long SetSslVerifyHostname(long value);
|
||||
static long GetSslVerifyHostname(void) { return S3fsCurl::ssl_verify_hostname; }
|
||||
static int SetMaxParallelCount(int value);
|
||||
static int GetMaxParallelCount(void) { return S3fsCurl::max_parallel_cnt; }
|
||||
static std::string SetIAMRole(const char* role);
|
||||
static const char* GetIAMRole(void) { return S3fsCurl::IAM_role.c_str(); }
|
||||
static bool SetMultipartSize(off_t size);
|
||||
static off_t GetMultipartSize(void) { return S3fsCurl::multipart_size; }
|
||||
|
||||
// methods
|
||||
bool CreateCurlHandle(bool force = false);
|
||||
bool DestroyCurlHandle(void);
|
||||
|
||||
bool AddSseKeyRequestHead(std::string& md5, bool is_copy);
|
||||
bool GetResponseCode(long& responseCode);
|
||||
int RequestPerform(void);
|
||||
int DeleteRequest(const char* tpath);
|
||||
bool PreHeadRequest(const char* tpath, const char* bpath = NULL, const char* savedpath = NULL);
|
||||
bool PreHeadRequest(std::string& tpath, std::string& bpath, std::string& savedpath) {
|
||||
return PreHeadRequest(tpath.c_str(), bpath.c_str(), savedpath.c_str());
|
||||
bool PreHeadRequest(const char* tpath, const char* bpath = NULL, const char* savedpath = NULL, int ssekey_pos = -1);
|
||||
bool PreHeadRequest(std::string& tpath, std::string& bpath, std::string& savedpath, int ssekey_pos = -1) {
|
||||
return PreHeadRequest(tpath.c_str(), bpath.c_str(), savedpath.c_str(), ssekey_pos);
|
||||
}
|
||||
int HeadRequest(const char* tpath, headers_t& meta);
|
||||
int PutHeadRequest(const char* tpath, headers_t& meta, bool ow_sse_flg);
|
||||
int PutRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
|
||||
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
|
||||
int PutHeadRequest(const char* tpath, headers_t& meta, bool is_copy);
|
||||
int PutRequest(const char* tpath, headers_t& meta, int fd);
|
||||
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, std::string& ssekeymd5);
|
||||
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
|
||||
int CheckBucket(void);
|
||||
int ListBucketRequest(const char* tpath, const char* query);
|
||||
int MultipartListRequest(std::string& body);
|
||||
int AbortMultipartUpload(const char* tpath, std::string& upload_id);
|
||||
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta);
|
||||
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
|
||||
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta, bool is_copy);
|
||||
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool is_copy);
|
||||
int MultipartRenameRequest(const char* from, const char* to, headers_t& meta, off_t size);
|
||||
|
||||
// methods(valiables)
|
||||
@ -318,6 +341,7 @@ class S3fsCurl
|
||||
int GetMultipartRetryCount(void) const { return retry_count; }
|
||||
void SetMultipartRetryCount(int retrycnt) { retry_count = retrycnt; }
|
||||
bool IsOverMultipartRetryCount(void) const { return (retry_count >= S3fsCurl::retries); }
|
||||
int GetLastPreHeadSeecKeyPos(void) const { return b_ssekey_pos; }
|
||||
};
|
||||
|
||||
//----------------------------------------------
|
||||
@ -400,3 +424,12 @@ struct curl_slist* curl_slist_sort_insert(struct curl_slist* list, const char* d
|
||||
bool MakeUrlResource(const char* realpath, std::string& resourcepath, std::string& url);
|
||||
|
||||
#endif // S3FS_CURL_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
@ -31,7 +31,6 @@
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
#include <curl/curl.h>
|
||||
#include <openssl/crypto.h>
|
||||
#include <string>
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
@ -51,8 +50,7 @@ using namespace std;
|
||||
//------------------------------------------------
|
||||
// Symbols
|
||||
//------------------------------------------------
|
||||
#define MAX_OBJECT_SIZE 68719476735LL // 64GB - 1L
|
||||
#define MULTIPART_LOWLIMIT (20 * 1024 * 1024) // 20MB
|
||||
#define MAX_MULTIPART_CNT 10000 // S3 multipart max count
|
||||
#define FDPAGE_SIZE (50 * 1024 * 1024) // 50MB(parallel uploading is 5 parallel(default) * 10 MB)
|
||||
|
||||
//------------------------------------------------
|
||||
@ -266,7 +264,7 @@ int PageList::Init(off_t size, bool is_init)
|
||||
{
|
||||
Clear();
|
||||
for(off_t total = 0; total < size; total += FdManager::GetPageSize()){
|
||||
size_t areasize = (total + FdManager::GetPageSize()) < size ? FdManager::GetPageSize() : static_cast<size_t>(size - total);
|
||||
size_t areasize = (total + static_cast<off_t>(FdManager::GetPageSize())) < size ? FdManager::GetPageSize() : static_cast<size_t>(size - total);
|
||||
fdpage* page = new fdpage(total, areasize, is_init);
|
||||
pages.push_back(page);
|
||||
}
|
||||
@ -798,7 +796,7 @@ int FdEntity::Load(off_t start, off_t size)
|
||||
break;
|
||||
}
|
||||
// download
|
||||
if((*iter)->bytes >= MULTIPART_LOWLIMIT && !nomultipart){ // 20MB
|
||||
if((*iter)->bytes >= static_cast<size_t>(2 * S3fsCurl::GetMultipartSize()) && !nomultipart){ // default 20MB
|
||||
// parallel request
|
||||
// Additional time is needed for large files
|
||||
time_t backup = 0;
|
||||
@ -857,7 +855,7 @@ bool FdEntity::LoadFull(off_t* size, bool force_load)
|
||||
return true;
|
||||
}
|
||||
|
||||
int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool force_sync)
|
||||
int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool force_sync)
|
||||
{
|
||||
int result;
|
||||
|
||||
@ -880,14 +878,14 @@ int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool
|
||||
* - 1 to 10,000 parts are allowed
|
||||
* - minimum size of parts is 5MB (expect for the last part)
|
||||
*
|
||||
* For our application, we will define part size to be 10MB (10 * 2^20 Bytes)
|
||||
* maximum file size will be ~64 GB - 2 ** 36
|
||||
* For our application, we will define minimum part size to be 10MB (10 * 2^20 Bytes)
|
||||
* minimum file size will be 64 GB - 2 ** 36
|
||||
*
|
||||
* Initially uploads will be done serially
|
||||
*
|
||||
* If file is > 20MB, then multipart will kick in
|
||||
*/
|
||||
if(pagelist.Size() > MAX_OBJECT_SIZE){ // 64GB - 1
|
||||
if(pagelist.Size() > (MAX_MULTIPART_CNT * S3fsCurl::GetMultipartSize())){
|
||||
// close f ?
|
||||
return -ENOTSUP;
|
||||
}
|
||||
@ -898,19 +896,19 @@ int FdEntity::RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool
|
||||
return -errno;
|
||||
}
|
||||
|
||||
if(pagelist.Size() >= MULTIPART_LOWLIMIT && !nomultipart){ // 20MB
|
||||
if(pagelist.Size() >= (2 * S3fsCurl::GetMultipartSize()) && !nomultipart){ // default 20MB
|
||||
// Additional time is needed for large files
|
||||
time_t backup = 0;
|
||||
if(120 > S3fsCurl::GetReadwriteTimeout()){
|
||||
backup = S3fsCurl::SetReadwriteTimeout(120);
|
||||
}
|
||||
result = S3fsCurl::ParallelMultipartUploadRequest(tpath ? tpath : path.c_str(), meta, fd, ow_sse_flg);
|
||||
result = S3fsCurl::ParallelMultipartUploadRequest(tpath ? tpath : path.c_str(), meta, fd);
|
||||
if(0 != backup){
|
||||
S3fsCurl::SetReadwriteTimeout(backup);
|
||||
}
|
||||
}else{
|
||||
S3fsCurl s3fscurl(true);
|
||||
result = s3fscurl.PutRequest(tpath ? tpath : path.c_str(), meta, fd, ow_sse_flg);
|
||||
result = s3fscurl.PutRequest(tpath ? tpath : path.c_str(), meta, fd);
|
||||
}
|
||||
|
||||
// seek to head of file.
|
||||
@ -1198,3 +1196,11 @@ bool FdManager::Close(FdEntity* ent)
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef FD_CACHE_H_
|
||||
#define FD_CACHE_H_
|
||||
|
||||
@ -110,8 +129,8 @@ class FdEntity
|
||||
bool SetAllDisable(void) { return SetAllStatus(false); }
|
||||
bool LoadFull(off_t* size = NULL, bool force_load = false);
|
||||
int Load(off_t start, off_t size);
|
||||
int RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool force_sync = false);
|
||||
int Flush(headers_t& meta, bool ow_sse_flg, bool force_sync = false) { return RowFlush(NULL, meta, ow_sse_flg, force_sync); }
|
||||
int RowFlush(const char* tpath, headers_t& meta, bool force_sync = false);
|
||||
int Flush(headers_t& meta, bool force_sync = false) { return RowFlush(NULL, meta, force_sync); }
|
||||
ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false);
|
||||
ssize_t Write(const char* bytes, off_t start, size_t size);
|
||||
};
|
||||
@ -154,3 +173,12 @@ class FdManager
|
||||
};
|
||||
|
||||
#endif // FD_CACHE_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
266
src/gnutls_auth.cpp
Normal file
266
src/gnutls_auth.cpp
Normal file
@ -0,0 +1,266 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <errno.h>
|
||||
#include <pthread.h>
|
||||
#include <unistd.h>
|
||||
#include <syslog.h>
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
#include <string.h>
|
||||
#include <gcrypt.h>
|
||||
#include <gnutls/gnutls.h>
|
||||
#include <gnutls/crypto.h>
|
||||
#ifdef USE_GNUTLS_NETTLE
|
||||
#include <nettle/md5.h>
|
||||
#include <nettle/sha1.h>
|
||||
#include <nettle/hmac.h>
|
||||
#endif
|
||||
#include <string>
|
||||
#include <map>
|
||||
|
||||
#include "common.h"
|
||||
#include "s3fs_auth.h"
|
||||
|
||||
using namespace std;
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for version
|
||||
//-------------------------------------------------------------------
|
||||
#ifdef USE_GNUTLS_NETTLE
|
||||
|
||||
const char* s3fs_crypt_lib_name(void)
|
||||
{
|
||||
static const char version[] = "GnuTLS(nettle)";
|
||||
|
||||
return version;
|
||||
}
|
||||
|
||||
#else // USE_GNUTLS_NETTLE
|
||||
|
||||
const char* s3fs_crypt_lib_name(void)
|
||||
{
|
||||
static const char version[] = "GnuTLS(gcrypt)";
|
||||
|
||||
return version;
|
||||
}
|
||||
|
||||
#endif // USE_GNUTLS_NETTLE
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for global init
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_init_global_ssl(void)
|
||||
{
|
||||
if(GNUTLS_E_SUCCESS != gnutls_global_init()){
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool s3fs_destroy_global_ssl(void)
|
||||
{
|
||||
gnutls_global_deinit();
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for crypt lock
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_init_crypt_mutex(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
bool s3fs_destroy_crypt_mutex(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for HMAC
|
||||
//-------------------------------------------------------------------
|
||||
#ifdef USE_GNUTLS_NETTLE
|
||||
|
||||
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
|
||||
{
|
||||
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
|
||||
return false;
|
||||
}
|
||||
|
||||
if(NULL == (*digest = (unsigned char*)malloc(SHA1_DIGEST_SIZE))){
|
||||
return false;
|
||||
}
|
||||
|
||||
struct hmac_sha1_ctx ctx_hmac;
|
||||
hmac_sha1_set_key(&ctx_hmac, keylen, reinterpret_cast<const uint8_t*>(key));
|
||||
hmac_sha1_update(&ctx_hmac, datalen, reinterpret_cast<const uint8_t*>(data));
|
||||
hmac_sha1_digest(&ctx_hmac, SHA1_DIGEST_SIZE, reinterpret_cast<uint8_t*>(*digest));
|
||||
*digestlen = SHA1_DIGEST_SIZE;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
#else // USE_GNUTLS_NETTLE
|
||||
|
||||
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
|
||||
{
|
||||
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
|
||||
return false;
|
||||
}
|
||||
|
||||
if(0 >= (*digestlen = gnutls_hmac_get_len(GNUTLS_MAC_SHA1))){
|
||||
return false;
|
||||
}
|
||||
if(NULL == (*digest = (unsigned char*)malloc(*digestlen + 1))){
|
||||
return false;
|
||||
}
|
||||
if(0 > gnutls_hmac_fast(GNUTLS_MAC_SHA1, key, keylen, data, datalen, *digest)){
|
||||
free(*digest);
|
||||
*digest = NULL;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
#endif // USE_GNUTLS_NETTLE
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for MD5
|
||||
//-------------------------------------------------------------------
|
||||
#define MD5_DIGEST_LENGTH 16
|
||||
|
||||
size_t get_md5_digest_length(void)
|
||||
{
|
||||
return MD5_DIGEST_LENGTH;
|
||||
}
|
||||
|
||||
#ifdef USE_GNUTLS_NETTLE
|
||||
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
|
||||
{
|
||||
struct md5_ctx ctx_md5;
|
||||
unsigned char buf[512];
|
||||
ssize_t bytes;
|
||||
unsigned char* result;
|
||||
|
||||
// seek to top of file.
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
return NULL;
|
||||
}
|
||||
|
||||
memset(buf, 0, 512);
|
||||
md5_init(&ctx_md5);
|
||||
|
||||
for(ssize_t total = 0; total < size; total += bytes){
|
||||
bytes = 512 < (size - total) ? 512 : (size - total);
|
||||
bytes = read(fd, buf, bytes);
|
||||
if(0 == bytes){
|
||||
// end of file
|
||||
break;
|
||||
}else if(-1 == bytes){
|
||||
// error
|
||||
DPRNNN("file read error(%d)", errno);
|
||||
return NULL;
|
||||
}
|
||||
md5_update(&ctx_md5, bytes, buf);
|
||||
memset(buf, 0, 512);
|
||||
}
|
||||
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
|
||||
return NULL;
|
||||
}
|
||||
md5_digest(&ctx_md5, get_md5_digest_length(), result);
|
||||
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
free(result);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
#else // USE_GNUTLS_NETTLE
|
||||
|
||||
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
|
||||
{
|
||||
gcry_md_hd_t ctx_md5;
|
||||
gcry_error_t err;
|
||||
char buf[512];
|
||||
ssize_t bytes;
|
||||
unsigned char* result;
|
||||
|
||||
if(-1 == size){
|
||||
struct stat st;
|
||||
if(-1 == fstat(fd, &st)){
|
||||
return NULL;
|
||||
}
|
||||
size = static_cast<ssize_t>(st.st_size);
|
||||
}
|
||||
|
||||
// seek to top of file.
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
return NULL;
|
||||
}
|
||||
|
||||
memset(buf, 0, 512);
|
||||
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_md5, GCRY_MD_MD5, 0))){
|
||||
DPRNN("MD5 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
|
||||
return NULL;
|
||||
}
|
||||
|
||||
for(ssize_t total = 0; total < size; total += bytes){
|
||||
bytes = 512 < (size - total) ? 512 : (size - total);
|
||||
bytes = read(fd, buf, bytes);
|
||||
if(0 == bytes){
|
||||
// end of file
|
||||
break;
|
||||
}else if(-1 == bytes){
|
||||
// error
|
||||
DPRNNN("file read error(%d)", errno);
|
||||
return NULL;
|
||||
}
|
||||
gcry_md_write(ctx_md5, buf, bytes);
|
||||
memset(buf, 0, 512);
|
||||
}
|
||||
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
|
||||
return NULL;
|
||||
}
|
||||
memcpy(result, gcry_md_read(ctx_md5, 0), get_md5_digest_length());
|
||||
gcry_md_close(ctx_md5);
|
||||
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
free(result);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
#endif // USE_GNUTLS_NETTLE
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
203
src/nss_auth.cpp
Normal file
203
src/nss_auth.cpp
Normal file
@ -0,0 +1,203 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <errno.h>
|
||||
#include <pthread.h>
|
||||
#include <unistd.h>
|
||||
#include <syslog.h>
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
#include <string.h>
|
||||
#include <nss.h>
|
||||
#include <pk11pub.h>
|
||||
#include <hasht.h>
|
||||
#include <prinit.h>
|
||||
#include <string>
|
||||
#include <map>
|
||||
|
||||
#include "common.h"
|
||||
#include "s3fs_auth.h"
|
||||
|
||||
using namespace std;
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for version
|
||||
//-------------------------------------------------------------------
|
||||
const char* s3fs_crypt_lib_name(void)
|
||||
{
|
||||
static const char version[] = "NSS";
|
||||
|
||||
return version;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for global init
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_init_global_ssl(void)
|
||||
{
|
||||
NSS_Init(NULL);
|
||||
NSS_NoDB_Init(NULL);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool s3fs_destroy_global_ssl(void)
|
||||
{
|
||||
NSS_Shutdown();
|
||||
PL_ArenaFinish();
|
||||
PR_Cleanup();
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for crypt lock
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_init_crypt_mutex(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
bool s3fs_destroy_crypt_mutex(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for HMAC
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
|
||||
{
|
||||
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
|
||||
return false;
|
||||
}
|
||||
|
||||
PK11SlotInfo* Slot;
|
||||
PK11SymKey* pKey;
|
||||
PK11Context* Context;
|
||||
SECStatus SecStatus;
|
||||
unsigned char tmpdigest[64];
|
||||
SECItem KeySecItem = {siBuffer, reinterpret_cast<unsigned char*>(const_cast<void*>(key)), keylen};
|
||||
SECItem NullSecItem = {siBuffer, NULL, 0};
|
||||
|
||||
if(NULL == (Slot = PK11_GetInternalKeySlot())){
|
||||
return false;
|
||||
}
|
||||
if(NULL == (pKey = PK11_ImportSymKey(Slot, CKM_SHA_1_HMAC, PK11_OriginUnwrap, CKA_SIGN, &KeySecItem, NULL))){
|
||||
PK11_FreeSlot(Slot);
|
||||
return false;
|
||||
}
|
||||
if(NULL == (Context = PK11_CreateContextBySymKey(CKM_SHA_1_HMAC, CKA_SIGN, pKey, &NullSecItem))){
|
||||
PK11_FreeSymKey(pKey);
|
||||
PK11_FreeSlot(Slot);
|
||||
return false;
|
||||
}
|
||||
|
||||
*digestlen = 0;
|
||||
if(SECSuccess != (SecStatus = PK11_DigestBegin(Context)) ||
|
||||
SECSuccess != (SecStatus = PK11_DigestOp(Context, data, datalen)) ||
|
||||
SECSuccess != (SecStatus = PK11_DigestFinal(Context, tmpdigest, digestlen, sizeof(tmpdigest))) )
|
||||
{
|
||||
PK11_DestroyContext(Context, PR_TRUE);
|
||||
PK11_FreeSymKey(pKey);
|
||||
PK11_FreeSlot(Slot);
|
||||
return false;
|
||||
}
|
||||
PK11_DestroyContext(Context, PR_TRUE);
|
||||
PK11_FreeSymKey(pKey);
|
||||
PK11_FreeSlot(Slot);
|
||||
|
||||
if(NULL == (*digest = (unsigned char*)malloc(*digestlen))){
|
||||
return false;
|
||||
}
|
||||
memcpy(*digest, tmpdigest, *digestlen);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for MD5
|
||||
//-------------------------------------------------------------------
|
||||
size_t get_md5_digest_length(void)
|
||||
{
|
||||
return MD5_LENGTH;
|
||||
}
|
||||
|
||||
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
|
||||
{
|
||||
PK11Context* md5ctx;
|
||||
unsigned char buf[512];
|
||||
ssize_t bytes;
|
||||
unsigned char* result;
|
||||
unsigned int md5outlen;
|
||||
|
||||
if(-1 == size){
|
||||
struct stat st;
|
||||
if(-1 == fstat(fd, &st)){
|
||||
return NULL;
|
||||
}
|
||||
size = static_cast<ssize_t>(st.st_size);
|
||||
}
|
||||
|
||||
// seek to top of file.
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
return NULL;
|
||||
}
|
||||
|
||||
memset(buf, 0, 512);
|
||||
md5ctx = PK11_CreateDigestContext(SEC_OID_MD5);
|
||||
|
||||
for(ssize_t total = 0; total < size; total += bytes){
|
||||
bytes = 512 < (size - total) ? 512 : (size - total);
|
||||
bytes = read(fd, buf, bytes);
|
||||
if(0 == bytes){
|
||||
// end of file
|
||||
break;
|
||||
}else if(-1 == bytes){
|
||||
// error
|
||||
DPRNNN("file read error(%d)", errno);
|
||||
return NULL;
|
||||
}
|
||||
PK11_DigestOp(md5ctx, buf, bytes);
|
||||
memset(buf, 0, 512);
|
||||
}
|
||||
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
|
||||
PK11_DestroyContext(md5ctx, PR_TRUE);
|
||||
return NULL;
|
||||
}
|
||||
PK11_DigestFinal(md5ctx, result, &md5outlen, get_md5_digest_length());
|
||||
PK11_DestroyContext(md5ctx, PR_TRUE);
|
||||
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
free(result);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
263
src/openssl_auth.cpp
Normal file
263
src/openssl_auth.cpp
Normal file
@ -0,0 +1,263 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <errno.h>
|
||||
#include <pthread.h>
|
||||
#include <unistd.h>
|
||||
#include <syslog.h>
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
#include <string.h>
|
||||
#include <openssl/bio.h>
|
||||
#include <openssl/buffer.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <openssl/hmac.h>
|
||||
#include <openssl/md5.h>
|
||||
#include <openssl/crypto.h>
|
||||
#include <openssl/err.h>
|
||||
#include <string>
|
||||
#include <map>
|
||||
|
||||
#include "common.h"
|
||||
#include "s3fs_auth.h"
|
||||
|
||||
using namespace std;
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for version
|
||||
//-------------------------------------------------------------------
|
||||
const char* s3fs_crypt_lib_name(void)
|
||||
{
|
||||
static const char version[] = "OpenSSL";
|
||||
|
||||
return version;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for global init
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_init_global_ssl(void)
|
||||
{
|
||||
ERR_load_crypto_strings();
|
||||
ERR_load_BIO_strings();
|
||||
OpenSSL_add_all_algorithms();
|
||||
return true;
|
||||
}
|
||||
|
||||
bool s3fs_destroy_global_ssl(void)
|
||||
{
|
||||
EVP_cleanup();
|
||||
ERR_free_strings();
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for crypt lock
|
||||
//-------------------------------------------------------------------
|
||||
// internal use struct for openssl
|
||||
struct CRYPTO_dynlock_value
|
||||
{
|
||||
pthread_mutex_t dyn_mutex;
|
||||
};
|
||||
|
||||
static pthread_mutex_t* s3fs_crypt_mutex = NULL;
|
||||
|
||||
static void s3fs_crypt_mutex_lock(int mode, int pos, const char* file, int line)
|
||||
{
|
||||
if(s3fs_crypt_mutex){
|
||||
if(mode & CRYPTO_LOCK){
|
||||
pthread_mutex_lock(&s3fs_crypt_mutex[pos]);
|
||||
}else{
|
||||
pthread_mutex_unlock(&s3fs_crypt_mutex[pos]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static unsigned long s3fs_crypt_get_threadid(void)
|
||||
{
|
||||
return static_cast<unsigned long>(pthread_self());
|
||||
}
|
||||
|
||||
static struct CRYPTO_dynlock_value* s3fs_dyn_crypt_mutex(const char* file, int line)
|
||||
{
|
||||
struct CRYPTO_dynlock_value* dyndata;
|
||||
|
||||
if(NULL == (dyndata = static_cast<struct CRYPTO_dynlock_value*>(malloc(sizeof(struct CRYPTO_dynlock_value))))){
|
||||
DPRNCRIT("Could not allocate memory for CRYPTO_dynlock_value");
|
||||
return NULL;
|
||||
}
|
||||
pthread_mutex_init(&(dyndata->dyn_mutex), NULL);
|
||||
return dyndata;
|
||||
}
|
||||
|
||||
static void s3fs_dyn_crypt_mutex_lock(int mode, struct CRYPTO_dynlock_value* dyndata, const char* file, int line)
|
||||
{
|
||||
if(dyndata){
|
||||
if(mode & CRYPTO_LOCK){
|
||||
pthread_mutex_lock(&(dyndata->dyn_mutex));
|
||||
}else{
|
||||
pthread_mutex_unlock(&(dyndata->dyn_mutex));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static void s3fs_destroy_dyn_crypt_mutex(struct CRYPTO_dynlock_value* dyndata, const char* file, int line)
|
||||
{
|
||||
if(dyndata){
|
||||
pthread_mutex_destroy(&(dyndata->dyn_mutex));
|
||||
free(dyndata);
|
||||
}
|
||||
}
|
||||
|
||||
bool s3fs_init_crypt_mutex(void)
|
||||
{
|
||||
if(s3fs_crypt_mutex){
|
||||
FPRNNN("s3fs_crypt_mutex is not NULL, destory it.");
|
||||
if(!s3fs_destroy_crypt_mutex()){
|
||||
DPRN("Failed to s3fs_crypt_mutex");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if(NULL == (s3fs_crypt_mutex = static_cast<pthread_mutex_t*>(malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t))))){
|
||||
DPRNCRIT("Could not allocate memory for s3fs_crypt_mutex");
|
||||
return false;
|
||||
}
|
||||
for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){
|
||||
pthread_mutex_init(&s3fs_crypt_mutex[cnt], NULL);
|
||||
}
|
||||
// static lock
|
||||
CRYPTO_set_locking_callback(s3fs_crypt_mutex_lock);
|
||||
CRYPTO_set_id_callback(s3fs_crypt_get_threadid);
|
||||
// dynamic lock
|
||||
CRYPTO_set_dynlock_create_callback(s3fs_dyn_crypt_mutex);
|
||||
CRYPTO_set_dynlock_lock_callback(s3fs_dyn_crypt_mutex_lock);
|
||||
CRYPTO_set_dynlock_destroy_callback(s3fs_destroy_dyn_crypt_mutex);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
bool s3fs_destroy_crypt_mutex(void)
|
||||
{
|
||||
if(!s3fs_crypt_mutex){
|
||||
return true;
|
||||
}
|
||||
|
||||
CRYPTO_set_dynlock_destroy_callback(NULL);
|
||||
CRYPTO_set_dynlock_lock_callback(NULL);
|
||||
CRYPTO_set_dynlock_create_callback(NULL);
|
||||
CRYPTO_set_id_callback(NULL);
|
||||
CRYPTO_set_locking_callback(NULL);
|
||||
|
||||
for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){
|
||||
pthread_mutex_destroy(&s3fs_crypt_mutex[cnt]);
|
||||
}
|
||||
CRYPTO_cleanup_all_ex_data();
|
||||
free(s3fs_crypt_mutex);
|
||||
s3fs_crypt_mutex = NULL;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for HMAC
|
||||
//-------------------------------------------------------------------
|
||||
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen)
|
||||
{
|
||||
if(!key || 0 >= keylen || !data || 0 >= datalen || !digest || !digestlen){
|
||||
return false;
|
||||
}
|
||||
(*digestlen) = EVP_MAX_MD_SIZE * sizeof(unsigned char);
|
||||
if(NULL == ((*digest) = (unsigned char*)malloc(*digestlen))){
|
||||
return false;
|
||||
}
|
||||
HMAC(EVP_sha1(), key, keylen, data, datalen, *digest, digestlen);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility Function for MD5
|
||||
//-------------------------------------------------------------------
|
||||
size_t get_md5_digest_length(void)
|
||||
{
|
||||
return MD5_DIGEST_LENGTH;
|
||||
}
|
||||
|
||||
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
|
||||
{
|
||||
MD5_CTX md5ctx;
|
||||
char buf[512];
|
||||
ssize_t bytes;
|
||||
unsigned char* result;
|
||||
|
||||
if(-1 == size){
|
||||
struct stat st;
|
||||
if(-1 == fstat(fd, &st)){
|
||||
return NULL;
|
||||
}
|
||||
size = static_cast<ssize_t>(st.st_size);
|
||||
}
|
||||
|
||||
// seek to top of file.
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
return NULL;
|
||||
}
|
||||
|
||||
memset(buf, 0, 512);
|
||||
MD5_Init(&md5ctx);
|
||||
|
||||
for(ssize_t total = 0; total < size; total += bytes){
|
||||
bytes = 512 < (size - total) ? 512 : (size - total);
|
||||
bytes = read(fd, buf, bytes);
|
||||
if(0 == bytes){
|
||||
// end of file
|
||||
break;
|
||||
}else if(-1 == bytes){
|
||||
// error
|
||||
DPRNNN("file read error(%d)", errno);
|
||||
return NULL;
|
||||
}
|
||||
MD5_Update(&md5ctx, buf, bytes);
|
||||
memset(buf, 0, 512);
|
||||
}
|
||||
|
||||
if(NULL == (result = (unsigned char*)malloc(get_md5_digest_length()))){
|
||||
return NULL;
|
||||
}
|
||||
MD5_Final(result, &md5ctx);
|
||||
|
||||
if(-1 == lseek(fd, start, SEEK_SET)){
|
||||
free(result);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
228
src/s3fs.cpp
228
src/s3fs.cpp
@ -31,7 +31,6 @@
|
||||
#include <libxml/xpathInternals.h>
|
||||
#include <libxml/tree.h>
|
||||
#include <curl/curl.h>
|
||||
#include <openssl/crypto.h>
|
||||
#include <pwd.h>
|
||||
#include <grp.h>
|
||||
#include <getopt.h>
|
||||
@ -50,6 +49,7 @@
|
||||
#include "string_util.h"
|
||||
#include "s3fs_util.h"
|
||||
#include "fdcache.h"
|
||||
#include "s3fs_auth.h"
|
||||
|
||||
using namespace std;
|
||||
|
||||
@ -83,6 +83,7 @@ bool debug = false;
|
||||
bool foreground = false;
|
||||
bool foreground2 = false;
|
||||
bool nomultipart = false;
|
||||
bool pathrequeststyle = false;
|
||||
std::string program_name;
|
||||
std::string service_path = "/";
|
||||
std::string host = "http://s3.amazonaws.com";
|
||||
@ -134,7 +135,7 @@ static xmlChar* get_base_exp(xmlDocPtr doc, const char* exp);
|
||||
static xmlChar* get_prefix(xmlDocPtr doc);
|
||||
static xmlChar* get_next_marker(xmlDocPtr doc);
|
||||
static char* get_object_name(xmlDocPtr doc, xmlNodePtr node, const char* path);
|
||||
static int put_headers(const char* path, headers_t& meta, bool ow_sse_flg);
|
||||
static int put_headers(const char* path, headers_t& meta, bool is_copy);
|
||||
static int rename_large_object(const char* from, const char* to);
|
||||
static int create_file_object(const char* path, mode_t mode, uid_t uid, gid_t gid);
|
||||
static int create_directory_object(const char* path, mode_t mode, time_t time, uid_t uid, gid_t gid);
|
||||
@ -609,6 +610,30 @@ static int check_parent_object_access(const char* path, int mask)
|
||||
return 0;
|
||||
}
|
||||
|
||||
//
|
||||
// This function is global, is called fom curl class(GetObject).
|
||||
//
|
||||
char* get_object_sseckey_md5(const char* path)
|
||||
{
|
||||
if(!path){
|
||||
return NULL;
|
||||
}
|
||||
headers_t meta;
|
||||
|
||||
if(0 != get_object_attribute(path, NULL, &meta)){
|
||||
DPRNNN("Failed to get object(%s) headers", path);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
for(headers_t::iterator iter = meta.begin(); iter != meta.end(); ++iter){
|
||||
string key = (*iter).first;
|
||||
if(0 == strcasecmp(key.c_str(), "x-amz-server-side-encryption-customer-key-md5")){
|
||||
return strdup((*iter).second.c_str());
|
||||
}
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static FdEntity* get_local_fent(const char* path, bool is_load)
|
||||
{
|
||||
struct stat stobj;
|
||||
@ -643,7 +668,7 @@ static FdEntity* get_local_fent(const char* path, bool is_load)
|
||||
* ow_sse_flg is for over writing sse header by use_sse option.
|
||||
* @return fuse return code
|
||||
*/
|
||||
static int put_headers(const char* path, headers_t& meta, bool ow_sse_flg)
|
||||
static int put_headers(const char* path, headers_t& meta, bool is_copy)
|
||||
{
|
||||
int result;
|
||||
S3fsCurl s3fscurl(true);
|
||||
@ -658,11 +683,11 @@ static int put_headers(const char* path, headers_t& meta, bool ow_sse_flg)
|
||||
|
||||
if(buf.st_size >= FIVE_GB){
|
||||
// multipart
|
||||
if(0 != (result = s3fscurl.MultipartHeadRequest(path, buf.st_size, meta))){
|
||||
if(0 != (result = s3fscurl.MultipartHeadRequest(path, buf.st_size, meta, is_copy))){
|
||||
return result;
|
||||
}
|
||||
}else{
|
||||
if(0 != (result = s3fscurl.PutHeadRequest(path, meta, ow_sse_flg))){
|
||||
if(0 != (result = s3fscurl.PutHeadRequest(path, meta, is_copy))){
|
||||
return result;
|
||||
}
|
||||
}
|
||||
@ -708,6 +733,8 @@ static int s3fs_getattr(const char* path, struct stat* stbuf)
|
||||
}
|
||||
FdManager::get()->Close(ent);
|
||||
}
|
||||
stbuf->st_blksize = 4096;
|
||||
stbuf->st_blocks = get_blocks(stbuf->st_size);
|
||||
}
|
||||
FPRNINFO("[path=%s] uid=%u, gid=%u, mode=%04o", path, (unsigned int)(stbuf->st_uid), (unsigned int)(stbuf->st_gid), stbuf->st_mode);
|
||||
S3FS_MALLOCTRIM(0);
|
||||
@ -764,7 +791,7 @@ static int create_file_object(const char* path, mode_t mode, uid_t uid, gid_t gi
|
||||
meta["x-amz-meta-mtime"] = str(time(NULL));
|
||||
|
||||
S3fsCurl s3fscurl(true);
|
||||
return s3fscurl.PutRequest(path, meta, -1, false); // fd=-1 means for creating zero byte object.
|
||||
return s3fscurl.PutRequest(path, meta, -1); // fd=-1 means for creating zero byte object.
|
||||
}
|
||||
|
||||
static int s3fs_mknod(const char *path, mode_t mode, dev_t rdev)
|
||||
@ -849,7 +876,7 @@ static int create_directory_object(const char* path, mode_t mode, time_t time, u
|
||||
meta["x-amz-meta-mtime"] = str(time);
|
||||
|
||||
S3fsCurl s3fscurl;
|
||||
return s3fscurl.PutRequest(tpath.c_str(), meta, -1, false); // fd=-1 means for creating zero byte object.
|
||||
return s3fscurl.PutRequest(tpath.c_str(), meta, -1); // fd=-1 means for creating zero byte object.
|
||||
}
|
||||
|
||||
static int s3fs_mkdir(const char* path, mode_t mode)
|
||||
@ -1011,7 +1038,7 @@ static int s3fs_symlink(const char* from, const char* to)
|
||||
return -errno;
|
||||
}
|
||||
// upload
|
||||
if(0 != (result = ent->Flush(headers, true, true))){
|
||||
if(0 != (result = ent->Flush(headers, true))){
|
||||
DPRN("could not upload tmpfile(result=%d)", result);
|
||||
}
|
||||
FdManager::get()->Close(ent);
|
||||
@ -1047,7 +1074,7 @@ static int rename_object(const char* from, const char* to)
|
||||
meta["Content-Type"] = S3fsCurl::LookupMimeType(string(to));
|
||||
meta["x-amz-metadata-directive"] = "REPLACE";
|
||||
|
||||
if(0 != (result = put_headers(to, meta, false))){
|
||||
if(0 != (result = put_headers(to, meta, true))){
|
||||
return result;
|
||||
}
|
||||
result = s3fs_unlink(from);
|
||||
@ -1088,7 +1115,7 @@ static int rename_object_nocopy(const char* from, const char* to)
|
||||
}
|
||||
|
||||
// upload
|
||||
if(0 != (result = ent->RowFlush(to, meta, false, true))){
|
||||
if(0 != (result = ent->RowFlush(to, meta, true))){
|
||||
DPRN("could not upload file(%s): result=%d", to, result);
|
||||
FdManager::get()->Close(ent);
|
||||
return result;
|
||||
@ -1388,7 +1415,7 @@ static int s3fs_chmod(const char* path, mode_t mode)
|
||||
meta["x-amz-copy-source"] = urlEncode(service_path + bucket + get_realpath(strpath.c_str()));
|
||||
meta["x-amz-metadata-directive"] = "REPLACE";
|
||||
|
||||
if(put_headers(strpath.c_str(), meta, false) != 0){
|
||||
if(put_headers(strpath.c_str(), meta, true) != 0){
|
||||
return -EIO;
|
||||
}
|
||||
StatCache::getStatCacheData()->DelStat(nowcache);
|
||||
@ -1464,7 +1491,7 @@ static int s3fs_chmod_nocopy(const char* path, mode_t mode)
|
||||
}
|
||||
|
||||
// upload
|
||||
if(0 != (result = ent->Flush(meta, false, true))){
|
||||
if(0 != (result = ent->Flush(meta, true))){
|
||||
DPRN("could not upload file(%s): result=%d", strpath.c_str(), result);
|
||||
FdManager::get()->Close(ent);
|
||||
return result;
|
||||
@ -1550,7 +1577,7 @@ static int s3fs_chown(const char* path, uid_t uid, gid_t gid)
|
||||
meta["x-amz-copy-source"] = urlEncode(service_path + bucket + get_realpath(strpath.c_str()));
|
||||
meta["x-amz-metadata-directive"] = "REPLACE";
|
||||
|
||||
if(put_headers(strpath.c_str(), meta, false) != 0){
|
||||
if(put_headers(strpath.c_str(), meta, true) != 0){
|
||||
return -EIO;
|
||||
}
|
||||
StatCache::getStatCacheData()->DelStat(nowcache);
|
||||
@ -1636,7 +1663,7 @@ static int s3fs_chown_nocopy(const char* path, uid_t uid, gid_t gid)
|
||||
}
|
||||
|
||||
// upload
|
||||
if(0 != (result = ent->Flush(meta, false, true))){
|
||||
if(0 != (result = ent->Flush(meta, true))){
|
||||
DPRN("could not upload file(%s): result=%d", strpath.c_str(), result);
|
||||
FdManager::get()->Close(ent);
|
||||
return result;
|
||||
@ -1708,7 +1735,7 @@ static int s3fs_utimens(const char* path, const struct timespec ts[2])
|
||||
meta["x-amz-copy-source"] = urlEncode(service_path + bucket + get_realpath(strpath.c_str()));
|
||||
meta["x-amz-metadata-directive"] = "REPLACE";
|
||||
|
||||
if(put_headers(strpath.c_str(), meta, false) != 0){
|
||||
if(put_headers(strpath.c_str(), meta, true) != 0){
|
||||
return -EIO;
|
||||
}
|
||||
StatCache::getStatCacheData()->DelStat(nowcache);
|
||||
@ -1793,7 +1820,7 @@ static int s3fs_utimens_nocopy(const char* path, const struct timespec ts[2])
|
||||
}
|
||||
|
||||
// upload
|
||||
if(0 != (result = ent->Flush(meta, false, true))){
|
||||
if(0 != (result = ent->Flush(meta, true))){
|
||||
DPRN("could not upload file(%s): result=%d", strpath.c_str(), result);
|
||||
FdManager::get()->Close(ent);
|
||||
return result;
|
||||
@ -1844,7 +1871,7 @@ static int s3fs_truncate(const char* path, off_t size)
|
||||
}
|
||||
|
||||
// upload
|
||||
if(0 != (result = ent->Flush(meta, false, true))){
|
||||
if(0 != (result = ent->Flush(meta, true))){
|
||||
DPRN("could not upload file(%s): result=%d", path, result);
|
||||
FdManager::get()->Close(ent);
|
||||
return result;
|
||||
@ -1999,7 +2026,7 @@ static int s3fs_flush(const char* path, struct fuse_file_info* fi)
|
||||
meta["x-amz-meta-mtime"] = str(ent_mtime);
|
||||
}
|
||||
}
|
||||
result = ent->Flush(meta, true, false);
|
||||
result = ent->Flush(meta, false);
|
||||
FdManager::get()->Close(ent);
|
||||
}
|
||||
S3FS_MALLOCTRIM(0);
|
||||
@ -2075,9 +2102,23 @@ static S3fsCurl* multi_head_retry_callback(S3fsCurl* s3fscurl)
|
||||
if(!s3fscurl){
|
||||
return NULL;
|
||||
}
|
||||
int ssec_key_pos = s3fscurl->GetLastPreHeadSeecKeyPos();
|
||||
int next_retry_count = s3fscurl->GetMultipartRetryCount() + 1;
|
||||
|
||||
if(s3fscurl->IsOverMultipartRetryCount()){
|
||||
DPRN("Over retry count(%d) limit(%s).", s3fscurl->GetMultipartRetryCount(), s3fscurl->GetSpacialSavedPath().c_str());
|
||||
return NULL;
|
||||
if(S3fsCurl::IsSseCustomMode()){
|
||||
// If sse-c mode, start check not sse-c(ssec_key_pos = -1).
|
||||
// do increment ssec_key_pos for checking all sse-c key.
|
||||
next_retry_count = 0;
|
||||
ssec_key_pos++;
|
||||
if(S3fsCurl::GetSseKeyCount() <= ssec_key_pos){
|
||||
DPRN("Over retry count(%d) limit(%s).", s3fscurl->GetMultipartRetryCount(), s3fscurl->GetSpacialSavedPath().c_str());
|
||||
return NULL;
|
||||
}
|
||||
}else{
|
||||
DPRN("Over retry count(%d) limit(%s).", s3fscurl->GetMultipartRetryCount(), s3fscurl->GetSpacialSavedPath().c_str());
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
|
||||
S3fsCurl* newcurl = new S3fsCurl(s3fscurl->IsUseAhbe());
|
||||
@ -2085,12 +2126,12 @@ static S3fsCurl* multi_head_retry_callback(S3fsCurl* s3fscurl)
|
||||
string base_path = s3fscurl->GetBasePath();
|
||||
string saved_path = s3fscurl->GetSpacialSavedPath();
|
||||
|
||||
if(!newcurl->PreHeadRequest(path, base_path, saved_path)){
|
||||
if(!newcurl->PreHeadRequest(path, base_path, saved_path, ssec_key_pos)){
|
||||
DPRN("Could not duplicate curl object(%s).", saved_path.c_str());
|
||||
delete newcurl;
|
||||
return NULL;
|
||||
}
|
||||
newcurl->SetMultipartRetryCount(s3fscurl->GetMultipartRetryCount() + 1);
|
||||
newcurl->SetMultipartRetryCount(next_retry_count);
|
||||
|
||||
return newcurl;
|
||||
}
|
||||
@ -2132,6 +2173,8 @@ static int readdir_multi_head(const char* path, S3ObjList& head, void* buf, fuse
|
||||
continue;
|
||||
}
|
||||
|
||||
// First check for directory, start checking "not sse-c".
|
||||
// If checking failed, retry to check with "sse-c" by retry callback func when sse-c mode.
|
||||
S3fsCurl* s3fscurl = new S3fsCurl();
|
||||
if(!s3fscurl->PreHeadRequest(disppath, (*iter), disppath)){ // target path = cache key path.(ex "dir/")
|
||||
DPRNNN("Could not make curl object for head request(%s).", disppath.c_str());
|
||||
@ -2269,8 +2312,20 @@ static int list_bucket(const char* path, S3ObjList& head, const char* delimiter)
|
||||
next_marker = (char*)tmpch;
|
||||
xmlFree(tmpch);
|
||||
}else{
|
||||
DPRN("Could not find next marker, thus break loop.");
|
||||
truncated = false;
|
||||
// If did not specify "delimiter", s3 did not return "NextMarker".
|
||||
// On this case, can use lastest name for next marker.
|
||||
//
|
||||
string lastname;
|
||||
if(!head.GetLastName(lastname)){
|
||||
DPRN("Could not find next marker, thus break loop.");
|
||||
truncated = false;
|
||||
}else{
|
||||
next_marker = s3_realpath.substr(1);
|
||||
if(0 == s3_realpath.length() || '/' != s3_realpath[s3_realpath.length() - 1]){
|
||||
next_marker += "/";
|
||||
}
|
||||
next_marker += lastname;
|
||||
}
|
||||
}
|
||||
}
|
||||
S3FS_XMLFREEDOC(doc);
|
||||
@ -2593,7 +2648,13 @@ static int remote_mountpath_exists(const char* path)
|
||||
static void* s3fs_init(struct fuse_conn_info* conn)
|
||||
{
|
||||
FPRN("init");
|
||||
LOWSYSLOGPRINT(LOG_ERR, "init $Rev$");
|
||||
LOWSYSLOGPRINT(LOG_ERR, "init v%s (%s)", VERSION, s3fs_crypt_lib_name());
|
||||
|
||||
// ssl init
|
||||
if(!s3fs_init_global_ssl()){
|
||||
fprintf(stderr, "%s: could not initialize for ssl libraries.\n", program_name.c_str());
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
// init curl
|
||||
if(!S3fsCurl::InitS3fsCurl("/etc/mime.types")){
|
||||
@ -2635,6 +2696,8 @@ static void s3fs_destroy(void*)
|
||||
if(is_remove_cache && !FdManager::DeleteCacheDirectory()){
|
||||
DPRN("Could not remove cache directory.");
|
||||
}
|
||||
// ssl
|
||||
s3fs_destroy_global_ssl();
|
||||
}
|
||||
|
||||
static int s3fs_access(const char* path, int mask)
|
||||
@ -2710,7 +2773,7 @@ static bool abort_uncomp_mp_list(uncomp_mp_list_t& list)
|
||||
char buff[1024];
|
||||
|
||||
if(0 >= list.size()){
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
memset(buff, 0, sizeof(buff));
|
||||
|
||||
@ -2838,10 +2901,17 @@ static int s3fs_utility_mode(void)
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
// ssl init
|
||||
if(!s3fs_init_global_ssl()){
|
||||
fprintf(stderr, "%s: could not initialize for ssl libraries.\n", program_name.c_str());
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
// init curl
|
||||
if(!S3fsCurl::InitS3fsCurl("/etc/mime.types")){
|
||||
fprintf(stderr, "%s: Could not initiate curl library.\n", program_name.c_str());
|
||||
LOWSYSLOGPRINT(LOG_ERR, "Could not initiate curl library.");
|
||||
s3fs_destroy_global_ssl();
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
@ -2886,6 +2956,10 @@ static int s3fs_utility_mode(void)
|
||||
if(!S3fsCurl::DestroyS3fsCurl()){
|
||||
DPRN("Could not release curl library.");
|
||||
}
|
||||
|
||||
// ssl
|
||||
s3fs_destroy_global_ssl();
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
@ -2955,10 +3029,18 @@ static int check_for_aws_format(void)
|
||||
ifstream PF(passwd_file.c_str());
|
||||
if(PF.good()){
|
||||
while (getline(PF, line)){
|
||||
if(line[0]=='#')
|
||||
if(line[0]=='#'){
|
||||
continue;
|
||||
if(line.size() == 0)
|
||||
}
|
||||
if(line.size() == 0){
|
||||
continue;
|
||||
}
|
||||
if('\r' == line[line.size() - 1]){
|
||||
line = line.substr(0, line.size() - 1);
|
||||
if(line.size() == 0){
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
first_pos = line.find_first_of(" \t");
|
||||
if(first_pos != string::npos){
|
||||
@ -3108,6 +3190,12 @@ static int read_passwd_file(void)
|
||||
if(line.size() == 0){
|
||||
continue;
|
||||
}
|
||||
if('\r' == line[line.size() - 1]){
|
||||
line = line.substr(0, line.size() - 1);
|
||||
if(line.size() == 0){
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
first_pos = line.find_first_of(" \t");
|
||||
if(first_pos != string::npos){
|
||||
@ -3474,22 +3562,44 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
|
||||
return 0;
|
||||
}
|
||||
if(0 == strcmp(arg, "use_sse") || 0 == STR2NCMP(arg, "use_sse=")){
|
||||
off_t sse = 1;
|
||||
// for an old format.
|
||||
if(0 == STR2NCMP(arg, "use_sse=")){
|
||||
sse = s3fs_strtoofft(strchr(arg, '=') + sizeof(char));
|
||||
}
|
||||
if(0 == sse){
|
||||
S3fsCurl::SetUseSse(false);
|
||||
}else if(1 == sse){
|
||||
if(S3fsCurl::GetUseRrs()){
|
||||
fprintf(stderr, "%s: use_sse option could not be specified with use_rrs.\n", program_name.c_str());
|
||||
return -1;
|
||||
}
|
||||
S3fsCurl::SetUseSse(true);
|
||||
const char* ssecfile = &arg[strlen("use_sse=")];
|
||||
if(0 == strcmp(ssecfile, "1")){
|
||||
if(S3fsCurl::IsSseCustomMode()){
|
||||
fprintf (stderr, "%s: already set SSE-C key by environment, and confrict use_sse option.\n", program_name.c_str());
|
||||
return -1;
|
||||
}
|
||||
S3fsCurl::SetUseSse(true);
|
||||
}else{
|
||||
// testing sse-c, try to load AES256 keys
|
||||
struct stat st;
|
||||
if(0 != stat(ssecfile, &st)){
|
||||
fprintf (stderr, "%s: could not open use_sse keys file(%s)\n", program_name.c_str(), ssecfile);
|
||||
return -1;
|
||||
}
|
||||
if(st.st_mode & (S_IXUSR | S_IRWXG | S_IRWXO)){
|
||||
fprintf (stderr, "%s: use_sse keys file %s should be 0600 permissions\n", program_name.c_str(), ssecfile);
|
||||
return -1;
|
||||
}
|
||||
if(!S3fsCurl::SetSseKeys(ssecfile)){
|
||||
fprintf (stderr, "%s: failed to load use_sse keys file %s\n", program_name.c_str(), ssecfile);
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
}else{
|
||||
fprintf(stderr, "%s: poorly formed argument to option: use_sse\n", program_name.c_str());
|
||||
return -1;
|
||||
if(S3fsCurl::GetUseRrs()){
|
||||
fprintf(stderr, "%s: use_sse option could not be specified with use_rrs.\n", program_name.c_str());
|
||||
return -1;
|
||||
}
|
||||
if(S3fsCurl::IsSseCustomMode()){
|
||||
fprintf (stderr, "%s: already set SSE-C key by environment, and confrict use_sse option.\n", program_name.c_str());
|
||||
return -1;
|
||||
}
|
||||
S3fsCurl::SetUseSse(true);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@ -3572,11 +3682,15 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
|
||||
return -1;
|
||||
}
|
||||
S3fsCurl::SetMaxParallelCount(maxpara);
|
||||
|
||||
if(FdManager::GetPageSize() < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
|
||||
FdManager::SetPageSize(static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount()));
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
if(0 == STR2NCMP(arg, "fd_page_size=")){
|
||||
size_t pagesize = static_cast<size_t>(s3fs_strtoofft(strchr(arg, '=') + sizeof(char)));
|
||||
if((1024 * 1024) >= pagesize){
|
||||
if(pagesize < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
|
||||
fprintf(stderr, "%s: argument should be over 1MB: fd_page_size\n",
|
||||
program_name.c_str());
|
||||
return -1;
|
||||
@ -3584,6 +3698,17 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
|
||||
FdManager::SetPageSize(pagesize);
|
||||
return 0;
|
||||
}
|
||||
if(0 == STR2NCMP(arg, "multipart_size=")){
|
||||
off_t size = static_cast<off_t>(s3fs_strtoofft(strchr(arg, '=') + sizeof(char)));
|
||||
if(!S3fsCurl::SetMultipartSize(size)){
|
||||
fprintf(stderr, "%s: multipart_size option could not be specified over 10(MB)\n", program_name.c_str());
|
||||
return -1;
|
||||
}
|
||||
if(FdManager::GetPageSize() < static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount())){
|
||||
FdManager::SetPageSize(static_cast<size_t>(S3fsCurl::GetMultipartSize() * S3fsCurl::GetMaxParallelCount()));
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
if(0 == STR2NCMP(arg, "ahbe_conf=")){
|
||||
string ahbe_conf = strchr(arg, '=') + sizeof(char);
|
||||
if(!AdditionalHeader::get()->Load(ahbe_conf.c_str())){
|
||||
@ -3624,6 +3749,10 @@ static int my_fuse_opt_proc(void* data, const char* arg, int key, struct fuse_ar
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
if(0 == strcmp(arg, "use_path_request_style")){
|
||||
pathrequeststyle = true;
|
||||
return 0;
|
||||
}
|
||||
|
||||
// debug option
|
||||
//
|
||||
@ -3723,6 +3852,9 @@ int main(int argc, char* argv[])
|
||||
}
|
||||
}
|
||||
|
||||
// Load SSE-C Key from env
|
||||
S3fsCurl::LoadEnvSseKeys();
|
||||
|
||||
// clear this structure
|
||||
memset(&s3fs_oper, 0, sizeof(s3fs_oper));
|
||||
|
||||
@ -3855,15 +3987,17 @@ int main(int argc, char* argv[])
|
||||
s3fs_oper.access = s3fs_access;
|
||||
s3fs_oper.create = s3fs_create;
|
||||
|
||||
// init NSS
|
||||
S3FS_INIT_NSS();
|
||||
if(!s3fs_init_global_ssl()){
|
||||
fprintf(stderr, "%s: could not initialize for ssl libraries.\n", program_name.c_str());
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
// now passing things off to fuse, fuse will finish evaluating the command line args
|
||||
fuse_res = fuse_main(custom_args.argc, custom_args.argv, &s3fs_oper, NULL);
|
||||
fuse_opt_free_args(&custom_args);
|
||||
|
||||
// cleanup NSS
|
||||
S3FS_CLEANUP_NSS();
|
||||
s3fs_destroy_global_ssl();
|
||||
|
||||
// cleanup xml2
|
||||
xmlCleanupParser();
|
||||
S3FS_MALLOCTRIM(0);
|
||||
@ -3871,3 +4005,11 @@ int main(int argc, char* argv[])
|
||||
exit(fuse_res);
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
60
src/s3fs.h
60
src/s3fs.h
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_S3_H_
|
||||
#define S3FS_S3_H_
|
||||
|
||||
@ -65,36 +84,15 @@
|
||||
|
||||
#endif // HAVE_MALLOC_TRIM
|
||||
|
||||
//
|
||||
// For initializing libcurl with NSS
|
||||
// Normally libcurl initializes the NSS library, but usually allows
|
||||
// you to initialize s3fs forcibly. Because Memory leak is reported
|
||||
// in valgrind(about curl_global_init() function), and this is for
|
||||
// the cancellation. When "--enable-nss-init" option is specified
|
||||
// at configurarion, it makes NSS_INIT_ENABLED flag into Makefile.
|
||||
// NOTICE
|
||||
// This defines and macros is temporary, and this should be deleted.
|
||||
//
|
||||
#ifdef NSS_INIT_ENABLED
|
||||
#include <nss.h>
|
||||
#include <prinit.h>
|
||||
|
||||
#define S3FS_INIT_NSS() \
|
||||
{ \
|
||||
NSS_NoDB_Init(NULL); \
|
||||
}
|
||||
#define S3FS_CLEANUP_NSS() \
|
||||
{ \
|
||||
NSS_Shutdown(); \
|
||||
PL_ArenaFinish(); \
|
||||
PR_Cleanup(); \
|
||||
}
|
||||
|
||||
#else // NSS_INIT_ENABLED
|
||||
|
||||
#define S3FS_INIT_NSS()
|
||||
#define S3FS_CLEANUP_NSS()
|
||||
|
||||
#endif // NSS_INIT_ENABLED
|
||||
char* get_object_sseckey_md5(const char* path);
|
||||
|
||||
#endif // S3FS_S3_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
54
src/s3fs_auth.h
Normal file
54
src/s3fs_auth.h
Normal file
@ -0,0 +1,54 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_AUTH_H_
|
||||
#define S3FS_AUTH_H_
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility functions for Authentication
|
||||
//-------------------------------------------------------------------
|
||||
//
|
||||
// in common_auth.cpp
|
||||
//
|
||||
char* s3fs_base64(unsigned char* input, size_t length);
|
||||
std::string s3fs_get_content_md5(int fd);
|
||||
std::string s3fs_md5sum(int fd, off_t start, ssize_t size);
|
||||
|
||||
//
|
||||
// in xxxxxx_auth.cpp
|
||||
//
|
||||
const char* s3fs_crypt_lib_name(void);
|
||||
bool s3fs_init_global_ssl(void);
|
||||
bool s3fs_destroy_global_ssl(void);
|
||||
bool s3fs_init_crypt_mutex(void);
|
||||
bool s3fs_destroy_crypt_mutex(void);
|
||||
bool s3fs_HMAC(const void* key, size_t keylen, const unsigned char* data, size_t datalen, unsigned char** digest, unsigned int* digestlen);
|
||||
size_t get_md5_digest_length(void);
|
||||
unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size);
|
||||
|
||||
#endif // S3FS_AUTH_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
@ -41,6 +41,7 @@
|
||||
#include "s3fs_util.h"
|
||||
#include "string_util.h"
|
||||
#include "s3fs.h"
|
||||
#include "s3fs_auth.h"
|
||||
|
||||
using namespace std;
|
||||
|
||||
@ -228,6 +229,26 @@ bool S3ObjList::IsDir(const char* name) const
|
||||
return ps3obj->is_dir;
|
||||
}
|
||||
|
||||
bool S3ObjList::GetLastName(std::string& lastname) const
|
||||
{
|
||||
bool result = false;
|
||||
lastname = "";
|
||||
for(s3obj_t::const_iterator iter = objects.begin(); iter != objects.end(); iter++){
|
||||
if((*iter).second.orgname.length()){
|
||||
if(0 > strcmp(lastname.c_str(), (*iter).second.orgname.c_str())){
|
||||
lastname = (*iter).second.orgname;
|
||||
result = true;
|
||||
}
|
||||
}else{
|
||||
if(0 > strcmp(lastname.c_str(), (*iter).second.normalname.c_str())){
|
||||
lastname = (*iter).second.normalname;
|
||||
result = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
bool S3ObjList::GetNameList(s3obj_list_t& list, bool OnlyNormalized, bool CutSlash) const
|
||||
{
|
||||
s3obj_t::const_iterator iter;
|
||||
@ -311,7 +332,7 @@ MVNODE *create_mvnode(const char *old_path, const char *new_path, bool is_dir, b
|
||||
return NULL;
|
||||
}
|
||||
|
||||
if(NULL == (p_new_path = strdup(new_path))){
|
||||
if(NULL == (p_new_path = strdup(new_path))){
|
||||
free(p);
|
||||
free(p_old_path);
|
||||
printf("create_mvnode: could not allocation memory for p_new_path\n");
|
||||
@ -329,7 +350,7 @@ MVNODE *create_mvnode(const char *old_path, const char *new_path, bool is_dir, b
|
||||
}
|
||||
|
||||
//
|
||||
// Add sorted MVNODE data(Ascending order)
|
||||
// Add sorted MVNODE data(Ascending order)
|
||||
//
|
||||
MVNODE *add_mvnode(MVNODE** head, MVNODE** tail, const char *old_path, const char *new_path, bool is_dir, bool normdir)
|
||||
{
|
||||
@ -670,9 +691,9 @@ mode_t get_mode(headers_t& meta, const char* path, bool checkdir, bool forcedir)
|
||||
}
|
||||
}
|
||||
// Checking the bitmask, if the last 3 bits are all zero then process as a regular
|
||||
// file type (S_IFDIR or S_IFREG), otherwise return mode unmodified so that S_IFIFO,
|
||||
// file type (S_IFDIR or S_IFREG), otherwise return mode unmodified so that S_IFIFO,
|
||||
// S_IFSOCK, S_IFCHR, S_IFLNK and S_IFBLK devices can be processed properly by fuse.
|
||||
if(!(mode & S_IFMT)){
|
||||
if(!(mode & S_IFMT)){
|
||||
if(!isS3sync){
|
||||
if(checkdir){
|
||||
if(forcedir){
|
||||
@ -752,7 +773,7 @@ time_t cvtIAMExpireStringToTime(const char* s)
|
||||
}
|
||||
memset(&tm, 0, sizeof(struct tm));
|
||||
strptime(s, "%Y-%m-%dT%H:%M:%S", &tm);
|
||||
return mktime(&tm); // GMT
|
||||
return timegm(&tm); // GMT
|
||||
}
|
||||
|
||||
time_t get_lastmodified(const char* s)
|
||||
@ -763,7 +784,7 @@ time_t get_lastmodified(const char* s)
|
||||
}
|
||||
memset(&tm, 0, sizeof(struct tm));
|
||||
strptime(s, "%a, %d %b %Y %H:%M:%S %Z", &tm);
|
||||
return mktime(&tm); // GMT
|
||||
return timegm(&tm); // GMT
|
||||
}
|
||||
|
||||
time_t get_lastmodified(headers_t& meta)
|
||||
@ -822,7 +843,7 @@ void show_usage (void)
|
||||
void show_help (void)
|
||||
{
|
||||
show_usage();
|
||||
printf(
|
||||
printf(
|
||||
"\n"
|
||||
"Mount an Amazon S3 bucket as a file system.\n"
|
||||
"\n"
|
||||
@ -854,7 +875,24 @@ void show_help (void)
|
||||
" - this option makes Amazon's Reduced Redundancy Storage enable.\n"
|
||||
"\n"
|
||||
" use_sse (default is disable)\n"
|
||||
" - this option makes Amazon's Server Site Encryption enable.\n"
|
||||
" - use Amazon<EFBFBD>fs Server-Site Encryption or Server-Side Encryption\n"
|
||||
" with Customer-Provided Encryption Keys.\n"
|
||||
" this option can not be specified with use_rrs. specifying only \n"
|
||||
" \"use_sse\" or \"use_sse=1\" enables Server-Side Encryption.\n"
|
||||
" (use_sse=1 for old version)\n"
|
||||
" specifying this option with file path which has some SSE-C\n"
|
||||
" secret key enables Server-Side Encryption with Customer-Provided\n"
|
||||
" Encryption Keys.(use_sse=file)\n"
|
||||
" the file must be 600 permission. the file can have some lines,\n"
|
||||
" each line is one SSE-C key. the first line in file is used as\n"
|
||||
" Customer-Provided Encryption Keys for uploading and changing\n"
|
||||
" headers etc.\n"
|
||||
" if there are some keys after first line, those are used\n"
|
||||
" downloading object which are encripted by not first key.\n"
|
||||
" so that, you can keep all SSE-C keys in file, that is SSE-C\n"
|
||||
" key history.\n"
|
||||
" if AWSSSECKEYS environment is set, you can set SSE-C key instead\n"
|
||||
" of this option.\n"
|
||||
"\n"
|
||||
" public_bucket (default=\"\" which means disabled)\n"
|
||||
" - anonymously mount a public bucket when set to 1\n"
|
||||
@ -889,7 +927,7 @@ void show_help (void)
|
||||
" readwrite_timeout (default=\"30\" seconds)\n"
|
||||
" - time to wait between read/write activity before giving up\n"
|
||||
"\n"
|
||||
" max_stat_cache_size (default=\"10000\" entries (about 4MB))\n"
|
||||
" max_stat_cache_size (default=\"1000\" entries (about 4MB))\n"
|
||||
" - maximum number of entries in the stat cache\n"
|
||||
"\n"
|
||||
" stat_cache_expire (default is no expire)\n"
|
||||
@ -966,6 +1004,11 @@ void show_help (void)
|
||||
" only rename command(ex. mv). If this option is specified with\n"
|
||||
" nocopapi, the s3fs ignores it.\n"
|
||||
"\n"
|
||||
" use_path_request_style (use legacy API calling style)\n"
|
||||
" Enble compatibility with S3-like APIs which do not support\n"
|
||||
" the virtual-host request style, by using the older path request\n"
|
||||
" style.\n"
|
||||
"\n"
|
||||
"FUSE/mount Options:\n"
|
||||
"\n"
|
||||
" Most of the generic mount options described in 'man mount' are\n"
|
||||
@ -997,11 +1040,20 @@ void show_help (void)
|
||||
void show_version(void)
|
||||
{
|
||||
printf(
|
||||
"Amazon Simple Storage Service File System %s\n"
|
||||
"Amazon Simple Storage Service File System V%s with %s\n"
|
||||
"Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>\n"
|
||||
"License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>\n"
|
||||
"This is free software: you are free to change and redistribute it.\n"
|
||||
"There is NO WARRANTY, to the extent permitted by law.\n", VERSION );
|
||||
"There is NO WARRANTY, to the extent permitted by law.\n",
|
||||
VERSION, s3fs_crypt_lib_name());
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_S3FS_UTIL_H_
|
||||
#define S3FS_S3FS_UTIL_H_
|
||||
|
||||
@ -51,6 +70,7 @@ class S3ObjList
|
||||
std::string GetETag(const char* name) const;
|
||||
bool IsDir(const char* name) const;
|
||||
bool GetNameList(s3obj_list_t& list, bool OnlyNormalized = true, bool CutSlash = true) const;
|
||||
bool GetLastName(std::string& lastname) const;
|
||||
|
||||
static bool MakeHierarchizedList(s3obj_list_t& list, bool haveSlash);
|
||||
};
|
||||
@ -116,3 +136,12 @@ void show_help(void);
|
||||
void show_version(void);
|
||||
|
||||
#endif // S3FS_S3FS_UTIL_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
@ -130,7 +130,9 @@ string urlEncode(const string &s)
|
||||
} else if (s[i] == '.' || s[i] == '-' || s[i] == '*' || s[i] == '_') {
|
||||
result += s[i];
|
||||
} else if (s[i] == ' ') {
|
||||
result += '+';
|
||||
result += '%';
|
||||
result += '2';
|
||||
result += '0';
|
||||
} else {
|
||||
result += "%";
|
||||
result += hexAlphabet[static_cast<unsigned char>(s[i]) / 16];
|
||||
@ -184,8 +186,18 @@ string prepare_url(const char* url)
|
||||
uri_length = 8;
|
||||
}
|
||||
uri = url_str.substr(0, uri_length);
|
||||
host = bucket + "." + url_str.substr(uri_length, bucket_pos - uri_length).c_str();
|
||||
path = url_str.substr((bucket_pos + bucket_length));
|
||||
|
||||
if(!pathrequeststyle){
|
||||
host = bucket + "." + url_str.substr(uri_length, bucket_pos - uri_length).c_str();
|
||||
path = url_str.substr((bucket_pos + bucket_length));
|
||||
}else{
|
||||
host = url_str.substr(uri_length, bucket_pos - uri_length).c_str();
|
||||
string part = url_str.substr((bucket_pos + bucket_length));
|
||||
if('/' != part[0]){
|
||||
part = "/" + part;
|
||||
}
|
||||
path = "/" + bucket + part;
|
||||
}
|
||||
|
||||
url_str = uri + host + path;
|
||||
|
||||
@ -206,3 +218,11 @@ string get_date()
|
||||
return buf;
|
||||
}
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* s3fs - FUSE-based file system backed by Amazon S3
|
||||
*
|
||||
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU General Public License
|
||||
* as published by the Free Software Foundation; either version 2
|
||||
* of the License, or (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
*/
|
||||
#ifndef S3FS_STRING_UTIL_H_
|
||||
#define S3FS_STRING_UTIL_H_
|
||||
|
||||
@ -32,3 +51,12 @@ std::string prepare_url(const char* url);
|
||||
bool get_keyword_value(std::string& target, const char* keyword, std::string& value);
|
||||
|
||||
#endif // S3FS_STRING_UTIL_H_
|
||||
|
||||
/*
|
||||
* Local variables:
|
||||
* tab-width: 4
|
||||
* c-basic-offset: 4
|
||||
* End:
|
||||
* vim600: noet sw=4 ts=4 fdm=marker
|
||||
* vim<600: noet sw=4 ts=4
|
||||
*/
|
||||
|
||||
@ -1,3 +1,22 @@
|
||||
######################################################################
|
||||
# s3fs - FUSE-based file system backed by Amazon S3
|
||||
#
|
||||
# Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
######################################################################
|
||||
TESTS=small-integration-test.sh
|
||||
|
||||
EXTRA_DIST = \
|
||||
|
||||
Reference in New Issue
Block a user