youngkingdom/s3fs-fuse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,705 Commits 1 Branch 23 Tags
20a6c9d35d6ced8d7365d7c3c39a5fd04cd894e2
Commit Graph

1705 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Gaul
20a6c9d35d Update to S3Proxy 1.8.0 (#1598) 
Notably this fixes an issue with the filesystem backend which returned
the incorrect Content-Type for some directories.  It also reduces
memory use with GetObject using range requests with large objects.
References #1543.

Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.8.0
 2021-03-28 11:56:10 +09:00
Morteza Ghasempour
0a90a40569 remove duplicate line from README.md 2021-03-27 19:03:15 +09:00
Carsten Grohmann
ef079f4e94 Don't ignore nomultipart when storage is low 
When the temporary storage filled up, the old implementation uploaded
all data with multipart uploads, even if "nomultipart" was set.

The new implementation emits a warning and returns -ENOSPC instead.

Fixes #1595
 2021-03-10 08:23:54 +09:00
Takeshi Nakatani
b589ebec23 Output version and command parameters at startup 2021-03-06 19:11:57 +09:00
Andrew Gaul
8c58ba8ac0 Update ChangeLog and configure.ac for 1.89 (#1584) 
Also make formatting more consistent.  Fixes #1588.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
v1.89 		2021-02-24 08:03:55 +09:00
Takeshi Nakatani
a19d223434 Bypass an OSX test(no different from other OS) which takes very time consuming 2021-02-23 20:47:51 +09:00
Takeshi Nakatani
fc06419549 Fixed aws command error by rsa on ubuntu16.04 (#1585) 2021-02-23 11:41:52 +09:00
Andrew Gaul
032fcf2a47 Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
4fc92d59f3 Fixed a bug that filling NULL bytes when changing the attribute after renaming 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Carsten Grohmann
5debf523b0 Allow changing temporary directory used for tests 
Set temporary directory to $TMPDIR if set or use default /var/tmp
 2021-02-19 13:59:23 +09:00
Carsten Grohmann
1c8aadafd1 Fix typo in an error message of the test suite 2021-02-19 13:59:23 +09:00
Takeshi Nakatani
d33f252404 Fixed forgetting to clean up test files 2021-02-14 00:07:46 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Takeshi Nakatani
493802a605 Fixed osx mistype declaration and typo warnings 2021-02-13 12:04:35 +09:00
林千里
22b0ae9d51 set IsExpireTime to true by default 
fixes #1563
 2021-02-12 12:20:47 +09:00
Andrew Gaul
134a54b32f Allow configuring the multipart threshold (#1562) 
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 23:35:46 +09:00
Andrew Gaul
7f6fbb0021 Correct singlepart_copy_limit unit (#1561) 
This is documented to be in MB not KB.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 22:53:50 +09:00
Andrew Gaul
ec8bae9827 Do not periodically flush with nomultipartupload (#1559) 
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean.  Without multipart copy, s3fs would write
the now-zeroed data along with any new data.  s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support.  Fixes #1542.
 2021-02-11 22:24:39 +09:00
Andrew Gaul
a4d916af13 Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Takeshi Nakatani
32f096fa3f Use clock_gettime instead of gettimeofday 2021-02-08 10:19:28 +09:00
Andrew Gaul
c692093921 Copy at most 5 GB per multipart copy part request (#1553) 
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
 2021-02-07 23:10:07 +09:00
Andrew Gaul
1a6d0826b5 Allow -1 value to disable max_dirty_data (#1552) 
This matches the man page.
 2021-02-07 22:14:31 +09:00
Andrew Gaul
f2c5e38724 Allow integration tests to use larger files (#1548) 
Previously these failed due to dd only copying 32 MB from /dev/urandom
and exhausting the limited space in /tmp.  References #1543.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:50:02 +09:00
Andrew Gaul
bf33fe7f55 Increase buffer size to avoid warning (#1550) 
This can be up to 54 bytes.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:26:58 +09:00
Andrew Gaul
ff9d6a75c7 Parse more specific AWS error codes (#1549) 
Fixes #1545.
 2021-02-07 11:29:08 +09:00
Carsten Grohmann
4c6690f5f0 Add timestamps to debug log and cache integrity check (#1540) 
* Add UTC timestamp to debug output

for easier correlation with external events

Old output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF]     s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...

New output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF]     s3fs.cpp:set_mountpoint_attrib...

* Add UTC timestamp to cache integrity check

for easier correlation with external events.

$ s3fs mybucket /mybucket  -oset_check_cache_sigusr1=/tmp/check.cache

Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
------------------------------------------------------------

New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
---------------------------------------------------------------------------

* Fix indentation of S3fsLog::GetCurrentTime()
 2021-02-04 10:41:29 +09:00
Andrew Gaul
67b9381825 Remove limit for single-part objects 
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects.  Fixes #1542.
 2021-02-04 09:19:37 +09:00
Andrew Gaul
4b53d4bf6b Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Carsten Grohmann
ed85b72bf5 Add warning for failing metadata updates of large files 
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.

This change is related to #1528
 2021-01-28 10:20:32 +09:00
Andrew Gaul
17fda89ae9 Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) 
Follows on to 4d833a4fb9.
 2021-01-25 19:08:14 +09:00
Andrew Gaul
1987bcbea3 Re-fix retry logic (#1533) 
This logic has always been broken but happened to try one time due to
&& operator precedence.  7158e50ee2
broke this further when quoting && since the command was not
evaluated.
 2021-01-25 18:32:40 +09:00
Andrew Gaul
d019dda4f7 Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63 Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Takeshi Nakatani
6d65e30dd5 Fixed brew install with cask in github actions workflow 2021-01-21 17:34:03 +09:00
Andrew Gaul
b70f8db037 Propagate errno from get_local_fent (#1525) 
Follows on to 4d833a4fb9.
References #1523.
 2021-01-20 23:21:14 +09:00
Andrew Gaul
4d833a4fb9 Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
Andrew Gaul
a6563211af Update ChangeLog and configure.ac for 1.88 v1.88 2021-01-11 17:59:28 +09:00
Andrew Gaul
8d66b0e4a8 Updated ChangeLog for release 1.88 (#1514) 
References #1511.
 2021-01-05 16:46:41 +09:00
Takeshi Nakatani
555c1dde3d Use Gtihub Actions instead of Travis CI 2021-01-04 23:46:54 +09:00
Andrew Gaul
b04bca37a5 Add configuration for Chaos HTTP Proxy (#1508) 
This can find errors in retry logic.  Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:

CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test

It can also be helpful to increase retries and reduce sleep times.
References #1504.
 2021-01-04 23:32:04 +09:00
fly3366
168e588ac7 fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Andrew Gaul
7158e50ee2 Quote | and && to allow passing it to retry (#1506) 2021-01-04 22:28:40 +09:00
Andrew Gaul
bd0fadbe5f Remove authorization header when remaking handle (#1505) 
This avoids including Authorization in SignedHeaders.  s3fs will
recreate the Authorization header before sending the request.
 2021-01-04 21:37:34 +09:00
kontrollanten
d1c638ab7a fix 404 error message 2020-12-30 14:10:17 +09:00
fly3366
51f65d7b14 fix: miss header when retry 2020-12-23 22:16:07 +09:00
Andrew Gaul
a16d00d673 Simply curl_slist_sort_insert (#1494) 
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
 2020-12-23 20:29:33 +09:00