15e89b78de
Add a partial page_list unit test ( #1735 )
2021-08-04 07:36:32 +09:00
66006ba48d
Add dedicated upper- and lower-case hex functions ( #1734 )
...
This makes the call sites more readable than a boolean parameter.
2021-08-04 07:28:51 +09:00
18e9c62087
Make string constants read-only const ( #1733 )
...
This removes some global constructors. Also use a consistent ALL_CAPS
style.
2021-08-03 00:10:27 +09:00
34f89e5936
Added fake_diskfree option to deceive free disk space for test
2021-08-02 11:04:42 +09:00
e1f3b9d8c1
Add support for AWS-style environment variables ( #1729 )
...
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.
The old environment variables are still supported, but they are
deprecated and no longer documented.
Close #1708
2021-07-26 23:29:45 +09:00
d3278f4886
Loosen CheckBucket to check only the bucket ( #1728 )
...
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460 . Fixes #1687 .
2021-07-25 12:29:00 +09:00
77f0b75d2f
Add jitter to avoid thundering herd ( #1723 )
2021-07-25 10:22:19 +09:00
012e6dd8a2
Fix README nit ( #1726 )
2021-07-22 13:54:20 +09:00
199b3d4709
Fixed a bug in disk free space calculation
2021-07-18 14:52:11 +09:00
7890989cbb
Address cppcheck 2.5 warnings ( #1720 )
...
This slightly simplifies memory management.
2021-07-14 22:18:09 +09:00
945cc2ac54
Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo
2021-07-12 21:45:29 +09:00
c30acbbf90
Splited some methods of FdEntity class by uploading mode
...
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
2021-07-10 17:24:34 +09:00
881025cc9e
Add initiator pid, uid and gid to a debug log ( #1716 )
2021-07-10 16:23:20 +09:00
4cc210c5ab
Prevents the etag string buffer from being destroyed during use
...
and absorbed MPPART_INFO structure into filepart structure
2021-07-03 22:42:18 +09:00
2d5316a334
Fixed 1710 - Added a flag for check in Linux
2021-07-02 11:34:51 +09:00
9cfa177af0
Add flag to run all tests ( #1710 )
...
This allows CI to run the full set of tests while making local
development faster.
2021-07-02 08:41:47 +09:00
fe44355d25
Report s3fs space as 18.4 EB ( #1709 )
...
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
2021-07-02 07:56:05 +09:00
487df27008
Return EREMOTE when reading Glacier objects ( #1705 )
...
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466 .
2021-06-30 09:25:36 +09:00
1965916f7a
Convert storage class values to uppercase ( #1704 )
...
AWS requires uppercase values. Fixes a regression from
bbcccd6e98#1613 .
2021-06-30 09:03:31 +09:00
8948eded09
Address clang-tidy warnings ( #1703 )
2021-06-30 08:42:44 +09:00
2f59cb5a0a
Remove unneeded volatile qualifiers ( #1702 )
...
We should use proper locking instead. Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
2021-06-30 08:20:44 +09:00
f505c8224e
Explicitly specify lock type ( #1701 )
...
This makes it more clear and type-safe if the caller already has the
lock. Follows on to 84174c560d
2021-06-27 15:15:48 +09:00
2154e898bc
Fix typos ( #1700 )
2021-06-27 11:22:33 +09:00
f9e80f995d
Fixed a bug about rename existing file of different sizes by mpcopy
2021-06-26 00:04:58 +09:00
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating ( #1681 )
...
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/ .
Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com >
2021-06-25 20:26:12 +09:00
6edf3d6427
Updated to clearly output message about the file descriptor type(pseudo/physical)
2021-06-25 19:18:08 +09:00
944c1fd148
Clean up litter in ut_test ( #1697 )
2021-06-25 19:15:03 +09:00
20281737b1
Set explicit Content-Length: 0 when initiating MPU ( #1696 )
...
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661 .
2021-06-25 18:36:40 +09:00
0555c4216e
Added test case for utimens is calling during multipart upload
2021-06-21 17:20:24 +09:00
5b6684ca19
Fixed a bug utimens is calling before flush
2021-06-21 17:20:24 +09:00
c4ac923b4c
Ensure NUL-terminated result after strncpy ( #1694 )
...
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
2021-06-21 08:08:56 +09:00
cc022a68f4
Fixed a bug in s3fs_write
2021-06-20 23:13:44 +09:00
1571379304
Set LC_ALL=en_US.UTF-8 in tests ( #1691 )
...
This allows tests to pass using the S3Proxy filesystem provider.
References #1665 .
2021-06-20 19:27:25 +09:00
858562ed53
Fixed comparison warning for size_t variable
2021-06-20 16:53:25 +09:00
b8724425d3
Require explicit base parameter in cvt_strtoofft ( #1683 )
...
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682 .
2021-06-20 11:00:15 +09:00
b9ce0faee2
Set Content-MD5 header even for empty objects
...
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
2021-06-19 16:10:58 +09:00
1e0e2752bf
Improve warning for terminated threads
...
with non-zero return code.
2021-06-18 07:54:38 +09:00
7a488b93d0
Set decimal base for converting subseconds to int
...
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.
Setting the base to 10 prevents this automatism and treats the value
as a decimal value.
Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.
Related to #1676
2021-06-17 06:27:02 +09:00
d67b83e671
Allow configuration for temporary files directory
2021-06-16 21:29:58 +09:00
a100be9dce
Prefer std::string::empty over length == 0 ( #1679 )
...
This is more concise.
2021-06-13 20:03:10 +09:00
4d39ea887e
Protect orgmeta with fdent_lock ( #1678 )
...
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer. Regression introduced by
ac578d188eFixes #1677 .
2021-06-13 16:14:24 +09:00
7638b5b3e3
Prefer std::string::rbegin over operator[] ( #1673 )
...
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
2021-06-13 13:26:38 +09:00
600cee118d
Fix implicit narrowing conversions ( #1672 )
...
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
2021-06-13 12:50:07 +09:00
c2c56d0263
Added info object about multipart uploading for each pseudo fd
...
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
ac578d188e
Introduced pseudo fd and separated fd for each file opening
2021-06-04 22:42:58 +09:00
53dfd48f59
Correct usage of istringstream ( #1670 )
...
Previously this looped one more time than necessary due to the eof
check:
https://isocpp.org/wiki/faq/input-output#istream-and-eof
Remove now redundant empty check.
2021-06-02 07:14:32 +09:00
f5701fa9ad
Correct typo in undefinedsanitizer config ( #1669 )
2021-05-29 22:07:33 +09:00
84174c560d
Fix data races caused by incorrect locking ( #1668 )
...
Found via Threadsanitizer. Fixes #1471 .
2021-05-29 00:11:55 +09:00
9bf525ee7a
Ensuring multipart size even when storage is low
...
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.
The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.
Fixes #1591
2021-05-27 22:55:52 +09:00
4b69d4b1bb
Fixed a bug when the disk capacity was insufficient in RowFlush
2021-05-22 23:11:46 +09:00
