Commit Graph

194 Commits

Author SHA1 Message Date
4b6e53223b Use std::shared_ptr to refer to FdEntity (#2541)
FdEntity may have multiple references due to ChangeEntityToTempPath.
This relies on the std::enable_shared_from_this helper to create a
std::shared_ptr from this.  Fixes #2532.
2024-10-20 14:56:29 +09:00
a505cebf9b Expand use of std::unique_ptr for FILE* (#2555) 2024-10-18 22:06:47 +09:00
141d74f187 Use auto for iterator variable types (#2554)
This touches a few other long type names.  Applied via clang-tidy
-fix.
2024-10-18 21:57:52 +09:00
5594106351 Add miscellaneous locking annotations (#2551) 2024-10-18 00:34:57 +09:00
4605cc2035 Fixed fake_diskfree option 2024-10-07 01:24:35 +09:00
1a50b9a04a Fixed a deadlock in the FdManager::ChangeEntityToTempPath method call 2024-06-30 20:29:01 +05:30
622dc0a815 Convert pthread_mutex to std::mutex (#2476)
This simplifies resource management and improve Windows compatibility.
2024-06-24 00:48:01 +09:00
2841601ad5 Remove uses of AutoLock::ALREADY_LOCKED (#2466)
Instead annotate the methods with REQUIRES so that the caller knows if
they should lock.  For public interfaces, introduce HasLock wrappers.
This simplifies control flow, allows migration to std::mutex, and
eventually will enable use of static lock checking.
2024-06-23 11:24:51 +09:00
ebae5a302f Prefer std::string::clear where possible (#2467)
This is somewhat more clear and is declared noexcept.
2024-06-10 23:36:49 +09:00
a4f694c345 Pass const std::string by reference (#2461)
This is more idiomatic than by pointer.
2024-05-28 09:17:45 +09:00
ccdcccd44c Fix DeadLock in FdManager::ChangeEntityToTempPath (#2455)
commit e3b50ad introduce smart pointer to manage FdEntity

But in ChangeEntityToTempPath, we should not destroy the entity.

We should move the entry to the temp ky

Signed-off-by: liubingrun <liubr1@chinatelecom.cn>
2024-05-11 11:29:40 +09:00
54aa278df0 Fixed errors reported by cppcheck 2.13.0 (#2400) 2024-01-25 00:46:45 +09:00
e3b50ad3e1 Convert FdEntity to std::unique_ptr (#2383) 2023-12-07 23:56:35 +09:00
4845831f93 Convert some const to constexpr (#2342)
This guarantees that the function or value will resolve at compile-time.
2023-11-14 22:15:17 +09:00
3856637cd2 s3fs: add option free_space_ratio to control cache size (#2351)
* Try to cleanup cache directory when initing without enough disk space

Also optimize log messages to print detailed errors to the user.

Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>

* s3fs: add option free_space_ratio to control cache size

Since the ensure_diskfree option is not convenient enough, we have added
a new option "-o free_space_ratio" to control the space used by the s3fs
cache based on the current disk size.

The value of this option can be between 0 and 100. It will control the
size of the cache according to this ratio to ensure that the idle ratio
of the disk is greater than this value.

For example, when the value is 10 and the disk space is 50GB, it will
ensure that the disk will reserve at least 50GB * 10% = 5GB of remaining
space.

Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>

---------

Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
2023-10-20 18:11:47 +09:00
361e10d09c Add scope_guard for ad-hoc resource management (#2313)
References #2261.  Suggested by:
https://stackoverflow.com/questions/10270328/the-simplest-and-neatest-c11-scopeguard
2023-09-26 07:52:55 +09:00
61df7bf42c Use std::unique_ptr for fclose (#2318)
References #2261.
2023-09-25 23:55:11 +09:00
7bb9609827 Return errors from AutoFdEntity::Open (#2296)
Found via pjdfstest which creates a PATH_MAX path that should return
NAMETOOLONG.
2023-08-29 22:57:30 +09:00
b2bb12fd2c Remove unneeded explicit std::string constructors (#2273)
std::string(const char*) implicitly constructs these.  The remaining call sites
requires string literals from C++14.
2023-08-17 22:12:28 +09:00
e157d811cb Use std::string::compare and operator== where possible (#2256) 2023-08-15 21:22:36 +09:00
5b93765802 Fixed a warning for compiling C/C++ codes 2023-08-13 20:49:39 +09:00
36db898d01 Use C++11 std::map::erase return value (#2236) 2023-07-27 23:34:43 +09:00
a4a2841c05 Use C++11 nullptr instead of 0 or NULL (#2234)
This improves type-safety.
2023-07-27 21:56:58 +09:00
faec0d9d15 Refixed for cppcheck 2.1x 2023-07-26 07:55:33 +09:00
cb3dc28e6e Supported cppcheck 2.10(and changed std from c++03 to c++11 for RHEL7) 2023-07-25 08:08:06 +09:00
2e51908bec Check FdEntity::Open() status correctly
FdEntity::Open() returns -errno on error, but FdManager::Open() only
checks if its ret is -1. This may lead to use '-errno' as pseudo fd
in next read or write, which would fail due to '-errno' is not in
fent map.

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-07-05 08:50:12 +09:00
3b6688253f Address cppcheck 2.10 warnings (#2163)
Disable newer cppcheck until we can diagnose this further.
References #2162.
2023-06-25 16:04:16 +09:00
c491fbeabc Replace uses of lock_already_held flag with AutoLock::Type 2022-07-30 16:20:43 +09:00
48e9e51f4f Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
4b2f3fecb5 Set mtime/ctime/atime of all objects as nanosecond 2022-07-28 13:47:03 +09:00
d75c6d6538 Changed the message level(ERR to INFO) of skip cleanup cache (#1946) 2022-05-22 14:36:56 +09:00
d842d45b2b Fixed a bug about truncation for shrinking file 2022-03-02 22:41:10 +09:00
839a33de49 Fixed not to call Flush even if the file size is increased (#1887)
Changed s3fs_truncate function.
This change reduces the number of file uploads if the file size is changed.

On macOS, I have found that the truncate call when "size=0" cannot reflect the file size.(This reason is not understood...)
To avoid this, only when "size=0", the flush method is called as before.

Other than that, I found a bug in FdEntity::Open() and fixed it.

Fixes #1875.
2022-02-15 21:29:07 +09:00
4dfe2bfdd7 Include climits to support musl libc
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.

fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
  381 |     char cfn[PATH_MAX];
      |              ^~~~~~~~

Fixes: d67b83e671 ("Allow configuration for temporary files directory")
2022-02-14 09:19:30 +09:00
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
beecf32dff fclose(FILE*) instead of close(fileno(FILE*)) (#1758)
This is the same thing but confuses Coverity.
2021-09-01 19:41:55 +09:00
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
f505c8224e Explicitly specify lock type (#1701)
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
2021-06-27 15:15:48 +09:00
2154e898bc Fix typos (#1700) 2021-06-27 11:22:33 +09:00
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
c2c56d0263 Added info object about multipart uploading for each pseudo fd
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
84174c560d Fix data races caused by incorrect locking (#1668)
Found via Threadsanitizer.  Fixes #1471.
2021-05-29 00:11:55 +09:00
bb6d2b1b74 Replace snprintf with string and ostringstream (#1649)
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
2021-05-08 02:48:47 +09:00
8a51a26819 Allow truncation of open and modified files
Regression introduced in f5bf41cf11.
Fixes #1575.
2021-04-25 12:35:35 +09:00
4c6690f5f0 Add timestamps to debug log and cache integrity check (#1540)
* Add UTC timestamp to debug output

for easier correlation with external events

Old output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF]     s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...

New output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF]     s3fs.cpp:set_mountpoint_attrib...

* Add UTC timestamp to cache integrity check

for easier correlation with external events.

$ s3fs mybucket /mybucket  -oset_check_cache_sigusr1=/tmp/check.cache

Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
------------------------------------------------------------

New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
---------------------------------------------------------------------------

* Fix indentation of S3fsLog::GetCurrentTime()
2021-02-04 10:41:29 +09:00
b0e8758b63 Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
05863a3178 Fix clang-tidy warnings 2020-09-21 07:03:42 +09:00
3958450c05 Added a class for automating fdentity reference counts 2020-09-14 22:02:28 +09:00