root
/
sftp
mirror of https://github.com/pkg/sftp.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity
908 Commits 14 Branches 33 Tags 3.9 MiB
Commit Graph

908 Commits

Author SHA1 Message Date
Nicola Murino 3f969fcd59 add optional AllocationModeOptimized 
after processing a packet we keep in memory the allocated slices and we reuse
them for new packets.
Slices are allocated in:

- recvPacket
- when we receive a sshFxpReadPacket (downloads)

The allocated slices have a fixed size = maxMsgLength.

Allocated slices are referenced to the request order id and are marked for reuse
after a request is served in maybeSendPackets.

The allocator is added to the packetManager struct and it is cleaned at the end
of the Serve() function.

This allocation mode is optional and disabled by default
 2020-03-14 19:42:19 +01:00
John Eikenberry 18dc4db7a4
Merge pull request #343 from drakkan/allocations 
fileget: allocate a slice with enough capacity
 2020-03-10 15:48:33 -07:00
Nicola Murino 0a45bc4b6d sshFxpDataPacket: document the required capacity to avoid a new allocation 2020-03-10 22:56:41 +01:00
Nicola Murino 0f0e40a3a6 minor changes as requested in the review 2020-03-10 15:35:56 +01:00
Nicola Murino 3ea9f24c15 server: remove now unused maxTxPacket 
maxTxPacket was only used to get the size for the read packet it is not
needed anymore
 2020-03-10 11:48:52 +01:00
Nicola Murino eeafeeff60 FxpReadPacket: add an helper method for slice allocation 
This way we can use the same method in both server and request-server
 2020-03-10 11:46:46 +01:00
Nicola Murino 44e44d716f fileget: allocate a slice with enough capacity 
so a new allocation is not needed in MarshalBinary and sendPacket.

Here are some profiling results while downloading a file (file size is about 1GB),

before this patch:

1254.24MB 55.18% 55.18%  1254.24MB 55.18%  github.com/pkg/sftp.sshFxpDataPacket.MarshalBinary
  991.81MB 43.63% 98.81%   991.81MB 43.63%  github.com/pkg/sftp.fileget
       1MB 0.044% 98.86%  1255.24MB 55.22%  github.com/pkg/sftp.(*packetManager).maybeSendPackets
    0.50MB 0.022% 98.88%  1260.24MB 55.44%  github.com/pkg/sftp.(*packetManager).controller
         0     0% 98.88%   991.81MB 43.63%  github.com/pkg/sftp.(*Request).call
         0     0% 98.88%   993.31MB 43.70%  github.com/pkg/sftp.(*RequestServer).Serve.func1.1
         0     0% 98.88%   993.31MB 43.70%  github.com/pkg/sftp.(*RequestServer).packetWorker
         0     0% 98.88%  1254.24MB 55.18%  github.com/pkg/sftp.(*conn).sendPacket
         0     0% 98.88%  1254.24MB 55.18%  github.com/pkg/sftp.sendPacket

with this patch:

1209.48MB 98.46% 98.46%  1209.48MB 98.46%  github.com/pkg/sftp.fileget
       2MB  0.16% 98.63%     7.50MB  0.61%  github.com/pkg/sftp.recvPacket
         0     0% 98.63%        8MB  0.65%  github.com/drakkan/sftpgo/sftpd.Configuration.handleSftpConnection
         0     0% 98.63%  1209.48MB 98.46%  github.com/pkg/sftp.(*Request).call
         0     0% 98.63%        8MB  0.65%  github.com/pkg/sftp.(*RequestServer).Serve
         0     0% 98.63%  1209.98MB 98.50%  github.com/pkg/sftp.(*RequestServer).Serve.func1.1
         0     0% 98.63%  1209.98MB 98.50%  github.com/pkg/sftp.(*RequestServer).packetWorker
         0     0% 98.63%     7.50MB  0.61%  github.com/pkg/sftp.(*conn).recvPacket (inline)
 2020-03-09 19:22:48 +01:00
John Eikenberry 0e1232311c
Merge pull request #342 from pkg/issue-333-conn-close-race 
fix connection close race
 2020-03-08 14:49:16 -07:00
John Eikenberry c31b3c3b22 fix connection close race 
Need to prevent Close from being called at the same time. The lock
already existed, but was only used in one place where it should be
applied generally.
 2020-03-07 15:05:46 -08:00
John Eikenberry 8f77623878
Merge pull request #338 from drakkan/copy 
data packet unmarshal binary: remove uneeded copy
 2020-03-01 15:36:14 -08:00
Cassondra Foesch 21ab0e5222
Merge pull request #340 from twmb/twmb/update_ssh 
bump ssh dep to fix vuln
 2020-02-27 08:00:34 +00:00
Nicola Murino 01ec2e2e14 write packet UnmarshalBinary: reuse already allocated data 2020-02-26 12:04:13 +01:00
Travis Bischel 26f1844016 bump ssh dep to fix vuln 
https://groups.google.com/forum/#!topic/golang-dev/bhd3Qn9_Q1k
 2020-02-25 12:06:54 -07:00
Nicola Murino 4eda1f42bb data packet unmarshal binary: remove uneeded copy 2020-02-16 21:52:13 +01:00
John Eikenberry f1f62a1b95
Merge pull request #337 from drakkan/servertest 
server integration test: add /usr/lib/ssh/sftp-server to search paths
 2020-02-16 00:12:50 +00:00
Nicola Murino 07905cb843 server integration test: add /usr/lib/ssh/sftp-server to search paths 
os some distro, for example ArchLinux, the sftp-server CLI is in this
path
 2020-02-14 17:27:18 +01:00
John Eikenberry 5616182052
Merge pull request #332 from peachfinance/Fstat-stat-options 
Added a ClientOption to determine whetehr to use Fstat or Stat when File.WriteTo is being called to support strange behaviour on some servers
 2020-01-11 00:44:58 +00:00
John Eikenberry a5f5a4cb21
Merge pull request #330 from pkg/issue-295-corrupt-resumes 
Issue 295 corrupt resumes
 2020-01-11 00:44:14 +00:00
John Eikenberry e6f382575a
Merge pull request #329 from pkg/issue-291-ifmt-platform-fix 
fix S_IFMT value on windows, plan9 and js/wasm
 2020-01-11 00:43:47 +00:00
Eran Sandler 77a4076c35 fixes based on comments + minor test added 2020-01-06 14:54:28 -08:00
John Eikenberry 1bc3ea14d1 fix issue with file put resume/append 
Fixes issue with append uploads. Was opening the file with O_APPEND, but
it uses WriteAt() to write the data which doesn't work with a file
opened in append mode. Removing the append flag fixes the issue as the
client is sending the offsets anyways.

Also added a note to the Request server's FileWriter interface on
handling append flags.
 2020-01-05 14:23:49 -08:00
John Eikenberry 3c6b3a4ff1 add test to exercise file upload resume 
Simulates resuming a file upload where more data is appended to the same
file (eg. network error, uploading log files, etc). Currently broken in
the server as it uses O_APPEND + WriteAt() which don't work together.
 2020-01-03 12:44:31 -08:00
Eran Sandler 09f03e7bff Added a ClientOption to determine whetehr to use Fstat or Stat when File.WriteTo is being called to support strange behaviour on some servers 2020-01-03 11:29:41 -08:00
John Eikenberry f3d13089ef add makefile to simplify running integration tests 2020-01-02 14:49:23 -08:00
John Eikenberry 0277caa990 fix S_IFMT value on windows and plan9 
Go defines S_IFMT on windows, plan9 and js/wasm as 0x1f000 instead of
0xf000. None of the the other S_IFxyz values (even on those mentioned)
include the "1" (in 0x1f000) which prevents them from matching the
bitmask.

This fixes that by overriding the S_IFMT value on the effected platforms
to be 0xf000, as it it on all others.
 2020-01-02 12:59:24 -08:00
John Eikenberry 939a6934bc
Merge pull request #328 from vansante/fix-empty-attributes-panicking 
Fix panics when attempting to get file attributes from too small byte slices
 2019-12-30 12:17:42 -08:00
Paul van Santen f94839fd59
Add const for all attribute flags and add missing newline 2019-12-27 10:56:42 +01:00
Paul van Santen 1bb2f2849b
Fix panics when attempting to get file attributes from too small byte slice 2019-12-24 09:37:32 +01:00
John Eikenberry 8488d36ede
Merge pull request #315 from drakkan/extensions 
make SFTP extensions configurable

Thanks @drakkan
 2019-11-11 15:44:05 -08:00
Nicola Murino 429ee173fb make SFTP extensions configurable 2019-11-11 18:04:56 +01:00
John Eikenberry 6d50bf4a21
Merge pull request #316 from drakkan/posixrename 
Fix posixrename and add a test case
 2019-11-02 14:07:27 -07:00
John Eikenberry 4033788e3d
Merge pull request #317 from pkg/revert-313-master 
Revert "Bug: windows specific issue on request-server.go"
 2019-11-02 14:05:18 -07:00
John Eikenberry 5a3a7572f5
Revert "Bug: windows specific issue on request-server.go" 2019-11-02 21:04:45 +00:00
John Eikenberry 73241350c5
Merge pull request #313 from danfletcher1/master 
Bug: windows specific issue on request-server.go
 2019-11-02 14:03:59 -07:00
Nicola Murino f9e61c9b8f request-server: add test case for posixrename 2019-10-28 10:34:49 +01:00
Daniel Fletcher 5df8e92923 request-server: fix posixrename 2019-10-27 06:59:47 +01:00
Daniel Fletcher 313f3c181d Bug: InMemoryHandler() mimics a linux file system. When build for 
windows filepath.Dir(), more specifically the clean function preforms
incorrectly in our situation. As the filesystem is constant we can do a
simplified dir() function to resolve the issue.
 2019-10-23 10:45:52 +01:00
Daniel Fletcher 26d6af7a59 Bug: Rename function wasn't working for me, the type was not being 
selected. I am unsure if this is the final solution as it will undo a
previous update that may resolve a differen issue.
 2019-10-23 10:42:02 +01:00
John Eikenberry 515578a967
Merge pull request #311 from danfletcher1/master 
Bug with sub folders/files of a renamed/removed path
 2019-10-21 22:40:01 -07:00
Daniel Fletcher e9514f4967 Removed some unnecessary logging, and previous merge duplicated some code 2019-10-21 12:43:45 +01:00
Daniel Fletcher 4adf56b4e4 Merge branch 'master' of https://github.com/danfletcher1/sftp 2019-10-21 11:54:57 +01:00
Daniel Fletcher 67f906c100 When removing or renaming a folder, subfiles/folders of the path was not being updated 2019-10-21 11:52:48 +01:00
John Eikenberry 4350932b98
Merge pull request #312 from LeeEirc/bugfix_request 
fix nil panic error
 2019-10-13 20:02:35 -07:00
John Eikenberry 463bb932d3
Merge pull request #310 from pkg/issue-305-client-write 
Issue 305 client write

Fixes #305
 2019-10-13 19:30:09 -07:00
John Eikenberry c732bf9952 add method to get fxerr value for StatusError code 
Need to get StatusError.Code (uint32) converted to fxerr type so you can
compare it to public (exported) ErrSSHFx[..] error types. This adds a
new method, FxCode(), to the StatusError objects to handle this case.

Did this vs trying to change types of existing fields to not break
backwards compatibility.
 2019-10-13 19:18:35 -07:00
John Eikenberry 8067637962 add note in Create() docs about AWS + O_RDWR 2019-10-13 19:18:30 -07:00
Eric 3f23d5935f fix nil panic error when sshFxpExtendedPacket could not UnmarshalBinary ExtendedRequest 2019-10-11 14:31:08 +08:00
Daniel Fletcher b4e1be7c5d When removing or renaming a folder, subfiles/folders of the path was not being updated 2019-10-09 16:40:31 +01:00
John Eikenberry 84e6527392
Merge pull request #309 from wutz/master 
fix: received packet too long
 2019-09-29 14:37:50 -07:00
John Eikenberry 0852dbaaaf update go versions in travis 2019-09-29 14:00:25 -07:00