82ef5086ee
server integration test scaffolding
2015-08-04 23:37:18 -07:00
348ee1a469
truncate + darwin VFS test
2015-08-04 22:21:35 -07:00
2f3ef8b897
Merge branch 'master' into server
2015-08-04 21:11:01 -07:00
615df6108a
Merge branch 'master' of https://github.com/pkg/sftp
2015-08-04 21:00:34 -07:00
f4c4138a0e
update client integration tests for more coverage
2015-08-04 20:47:26 -07:00
0f2bc1aa17
server is passing the client integration tests now. I don't understand the ATTRs field, it has a name in it.
2015-07-31 23:09:51 -07:00
435f753792
readdir, rename, remove
2015-07-31 15:46:13 -07:00
7ab0966023
readonly (only for open right now)
2015-07-30 23:43:00 -07:00
bf6b5bce28
fstat
2015-07-30 09:21:59 -07:00
c9cee8ac6f
implement write
2015-07-29 17:37:58 -07:00
2888b4a6b1
implement read
2015-07-29 17:24:24 -07:00
058e1bee58
open & close
2015-07-26 01:32:19 -07:00
91d0c9e68a
mkdir
2015-07-25 19:07:33 -07:00
5b6348f034
version, lstat
2015-07-25 01:19:29 -07:00
d2d0a435b7
fix linux tests
2015-07-15 16:38:33 -07:00
587af18cfc
Fix integration tests on darwin
2015-07-15 16:36:15 -07:00
518aed2757
Merge pull request #44 from pkg/fixedbugs/43
...
Fix typo in slice creation.
2015-06-16 07:04:13 +10:00
c1748e2776
Fix typo in slice creation.
...
Fixes #43
2015-06-15 20:12:08 +10:00
f234c3c654
Merge branch 'master' of https://github.com/pkg/sftp
2015-06-05 10:48:03 +10:00
97cfc9933a
removed gsftp, moved to github.com/constabulary/example-gsftp
2015-06-05 10:47:18 +10:00
aef29b2c34
Merge pull request #41 from ggriffiniii/master
...
Increase throughput
2015-06-04 07:27:19 +10:00
34978efaaf
Address review comments and fix a data race.
...
Previously sftp.Close() would only shutdown the write side of the
connection and return. The sftp server would then shutdown the read side
of the connection at which point the recv goroutine would receive a
failure and return. When using a local sftp server in the intergration
tests we first call sftp.Close() followed by cmd.Wait(). cmd.Wait()
calls close on stdout while the sftp recv goroutine is still reading
from it. The fix is to block in sftp.Close() until the sftp server has
closed the receive end of the connection and the recv goroutine has
subsequently exited.
2015-06-03 09:33:09 -07:00
29600148e4
Handle recvPacket in a single goroutine.
...
Previously recvPacket would be invoked in several goroutines. This meant
that when multiple concurrent requests were in flight there were N
goroutines each waiting on recvPacket. For optimal throughput the goal
is to send a new request as quickly as possible once a response is
received. The previous mechanism worked counter to this because the
goroutine sending new requests would be competing against N recvPacket
goroutines that may become runnable as data streams in. Having a single
goroutine responsible for recvPacket means that the recv and send
goroutines will ping-pong back and forth optimizing throughput.
This changes shows a ~10-25% increase in throughput in the the *Delay*
benchmark tests.
$ go test -bench=. -integration
PASS
BenchmarkRead1k 2 840068631 ns/op 12.48 MB/s
BenchmarkRead16k 20 72968548 ns/op 143.70 MB/s
BenchmarkRead32k 30 56871347 ns/op 184.38 MB/s
BenchmarkRead128k 100 34150953 ns/op 307.05 MB/s
BenchmarkRead512k 100 15730685 ns/op 666.59 MB/s
BenchmarkRead1MiB 200 10462421 ns/op 1002.24 MB/s
BenchmarkRead4MiB 200 7325236 ns/op 1431.47 MB/s
BenchmarkRead4MiBDelay10Msec 10 186893765 ns/op 56.11 MB/s
BenchmarkRead4MiBDelay50Msec 2 907127114 ns/op 11.56 MB/s
BenchmarkRead4MiBDelay150Msec 1 2708025060 ns/op 3.87 MB/s
BenchmarkWrite1k 1 1623940932 ns/op 6.46 MB/s
BenchmarkWrite16k 10 174293843 ns/op 60.16 MB/s
BenchmarkWrite32k 10 120377272 ns/op 87.11 MB/s
BenchmarkWrite128k 20 54592205 ns/op 192.08 MB/s
BenchmarkWrite512k 50 66449591 ns/op 157.80 MB/s
BenchmarkWrite1MiB 50 70965660 ns/op 147.76 MB/s
BenchmarkWrite4MiB 50 69234861 ns/op 151.45 MB/s
BenchmarkWrite4MiBDelay10Msec 5 276624260 ns/op 37.91 MB/s
BenchmarkWrite4MiBDelay50Msec 1 1318396552 ns/op 7.95 MB/s
BenchmarkWrite4MiBDelay150Msec 1 3918416658 ns/op 2.68 MB/s
BenchmarkCopyDown10MiBDelay10Msec 10 152240808 ns/op 68.88 MB/s
BenchmarkCopyDown10MiBDelay50Msec 2 715003188 ns/op 14.67 MB/s
BenchmarkCopyDown10MiBDelay150Msec 1 2116878801 ns/op 4.95 MB/s
BenchmarkCopyUp10MiBDelay10Msec 10 192748258 ns/op 54.40 MB/s
BenchmarkCopyUp10MiBDelay50Msec 2 691486538 ns/op 15.16 MB/s
BenchmarkCopyUp10MiBDelay150Msec 1 1997162991 ns/op 5.25 MB/s
BenchmarkMarshalInit 2000000 644 ns/op
BenchmarkMarshalOpen 3000000 562 ns/op
BenchmarkMarshalWriteWorstCase 20000 75166 ns/op
BenchmarkMarshalWrite1k 500000 3862 ns/op
ok github.com/pkg/sftp 71.174s
2015-06-02 13:13:32 -07:00
de236e835b
Implement ReadFrom for *sftp.File
...
Improve the naive io.Copy case by splitting up the transfer into
multiple concurrent chunks similar to how large Write's are performed. This
improves the throughput on the BenchmarkCopyUp tests by 15-20x.
$ go test -bench=. -integration
PASS
BenchmarkRead1k 20 78382052 ns/op 133.78 MB/s
BenchmarkRead16k 100 14038681 ns/op 746.93 MB/s
BenchmarkRead32k 100 12076514 ns/op 868.29 MB/s
BenchmarkRead128k 200 8892708 ns/op 1179.16 MB/s
BenchmarkRead512k 300 5937224 ns/op 1766.13 MB/s
BenchmarkRead1MiB 300 5383775 ns/op 1947.68 MB/s
BenchmarkRead4MiB 300 5896306 ns/op 1778.38 MB/s
BenchmarkRead4MiBDelay10Msec 5 213987487 ns/op 49.00 MB/s
BenchmarkRead4MiBDelay50Msec 1 1013717329 ns/op 10.34 MB/s
BenchmarkRead4MiBDelay150Msec 1 3012666692 ns/op 3.48 MB/s
BenchmarkWrite1k 10 189878293 ns/op 55.22 MB/s
BenchmarkWrite16k 50 57726712 ns/op 181.65 MB/s
BenchmarkWrite32k 30 79804300 ns/op 131.39 MB/s
BenchmarkWrite128k 20 71296126 ns/op 147.08 MB/s
BenchmarkWrite512k 20 101823875 ns/op 102.98 MB/s
BenchmarkWrite1MiB 50 70351842 ns/op 149.05 MB/s
BenchmarkWrite4MiB 20 70187426 ns/op 149.40 MB/s
BenchmarkWrite4MiBDelay10Msec 5 333251686 ns/op 31.47 MB/s
BenchmarkWrite4MiBDelay50Msec 1 1576708254 ns/op 6.65 MB/s
BenchmarkWrite4MiBDelay150Msec 1 4823796059 ns/op 2.17 MB/s
BenchmarkCopyDown10MiBDelay10Msec 10 196175368 ns/op 53.45 MB/s
BenchmarkCopyDown10MiBDelay50Msec 2 918624682 ns/op 11.41 MB/s
BenchmarkCopyDown10MiBDelay150Msec 1 2880111274 ns/op 3.64 MB/s
BenchmarkCopyUp10MiBDelay10Msec 5 246048181 ns/op 42.62 MB/s
BenchmarkCopyUp10MiBDelay50Msec 2 872059111 ns/op 12.02 MB/s
BenchmarkCopyUp10MiBDelay150Msec 1 2516801139 ns/op 4.17 MB/s
BenchmarkMarshalInit 2000000 690 ns/op
BenchmarkMarshalOpen 3000000 579 ns/op
BenchmarkMarshalWriteWorstCase 20000 60438 ns/op
BenchmarkMarshalWrite1k 300000 4318 ns/op
ok github.com/pkg/sftp 70.210s
2015-06-02 13:13:32 -07:00
828850be97
Implement WriteTo for *sftp.File
...
Improve the naive io.Copy case by splitting up the transfer into
multiple concurrent chunks similar to how large Read's are performed. This
improves the throughput on the BenchmarkCopyDown tests by 15-20x.
$ go test -bench=. -integration
PASS
BenchmarkRead1k 20 80039871 ns/op 131.01 MB/s
BenchmarkRead16k 100 13109576 ns/op 799.86 MB/s
BenchmarkRead32k 100 13002925 ns/op 806.42 MB/s
BenchmarkRead128k 200 9189480 ns/op 1141.07 MB/s
BenchmarkRead512k 300 5863892 ns/op 1788.21 MB/s
BenchmarkRead1MiB 300 5350731 ns/op 1959.71 MB/s
BenchmarkRead4MiB 300 5880209 ns/op 1783.25 MB/s
BenchmarkRead4MiBDelay10Msec 5 211600615 ns/op 49.56 MB/s
BenchmarkRead4MiBDelay50Msec 1 1014580728 ns/op 10.34 MB/s
BenchmarkRead4MiBDelay150Msec 1 3015748763 ns/op 3.48 MB/s
BenchmarkWrite1k 10 210602614 ns/op 49.79 MB/s
BenchmarkWrite16k 30 53914210 ns/op 194.49 MB/s
BenchmarkWrite32k 20 68630676 ns/op 152.79 MB/s
BenchmarkWrite128k 50 70518854 ns/op 148.70 MB/s
BenchmarkWrite512k 30 69846510 ns/op 150.13 MB/s
BenchmarkWrite1MiB 30 70971873 ns/op 147.75 MB/s
BenchmarkWrite4MiB 20 68902426 ns/op 152.18 MB/s
BenchmarkWrite4MiBDelay10Msec 5 334770724 ns/op 31.32 MB/s
BenchmarkWrite4MiBDelay50Msec 1 1439154435 ns/op 7.29 MB/s
BenchmarkWrite4MiBDelay150Msec 1 4381710538 ns/op 2.39 MB/s
BenchmarkCopyDown10MiBDelay10Msec 10 161331837 ns/op 64.99 MB/s
BenchmarkCopyDown10MiBDelay50Msec 2 844679071 ns/op 12.41 MB/s
BenchmarkCopyDown10MiBDelay150Msec 1 2721133400 ns/op 3.85 MB/s
BenchmarkCopyUp10MiBDelay10Msec 1 3410147635 ns/op 3.07 MB/s
BenchmarkCopyUp10MiBDelay50Msec 1 16310789039 ns/op 0.64 MB/s
BenchmarkCopyUp10MiBDelay150Msec 1 48479031068 ns/op 0.22 MB/s
BenchmarkMarshalInit 2000000 685 ns/op
BenchmarkMarshalOpen 3000000 606 ns/op
BenchmarkMarshalWriteWorstCase 20000 81904 ns/op
BenchmarkMarshalWrite1k 300000 4646 ns/op
ok github.com/pkg/sftp 128.842s
2015-06-02 13:13:32 -07:00
d9e7820587
Increase throughput of Read/Write.
...
Break up Read/Write calls into multiple concurrent requests to allow the
roundtrip time to the server to overlap. This provides a roughly 10x
throughput increase when using large buffers over a high latency link. This
does not help the naive io.Copy case since io.Copy defaults to an 8k buffer.
$ go test -bench=. -integration
PASS
BenchmarkRead1k 20 82017395 ns/op 127.85 MB/s
BenchmarkRead16k 100 14634723 ns/op 716.51 MB/s
BenchmarkRead32k 100 13706765 ns/op 765.02 MB/s
BenchmarkRead128k 200 9614364 ns/op 1090.65 MB/s
BenchmarkRead512k 200 5778457 ns/op 1814.65 MB/s
BenchmarkRead1MiB 300 5624251 ns/op 1864.41 MB/s
BenchmarkRead4MiB 200 5798324 ns/op 1808.43 MB/s
BenchmarkRead4MiBDelay10Msec 5 214369945 ns/op 48.91 MB/s
BenchmarkRead4MiBDelay50Msec 1 1014850552 ns/op 10.33 MB/s
BenchmarkRead4MiBDelay150Msec 1 3016993337 ns/op 3.48 MB/s
BenchmarkWrite1k 10 200740041 ns/op 52.24 MB/s
BenchmarkWrite16k 50 74597799 ns/op 140.57 MB/s
BenchmarkWrite32k 20 63229429 ns/op 165.84 MB/s
BenchmarkWrite128k 20 78691019 ns/op 133.25 MB/s
BenchmarkWrite512k 20 64372711 ns/op 162.89 MB/s
BenchmarkWrite1MiB 20 95393443 ns/op 109.92 MB/s
BenchmarkWrite4MiB 20 72211301 ns/op 145.21 MB/s
BenchmarkWrite4MiBDelay10Msec 3 335329748 ns/op 31.27 MB/s
BenchmarkWrite4MiBDelay50Msec 1 1668562466 ns/op 6.28 MB/s
BenchmarkWrite4MiBDelay150Msec 1 4535944414 ns/op 2.31 MB/s
BenchmarkCopyDown10MiBDelay10Msec 1 3371273197 ns/op 3.11 MB/s
BenchmarkCopyDown10MiBDelay50Msec 1 16250399252 ns/op 0.65 MB/s
BenchmarkCopyDown10MiBDelay150Msec 1 48459210755 ns/op 0.22 MB/s
BenchmarkCopyUp10MiBDelay10Msec 1 3410202609 ns/op 3.07 MB/s
BenchmarkCopyUp10MiBDelay50Msec 1 16291168491 ns/op 0.64 MB/s
BenchmarkCopyUp10MiBDelay150Msec 1 48478335678 ns/op 0.22 MB/s
BenchmarkMarshalInit 2000000 716 ns/op
BenchmarkMarshalOpen 2000000 638 ns/op
BenchmarkMarshalWriteWorstCase 20000 61127 ns/op
BenchmarkMarshalWrite1k 300000 4724 ns/op
ok github.com/pkg/sftp 186.575s
2015-06-02 13:13:15 -07:00
fdbec907ff
Add integration tests that measure how throughput varies with latency.
...
This clearly shows the impact high latency networks have on the
throughput of sftp transfers. The current benchmark results on my
machine are:
$ go test -bench=. -integration
PASS
BenchmarkRead1k 20 93028268 ns/op 112.72 MB/s
BenchmarkRead16k 100 14912438 ns/op 703.16 MB/s
BenchmarkRead32k 100 12282661 ns/op 853.71 MB/s
BenchmarkRead128k 100 10550935 ns/op 993.83 MB/s
BenchmarkRead512k 100 12970518 ns/op 808.44 MB/s
BenchmarkRead1MiB 100 11526693 ns/op 909.70 MB/s
BenchmarkRead4MiB 100 15761135 ns/op 665.30 MB/s
BenchmarkRead4MiBDelay10Msec 1 2662626928 ns/op 3.94 MB/s
BenchmarkRead4MiBDelay50Msec 1 12977334971 ns/op 0.81 MB/s
BenchmarkRead4MiBDelay150Msec 1 38777294968 ns/op 0.27 MB/s
BenchmarkWrite1k 5 235318220 ns/op 44.56 MB/s
BenchmarkWrite16k 50 57347520 ns/op 182.85 MB/s
BenchmarkWrite32k 30 66151560 ns/op 158.51 MB/s
BenchmarkWrite128k 20 86642733 ns/op 121.02 MB/s
BenchmarkWrite512k 20 72648883 ns/op 144.34 MB/s
BenchmarkWrite1MiB 20 87145317 ns/op 120.33 MB/s
BenchmarkWrite4MiB 50 64098344 ns/op 163.59 MB/s
BenchmarkWrite4MiBDelay10Msec 1 3360676836 ns/op 3.12 MB/s
BenchmarkWrite4MiBDelay50Msec 1 16321294488 ns/op 0.64 MB/s
BenchmarkWrite4MiBDelay150Msec 1 48731747397 ns/op 0.22 MB/s
BenchmarkCopyDown10MiBDelay10Msec 1 3369680836 ns/op 3.11 MB/s
BenchmarkCopyDown10MiBDelay50Msec 1 16228310257 ns/op 0.65 MB/s
BenchmarkCopyDown10MiBDelay150Msec 1 48505306499 ns/op 0.22 MB/s
BenchmarkCopyUp10MiBDelay10Msec 1 3393474907 ns/op 3.09 MB/s
BenchmarkCopyUp10MiBDelay50Msec 1 16273951972 ns/op 0.64 MB/s
BenchmarkCopyUp10MiBDelay150Msec 1 48461724780 ns/op 0.22 MB/s
BenchmarkMarshalInit 2000000 779 ns/op
BenchmarkMarshalOpen 2000000 645 ns/op
BenchmarkMarshalWriteWorstCase 20000 70954 ns/op
BenchmarkMarshalWrite1k 300000 5072 ns/op
ok github.com/pkg/sftp 296.179s
2015-06-02 10:48:10 -07:00
db7130eb68
Merge pull request #40 from pkg/make-send-request-concurrent
...
Make c.sendRequest concurrent
2015-05-24 14:57:27 +10:00
2fc1e77011
Final cleanup
2015-05-24 14:56:15 +10:00
a17c010174
use a named type for readability
2015-05-24 14:51:18 +10:00
c020667ac1
Make c.sendRequest concurrent
...
This is an attempt to permit overlapping requests.
For each request sent, we spin off a goroutine to send the request for us
then handle _a_ response. That goroutine will try to dispatch the request
to the original sender. As every request is initiated from the client, there
will always be a matching number of replies from the server so eventally
every worker goroutine will process a request and exit.
2015-05-23 22:51:29 +10:00
2c28678ca1
Merge pull request #39 from pkg/add-functional-options
...
Add function options to sftp.NewClient and a MaxPacket option
2015-05-23 22:04:35 +10:00
750628aee1
Added -s flag to all benchmark programs
2015-05-23 22:03:37 +10:00
be8bb997e5
Add MaxPacket option to increase the size of the payload
...
% buffered-read-benchmark
2015/05/23 21:51:27 reading 1e+09 bytes
2015/05/23 21:51:53 read 1000000000 bytes in 25.668237836s
% buffered-read-benchmark -s 65536
2015/05/23 21:52:02 reading 1e+09 bytes
2015/05/23 21:52:20 read 1000000000 bytes in 18.419390056s
% buffered-read-benchmark -s 262144
2015/05/23 21:52:50 reading 1e+09 bytes
2015/05/23 21:53:08 read 1000000000 bytes in 18.367038236s
2015-05-23 21:53:59 +10:00
de1627afda
Merge pull request #38 from pkg/add-mux
...
Push c.mu.Lock/Unlock down to sendRequest
2015-05-23 21:36:05 +10:00
50631306d7
Push c.mu.Lock/Unlock down to sendRequest
...
Also includes gofmt cleanups
2015-05-23 17:10:22 +10:00
e92381e4cb
Merge pull request #37 from faryon93/master
...
Add statvfs@openssh.com extension request
2015-05-18 08:46:45 +10:00
267a179e16
added integration test for client.StatVFS
2015-05-16 00:54:02 +02:00
0b805e96f9
added statvfs@openssh.com extension request
2015-05-15 20:37:52 +02:00
1165da51c7
Updated to use github.com/Scriptrock/crypto/ssh
2015-02-13 09:59:47 -08:00
074b201d7d
Reference ScriptRock ssh library, not current google one
2015-01-08 20:33:06 -08:00
506297c901
Merge pull request #31 from jblazquez/master
...
Fixed write errors not bubbling up correctly
2014-12-19 12:46:30 +11:00
62281627fa
Fixed write errors not bubbling up correctly
2014-12-18 17:22:24 -08:00
873d14fe79
Merge pull request #30 from ottob/master
...
New import paths for Go sub-respositories
2014-11-10 11:18:18 +11:00
1c05ca70ba
New import paths for Go sub-respositories
...
see https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/eD8dh3T9yyA
2014-11-06 09:30:12 +01:00
bc39340f06
Update wercker config
2014-11-02 13:25:43 +11:00
64b46dd973
Merge pull request #29 from gbbr/patch-1
...
Rename LICENCE to LICENSE
2014-10-28 09:27:43 +11:00
aa431f9a21
Rename LICENCE to LICENSE
2014-10-27 21:31:41 +00:00
d72f41ea0b
remove tt
2014-10-22 20:49:39 +11:00
bcbcc6600d
Merge branch 'master' of https://github.com/pkg/sftp
2014-10-22 20:49:21 +11:00
Mark Sheahan
Dave Cheney
Glenn Griffin
Maximilian Pachl
Javier Blazquez
Otto Bretz
Gabriel Aszalos