7fe159edef
Yolo-replace format strings
...
Replaces `~s` and `~p` with their unicode-friendly counterparts.
```
git ls-files *.erl | xargs sed -i.ORIG -e s/~s>/~ts/g -e s/~p>/~tp/g
```
2022-10-10 10:32:03 +04:00
963b54d283
Use credit flow for dest side of on-publish shovels
...
This avoids message queue build up of the rabbit_writer process in case
the destination is slow (or blocked by a resource alarm) and the sending
to the socket blocks.
This is applied only for one of the ack-modes:
- on-confirm: already has acknowledgment for each message
- on-publish: now has credit based flow control - which allows multiple
but finite number of on-the-fly messages at any time
- no-ack: highest performance and asynchronicity without any guarantees
fixes #3407
2022-09-04 12:20:13 +02:00
c38a3d697d
Bump (c) year
2022-03-21 01:21:56 +04:00
f934fde55a
fix shovels crashing with delete-after=0
...
When a shovel was added with delete-after=0 it crashed immediatelly,
causing to never be cleaned up. This caused RabbitMQ to not boot if
the shovel parameter existed.
The fix makes sure the validation is correct on this value, and when
a shovel starts it will let it start with a 0 delete-after.
2022-02-03 11:11:59 +01:00
1cabd3ccd4
Naming
2021-09-21 20:39:35 +03:00
7aedc45d1f
convert property map to proplists and vice versa for uri obfucation in shovel config
2021-09-21 19:39:22 +03:00
6dbdc991c3
Obfuscate credentials in shovel worker states to avoid plaintext passwords being logged on crashes
2021-09-21 19:39:22 +03:00
a3c97d491f
Update additional test skipping for 3.8/3.9 mixed versions
2021-06-25 11:17:46 +02:00
de8dd5fb69
Move messages from stream queues on the UI
...
Requires support of consumer args in shovel
2021-04-27 12:07:23 +02:00
cdcf602749
Switch from Lager to the new Erlang Logger API for logging
...
The configuration remains the same for the end-user. The only exception
is the log root directory: it is now set through the `log_root`
application env. variable in `rabbit`. People using the Cuttlefish-based
configuration file are not affected by this exception.
The main change is how the logging facility is configured. It now
happens in `rabbit_prelaunch_logging`. The `rabbit_lager` module is
removed.
The supported outputs remain the same: the console, text files, the
`amq.rabbitmq.log` exchange and syslog.
The message text format slightly changed: the timestamp is more precise
(now to the microsecond) and the level can be abbreviated to always be
4-character long to align all messages and improve readability. Here is
an example:
2021-03-03 10:22:30.377392+01:00 [dbug] <0.229.0> == Prelaunch DONE ==
2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>
2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> Starting RabbitMQ 3.8.10+115.g071f3fb on Erlang 23.2.5
2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
The example above also shows that multiline messages are supported and
each line is prepended with the same prefix (the timestamp, the level
and the Erlang process PID).
JSON is also supported as a message format and now for any outputs.
Indeed, it is possible to use it with e.g. syslog or the exchange. Here
is an example of a JSON-formatted message sent to syslog:
Mar 3 11:23:06 localhost rabbitmq-server[27908] <0.229.0> - {"time":"2021-03-03T11:23:06.998466+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","meta":{"domain":"rabbitmq.prelaunch","file":"src/rabbit_prelaunch_logging.erl","gl":"<0.228.0>","line":311,"mfa":["rabbit_prelaunch_logging","configure_logger",1],"pid":"<0.229.0>"}}
For quick testing, the values accepted by the `$RABBITMQ_LOGS`
environment variables were extended:
* `-` still means stdout
* `-stderr` means stderr
* `syslog:` means syslog on localhost
* `exchange:` means logging to `amq.rabbitmq.log`
`$RABBITMQ_LOG` was also extended. It now accepts a `+json` modifier (in
addition to the existing `+color` one). With that modifier, messages are
formatted as JSON intead of plain text.
The `rabbitmqctl rotate_logs` command is deprecated. The reason is
Logger does not expose a function to force log rotation. However, it
will detect when a file was rotated by an external tool.
From a developer point of view, the old `rabbit_log*` API remains
supported, though it is now deprecated. It is implemented as regular
modules: there is no `parse_transform` involved anymore.
In the code, it is recommended to use the new Logger macros. For
instance, `?LOG_INFO(Format, Args)`. If possible, messages should be
augmented with some metadata. For instance (note the map after the
message):
?LOG_NOTICE("Logging: switching to configured handler(s); following "
"messages may not be visible in this log output",
#{domain => ?RMQLOG_DOMAIN_PRELAUNCH}),
Domains in Erlang Logger parlance are the way to categorize messages.
Some predefined domains, matching previous categories, are currently
defined in `rabbit_common/include/logging.hrl` or headers in the
relevant plugins for plugin-specific categories.
At this point, very few messages have been converted from the old
`rabbit_log*` API to the new macros. It can be done gradually when
working on a particular module or logging.
The Erlang builtin console/file handler, `logger_std_h`, has been forked
because it lacks date-based file rotation. The configuration of
date-based rotation is identical to Lager. Once the dust has settled for
this feature, the goal is to submit it upstream for inclusion in Erlang.
The forked module is calld `rabbit_logger_std_h` and is based
`logger_std_h` in Erlang 23.0.
2021-03-11 15:17:36 +01:00
9e0bc75844
Skip QQ-specific tests in mixed version mode
...
It's not very important to run these in mixed
version clusters and mess with feature flags.
References #2798 .
2021-02-11 03:15:39 +03:00
927a9ddb52
Make it possible to specify optional queue arguments for dynamic Shovels
...
when shovels declare queues, it is currently not possible to declare
a quorum queue.
Closes #2798 .
2021-02-10 06:13:18 +03:00
52479099ec
Bump (c) year
2021-01-22 09:00:14 +03:00
45b6b223d8
Switch to MPL2
2020-07-14 21:39:08 +03:00
5736984b86
Fall back to delete-after where src-delete-after is used in a few more places
...
064a7ff49fCloses #72 , spotted and suggested by @inikulshin.
2020-04-27 06:45:42 +03:00
9c1e914444
Merge pull request #71 from rabbitmq/shovel-issue-70
...
Increase meta-data fidelity
2020-04-22 17:48:08 +03:00
2d22a36ab1
Increase meta-data fidelity
...
When shovelling 0.9.1 messages to 1.0.
Map any basic properties that map directly to AMQP 1.0 properties and
add the remaining properties (including routing key) to application
properties if they have compatible types.
2020-04-22 14:37:27 +01:00
71e8e50224
Change error_string to error as the return is not from the parameters module any more
2020-03-13 11:12:58 -07:00
bdccd8c21e
Update copyright (year 2020)
2020-03-10 16:50:05 +01:00
2a0719db88
(c) bump
2019-12-29 05:50:35 +03:00
7dc9ee4a15
amqp10_SUITE: Don't ack in `no_ack` mode
...
Otherwise we get a "precondition failed" error and the channel is
closed. This error occurred rarely because the client also closes the
channel in parallel. So depending on the order both close messages are
handled, the client might be happy (which is the case most of the time).
2019-09-26 12:34:58 +02:00
d1ab9cb4c4
Handle Shovel names that are a single term
2019-05-17 19:36:33 +03:00
668d714e3c
URL Cleanup
...
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).
# HTTP URLs that Could Not Be Fixed
These URLs were unable to be fixed. Please review them to see if they can be manually resolved.
* http://blog.listincomprehension.com/search/label/procket (200) with 1 occurrences could not be migrated:
([https](https://blog.listincomprehension.com/search/label/procket ) result ClosedChannelException).
* http://dozzie.jarowit.net/trac/wiki/TOML (200) with 1 occurrences could not be migrated:
([https](https://dozzie.jarowit.net/trac/wiki/TOML ) result SSLHandshakeException).
* http://dozzie.jarowit.net/trac/wiki/subproc (200) with 1 occurrences could not be migrated:
([https](https://dozzie.jarowit.net/trac/wiki/subproc ) result SSLHandshakeException).
* http://e2project.org (200) with 1 occurrences could not be migrated:
([https](https://e2project.org ) result AnnotatedConnectException).
* http://nitrogenproject.com/ (200) with 2 occurrences could not be migrated:
([https](https://nitrogenproject.com/ ) result ConnectTimeoutException).
* http://proper.softlab.ntua.gr (200) with 1 occurrences could not be migrated:
([https](https://proper.softlab.ntua.gr ) result SSLHandshakeException).
* http://yaws.hyber.org (200) with 1 occurrences could not be migrated:
([https](https://yaws.hyber.org ) result AnnotatedConnectException).
* http://choven.ca (503) with 1 occurrences could not be migrated:
([https](https://choven.ca ) result ConnectTimeoutException).
# Fixed URLs
## Fixed But Review Recommended
These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended.
* http://fixprotocol.org/ (301) with 1 occurrences migrated to:
https://fixtrading.org ([https](https://fixprotocol.org/ ) result SSLHandshakeException).
* http://erldb.org (UnknownHostException) with 1 occurrences migrated to:
https://erldb.org ([https](https://erldb.org ) result UnknownHostException).
## Fixed Success
These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended.
* http://cloudi.org/ with 27 occurrences migrated to:
https://cloudi.org/ ([https](https://cloudi.org/ ) result 200).
* http://erlware.org/ with 1 occurrences migrated to:
https://erlware.org/ ([https](https://erlware.org/ ) result 200).
* http://inaka.github.io/cowboy-trails/ with 1 occurrences migrated to:
https://inaka.github.io/cowboy-trails/ ([https](https://inaka.github.io/cowboy-trails/ ) result 200).
* http://ninenines.eu with 6 occurrences migrated to:
https://ninenines.eu ([https](https://ninenines.eu ) result 200).
* http://www.actordb.com/ with 2 occurrences migrated to:
https://www.actordb.com/ ([https](https://www.actordb.com/ ) result 200).
* http://www.cs.kent.ac.uk/projects/wrangler/Home.html with 1 occurrences migrated to:
https://www.cs.kent.ac.uk/projects/wrangler/Home.html ([https](https://www.cs.kent.ac.uk/projects/wrangler/Home.html ) result 200).
* http://www.rabbitmq.com/shovel.html with 1 occurrences migrated to:
https://www.rabbitmq.com/shovel.html ([https](https://www.rabbitmq.com/shovel.html ) result 200).
* http://www.rebar3.org with 1 occurrences migrated to:
https://www.rebar3.org ([https](https://www.rebar3.org ) result 200).
* http://contributor-covenant.org with 1 occurrences migrated to:
https://contributor-covenant.org ([https](https://contributor-covenant.org ) result 301).
* http://contributor-covenant.org/version/1/3/0/ with 1 occurrences migrated to:
https://contributor-covenant.org/version/1/3/0/ ([https](https://contributor-covenant.org/version/1/3/0/ ) result 301).
* http://inaka.github.com/apns4erl with 1 occurrences migrated to:
https://inaka.github.com/apns4erl ([https](https://inaka.github.com/apns4erl ) result 301).
* http://inaka.github.com/edis/ with 1 occurrences migrated to:
https://inaka.github.com/edis/ ([https](https://inaka.github.com/edis/ ) result 301).
* http://lasp-lang.org/ with 1 occurrences migrated to:
https://lasp-lang.org/ ([https](https://lasp-lang.org/ ) result 301).
* http://saleyn.github.com/erlexec with 1 occurrences migrated to:
https://saleyn.github.com/erlexec ([https](https://saleyn.github.com/erlexec ) result 301).
* http://www.mozilla.org/MPL/ with 25 occurrences migrated to:
https://www.mozilla.org/MPL/ ([https](https://www.mozilla.org/MPL/ ) result 301).
* http://zhongwencool.github.io/observer_cli with 1 occurrences migrated to:
https://zhongwencool.github.io/observer_cli ([https](https://zhongwencool.github.io/observer_cli ) result 301).
2019-03-20 03:23:23 -05:00
3a0aaa6ad6
Add tests to for AMQP1.0 shovel message properties parsing.
2019-02-11 17:42:35 -05:00
6edda5381e
Update (c) year
2019-01-30 04:32:59 +03:00
9431393460
More tests that use empty publish properties
...
References #46 .
2018-11-29 12:38:23 +08:00
50fad9f2f8
Add a few basic tests that use blank AMQP 0-9-1 publish properties
...
References #46 .
2018-11-27 06:22:11 +08:00
7f1bf736d9
Handle amqp encoded body values.
...
Use new amqp10_msg:body_bin to get a binary representation of the amqp
1.0 encoded body.
[#159434026 ]
2018-08-13 15:34:13 +01:00
ad328ee6e4
More coercion to proplists, more cases in tests
2018-03-29 20:31:29 +03:00
d8e28fb133
Convert Shovel definitions to proplists before validating
...
Validation historically expects proplists and doesn't handle
maps.
Closes #38 .
[#156367916 ]
2018-03-29 13:33:53 +03:00
c352fe5b4e
Fix off-by-one error in amqp10 shovel
...
This bug only manifested itself when the source used amqp10 and the
destination used amqp091 and the first publisher confirm was a multiple.
In this scenario the first message would not have been acked back to the
source.
2017-06-09 19:12:11 +01:00
d17ef783a1
Refactor of duplicated code
2017-05-31 14:49:18 +01:00
bbce661013
Refactor configuration validation
...
To get better separation between validation
and parsing and also move some of the protocol
specific parsing into the relevant shovel
implementation.
[#138600475 ]
2017-05-22 11:24:17 +01:00
dc39ca61cc
Add license headers
...
Only decrement counters for known tags
2017-05-17 14:37:05 +01:00
e95fd3e5bd
Tidy up logging
2017-05-11 10:43:09 +01:00
bc10689e3c
Refactor static configuration validation.
2017-05-10 12:42:24 +01:00
1b7a72b64d
Translate multi-acks into amqp 1.0 disposition
...
range.
2017-05-09 10:59:35 +01:00
645b50a0d9
mixed dynamic tests
2017-05-08 17:16:22 +01:00
0f6976994f
amqp10 autodelete
2017-05-05 12:29:41 +01:00
f180ba2bf7
Move dynamic calls to the module defining the behaviour.
2017-05-04 11:07:27 +01:00
9e03d3e7dc
amqp10 dynamic tests
2017-04-28 11:13:22 +01:00
3954c33703
simple dynamic ampq10 shovel
2017-04-26 16:29:53 +01:00
9cac5a23bc
Fix flaky dynamic test.
2017-04-26 11:10:09 +01:00
064a7ff49f
refactor parameter validation towards multi-protocol
2017-04-25 10:23:54 +01:00
6eed57f06c
amqp10 parameters configuration parsing
2017-04-24 13:20:33 +01:00
6f9e73ed1b
Refactor parameter parsing
2017-04-24 10:15:20 +01:00
e34c9436ab
merge from master
2017-04-19 11:33:25 +01:00
9d1feb2dc6
Refactor towards multi-protocol
...
Amqp10 static shovel
2017-04-19 11:26:30 +01:00
157557085f
Merge branch 'stable'
2017-04-02 22:03:22 +03:00
8bc5085cb8
(c) year
2017-04-02 21:49:21 +03:00
4e996e6c56
Add suport for username in parameters and vhost events
2017-01-17 07:52:21 +00:00
a07127e917
Merge branch 'stable'
2016-12-13 13:09:31 +01:00
f0f366f8f2
configuration_SUITE: Move plugin stop to a setup step
...
rabbit_ct_helpers:run_setup_steps/2 may not return a configuration: it
can be `{skip, _}`. Thus, move the plugin stop to a proper setup step
instead of assuming we can use the return value as a configuration.
2016-12-13 13:07:23 +01:00
62d8fd4c41
Merge branch 'master' into rabbitmq-shovel-22
2016-11-21 16:56:14 +03:00
556ff23517
Include empty fields for csv and table formatters
2016-11-21 11:34:53 +00:00
b8a864313c
Tests for delete shovel command
2016-11-18 12:29:47 +00:00
1a5c214e9b
Tests for shovel status command
2016-11-18 12:19:19 +00:00
d1060f08d1
Move some auxiliary test functions to utils module
2016-11-18 12:19:12 +00:00
1d8e4aa106
Merge branch 'stable'
2016-10-10 17:36:39 +02:00
dbf3a4a240
Make Shovel connections specify a name
...
Fixes #19
2016-10-07 13:16:00 +02:00
97b2199b3e
Merge branch 'stable'
2016-06-15 16:47:14 +02:00
64f46734fa
Testsuite: Do not stop amqp_client during teardown
2016-06-15 16:35:11 +02:00
555298e7de
Switch testsuite to common_test
...
[#121411939 ]
2016-06-14 16:11:58 +02:00
bd02c475f0
Do not use time_compat
2016-05-12 10:24:08 +01:00
b959605a26
Wording
2016-01-11 17:03:06 +03:00
b4b3f46dc3
Tests for adding separate headers
2016-01-11 12:02:58 +00:00
3bc0bdea07
Shovel headers test
2016-01-11 11:48:14 +00:00
1db6e79388
Update (c) info
2016-01-01 12:59:18 +03:00
c65a783492
(c) year
2015-05-24 05:16:44 +03:00
f323b605fc
API update and don't drive the auth_backend API directly, we have rabbit_access_control for that!
2014-11-14 11:39:18 +00:00
50069cf286
Rename to match static shovel.
2014-07-21 16:40:30 +01:00
5dd6350924
Support add_forward_headers in static shovels.
2014-07-21 16:25:02 +01:00
392d5cc20c
Oops
2014-07-21 15:35:20 +01:00
f151087fc2
Allow setting properties in dynamic shovels.
2014-07-21 14:59:48 +01:00
b187f66350
fixes test for older versions of Erlang
2014-06-16 17:40:53 +02:00
2b640acda8
Allow 'none' user (i.e. rabbitmqctl) to set any direct connection. And update the tests so that security_validation_test tests explicit users but the other tests test 'none'.
2014-06-03 10:34:44 +01:00
4889996556
Ensure users can only create shovels which can access local vhosts they are allowed to access.
2014-04-08 17:33:35 +01:00
b6de90a3c9
Update copyright for 2014
2014-03-17 17:25:24 +00:00
00ded798ae
Auto-delete shovels
2014-02-28 16:58:32 +00:00
9cf8bb29e8
The shovel tests have always been racy, starting the shovel application, and expecting that to create an exchange which we can then bind to. But it's only been tickled recently. Anyway, wait for the shovel to start running before proceeding.
2014-02-07 13:14:01 +00:00
79dad4cc9e
Test with eunit for better error reporting.
2014-02-07 13:03:25 +00:00
2c3cce665e
stable to default
2014-01-06 13:42:36 +00:00
0eefd9e825
Allow publish_properties and publish_fields to be delivered in any order.
2014-01-06 12:37:39 +00:00
db98482b34
Fix the tests, and fix bug found by the tests :/
2013-12-12 16:23:56 +00:00
c839012148
Tighten up validation of exchange vs queue. Add validation tests. Remove list_to_binary/1 festival.
2013-11-27 14:42:01 +00:00
35ecea5f50
And a bit more testing
2013-11-27 12:56:26 +00:00
6abe5beab1
Make the test helpers assume slightly more so tests are less verbose, add another test, fix a bug thus found.
2013-11-27 12:53:53 +00:00
fe9c36da40
More tests, less races, less shelling out complexity.
2013-11-26 18:28:45 +00:00
0d703d8501
Simple test.
2013-11-26 17:48:37 +00:00
f58f18bc88
Fix test
2013-11-26 17:15:48 +00:00
3e58781f9a
s/VMware/GoPivotal/g
2013-07-01 10:49:14 +01:00
8e2c02ff22
Update copyright 2013
2013-01-23 12:05:48 +00:00
098db93771
Update copyright 2012
2012-02-02 13:42:46 +00:00
77288717dd
remove missing ssl parameter test
...
You can establish SSL connections without specifing all those parameters
(provided the broker is permisive enough).
2011-10-17 11:16:39 +01:00
c3f79c285a
rename url -> uri; move params_network conversion to amqp_client
2011-09-27 17:34:19 +01:00
5d5a080b2a
tweak tests to work with the new AMQP URLs
2011-09-27 16:35:02 +01:00
89b6e021a9
handle the return of amqp_url:parse better
2011-09-26 16:53:29 +01:00
1fe0201997
uri -> url
2011-09-26 12:49:40 +01:00
797ce51070
Undo the depressing idiocy of 978410e62a1c
2011-07-25 16:10:59 +01:00
48bd3f2cc2
Rename ack_on to ack_mode and associated changes to possible values
2011-06-29 11:59:55 +01:00
39b3a40413
space, white
2011-06-27 14:36:26 +01:00
cf9a248566
Having a single non-boolean is much nicer than multiple booleans, with mutually exclusive combinations
2011-06-27 14:27:58 +01:00
f4bf3614e7
Add support for confirms
2011-06-27 13:31:13 +01:00
f1dcecc8fe
Rip out the rest of txn support
2011-06-27 11:08:05 +01:00
1d61a89574
Updates for the new Erlang client API.
2011-05-03 16:50:51 +01:00
1f31c0f06b
Fix tests
2011-04-13 17:10:41 +01:00
cfcbad30cf
Make test return ok on success
...
For consistency with eunit, which generates test functions that don't
crash on failures.
2011-02-22 04:21:10 +00:00
7856dc317d
Merge default into bug23568
2011-01-28 17:10:06 +00:00
4189dc7127
Added support for configuring heartbeat, channel_max and frame_max in broker endpoint uri
2011-01-27 14:25:11 +00:00
69c80d15c7
Update license boiler plate copyright dodats
2011-01-19 17:14:59 +00:00
101b1e19b5
don't match on return code
2010-09-29 13:38:45 +01:00
7cce1af419
Make shovel tests work with new build
2010-09-28 17:22:08 +01:00
bb25fe252c
updating use of Erlang client's API
2010-09-08 15:31:52 +01:00
8f0e146757
Correct tests so that they don't complain now that the shovel doesn't blow up with no shovels configured
2010-07-08 16:34:14 +01:00
bb01670b99
simplify config parsing by making more of it monadic
...
and make error messages more consistent in the process
2010-03-01 21:08:30 +00:00
164e375efd
allow sparse proplists as configs and reject non proplists
2010-03-01 20:22:09 +00:00
3858216cf6
detect duplicate config params
2010-03-01 19:43:13 +00:00
638f515bcf
rename some config params
2010-02-25 18:19:11 +00:00
001fe2246b
test with tx
...
this provides better code coverage
2010-02-24 21:14:11 +00:00
063d3aed80
clean up config parsing error handling
...
- remove unreachable clauses
- catch a few more errors
- better error messages
- lots of tests of error handling
2010-02-24 17:48:32 +00:00
9f235abcc2
Minor tweaks to tests
2010-02-23 12:01:09 +00:00
77d961c0dd
Made the test obtain the message both before and after the shovel has had its evil way with it...
2010-02-22 17:39:56 +00:00
4821b7aaf4
Added a test
2010-02-22 17:37:27 +00:00
Luke Bakken
Péter Gömöri
Michael Klishin
Lajos Gerecs
Michael Klishin
Thuan Duong Ba
Philip Kuryloski
dcorbacho
Jean-Sébastien Pédron
Michael Klishin
kjnilsson
Luke Bakken
Jean-Sébastien Pédron
Spring Operator
Daniil Fedotov
Diana Corbacho
Arnaud Cogoluègnes
Daniil Fedotov
Simon MacMullen
Alvaro Videla
Emile Joubert
Alexandru Scvortov
Matthew Sackman
David Wragg
Vlad Alexandru Ionescu
Matthias Radestock
Matthew Sackman