Discussion:
[akka-user] Akka-remote docker<->docker works fine with netty.tcp, but fails with artery - ideas?
j***@gmail.com
2016-11-09 17:11:12 UTC
Permalink
I'm pretty new to Akka and am currently trying out Akka's new Artery
backend for akka-remoting
(http://doc.akka.io/docs/akka/2.4/scala/remoting-artery.html) while also
testing the regular netty.tcp transport
(http://doc.akka.io/docs/akka/2.4/scala/remoting.html).

It looks fairly straight forward and I have two test applications running:
* A producer application which sends a message to the consumer every 3
seconds, and prints some stuff when the reply comes back
* A consumer application which prints a message when it receives one, and
then sends back a reply to sender()

However I cannot get the communication to work when running both my
applications with artery inside docker containers on the same host - it
perfectly when I use netty.tcp though!.
Below are the results I achieved:

Results using artery
Producer Consumer Result
-------------------------
IntelliJ Docker ok
Docker IntelliJ ok
IntelliJ IntelliJ ok
Docker Docker fail (no message reaches the consumer, and the producer
eventually prints handshake timeouts after 20 seconds/default timeout in
akka-remote-artery - the same error message I get when not running any
Consumer at all)

Results using netty.tcp
Producer Consumer Result
-------------------------
IntelliJ Docker ok
Docker IntelliJ ok
IntelliJ IntelliJ ok
Docker Docker ok

As you see above, I got all combinations to work except when using artery
to communicate between two local docker containers.
The akka configurations for the failing artery case are as follows

Consumer conf (prettified output of typesafe Config object logged):
{
"akka": {
"actor": {
"provider": "remote"
},
"remote": {
"artery": {
"bind": {
"hostname": "0.0.0.0",
"port": 12345
},
"canonical": {
"hostname": "192.168.99.100",
"port": 12345
},
"enabled": "on"
}
}
}
}

Producer conf (prettified output of typesafe Config object logged):
{
"akka": {
"actor": {
"provider": "remote"
},
"remote": {
"artery": {
"bind": {
"hostname": "0.0.0.0",
"port": 12346
},
"canonical": {
"hostname": "192.168.99.100",
"port": 12346
},
"enabled": "on"
}
}
}
}


192.168.99.100 is the IP of my docker machine executing both containers.
The equivalent netty.tcp config works fine (as seen in the results table
earlier in this message)

*However using artery I just get zero activity on the consumer, and
eventually the producer starts logging errors:*
[ERROR] [11/09/2016 16:24:17.201]
[producer-system-akka.remote.default-remote-dispatcher-6]
[akka.remote.artery.Association(akka://producer-system)] Outbound message
stream to [akka://consumer-***@192.168.99.100:12345] failed. Restarting
it. Handshake with [akka://consumer-***@192.168.99.100:12345] did not
complete within 20000 ms
(akka.remote.artery.OutboundHandshake$HandshakeTimeoutException) after
about 20 seconds, indicating what I assume is a timeout.

When I use netty.tcp it works just fine between the containers.
What is also strange is that it works with artery when I move one of the
applications outside of a docker container. Both

When running each application outside docker (e.g. from within intelliJ
during development) they can communicate with each other fine, but as soon
as I try to make them talk to each other with artery from docker container
to docker container, no messages get through to the consumer
Each application can be configured (cmd line args) to communicate with
either netty.tcp or artery.

I've repeatedly gone over the configurations, code and startup arguments
from scratch now so many times, and had colleagues look at it as well. We
cannot figure out at all what could be wrong here :S.

The following are the docker startup commands. The command line arguments
you see here are used to generate the conf you see above.

Consumer:
docker run --shm-size 1024M -p 12345:12345 -p 12345:12345/udp --rm
akka-remote-eval-consumer:0.1.0-SNAPSHOT --transport tcp --host
192.168.99.100 --port 12345 --bind-host 0.0.0.0 --bind-port 12345

Producer:
docker run --shm-size 1024M -p 11001:11001/udp -p 12346:12346/udp --rm
akka-remote-eval-producer:0.1.0-SNAPST --transport artery --host
192.168.99.100 --port 12346 --bind-host 0.0.0.0 --bind-port 12346
--consumer-host 192.168.99.100 --consumer-port 12345

Running this on a regular macbook pro with what I think are the latest
updates, oracle jdk8, docker running with docker-machine.
Using akka-actor and akka-remote 2.4.12
Note: We are running an older docker version if that is of any importance:
1.10.2

Any ideas of what I could be doing wrong?

/ Johan
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Akka Team
2016-11-09 17:57:06 UTC
Permalink
Hi Johan (good name!),

Artery/Aeron uses UDP, while the old remoting is TCP-based, did you
configure your docker nodes with udp port mappings?

--
Johan
Akka Team
Post by j***@gmail.com
I'm pretty new to Akka and am currently trying out Akka's new Artery
backend for akka-remoting (http://doc.akka.io/docs/akka/
2.4/scala/remoting-artery.html) while also testing the regular netty.tcp
transport (http://doc.akka.io/docs/akka/2.4/scala/remoting.html).
* A producer application which sends a message to the consumer every 3
seconds, and prints some stuff when the reply comes back
* A consumer application which prints a message when it receives one, and
then sends back a reply to sender()
However I cannot get the communication to work when running both my
applications with artery inside docker containers on the same host - it
perfectly when I use netty.tcp though!.
Results using artery
Producer Consumer Result
-------------------------
IntelliJ Docker ok
Docker IntelliJ ok
IntelliJ IntelliJ ok
Docker Docker fail (no message reaches the consumer, and the
producer eventually prints handshake timeouts after 20 seconds/default
timeout in akka-remote-artery - the same error message I get when not
running any Consumer at all)
Results using netty.tcp
Producer Consumer Result
-------------------------
IntelliJ Docker ok
Docker IntelliJ ok
IntelliJ IntelliJ ok
Docker Docker ok
As you see above, I got all combinations to work except when using artery
to communicate between two local docker containers.
The akka configurations for the failing artery case are as follows
{
"akka": {
"actor": {
"provider": "remote"
},
"remote": {
"artery": {
"bind": {
"hostname": "0.0.0.0",
"port": 12345
},
"canonical": {
"hostname": "192.168.99.100",
"port": 12345
},
"enabled": "on"
}
}
}
}
{
"akka": {
"actor": {
"provider": "remote"
},
"remote": {
"artery": {
"bind": {
"hostname": "0.0.0.0",
"port": 12346
},
"canonical": {
"hostname": "192.168.99.100",
"port": 12346
},
"enabled": "on"
}
}
}
}
192.168.99.100 is the IP of my docker machine executing both containers.
The equivalent netty.tcp config works fine (as seen in the results table
earlier in this message)
*However using artery I just get zero activity on the consumer, and
eventually the producer starts logging errors:*
[ERROR] [11/09/2016 16:24:17.201] [producer-system-akka.remote.default-remote-dispatcher-6]
[akka.remote.artery.Association(akka://producer-system)] Outbound message
did not complete within 20000 ms (akka.remote.artery.OutboundHandshake$HandshakeTimeoutException)
after about 20 seconds, indicating what I assume is a timeout.
When I use netty.tcp it works just fine between the containers.
What is also strange is that it works with artery when I move one of the
applications outside of a docker container. Both
When running each application outside docker (e.g. from within intelliJ
during development) they can communicate with each other fine, but as soon
as I try to make them talk to each other with artery from docker container
to docker container, no messages get through to the consumer
Each application can be configured (cmd line args) to communicate with
either netty.tcp or artery.
I've repeatedly gone over the configurations, code and startup arguments
from scratch now so many times, and had colleagues look at it as well. We
cannot figure out at all what could be wrong here :S.
The following are the docker startup commands. The command line arguments
you see here are used to generate the conf you see above.
docker run --shm-size 1024M -p 12345:12345 -p 12345:12345/udp --rm
akka-remote-eval-consumer:0.1.0-SNAPSHOT --transport tcp --host
192.168.99.100 --port 12345 --bind-host 0.0.0.0 --bind-port 12345
docker run --shm-size 1024M -p 11001:11001/udp -p 12346:12346/udp --rm
akka-remote-eval-producer:0.1.0-SNAPST --transport artery --host
192.168.99.100 --port 12346 --bind-host 0.0.0.0 --bind-port 12346
--consumer-host 192.168.99.100 --consumer-port 12345
Running this on a regular macbook pro with what I think are the latest
updates, oracle jdk8, docker running with docker-machine.
Using akka-actor and akka-remote 2.4.12
1.10.2
Any ideas of what I could be doing wrong?
/ Johan
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/
current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 18:14:20 UTC
Permalink
Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Akka Team
2016-11-09 18:29:19 UTC
Permalink
Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable using
the setting akka.remote.artery.advanced.flight-recorder.enabled, it will
write artery related events to a binary logfile, which you can then feed to
akka.remote.artery.FlightRecorderDump in akka-remote to get human readable
output, it may also contain some hints about what/why/how it is not working.

--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/
current/additional/faq.html
Search the archives: https://groups.google.com/
group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 19:13:13 UTC
Permalink
Well, host <-> docker container artery works (both permutations). It's just
docker <-> docker on the same host that doesn't.
So udp forwarding seems to work. I will try to figure out a non-artery way
of testing udp. Might end up writing a small scala echo server + client
with udp (im terrible at linux/unix/mac command line tools :))

Also found this, but I'm confused as to what it says. Maybe it is related?
https://github.com/docker/docker/issues/15127

/ Johan
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Viktor Klang
2016-11-09 19:33:34 UTC
Permalink
looks related
--
Cheers,
√
Post by j***@gmail.com
Well, host <-> docker container artery works (both permutations). It's
just docker <-> docker on the same host that doesn't.
So udp forwarding seems to work. I will try to figure out a non-artery way
of testing udp. Might end up writing a small scala echo server + client
with udp (im terrible at linux/unix/mac command line tools :))
Also found this, but I'm confused as to what it says. Maybe it is related?
https://github.com/docker/docker/issues/15127
/ Johan
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled,
it will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get
human readable output, it may also contain some hints about what/why/how it
is not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that
I use the udp parameter for my docker port forwarding. Also please note
that I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/c
urrent/additional/faq.html
Search the archives: https://groups.google.com/grou
p/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/
current/additional/faq.html
Post by Akka Team
Post by j***@gmail.com
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by j***@gmail.com
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 19:56:12 UTC
Permalink
New info.

I have now tested regular udp docker <-> docker - with the exact same
application repurposed for raw java udp sockets. Using the same command
line arguments.
*It works fine with regular udp docker<->docker*. It's just artery that
doesn't want to work for me I suppose :(.

Could it be an artery bug?

/ Johan
Post by Viktor Klang
looks related
--
Cheers,
√
Post by j***@gmail.com
Well, host <-> docker container artery works (both permutations). It's
just docker <-> docker on the same host that doesn't.
So udp forwarding seems to work. I will try to figure out a non-artery
way of testing udp. Might end up writing a small scala echo server + client
with udp (im terrible at linux/unix/mac command line tools :))
Also found this, but I'm confused as to what it says. Maybe it is
related? https://github.com/docker/docker/issues/15127
/ Johan
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that
I use the udp parameter for my docker port forwarding. Also please note
that I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
Post by Akka Team
Post by j***@gmail.com
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Viktor Klang
Post by j***@gmail.com
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 20:02:33 UTC
Permalink
Ok hm sorry.. this is a lot of new things to take in for me :).
How do I change debug level for akka? Is there a typesafe/akka conf
parameter?

I may try the FlightRecorder later if I have time. I think we're probably
going to stick with netty.tcp for now though while artery has some time to
mature. Seems nice though.

/ Johan
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 20:08:41 UTC
Permalink
Even more new info :).
If I for fun launch the consumer as a regular udp socket I actually receive
something. So the artery udp layer seems to work. However I suspect there
could be some id/address mismatch in the handshake which makes akka-artery
silently discard the info. Will see if I can get some debug info out of
this..
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 20:33:21 UTC
Permalink
With akka.loglevel = DEBUG i only get two more lines on the consumer side:

First
[DEBUG] [11/09/2016 20:29:02.937] [aeron-client-conductor] [akka.remote.
artery.ArteryTransport(akka://consumer-system)] onAvailableImage from
172.17.0.1:52515 session 2110811048

And then a few seconds later
[DEBUG] [11/09/2016 20:29:13.417] [aeron-client-conductor] [akka.remote.
artery.ArteryTransport(akka://consumer-system)] onUnavailableImage from
172.17.0.1:52515 session 2110811048
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-10 09:06:03 UTC
Permalink
One additional piece of info that could be useful:

It works if I let the docker containers call each other by container IP
instead of host IP+fowarding.
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-10 10:37:59 UTC
Permalink
Another piece to the puzzle:
docker <-> docker does work routed through the host as long as the
containers run on different hosts.
It's only when doing docker <-> docker routed through host on the same host
that it doesn't work.

Seems like an issue in the artery or aeron protocol.
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that I
use the udp parameter for my docker port forwarding. Also please note that
I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Vishnu Vardhan
2017-05-28 06:03:05 UTC
Permalink
Hi Johan,

Was there a defect logged for this issue. I seem to be suffering from
exactly the same symptoms. I am now checking the docker / docker on
different hosts.

A related issue is the shutdown behavior if one of my producers is on bare
metal, and the consumers is on a docker container. I am unable to get a
clean shutdown. If both are running native (without containers ) shutdown
is clean, but if not, then the cluster master does not shut. I have
truncated and attached the two logs
1. Good log - non containerized (master and slave)
2. Bad log - cluster seed is not containerized, slave is containerized.
Master does not exit (log for master only)


On Thursday, November 10, 2016 at 2:38:00 AM UTC-8,
Post by j***@gmail.com
docker <-> docker does work routed through the host as long as the
containers run on different hosts.
It's only when doing docker <-> docker routed through host on the same
host that it doesn't work.
Seems like an issue in the artery or aeron protocol.
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that
I use the udp parameter for my docker port forwarding. Also please note
that I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Post by j***@gmail.com
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Donovan Kim
2018-11-08 20:32:09 UTC
Permalink
Is this resolved problem?
Post by Vishnu Vardhan
Hi Johan,
Was there a defect logged for this issue. I seem to be suffering from
exactly the same symptoms. I am now checking the docker / docker on
different hosts.
A related issue is the shutdown behavior if one of my producers is on bare
metal, and the consumers is on a docker container. I am unable to get a
clean shutdown. If both are running native (without containers ) shutdown
is clean, but if not, then the cluster master does not shut. I have
truncated and attached the two logs
1. Good log - non containerized (master and slave)
2. Bad log - cluster seed is not containerized, slave is containerized.
Master does not exit (log for master only)
On Thursday, November 10, 2016 at 2:38:00 AM UTC-8,
Post by j***@gmail.com
docker <-> docker does work routed through the host as long as the
containers run on different hosts.
It's only when doing docker <-> docker routed through host on the same
host that it doesn't work.
Seems like an issue in the artery or aeron protocol.
Post by Akka Team
Ah, sorry, didn't notice it in the command line.
Can you verify with some other tool that you can communicate using udp
between the nodes? (using netcat for example)
Set logs to debug to see if there is anything helpful in there.
Additionally artery contains a "flight recorder" which you can enable
using the setting akka.remote.artery.advanced.flight-recorder.enabled, it
will write artery related events to a binary logfile, which you can then
feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human
readable output, it may also contain some hints about what/why/how it is
not working.
--
Johan
Akka Team
Post by j***@gmail.com
Yes - i Did. Please see my original post for details. You will see that
I use the udp parameter for my docker port forwarding. Also please note
that I've tested artery successfully in docker as long as one of the
applications is running outside docker. However the problem occurs only
when both producer and consumer run in containers. See the test results
table and error log in my previous message for more details.
/Johan
--
Read the docs: http://akka.io/docs/
http://doc.akka.io/docs/akka/current/additional/faq.html
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************
Post by Vishnu Vardhan
Post by j***@gmail.com
Post by Akka Team
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2016-11-09 19:55:37 UTC
Permalink
New info.

I have now tested regular udp docker <-> docker - with the exact same
application repurposed for raw java udp sockets. Using the same command
line arguments.
*It works fine with regular udp docker<->docker*. It's just artery that
doesn't want to work for me I suppose :(.

Could it be an artery bug?

/ Johan


Den onsdag 9 november 2016 kl. 18:22:35 UTC+1 skrev
Post by j***@gmail.com
I'm pretty new to Akka and am currently trying out Akka's new Artery
backend for akka-remoting (
http://doc.akka.io/docs/akka/2.4/scala/remoting-artery.html) while also
testing the regular netty.tcp transport (
http://doc.akka.io/docs/akka/2.4/scala/remoting.html).
* A producer application which sends a message to the consumer every 3
seconds, and prints some stuff when the reply comes back
* A consumer application which prints a message when it receives one, and
then sends back a reply to sender()
However I cannot get the communication to work when running both my
applications with artery inside docker containers on the same host - it
perfectly when I use netty.tcp though!.
Results using artery
Producer Consumer Result
-------------------------
IntelliJ Docker ok
Docker IntelliJ ok
IntelliJ IntelliJ ok
Docker Docker fail (no message reaches the consumer, and the
producer eventually prints handshake timeouts after 20 seconds/default
timeout in akka-remote-artery - the same error message I get when not
running any Consumer at all)
Results using netty.tcp
Producer Consumer Result
-------------------------
IntelliJ Docker ok
Docker IntelliJ ok
IntelliJ IntelliJ ok
Docker Docker ok
As you see above, I got all combinations to work except when using artery
to communicate between two local docker containers.
The akka configurations for the failing artery case are as follows
{
"akka": {
"actor": {
"provider": "remote"
},
"remote": {
"artery": {
"bind": {
"hostname": "0.0.0.0",
"port": 12345
},
"canonical": {
"hostname": "192.168.99.100",
"port": 12345
},
"enabled": "on"
}
}
}
}
{
"akka": {
"actor": {
"provider": "remote"
},
"remote": {
"artery": {
"bind": {
"hostname": "0.0.0.0",
"port": 12346
},
"canonical": {
"hostname": "192.168.99.100",
"port": 12346
},
"enabled": "on"
}
}
}
}
192.168.99.100 is the IP of my docker machine executing both containers.
The equivalent netty.tcp config works fine (as seen in the results table
earlier in this message)
*However using artery I just get zero activity on the consumer, and
eventually the producer starts logging errors:*
[ERROR] [11/09/2016 16:24:17.201]
[producer-system-akka.remote.default-remote-dispatcher-6]
[akka.remote.artery.Association(akka://producer-system)] Outbound message
did not complete within 20000 ms
(akka.remote.artery.OutboundHandshake$HandshakeTimeoutException) after
about 20 seconds, indicating what I assume is a timeout.
When I use netty.tcp it works just fine between the containers.
What is also strange is that it works with artery when I move one of the
applications outside of a docker container. Both
When running each application outside docker (e.g. from within intelliJ
during development) they can communicate with each other fine, but as soon
as I try to make them talk to each other with artery from docker container
to docker container, no messages get through to the consumer
Each application can be configured (cmd line args) to communicate with
either netty.tcp or artery.
I've repeatedly gone over the configurations, code and startup arguments
from scratch now so many times, and had colleagues look at it as well. We
cannot figure out at all what could be wrong here :S.
The following are the docker startup commands. The command line arguments
you see here are used to generate the conf you see above.
docker run --shm-size 1024M -p 12345:12345 -p 12345:12345/udp --rm
akka-remote-eval-consumer:0.1.0-SNAPSHOT --transport tcp --host
192.168.99.100 --port 12345 --bind-host 0.0.0.0 --bind-port 12345
docker run --shm-size 1024M -p 11001:11001/udp -p 12346:12346/udp --rm
akka-remote-eval-producer:0.1.0-SNAPST --transport artery --host
192.168.99.100 --port 12346 --bind-host 0.0.0.0 --bind-port 12346
--consumer-host 192.168.99.100 --consumer-port 12345
Running this on a regular macbook pro with what I think are the latest
updates, oracle jdk8, docker running with docker-machine.
Using akka-actor and akka-remote 2.4.12
1.10.2
Any ideas of what I could be doing wrong?
/ Johan
--
Post by j***@gmail.com
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Loading...