[akka-user][deprecated] Akka SSLSession leak when running Akka with native TLS

Discussion:

Sean Gibbons

2018-11-21 17:06:50 UTC

Hi all,

I have been working with a native TLS Akka HTTP service deployed to
Production. We have noticed memory increasing consistently throughout the
week until our Akka service died due to memory constraints.
Running a JProfiler locally I've managed to reproduce what I believe to be
a leak in SSL related classes just using Akka code.

I am using Akka HTTP version 10.1.5 and Akka version 2.5.18.

I ran a comparison of a non native TLS Akka server vs a native TLS Akka
server. The load test consisted of sending around 20 req/s to a dummy
endpoint that simply just returns a hardcoded "hello" string.

A *complete* *runnable* *example* of the Akka code used to produce this
leak can be found as *dummy-app.zip *along with JProfiler snapshots in this *Google
Drive link*
- https://drive.google.com/drive/folders/1Q1zgN4m5J4oI_S0TupMs1LO1UVlJfh87?usp=sharing

*With Native TLS:*

[image: Screen Shot 2018-11-21 at 12.09.17 PM.png] <about:invalid#zClosurez>

JProfiler snapshots associated with this image are *included *in the*
Google Drive link *above and named:
- TLSLocal1.jps
- TLSLocal2.jps
- TLSLocal3.jps
Each blue line above is when a snapshot was taken.

*Without Native TLS:*

<about:invalid#zClosurez>[image: Screen Shot 2018-11-21 at 12.12.58 PM.png]
<about:invalid#zClosurez>

JProfiler snapshots associated with this image are *included *in the* Google
Drive link *above and named:
- NoTLSLocal1.jps
- NoTLSLocal2.jps
- NoTLSLocal3.jps
Each blue line above is when a snapshot was taken.

Any help is much appreciated on this topic thanks.
--
__

SÃ©anadh RÃomhphoist/_

EmailÂ Disclaimer__
**

TÃ¡ an rÃomhphost seo agus
aon chomhad a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus
sin amhÃ¡in Ã©.Â Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>Â
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_

This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee.Â Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml>Â _
*_
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+***@googlegroups.com.
To post to this group, send email to akka-***@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

j***@lightbend.com

2018-11-22 10:12:36 UTC

Permalink

Hi Sean,

thanks for the comprehensive report. What do you mean with a native vs
non-native TLS server? Is the example app for the "native TLS" server?

Johannes

Post by Sean Gibbons
Hi all,
I have been working with a native TLS Akka HTTP service deployed to
Production. We have noticed memory increasing consistently throughout the
week until our Akka service died due to memory constraints.
Running a JProfiler locally I've managed to reproduce what I believe to be
a leak in SSL related classes just using Akka code.
I am using Akka HTTP version 10.1.5 and Akka version 2.5.18.
I ran a comparison of a non native TLS Akka server vs a native TLS Akka
server. The load test consisted of sending around 20 req/s to a dummy
endpoint that simply just returns a hardcoded "hello" string.
A *complete* *runnable* *example* of the Akka code used to produce this
leak can be found as *dummy-app.zip *along with JProfiler snapshots in
this *Google Drive link* -
https://drive.google.com/drive/folders/1Q1zgN4m5J4oI_S0TupMs1LO1UVlJfh87?usp=sharing
*With Native TLS:*
[image: Screen Shot 2018-11-21 at 12.09.17 PM.png]
JProfiler snapshots associated with this image are *included *in the*
- TLSLocal1.jps
- TLSLocal2.jps
- TLSLocal3.jps
Each blue line above is when a snapshot was taken.
*Without Native TLS:*
[image: Screen Shot 2018-11-21 at 12.12.58 PM.png]
JProfiler snapshots associated with this image are *included *in the* Google
- NoTLSLocal1.jps
- NoTLSLocal2.jps
- NoTLSLocal3.jps
Each blue line above is when a snapshot was taken.
Any help is much appreciated on this topic thanks.
*SÃ©anadh RÃomhphoist/Email DisclaimerTÃ¡ an rÃomhphost seo agus aon chomhad
a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus sin amhÃ¡in
Ã©. Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee. Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml> *

--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Post by Sean Gibbons

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Sean Gibbons

2018-11-22 11:13:19 UTC

Permalink

Hi Johannes thanks for the reply,

I used the example code for both the native and non-native TLS Akka
servers. In the case of the non-native TLS server setup I simply changed
the Http.bind() method to not include the setupTls parameter e.g. *Http().bindAndHandle(routes,
"0.0.0.0", 17715)*
--
__

SÃ©anadh RÃomhphoist/_

EmailÂ Disclaimer__
**

TÃ¡ an rÃomhphost seo agus
aon chomhad a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus
sin amhÃ¡in Ã©.Â Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>Â
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_

This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee.Â Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml>Â _
*_
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Sean Gibbons

2018-11-22 12:10:36 UTC

Permalink

Just to clarify by non-native TLS I meant to say "no TLS" i.e. simply an
insecure AKKA server.
--
__

SÃ©anadh RÃomhphoist/_

EmailÂ Disclaimer__
**

TÃ¡ an rÃomhphost seo agus
aon chomhad a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus
sin amhÃ¡in Ã©.Â Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>Â
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_

This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee.Â Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml>Â _
*_
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Sean Gibbons

2018-11-22 12:16:40 UTC

Permalink

And by native TLS I just mean a standard TLS AKKA Server, apologies for any
confusion.
--
__

SÃ©anadh RÃomhphoist/_

EmailÂ Disclaimer__
**

TÃ¡ an rÃomhphost seo agus
aon chomhad a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus
sin amhÃ¡in Ã©.Â Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>Â
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_

This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee.Â Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml>Â _
*_
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Johannes Rudolph

2018-11-22 13:31:52 UTC

Permalink

I see. Thanks.

With the provided code I couldn't reproduce the issue at least in the quick
tests I did. Could you run

jmap -histo:live <pid> on the command line when some memory has accrued and
send the output here (or in private)?

Johannes

Post by Sean Gibbons
And by native TLS I just mean a standard TLS AKKA Server, apologies for
any confusion.
*SÃ©anadh RÃomhphoist/Email DisclaimerTÃ¡ an rÃomhphost seo agus aon chomhad
a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus sin amhÃ¡in
Ã©. Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee. Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml> *
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/

http://doc.akka.io/docs/akka/current/additional/faq.html

Search the archives: https://groups.google.com/group/akka-user

---
You received this message because you are subscribed to a topic in the
Google Groups "Akka User List" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/akka-user/b6VtlNFLsr8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Post by Sean Gibbons

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Sean Gibbons

2018-11-22 15:28:30 UTC

Permalink

I've ran the test again locally using the example server and attached is my
output for 3 heap dumps over a 40 minutes time period

Notice the growth in SSLSessionImpl count between 'grep ssl snapShot1',
'grep ssl snapShot2' and 'grep ssl snapShot3'
snapShot1 was taken at the start of the process, snapShot2 was taken after
about 10 minutes and snapShot3 was taken about 30 minutes after snapShot2

How my test is setup:
- Start the dummy Akka service supplied in the google drive link with TLS
enabled
- Run the following script testEndpoint.sh 20 times so as to have 20 req/s
incoming

#!/bin/bash

set -B # enable brace expansion

for i in {1..100000}; do

curl -s -k 'GET' 'https://localhost:17715/test'

sleep 1

done
- I ran this script using './testEndpoint.sh &'

In the google drive link from the opening post I have supplied JProfiler
snapshots that also show the heap usage which I would recommend viewing for
additional verification.
--
__

SÃ©anadh RÃomhphoist/_

EmailÂ Disclaimer__
**

TÃ¡ an rÃomhphost seo agus
aon chomhad a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus
sin amhÃ¡in Ã©.Â Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>Â
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_

This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee.Â Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml>Â _
*_
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Patrik Nordwall

2018-11-27 13:01:21 UTC

Permalink

Hi Sean,

I tried your sample app and ran 20 clients as you described. Attached
profiler and looked at memory snapshot. There are a few thousand
SSLSessionImpl objects but most of them are not strong reachable, so they
will be garbage collected when needed.

To be convinced you can try to run with a smaller maximum heap, such
as -Xmx128m, which will trigger garbage collection more aggressively. If
there is a real memory leak you will eventually see OutOfMemoryError.

Regards,
Patrik

Post by Sean Gibbons
I've ran the test again locally using the example server and attached is
my output for 3 heap dumps over a 40 minutes time period
Notice the growth in SSLSessionImpl count between 'grep ssl snapShot1',
'grep ssl snapShot2' and 'grep ssl snapShot3'
snapShot1 was taken at the start of the process, snapShot2 was taken after
about 10 minutes and snapShot3 was taken about 30 minutes after snapShot2
- Start the dummy Akka service supplied in the google drive link with TLS
enabled
- Run the following script testEndpoint.sh 20 times so as to have 20 req/s
incoming
#!/bin/bash
set -B # enable brace expansion
for i in {1..100000}; do
curl -s -k 'GET' 'https://localhost:17715/test'
sleep 1
done
- I ran this script using './testEndpoint.sh &'
In the google drive link from the opening post I have supplied JProfiler
snapshots that also show the heap usage which I would recommend viewing for
additional verification.
*SÃ©anadh RÃomhphoist/Email DisclaimerTÃ¡ an rÃomhphost seo agus aon chomhad
a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus sin amhÃ¡in
Ã©. Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee. Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml> *
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/

http://doc.akka.io/docs/akka/current/additional/faq.html

Search the archives: https://groups.google.com/group/akka-user

---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
Patrik Nordwall
Akka Tech Lead
Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM
Twitter: @patriknw
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Post by Sean Gibbons

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Sean Gibbons

2018-11-30 12:41:21 UTC

Permalink

Thanks for the response Patrik, we have tested with a smaller heap size and
it appears our applications memory does remain stable with no appearance of
a leak.
I believe the reason we were seeing our instances fail was actually due to
setting the heap size too large and our instances were using up all
available memory on the box they were running on.

Appreciate the help debugging this issue.

All the best,
Sean
--
__

SÃ©anadh RÃomhphoist/_

EmailÂ Disclaimer__
**

TÃ¡ an rÃomhphost seo agus
aon chomhad a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus
sin amhÃ¡in Ã©.Â Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>Â
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_

This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee.Â Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml>Â _
*_
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user

Patrik Nordwall

2018-11-30 12:42:38 UTC

Permalink

You're welcome.

Post by Sean Gibbons
Thanks for the response Patrik, we have tested with a smaller heap size
and it appears our applications memory does remain stable with no
appearance of a leak.
I believe the reason we were seeing our instances fail was actually due to
setting the heap size too large and our instances were using up all
available memory on the box they were running on.
Appreciate the help debugging this issue.
All the best,
Sean
*SÃ©anadh RÃomhphoist/Email DisclaimerTÃ¡ an rÃomhphost seo agus aon chomhad
a sheoltar leis faoi rÃºn agus is lena ÃºsÃ¡id ag an seolaÃ agus sin amhÃ¡in
Ã©. Is fÃ©idir tuilleadh a lÃ©amh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee. Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml> *
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************

Read the docs: http://akka.io/docs/

http://doc.akka.io/docs/akka/current/additional/faq.html

Search the archives: https://groups.google.com/group/akka-user

Post by Sean Gibbons

Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user