Guest User

I occasionally noticed that certain servers are in full of threads from a few weeks ago. It delayed our service. please refer to our service environment below.

AWS Amazon EC2 Instance
OS Version ubuntu 14.04.1 LTS
JAVA JVM Version 1.7.0_75
Server version: Apache Tomcat/7.0.42
Jennifer 5.0.9.1

I checked the thread dump and noticed that 490 tomcat threads out of 500 were in the status of waiting of monitor. You may refer to its stack lace below.
at java.util.Hashtable.get(Hashtable.java:433)
- waiting to lock [0×000000074050d4a8] (a java.util.Hashtable)
at jennifer.runtime.tracer.b.d.read(SourceFile:84)
at jennifer.base.profile.ProfileTopology.read(ProfileTopology.java:59)
at jennifer.base.vm.SocketInputStream.read(SocketInputStream.java:104)
at org.apache.http.impl.io.AbstractSessiON-INPUTBuffer.read(AbstractSessiON-INPUTBuffer.java:204)
at org.apache.http.impl.conn.LoggingSessiON-INPUTBuffer.read(LoggingSessiON-INPUTBuffer.java:82)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:204)
at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:108)
at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:164)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:237)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:186)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.event.ProgressInputStream.close(ProgressInputStream.java:182)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.event.ProgressInputStream.close(ProgressInputStream.java:182)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.close(CipherLiteInputStream.java:153)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at com.amazonaws.services.s3.internal.crypto.AdjustedRangeInputStream.close(AdjustedRangeInputStream.java:157)
at com.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:89)
at org.apache.commons.net.io.Util.closeQuietly(Util.java:353)
at *..****.******..storage.provider.S3StorageManager.download(S3StorageManager.java:324)
at *.
..*...object.service.impl.DownloadServiceImpl.download(DownloadServiceImpl.java:286)
at *.
..*...object.service.impl.DownloadServiceImpl.downloadRequest(DownloadServiceImpl.java:199)
at *.
..*...object.service.impl.DownloadServiceImpl.downloadRequest(DownloadServiceImpl.java:62)
at *.
..*...object.manager.impl.DefaultObjectManagerImpl.publicDownload(DefaultObjectManagerImpl.java:420)
at *.
..*...object.manager.impl.VariedObjectManagerImpl.publicDownload(VariedObjectManagerImpl.java:182)
at *.
..*...controller.v2.V2Controller.publicDownload(V2Controller.java:871)
at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked [0×00000007858af5a8] (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
All of these threads were blocked by an single thread and I would like to share their stack lace below.
at java.util.Hashtable.get(Hashtable.java:433)
- locked [0×000000074050d4a8] (a java.util.Hashtable)
at jennifer.runtime.tracer.b.d.read(SourceFile:84)
at jennifer.base.profile.ProfileTopology.read(ProfileTopology.java:59)
at jennifer.base.vm.SocketInputStream.read(SocketInputStream.java:104)
at org.apache.http.impl.io.AbstractSessiON-INPUTBuffer.fillBuffer(AbstractSessiON-INPUTBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessiON-INPUTBuffer.readLine(AbstractSessiON-INPUTBuffer.java:281)
at org.apache.http.impl.conn.LoggingSessiON-INPUTBuffer.readLine(LoggingSessiON-INPUTBuffer.java:115)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:461)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:296)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3737)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1164)
at com.amazonaws.services.s3.AmazonS3EncryptionClient.access$201(AmazonS3EncryptionClient.java:79)
at com.amazonaws.services.s3.AmazonS3EncryptionClient$S3DirectImpl.getObject(AmazonS3EncryptionClient.java:604)
at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleAE.getObjectSecurely(S3CryptoModuleAE.java:116)
at com.amazonaws.services.s3.internal.crypto.CryptoModuleDispatcher.getObjectSecurely(CryptoModuleDispatcher.java:116)
at com.amazonaws.services.s3.AmazonS3EncryptionClient.getObject(AmazonS3EncryptionClient.java:483)
at 
..****.******..storage.provider.S3StorageManager.download(S3StorageManager.java:304)
at *.
..*...object.service.impl.DownloadServiceImpl.download(DownloadServiceImpl.java:286)
at *.
..*...object.service.impl.DownloadServiceImpl.downloadRequest(DownloadServiceImpl.java:199)
at *.
..*...object.service.impl.DownloadServiceImpl.downloadRequest(DownloadServiceImpl.java:62)
at *.
..*...object.manager.impl.DefaultObjectManagerImpl.publicDownload(DefaultObjectManagerImpl.java:420)
at *.
..*...object.manager.impl.VariedObjectManagerImpl.publicDownload(VariedObjectManagerImpl.java:182)
at *.
..*..*.controller.v2.V2Controller.publicDownload(V2Controller.java:871)
at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked [0×0000000785f1d090] (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at jennifer.runtime.tracer.b.d.read(SourceFile:84)
at jennifer.base.profile.ProfileTopology.read(ProfileTopology.java:59)
at jennifer.base.vm.SocketInputStream.read(SocketInputStream.java:1
I was wondering if this is in the range of normal process. If so, I would like to know how this process is related with the function of JENNIFER.
It would be much helpful if you could explain why over 500 catalina-exe threads have remained as waiting due to the thread. Thanks.

Khalid Saeed

It appears this is in normal condition. It occurs due to the socket monitoring and topology map. Without installing the JENNIFER, you would see such event. The reason you see the JENNIFER stack on the top is because the JENNIFER was wrapping the socket. From the status of stack, it is waiting for data received from the socket. In other words, it is waiting to see the data coming while it is connected to the socket.