[Weblogic Troubleshooting]WLS 10.0.1 Out of Memory Error on Admin Server instance of Weblogic Server

WLS 10.0.1 Out of Memory Error on Admin Server instance of Weblogic Server

 

1. 시스템 환경

1. System Environment

 

 - HP-UX(11.31)

 - Weblogic Server 10.0.1 (AdminServer start/shutdown managed Servers, using the node manager)

 - JDK 1.5_21 (32bit)

 - Admin Server Memory option: -Xms1024m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=128m

 - No deployment on AdminServer

 

2. 현상

2. Sympton

 

1주일 간격으로 Admin Server에 Out of Memory 현상 발생

got Out of Memory Error on AdminServer each a week.

 

3. 원인 분석

3. analyize cause

 

3.1 GC 그래프

3.1 GC graph

 

 

 

 

3.2 Heap dump 분석

3.2 Heap dump Analyzing

921,420,847 (94%) [36] 5 weblogic/timers/internal/TimerManagerImpl 0x48435938
921,420,606 (94%) [24] 2 weblogic/timers/internal/TimerThread 0x47d615c0
493,943,173 (50%) [36] 1 java/util/TreeMap 0x47e18098
493,943,137 (50%) [29] 4 java/util/TreeMap$Entry 0x72292510
...
427,477,409 (43%) [104] 11 weblogic/timers/internal/TimerThread$Thread 0x47d61558
427,476,798 (43%) [43] 4 java/lang/ThreadGroup 0x47d50030
427,469,841 (43%) [16] 3 array of [Ljava/lang/ThreadGroup; 0x47e12460
427,468,646 (43%) [43] 4 java/lang/ThreadGroup 0x47e0c8b0
427,457,775 (43%) [1,024] 196 array of [Ljava/lang/Thread; 0x47bcacd0
414,556,473 (42%) [161] 17 weblogic/work/ExecuteThread 0x47d4e6f8
354,535,065(36%)[30] 3 weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl 0x435675f0
354,534,825 (36%) [58] 3 weblogic/timers/internal/TimerImpl 0x58622f00
354,533,943 (36%) [36] 5 weblogic/timers/internal/TimerManagerImpl 0x48bab680
354,033,163 (36%) [36] 1 java/util/TreeMap 0x48e21bc8

  

 

3.3 로그분석

3.3 Log Analyzing

기존 AdminServer에 Notice 레벨을 Info 레벨로 낮춘 뒤 로그 확인 결과 아래의 현상이 5초 간격으로 managed Server의 개수 만큼 지속적으로 발생

generate logs below each 5 seconds as much as the number of managed server connecting with AdminServer

<Feb 9, 2012 10:36:14 AM KST> <Info> <ServletContext-/bea_wls_internal> <BEA-000000> <HTTPClntLogin: Login rejected with code: 'Failed', reason: java.net.ProtocolException: HTTP tunneling is disabled

        at weblogic.rjvm.http.HTTPServerJVMConnection.acceptJVMConnection(HTTPServerJVMConnection.java:88)

        at weblogic.rjvm.http.TunnelLoginServlet.service(TunnelLoginServlet.java:80)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)

        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395)

        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

        at weblogic.security.service.SecurityManager.runAs(Unknown Source)

        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)

        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)

        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

 

 

4. 결론

4. Conclusion

 

독립(MSI) 모드로 기동해온 managed Server가 admin Server와 통신(http 프로토콜)하려고 connect 할때 발생하는 에러 현상. Admin Server의 Out of Memory Error 현상을 초래하는 버그임

When Managed Server on Independent mode communicate to AdminServer, it happen with Error. it is known bug Issue 

 

5. 조치사항

5. Solution

 

bug# 8180780, 8138764, 8136767, 8183179을 merge 한 bug#8915871 버그 패치 적용

apply Bug patch #8915871 Merging (bug #8180780, #8138764, #8136767, #8183179)

 

 

 

6. 조치 후 모니터링 결과 (GC 상태 확인)

6. GC Monitoring after Bug patch apply

 

 

 

7. 적용하고자 하는 패치 정보는 아래와 같습니다.

7. the details of Bug patch is like below

----------------------------------------------------------------------------------------------------------------

#8180780

This CR fixes an periodic exception caused on the server when the admin protocol specified to start a managed server uses a http based protocol. This issue has been addressed.

 

#8138764

####<Sep 10, 2007 3:36:01 PM JST> <Warning> <Server> <akitada04>

<AdminServer> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1189406161464>

<BEA-002637> <Failed to register a disconnect listener because of weblogic.rmi.extensions.DisconnectMonitorUnavailableException

 

#8136767

This patch fixes a java.net.ProtocolException resulting when AdminServer is set to a listenAddress & managed servers us http URL to boot from the admin server.

 

#8183179

The java.net.ProtocolException exceptions occurred on an admin server if a managed server started in MSI-mode using the http based admin protocol. This issue has been addressed.

-------------------------------------------------------------------------------------------------------------------

 

8. 해당 버그 패치 적용시 주의사항

bug#8915871 버그 패치를 적용할 경우 Admin Server 뿐만 아니라 Admin Server와 통신하고 있는 모든 Managed Server에 적용되어야 하는 패치임 미적용시 적용하지 않은 managed Server의 개수 만큼 Admin Server를 Info 레벨에서 확인할 경우 5초 간격으로 지속 발생

Eng. The Bug patch # 8915871 have to be applied in even managed Server as well.