问题现象
NC65系统,在做某单录入时,出现卡死现象。
问题分析
首先需要先判断卡是出现在前台客户端卡还是后台代码、SQL层面卡。
查看NMC监控,发现NMC中无相关代码堆栈、sql卡住。说明卡顿发生在前台客户端。
客户端卡顿,首先需要排查电脑的CPU、内存等资源是否耗尽导致的。发现正常。
如何抓取前台客户端监控,可以使用Oracle-JDK自带的工具jconsole.exe
找到客户端的进程,连接进去。
直接连接即可
首先排查客户端资源状况,Uclient堆内存配置为4G。
发现资源调度正常。堆内存占比正常。
查看线程监控发现有相关线程死锁。
AWT-EventQueue-0与SwingWorker-pool-1-thread-1死锁。
AWT-EventQueue-0线程信息
### XXX为加密信息名称: AWT-EventQueue-0
状态: java.util.concurrent.ConcurrentHashMap@64329307上的BLOCKED, 拥有者: SwingWorker-pool-1-thread-1
总阻止数: 1,525, 总等待数: 20,375堆栈跟踪:
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:166)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:234)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
nc.ui.uif2.factory.UIF2BeanFactory.getBean(UIF2BeanFactory.java:176)
nc.ui.uif2.xxx.getBesideExtendLoader(xxx.java:1015)
nc.ui.uif2.xxx.loadBesideExtend(xxx.java:1025)
nc.ui.uif2.xxx.validateTree(xxx.java:1008)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validateTree(Unknown Source)
java.awt.Container.validate(Unknown Source)- 已锁定 java.awt.Component$AWTTreeLock@65b17009
java.awt.Container.validateUnconditionally(Unknown Source)- 已锁定 java.awt.Component$AWTTreeLock@65b17009
java.awt.Dialog.conditionalShow(Unknown Source)- 已锁定 java.awt.Component$AWTTreeLock@65b17009
java.awt.Dialog.show(Unknown Source)
java.awt.Component.show(Unknown Source)
java.awt.Component.setVisible(Unknown Source)
... ...
SwingWorker-pool-1-thread-1线程信息
### XXX 是把相关方法名加密名称: SwingWorker-pool-1-thread-1
状态: java.awt.Component$AWTTreeLock@65b17009上的BLOCKED, 拥有者: AWT-EventQueue-0
总阻止数: 260, 总等待数: 118堆栈跟踪:
java.awt.Component.invalidate(Unknown Source)
java.awt.Container.invalidate(Unknown Source)
nc.ui.plaf.xxx.xxx.validateActionsBar(BasicActionsBarUI.java:602)
nc.ui.plaf.xxx.xxx.access$600(BasicActionsBarUI.java:44)
nc.ui.plaf.xxx.xxx$ActionsBarPropertyChangeHandler.propertyChange(BasicActionsBarUI.java:154)
java.beans.xxx.fire(Unknown Source)
java.beans.xxx.firePropertyChange(Unknown Source)
java.beans.xxx.fireIndexedPropertyChange(Unknown Source)
nc.ui.pub.xxx.xxx.fireIndexedPropertyChange(xxx.java:233)
nc.ui.pub.xxx.xxx.addAction(xxx.java:183)
nc.ui.pub.xxx.xxx.addAction(xxx.java:161)
nc.ui.pub.xxx.xxx.setActions(xxx.java:169)
nc.ui.uif2.xxx.xxxx.addTabActions(TangramLayoutTabbedPane.java:165)
nc.ui.uif2.xxx.xxxx.addToolBarToTabbedPane(TangramLayoutTabbedPane.java:68)
nc.ui.uif2.xxx.TangramLayoutTabbedPane.setSelectedIndex(TangramLayoutTabbedPane.java:58)
javax.swing.JTabbedPane.insertTab(Unknown Source)
nc.ui.pub.beans.ExtTabbedPane.insertTab(ExtTabbedPane.java:91)- 已锁定 nc.ui.uif2.xxx.TangramLayoutTabbedPane@130a96ee
nc.ui.pub.beans.ExtTabbedPane.insertTab(ExtTabbedPane.java:86)
javax.swing.JTabbedPane.addTab(Unknown Source)
javax.swing.JTabbedPane.add(Unknown Source)
nc.ui.uif2.xxx.TangramLayoutTabbedPane.add(TangramLayoutTabbedPane.java:76)
nc.ui.uif2.xxx.xxx$DefaultContainerComponentStrategy.createByTabbedPane(xxx.java:1314)
nc.ui.uif2.xxx.xxx$DefaultContainerComponentStrategy.createTabbedPane(xxx.java:1273)
nc.ui.uif2.xxx.xxx.createTabbedNodeHelper(xxx.java:645)
nc.ui.uif2.xxx.xxx.createComponentHelper(xxx.java:612)
nc.ui.uif2.xxx.xxx.createComponentByShadowModel(xxx.java:575)
nc.ui.uif2.xxx.xxx.createComponent(xxx.java:163)
nc.ui.uif2.xxx.xxx.getComponent(xxx.java:106)
nc.ui.uif2.xxx.initUI(TangramContainer.java:50)
为了印证这个工具抓取线程无问题,又可以使用jmc.exe工具
检测死锁
解决办法
dump线程下来,与业务研发沟通,出具补丁修复死锁问题。