记一次OOM

一个服务器上布了好多相同的服务,在修改代码重新部署的时候,发现服务在启动之后马上会报OOM,因为配置了-XX:+HeapDumpOnOutOfMemoryError,所以报出OOM之后,dump了当前的内存情况,如下

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 167772160 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_solaris.cpp:2553), pid=21140, tid=0x0000000000000058
#
# JRE version: Java(TM) SE Runtime Environment (8.0_141-b15) (build 1.8.0_141-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.141-b15 mixed mode solaris-sparc compressed oops)
# Core dump written. Default location: /export/home/optuser/core or core.21140
#

---------------  T H R E A D  ---------------

Current thread (0x00000001003b8800):  VMThread [stack: 0xffffffff25700000,0xffffffff25800000] [id=88]

Stack: [0xffffffff25700000,0xffffffff25800000],  sp=0xffffffff257fe8e0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xdf58fc]  void VMError::report_and_die()+0x6dc
V  [libjvm.so+0x53e554]  void report_vm_out_of_memory(const char*,int,unsigned long,VMErrorType,const char*)+0x64
V  [libjvm.so+0xb83e20]  bool os::pd_commit_memory(char*,unsigned long,unsigned long,bool)+0x110
V  [libjvm.so+0xb765d4]  bool os::commit_memory(char*,unsigned long,unsigned long,bool)+0x2c
V  [libjvm.so+0xc27a70]  bool PSVirtualSpace::expand_by(unsigned long)+0x50
V  [libjvm.so+0xc29188]  bool PSYoungGen::resize_generation(unsigned long,unsigned long)+0x74
V  [libjvm.so+0xc290f0]  void PSYoungGen::resize(unsigned long,unsigned long)+0x1c
V  [libjvm.so+0xc22080]  bool PSScavenge::invoke_no_policy()+0x12b0
V  [libjvm.so+0xc20bec]  bool PSScavenge::invoke()+0x4c
V  [libjvm.so+0xbb0fc8]  HeapWord*ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x78
V  [libjvm.so+0xdf9ac8]  void VM_ParallelGCFailedAllocation::doit()+0xa0
V  [libjvm.so+0xe041ec]  void VM_Operation::evaluate()+0x54
V  [libjvm.so+0xe01424]  void VMThread::evaluate_operation(VM_Operation*)+0x11c
V  [libjvm.so+0xe01aa4]  void VMThread::loop()+0x434
V  [libjvm.so+0xe00fd4]  void VMThread::run()+0xa4
V  [libjvm.so+0xb7e5f0]  java_start+0x390

VM_Operation (0xffffffff159fe3a0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x000000010065a000


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  省略无数的项目线程.....

Other Threads:
=>0x00000001003b8800 VMThread [stack: 0xffffffff25700000,0xffffffff25800000] [id=88]
  0x00000001005be800 WatcherThread [stack: 0xffffffff1ff00000,0xffffffff20000000] [id=129]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00000001002dfe10] Threads_lock - owner thread: 0x00000001003b8800
[0x00000001002e03b0] Heap_lock - owner thread: 0x000000010065a000

Heap:
 PSYoungGen      total 1619456K, used 37811K [0x0000000715580000, 0x00000007b3080000, 0x00000007c0000000)
  eden space 1154560K, 0% used [0x0000000715580000,0x0000000715580000,0x000000075bd00000)
  from space 464896K, 8% used [0x0000000796a80000,0x0000000798f6cfd8,0x00000007b3080000)
  to   space 480768K, 0% used [0x000000075bd00000,0x000000075bd00000,0x0000000779280000)
 ParOldGen       total 1780736K, used 668626K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cf4b48,0x000000062cb00000)
 Metaspace       used 30543K, capacity 31006K, committed 31104K, reserved 1077248K
  class space    used 3033K, capacity 3196K, committed 3200K, reserved 1048576K

Card table byte_map: [0xffffffff61800000,0xffffffff62802000] byte_map_base: 0xffffffff5ea00000

Marking Bits: (ParMarkBitMap*) 0xffffffff74182c88
 Begin Bits: [0xffffffff40000000, 0xffffffff48000000)
 End Bits:   [0xffffffff48000000, 0xffffffff50000000)

Polling page: 0xffffffff7c800000

CodeCache: size=245760Kb used=26376Kb max_used=26383Kb free=219383Kb
 bounds [0xffffffff63000000, 0xffffffff649e0000, 0xffffffff72000000]
 total_blobs=5940 nmethods=5429 adapters=414
 compilation: enabled

Compilation events (10 events):
Event: 283.852 Thread 0x0000000100591800 7085       1       oracle.jdbc.driver.T4CConnection::isLoggedOn (5 bytes)
Event: 283.852 Thread 0x0000000100591800 nmethod 7085 0xffffffff649d9ad0 code [0xffffffff649d9c00, 0xffffffff649d9d08]
Event: 283.916 Thread 0x0000000100565000 7087       3       com.stock.framework.database.DBResultSet::close (61 bytes)
Event: 283.916 Thread 0x0000000100589000 7086       3       com.stock.framework.database.StatementList::close (39 bytes)
Event: 283.917 Thread 0x0000000100565000 nmethod 7087 0xffffffff649d9d50 code [0xffffffff649d9f00, 0xffffffff649da458]
Event: 283.917 Thread 0x0000000100589000 nmethod 7086 0xffffffff649da5d0 code [0xffffffff649da780, 0xffffffff649dacd0]
Event: 285.140 Thread 0x00000001004fc000 7088       4       java.lang.ref.Finalizer::access$100 (6 bytes)
Event: 285.147 Thread 0x000000010042a000 7089   !   4       java.lang.ref.Finalizer::runFinalizer (62 bytes)
Event: 285.157 Thread 0x00000001004fc000 nmethod 7088 0xffffffff649dd310 code [0xffffffff649dd4a0, 0xffffffff649dd970]
Event: 285.158 Thread 0x000000010042a000 nmethod 7089 0xffffffff649dbcd0 code [0xffffffff649dbe60, 0xffffffff649dc330]

GC Heap History (10 events):
Event: 101.467 GC heap after
Heap after GC invocations=4 (full 1):
 PSYoungGen      total 611840K, used 0K [0x0000000715580000, 0x0000000760080000, 0x00000007c0000000)
  eden space 524800K, 0% used [0x0000000715580000,0x0000000715580000,0x0000000735600000)
  from space 87040K, 0% used [0x0000000735600000,0x0000000735600000,0x000000073ab00000)
  to   space 87040K, 0% used [0x000000075ab80000,0x000000075ab80000,0x0000000760080000)
 ParOldGen       total 1780736K, used 668570K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8ce6b48,0x000000062cb00000)
 Metaspace       used 20855K, capacity 21072K, committed 21248K, reserved 1069056K
  class space    used 2131K, capacity 2230K, committed 2304K, reserved 1048576K
}
Event: 189.855 GC heap before
{Heap before GC invocations=5 (full 1):
 PSYoungGen      total 611840K, used 524800K [0x0000000715580000, 0x0000000760080000, 0x00000007c0000000)
  eden space 524800K, 100% used [0x0000000715580000,0x0000000735600000,0x0000000735600000)
  from space 87040K, 0% used [0x0000000735600000,0x0000000735600000,0x000000073ab00000)
  to   space 87040K, 0% used [0x000000075ab80000,0x000000075ab80000,0x0000000760080000)
 ParOldGen       total 1780736K, used 668570K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8ce6b48,0x000000062cb00000)
 Metaspace       used 29025K, capacity 29556K, committed 29824K, reserved 1075200K
  class space    used 2990K, capacity 3132K, committed 3200K, reserved 1048576K
Event: 189.996 GC heap after
Heap after GC invocations=5 (full 1):
 PSYoungGen      total 832000K, used 39954K [0x0000000715580000, 0x000000075d300000, 0x00000007c0000000)
  eden space 791552K, 0% used [0x0000000715580000,0x0000000715580000,0x0000000745a80000)
  from space 40448K, 98% used [0x000000075ab80000,0x000000075d284b88,0x000000075d300000)
  to   space 87040K, 0% used [0x0000000752900000,0x0000000752900000,0x0000000757e00000)
 ParOldGen       total 1780736K, used 668594K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cecb48,0x000000062cb00000)
 Metaspace       used 29025K, capacity 29556K, committed 29824K, reserved 1075200K
  class space    used 2990K, capacity 3132K, committed 3200K, reserved 1048576K
}
Event: 250.808 GC heap before
{Heap before GC invocations=6 (full 1):
 PSYoungGen      total 832000K, used 831506K [0x0000000715580000, 0x000000075d300000, 0x00000007c0000000)
  eden space 791552K, 100% used [0x0000000715580000,0x0000000745a80000,0x0000000745a80000)
  from space 40448K, 98% used [0x000000075ab80000,0x000000075d284b88,0x000000075d300000)
  to   space 87040K, 0% used [0x0000000752900000,0x0000000752900000,0x0000000757e00000)
 ParOldGen       total 1780736K, used 668594K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cecb48,0x000000062cb00000)
 Metaspace       used 29961K, capacity 30441K, committed 30592K, reserved 1077248K
  class space    used 3019K, capacity 3164K, committed 3200K, reserved 1048576K
Event: 251.911 GC heap after
Heap after GC invocations=6 (full 1):
 PSYoungGen      total 1090048K, used 37989K [0x0000000715580000, 0x0000000797280000, 0x00000007c0000000)
  eden space 1003008K, 0% used [0x0000000715580000,0x0000000715580000,0x0000000752900000)
  from space 87040K, 43% used [0x0000000752900000,0x0000000754e19630,0x0000000757e00000)
  to   space 491520K, 0% used [0x0000000779280000,0x0000000779280000,0x0000000797280000)
 ParOldGen       total 1780736K, used 668602K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8ceeb48,0x000000062cb00000)
 Metaspace       used 29961K, capacity 30441K, committed 30592K, reserved 1077248K
  class space    used 3019K, capacity 3164K, committed 3200K, reserved 1048576K
}
Event: 279.411 GC heap before
{Heap before GC invocations=7 (full 1):
 PSYoungGen      total 1090048K, used 1040997K [0x0000000715580000, 0x0000000797280000, 0x00000007c0000000)
  eden space 1003008K, 100% used [0x0000000715580000,0x0000000752900000,0x0000000752900000)
  from space 87040K, 43% used [0x0000000752900000,0x0000000754e19630,0x0000000757e00000)
  to   space 491520K, 0% used [0x0000000779280000,0x0000000779280000,0x0000000797280000)
 ParOldGen       total 1780736K, used 668602K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8ceeb48,0x000000062cb00000)
 Metaspace       used 30040K, capacity 30569K, committed 30848K, reserved 1077248K
  class space    used 3022K, capacity 3164K, committed 3200K, reserved 1048576K
Event: 279.480 GC heap after
Heap after GC invocations=7 (full 1):
 PSYoungGen      total 1646080K, used 37877K [0x0000000715580000, 0x00000007a5680000, 0x00000007c0000000)
  eden space 1154560K, 0% used [0x0000000715580000,0x0000000715580000,0x000000075bd00000)
  from space 491520K, 7% used [0x0000000779280000,0x000000077b77d640,0x0000000797280000)
  to   space 480768K, 0% used [0x000000075bd00000,0x000000075bd00000,0x0000000779280000)
 ParOldGen       total 1780736K, used 668610K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cf0b48,0x000000062cb00000)
 Metaspace       used 30040K, capacity 30569K, committed 30848K, reserved 1077248K
  class space    used 3022K, capacity 3164K, committed 3200K, reserved 1048576K
}
Event: 284.083 GC heap before
{Heap before GC invocations=8 (full 1):
 PSYoungGen      total 1646080K, used 1192437K [0x0000000715580000, 0x00000007a5680000, 0x00000007c0000000)
  eden space 1154560K, 100% used [0x0000000715580000,0x000000075bd00000,0x000000075bd00000)
  from space 491520K, 7% used [0x0000000779280000,0x000000077b77d640,0x0000000797280000)
  to   space 480768K, 0% used [0x000000075bd00000,0x000000075bd00000,0x0000000779280000)
 ParOldGen       total 1780736K, used 668610K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cf0b48,0x000000062cb00000)
 Metaspace       used 30540K, capacity 31006K, committed 31104K, reserved 1077248K
  class space    used 3032K, capacity 3196K, committed 3200K, reserved 1048576K
Event: 285.139 GC heap after
Heap after GC invocations=8 (full 1):
 PSYoungGen      total 1635328K, used 37969K [0x0000000715580000, 0x00000007b3080000, 0x00000007c0000000)
  eden space 1154560K, 0% used [0x0000000715580000,0x0000000715580000,0x000000075bd00000)
  from space 480768K, 7% used [0x000000075bd00000,0x000000075e2147e0,0x0000000779280000)
  to   space 464896K, 0% used [0x0000000796a80000,0x0000000796a80000,0x00000007b3080000)
 ParOldGen       total 1780736K, used 668618K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cf2b48,0x000000062cb00000)
 Metaspace       used 30540K, capacity 31006K, committed 31104K, reserved 1077248K
  class space    used 3032K, capacity 3196K, committed 3200K, reserved 1048576K
}
Event: 285.558 GC heap before
{Heap before GC invocations=9 (full 1):
 PSYoungGen      total 1635328K, used 1192529K [0x0000000715580000, 0x00000007b3080000, 0x00000007c0000000)
  eden space 1154560K, 100% used [0x0000000715580000,0x000000075bd00000,0x000000075bd00000)
  from space 480768K, 7% used [0x000000075bd00000,0x000000075e2147e0,0x0000000779280000)
  to   space 464896K, 0% used [0x0000000796a80000,0x0000000796a80000,0x00000007b3080000)
 ParOldGen       total 1780736K, used 668618K [0x00000005c0000000, 0x000000062cb00000, 0x0000000715580000)
  object space 1780736K, 37% used [0x00000005c0000000,0x00000005e8cf2b48,0x000000062cb00000)
 Metaspace       used 30543K, capacity 31006K, committed 31104K, reserved 1077248K
  class space    used 3033K, capacity 3196K, committed 3200K, reserved 1048576K

Deoptimization events (10 events):
Event: 237.110 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff642ba100 method=oracle.jdbc.driver.OracleSql.computeBasicInfo(Ljava/lang/String;)V @ 103
Event: 237.111 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff642b16a8 method=oracle.jdbc.driver.OracleSql.handleODBC(Loracle/jdbc/driver/OracleSql$ParseMode;)V @ 69
Event: 237.111 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff63bdd0a0 method=oracle.sql.CharacterSet.convertJavaCharsToUTFBytes([CI[BII)I @ 38
Event: 237.448 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff6409ad9c method=com.stock.businesslogic.global.StockManager.getStockInformationPickExchId(Ljava/util/HashMap;)Ljava/util/Vector; @ 35
Event: 280.464 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff648ddc24 method=com.stock.businesslogic.controller.LoginService.getStkCodeInfo3()V @ 128
Event: 280.465 Thread 0x000000010111e000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff6446b63c method=com.stock.framework.communication.DataPack.pack()S @ 112
Event: 280.776 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff63a496d4 method=com.stock.framework.communication.DataUnit.replaceAll(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; @ 16
Event: 280.778 Thread 0x000000010065a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff646fcb14 method=com.stock.framework.communication.DataUnit.replaceAll(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; @ 16
Event: 280.824 Thread 0x0000000101a4e800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff63db82d8 method=com.stock.framework.communication.ConvertDataType.toInt([B)I @ 51
Event: 280.828 Thread 0x0000000101a4e800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xffffffff63c22660 method=com.stock.framework.communication.DataUnit.replaceAll(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; @ 16

Internal exceptions (10 events):
Event: 283.906 Thread 0x0000000101459800 Exception  (0x00000007155aaa28) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 283.907 Thread 0x0000000101459800 Exception  (0x00000007155aba40) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 284.043 Thread 0x0000000101459800 Exception  (0x00000007155aca80) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.146 Thread 0x0000000101459800 Exception  (0x000000071626c320) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.152 Thread 0x0000000100b00000 Exception  (0x00000007161c1078) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.252 Thread 0x0000000101459800 Exception  (0x000000071626d338) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.415 Thread 0x0000000101459800 Exception  (0x000000071626e350) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.479 Thread 0x0000000101459800 Exception  (0x000000071626f368) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.480 Thread 0x0000000101459800 Exception  (0x00000007162703a8) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 285.503 Thread 0x0000000101459800 Exception  (0x00000007162713c0) thrown at [/HUDSON/workspace/8-2-build-solaris-sparcv9/jdk8u141/9370/hotspot/src/share/vm/prims/jni.cpp, line 709]

Events (10 events):
Event: 285.149 Executing VM operation: RevokeBias done
Event: 285.149 Executing VM operation: RevokeBias
Event: 285.149 Executing VM operation: RevokeBias done
Event: 285.149 Executing VM operation: RevokeBias
Event: 285.150 Executing VM operation: RevokeBias done
Event: 285.150 Executing VM operation: RevokeBias
Event: 285.151 Executing VM operation: RevokeBias done
Event: 285.151 Executing VM operation: RevokeBias
Event: 285.152 Executing VM operation: RevokeBias done
Event: 285.557 Executing VM operation: ParallelGCFailedAllocation


Dynamic libraries:
0x0000000100000000 	/usr/jdk/instances/jdk1.8.0_141/jre/bin/java
0xffffffff74300000 	/usr/jdk/instances/jdk1.8.0_141/jre/bin/../lib/sparcv9/jli/libjli.so
0xffffffff7effc000 	/lib/64/libthread.so.1
0xffffffff7edfa000 	/lib/64/libdl.so.1
0xffffffff7eb00000 	/lib/64/libc.so.1
0xffffffff72f00000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/server/libjvm.so
0xffffffff7e900000 	/lib/64/libsocket.so.1
0xffffffff7b2fe000 	/usr/lib/64/libsched.so.1
0xffffffff7d500000 	/lib/64/libm.so.1
0xffffffff72d00000 	/usr/lib/64/libCrun.so.1
0xffffffff7b2fc000 	/lib/64/libdoor.so.1
0xffffffff72b00000 	/usr/lib/64/libdemangle.so.1
0xffffffff7d700000 	/lib/64/libkstat.so.1
0xffffffff7e300000 	/lib/64/libm.so.2
0xffffffff7e100000 	/lib/64/libnsl.so.1
0xffffffff7bf00000 	/lib/64/libmd.so.1
0xffffffff7dd00000 	/lib/64/libcryptoutil.so.1
0xffffffff7d300000 	/lib/64/libmp.so.2
0xffffffff72900000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libverify.so
0xffffffff72700000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libjava.so
0xffffffff75600000 	/lib/64/libscf.so.1
0xffffffff75400000 	/lib/64/libuutil.so.1
0xffffffff7ca00000 	/lib/64/libgen.so.1
0xffffffff75200000 	/lib/64/libnvpair.so.1
0xffffffff72300000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libzip.so
0xffffffff25100000 	/usr/lib/locale/zh_CN.GBK/sparcv9/zh_CN.GBK.so.3
0xffffffff24e00000 	/usr/lib/locale/zh_CN.GBK/sparcv9/methods_zh_CN.GBK.so.3
0xffffffff1fa00000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libj2ucrypto.so
0xffffffff1f700000 	/usr/lib/64/libsoftcrypto.so
0xffffffff1f500000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libj2pkcs11.so
0xffffffff1f200000 	/usr/lib/sparcv9/libpkcs11.so
0xffffffff1f000000 	/usr/lib/security/sparcv9/pkcs11_kernel.so
0xffffffff1ed00000 	/usr/lib/security/sparcv9/pkcs11_softtoken.so
0xffffffff1ea00000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libsunec.so
0xffffffff1e200000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libnet.so
0xffffffff1e000000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libnio.so
0xffffffff7edf4000 	/lib/64/librt.so.1
0xffffffff1de00000 	/lib/64/libsendfile.so.1
0xffffffff1c800000 	/usr/jdk/instances/jdk1.8.0_141/jre/lib/sparcv9/libmanagement.so

VM Arguments:
jvm_args: -Dsysname=AppServer -Xmx8192m -XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -Dfile.encoding=gbk 
java_command: AppServer
java_class_path (initial): .:/export/home/optuser/lib/itext-asian-5.2.0.jar:/export/home/optuser/lib/javax.mail-1.5.6.jar:/export/home/optuser/lib/ojalgo-33.0.jar:/export/home/optuser/lib/itext5-itextpdf-5.5.12.jar:/export/home/optuser/lib/commons-configuration-1.10.jar:/export/home/optuser/lib/AppDataPack.jar:/export/home/optuser/lib/mina-core-2.0.21.jar:/export/home/optuser/lib/jeromq.jar:/export/home/optuser/lib/activation.jar:/export/home/optuser/lib/commons-logging.jar:/export/home/optuser/lib/snakeyaml-1.24.jar:/export/home/optuser/lib/protostuff-runtime-1.6.0.jar:/export/home/optuser/lib/sqljep-0.2.jar:/export/home/optuser/lib/jcl-over-slf4j-1.7.13.jar:/export/home/optuser/lib/app_notification.jar:/export/home/optuser/lib/disruptor-3.4.2.jar:/export/home/optuser/lib/commons-digester3-3.2.jar:/export/home/optuser/lib/mybatis-generator-core-1.3.7.jar:/export/home/optuser/lib/offer-comm-2.3.1-RELEASE.jar:/export/home/optuser/lib/commons-math3-3.0.jar:/export/home/optuser/lib/animal-sniffer-annotations-1.14.jar:/export/home/optuser/lib/json-20140107.jar:/export/home/optuser/lib/juniversalchardet-1.0.3.jar:/export/home/optuser/lib/protostuff-api-1.6.0.jar:/export/home/optuser/lib/commons-lang-2.6.jar:/export/home/optuser/lib/IMIXClientAPI.jar:/export/home/optuser/lib/oms-comm-common-1.3-RELEASE.jar:/export/home/optuser/lib/error_prone_annotations-2.0.18.jar:/export/home/optuser/lib/guava-23.0.jar:/export/home/optuser/lib/quickfixj-all-1.6.4.jar:/export/home/optuser/lib/j2objc-annotations-1.1.jar:/export/home/optuser/lib/IMIXProtocol.jar:/export/home/optuser/lib/protostuff-collectionschema-1.6.0.jar:/export/home/optuser/lib/protobuf-java-2.6.1.jar:/export/home/optuser/lib/commons-beanutils-1.8.3.jar:/export/home/optuser/lib/activemq-all-5.10.2.jar:/export/home/optuser/lib/mybatis-3.4.6.jar:/export/home/optuser/lib/commons-lang3-3.4.jar:/export/home/optuser/lib/jsr305-1.3.9.jar:/export/home/optuser/lib/ojdbc6.jar:/export/home/optuser/lib/communication.jar:/export/
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=.:/export/home/optuser/conf
PATH=.:/usr/bin:/usr/sbin:/bin:/usr/ucb:/usr/bin:/export/home/optuser/bin:/usr/local/bin
LD_LIBRARY_PATH=/etc/emc/rsa/cst/lib
SHELL=/bin/csh
DISPLAY=192.168.70.165:0.0

Signal Handlers:
SIGSEGV: [libjvm.so+0xdf7770], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xdf7770], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xb88f50], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xb88f50], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xb88f50], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xb88f50], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: [libjvm.so+0xb82af0], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART
SIGHUP: [libjvm.so+0xb82af0], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: [libjvm.so+0xb82af0], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART
SIG39: [libjvm.so+0xb88fa0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_SIGINFO
SIG40: [libjvm.so+0xb88f50], sa_mask[0]=11111111011111111111110111111111, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:                            Oracle Solaris 11.1 SPARC
  Copyright (c) 1983, 2012, Oracle and/or its affiliates.  All rights reserved.
                           Assembled 19 September 2012

uname:SunOS 5.11 11.1 sun4v
  (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:2.18 2.11 2.15

CPU:total 256 (initial active 256) v9, popc, vis1, vis2, vis3, blk_init, cbcond, aes, sha1, sha256, sha512, sun4v, niagara_plus

Memory: 8k page, physical 267386880k(41956136k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.141-b15) for solaris-sparc JRE (1.8.0_141-b15), built on Jul 12 2017 11:41:16 by "java_re" with Sun Studio 12u1

time: Thu Aug  8 11:09:13 2019
elapsed time: 299 seconds (0d 0h 4m 59s)

其中

列出了异常信息、可能原因以及可能的解决方法:

逐个翻译如下:

可能的原因:

系统没有物理RAM或交换空间

在32位模式下，进程大小限制

可能的解决方案:

减少系统的内存负载

增加物理内存或交换空间

检查交换备份存储是否已满

在64位操作系统上使用64位Java

减少Java堆大小(-Xmx/-Xms)

当时的第一反应就是先减小JVM内存分配的参数,但是之前分配的是8G,减少到4G之后依然不行,继续找原因

服务器内存情况

现在的free swap是60G,当时是30G,也就是说检查交换备份存储是否已满这项是有可能的,不过不是说看到满了就直接进行swap相关处理

1.增加物理内存的大小，减少swap使用的可能性；
2.杀掉使用swap最多的进程；
3.通过创建swapfile分区，来增加swap空间；
4.主动swapout:swapoff -a 执行结束后，再执行swapon -a。

满是有原因的,不能直接去释放空间,下次起来之后还是这么大,下图 

在没改线程的栈内存大小-Xss参数之前 Capacity是99%,现在恢复到1%;

查看到/tmp下全是这个项目的临时文件,但是一直删不掉,怀疑有线程卡死,

但是项目可以正常停止,就比较奇怪;

如果是通过reboot虚拟机的话,应该是可以解决这个问题的,因为会释放出一大部分的swap空间,不过服务器上的服务太多要用,马上重启不现实,所以选择了修改-Xss为256K,启动完成;

因为项目中起的线程特别多,线程申请内存分为横向和纵向,横向的是方法调用的深度,如果超出虚拟机所允许的最大深度,将抛出StackOverflowError异常,当分配新线程的时候,如果栈的大小不够用,会横向扩展栈,当无法申请足够的内存的时候会抛出OOM,这次遇到的问题属于第二种情况,启动的时候开辟的新线程不足以达到扩展栈的程度,但是差不多到了临界值,运行之后直接OOM,至于不能分配的原因,认为是碎片化太多,不足以支撑一次分配,又无法进行碎片整理,所以OOM