JVM源碼分析之消失的死鎖

問題描述

如果java層面發生了死鎖,當我們使用jstack命令的時候其實是可以將死鎖的信息給dump出來的,在dump結果的最后會有類似Found one Java-level deadlock:的關鍵字,接著會把發生死鎖的線程的堆棧及對應的同步鎖給打印出來,這次碰到一個系統就發生類似的問題,不過這個dump文檔里雖然提到了如下的死鎖信息:

Found one Java-level deadlock:
=============================
"worker-1-thread-121":
  waiting to lock monitor 0x00007f3758209dc8 (object 0x0000000764cd2b20, a java.util.concurrent.ConcurrentHashMap),
  which is held by "HSFBizProcessor-4-thread-4"
"HSFBizProcessor-4-thread-4":
  waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-5"
"HSFBizProcessor-4-thread-5":
  waiting to lock monitor 0x00007f3758253420 (object 0x00000007608e6fc8, a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-4"

但是我們在堆棧里搜索對應的鎖的時候并沒發現,也就是上面提到的

object 0x00000007608e6fc8 which is held by "HSFBizProcessor-4-thread-4"

我們在HSFBizProcessor-4-thread-4這個線程的堆棧里并沒有看到對應的持鎖信息。

附上線程dump詳情

Found one Java-level deadlock:
=============================
"worker-1-thread-121":
  waiting to lock monitor 0x00007f3758209dc8 (object 0x0000000764cd2b20, a java.util.concurrent.ConcurrentHashMap),
  which is held by "HSFBizProcessor-4-thread-4"
"HSFBizProcessor-4-thread-4":
  waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-5"
"HSFBizProcessor-4-thread-5":
  waiting to lock monitor 0x00007f3758253420 (object 0x00000007608e6fc8, a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-4"

Java stack information for the threads listed above:
===================================================
"worker-1-thread-121":
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)
    - waiting to lock <0x0000000764cd2b20> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:455)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:317)
    ......
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
"HSFBizProcessor-4-thread-4":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x000000076073ddc8> (a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.springframework.scripting.groovy.GroovyScriptFactory.executeScript(GroovyScriptFactory.java:238)
    ......
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
"HSFBizProcessor-4-thread-5":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x00000007608e6fc8> (a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)
    at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.createWithCustomLookup(MetaClassRegistry.java:127)
    at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.create(MetaClassRegistry.java:122)
    ......
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.

類加載的問題?

首先應該懷疑類加載的問題,因為我們看到導致死鎖的對象是一個classloader對象:

waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader)

然后我們再來分析下堆棧

HSFBizProcessor-4-thread-4

"HSFBizProcessor-4-thread-4":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x000000076073ddc8> (a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.springframework.scripting.groovy.GroovyScriptFactory.executeScript(GroovyScriptFactory.java:238)
    at org.springframework.scripting.groovy.GroovyScriptFactory.getScriptedObject(GroovyScriptFactory.java:185)

我這里只把關鍵的線程棧貼出來,從棧頂知道正在等一把鎖:

    - waiting to lock <0x000000076073ddc8> (a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader)

這把鎖的對象是一個ClassLoader對象,我們找到對應的代碼,確實存在synchronized的操作:

private Class<?> findLoadedClass(String classname) {
    if ((LOCK_CLASSNAME) || (this.isParallelClassLoader)) {
      boolean initialLock = lockClassName(classname);
      try {
        return this.classloader.publicFindLoaded(classname);
      } finally {
        if (initialLock)
          unlockClassName(classname);
      }
    }
    synchronized (this.classloader) {
      return this.classloader.publicFindLoaded(classname);
    }
  }

另外我們還知道它正在執行loadClass的動作,并且是從groovy調用來的,同樣找到對應的代碼:

protected Object executeScript(ScriptSource scriptSource, Class scriptClass)
    throws ScriptCompilationException
  {
    try
    {
      GroovyObject goo = (GroovyObject)scriptClass.newInstance();//line 238

      if (this.groovyObjectCustomizer != null)
      {
        this.groovyObjectCustomizer.customize(goo);
      }

      if ((goo instanceof Script))
      {
        return ((Script)goo).run();
      }

      return goo;
    }
    catch (InstantiationException ex)
    {
      throw new ScriptCompilationException(
        scriptSource, "Could not instantiate Groovy script class: " + scriptClass.getName(), ex);
    }
    catch (IllegalAccessException ex) {
      throw new ScriptCompilationException(
        scriptSource, "Could not access Groovy script constructor: " + scriptClass.getName(), ex);
    }
  }

執行到第238行的時候

GroovyObject goo = (GroovyObject)scriptClass.newInstance();//line 238

突然發現調用了

java.lang.ClassLoader.loadClass(ClassLoader.java:247)

而我們看到上面第238行的邏輯其實就是實例化一個對象,然后進行強轉,我們看看對應的字節碼:

 0: aload_2
 1: invokevirtual #164                // Method java/lang/Class.newInstance:()Ljava/lang/Object;
 4: checkcast     #168                // class groovy/lang/GroovyObject
 7: astore_3

其實就對應這么幾條字節碼指令,其實在jvm里當我們執行checkcast指令的時候會觸發類加載的動作:

void TemplateTable::checkcast() {
    ...
    call_VM(rax, CAST_FROM_FN_PTR(address, InterpreterRuntime::quicken_io_cc));
    ...
}

IRT_ENTRY(void, InterpreterRuntime::quicken_io_cc(JavaThread* thread))
  // Force resolving; quicken the bytecode
  int which = get_index_u2(thread, Bytecodes::_checkcast);
  constantPoolOop cpool = method(thread)->constants();
  // We'd expect to assert that we're only here to quicken bytecodes, but in a multithreaded
  // program we might have seen an unquick'd bytecode in the interpreter but have another
  // thread quicken the bytecode before we get here.
  // assert( cpool->tag_at(which).is_unresolved_klass(), "should only come here to quicken bytecodes" );
  klassOop klass = cpool->klass_at(which, CHECK);
  thread->set_vm_result(klass);
IRT_END

klassOop klass_at(int which, TRAPS) {
    constantPoolHandle h_this(THREAD, this);
    return klass_at_impl(h_this, which, CHECK_NULL);
}

klassOop constantPoolOopDesc::klass_at_impl(constantPoolHandle this_oop, int which, TRAPS) {
    ...
    klassOop k_oop = SystemDictionary::resolve_or_fail(name, loader, h_prot, true, THREAD);
    ...
}   

//SystemDictionary::resolve_or_fail最終會調用到下面這個方法
klassOop SystemDictionary::resolve_instance_class_or_null(Symbol* name, Handle class_loader, Handle protection_domain, TRAPS) {
  ...
  // Class is not in SystemDictionary so we have to do loading.
  // Make sure we are synchronized on the class loader before we proceed
  Handle lockObject = compute_loader_lock_object(class_loader, THREAD);
  check_loader_lock_contention(lockObject, THREAD);
  ObjectLocker ol(lockObject, THREAD, DoObjectLock);
  ...
  //此時會調用ClassLoader.loadClass來加載類了
  ...
}

Handle SystemDictionary::compute_loader_lock_object(Handle class_loader, TRAPS) {
  // If class_loader is NULL we synchronize on _system_loader_lock_obj
  if (class_loader.is_null()) {
    return Handle(THREAD, _system_loader_lock_obj);
  } else {
    return class_loader;
  }
}

SystemDictionary::resolve_instance_class_or_null這個方法非常關鍵了,在里面我們看到會獲取一把鎖ObjectLocker,其相當于我們java代碼里的synchronized關鍵字,而對象對應的是lockObject,這個對象是上面的SystemDictionary::compute_loader_lock_object方法返回的,從代碼可知只要不是bootstrapClassloader加載的類就會返回當前classloader對象,也就是說當我們在加載一個類的時候其實是會持有當前類加載對象的鎖的,在獲取了這把鎖之后就會調用ClassLoader.loadClass來加載類了。這其實就解釋了HSFBizProcessor-4-thread-4這個線程為什么持有了

object 0x00000007608e6fc8, a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader

這個類加載的鎖,不過遺憾的是因為這把鎖不是java層面來顯示加載的,因此我們在jstack線程dump的輸出里居然看不到這把鎖的存在.

HSFBizProcessor-4-thread-5

先上堆棧:

"HSFBizProcessor-4-thread-5":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x00000007608e6fc8> (a com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.alipay.cloudengine.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)

這個線程棧其實和之前那個線程差不多,只是等的鎖不一樣,另外觸發類加載的動作是Class.forName,獲取大家也猜到了,其實是在下面兩行堆棧之間同樣獲取了一把類加載器的鎖

    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)

這里的代碼我也不細貼了,最終調用的jvm里的方法都是一樣的,獲取鎖的邏輯也是一樣的

總結

想象下這種場景,兩個線程分別使用不同的classloader對兩個類進行類加載,然而由于osgi類加載機制的緣故,在loadClass過程中可能會委托給別的classloader去加載,而正巧,這兩個線程在獲取當前classloader的鎖之后,然后分別委托對方的classloader去加載,可以看到文章開頭列的那個findLoadedClass方法,而synchronized的那個classloader正好是對方的classloader,從而導致了死鎖

個人微信公眾號

gzh.jpg
最后編輯于
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。
  • 序言:七十年代末,一起剝皮案震驚了整個濱河市,隨后出現的幾起案子,更是在濱河造成了極大的恐慌,老刑警劉巖,帶你破解...
    沈念sama閱讀 227,967評論 6 531
  • 序言:濱河連續發生了三起死亡事件,死亡現場離奇詭異,居然都是意外死亡,警方通過查閱死者的電腦和手機,發現死者居然都...
    沈念sama閱讀 98,273評論 3 415
  • 文/潘曉璐 我一進店門,熙熙樓的掌柜王于貴愁眉苦臉地迎上來,“玉大人,你說我怎么就攤上這事?!?“怎么了?”我有些...
    開封第一講書人閱讀 175,870評論 0 373
  • 文/不壞的土叔 我叫張陵,是天一觀的道長。 經常有香客問我,道長,這世上最難降的妖魔是什么? 我笑而不...
    開封第一講書人閱讀 62,742評論 1 309
  • 正文 為了忘掉前任,我火速辦了婚禮,結果婚禮上,老公的妹妹穿的比我還像新娘。我一直安慰自己,他們只是感情好,可當我...
    茶點故事閱讀 71,527評論 6 407
  • 文/花漫 我一把揭開白布。 她就那樣靜靜地躺著,像睡著了一般。 火紅的嫁衣襯著肌膚如雪。 梳的紋絲不亂的頭發上,一...
    開封第一講書人閱讀 55,010評論 1 322
  • 那天,我揣著相機與錄音,去河邊找鬼。 笑死,一個胖子當著我的面吹牛,可吹牛的內容都是我干的。 我是一名探鬼主播,決...
    沈念sama閱讀 43,108評論 3 440
  • 文/蒼蘭香墨 我猛地睜開眼,長吁一口氣:“原來是場噩夢啊……” “哼!你這毒婦竟也來了?” 一聲冷哼從身側響起,我...
    開封第一講書人閱讀 42,250評論 0 288
  • 序言:老撾萬榮一對情侶失蹤,失蹤者是張志新(化名)和其女友劉穎,沒想到半個月后,有當地人在樹林里發現了一具尸體,經...
    沈念sama閱讀 48,769評論 1 333
  • 正文 獨居荒郊野嶺守林人離奇死亡,尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內容為張勛視角 年9月15日...
    茶點故事閱讀 40,656評論 3 354
  • 正文 我和宋清朗相戀三年,在試婚紗的時候發現自己被綠了。 大學時的朋友給我發了我未婚夫和他白月光在一起吃飯的照片。...
    茶點故事閱讀 42,853評論 1 369
  • 序言:一個原本活蹦亂跳的男人離奇死亡,死狀恐怖,靈堂內的尸體忽然破棺而出,到底是詐尸還是另有隱情,我是刑警寧澤,帶...
    沈念sama閱讀 38,371評論 5 358
  • 正文 年R本政府宣布,位于F島的核電站,受9級特大地震影響,放射性物質發生泄漏。R本人自食惡果不足惜,卻給世界環境...
    茶點故事閱讀 44,103評論 3 347
  • 文/蒙蒙 一、第九天 我趴在偏房一處隱蔽的房頂上張望。 院中可真熱鬧,春花似錦、人聲如沸。這莊子的主人今日做“春日...
    開封第一講書人閱讀 34,472評論 0 26
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽。三九已至,卻和暖如春,著一層夾襖步出監牢的瞬間,已是汗流浹背。 一陣腳步聲響...
    開封第一講書人閱讀 35,717評論 1 281
  • 我被黑心中介騙來泰國打工, 沒想到剛下飛機就差點兒被人妖公主榨干…… 1. 我叫王不留,地道東北人。 一個月前我還...
    沈念sama閱讀 51,487評論 3 390
  • 正文 我出身青樓,卻偏偏與公主長得像,于是被迫代替她去往敵國和親。 傳聞我的和親對象是個殘疾皇子,可洞房花燭夜當晚...
    茶點故事閱讀 47,815評論 2 372

推薦閱讀更多精彩內容