文章均遷移到我的主頁 http://zhenlianghe.com

my github: https://github.com/LynnHo

tf.Variable

1. 變量域？
   • tf.name_scope和tf.variable_scope都會對tf.Variable生成的變量域造成影響，tf.variable_scope中的reuse參數對tf.Variable沒有影響（本質上是因為tf.Variable受到了tf.variable_scope中同時創建的tf.name_scope的影響）
2. 重名？
   • 當變量名相同的時候，tf會自動打上序號

     with tf.name_scope('s'):    # or tf.variable_scope('s')
         a = tf.Variable(initial_value=10, name='a')
         b = tf.Variable(initial_value=10, name='a')
         print(a.name)
         print(b.name)
     
     [out]
     s/a:0
     s/a_1:0
     

3. 初始化？
   • tf.Variable是用一個tensor來初始化的，

     a = tf.Variable(initial_value=[1, 2])
     b = tf.Variable(initial_value=tf.constant([1, 2]))
     c = tf.Variable(initial_value=tf.random_uniform(shape=(1, 2)))
     d = tf.Variable(initial_value=tf.zeros_initializer()(shape=(1, 2), dtype=tf.int64))
     e = tf.Variable(initial_value=slim.xavier_initializer()(shape=(1, 2)))
     

   • tf.zeros_initializer()返回的是一個對象，對象對應的類有相應的call函數，這個call函數負責產生一個相應類型的tensor
   • slim.xavier_initializer()則返回的是一個函數，調用這個函數能夠產生一個相應類型的tensor
4. 變量共享？
   • 用生成的變量去干不同的事情不就共享了嘛
   • tf.Variable產生的變量不能用tf.variable_scope的reuse設置共享，否則會報錯

tf.get_variable

1. 變量域？
   • tf.get_variable產生的變量只會受到tf.variable_scope的影響，不受tf.name_scope的影響

     with tf.name_scope('s'):
         a = tf.get_variable(name='a', shape=(10, 10))
     with tf.variable_scope('s'):
         b = tf.get_variable(name='a', shape=(10, 10))
     print(a.name)
     print(b.name)
     
     [out]
     a:0
     s/a:0
     

2. 重名？變量共享？
   • 在同一個域下，重名是會報錯的。

     with tf.variable_scope('s'):
         a = tf.get_variable(name='a', shape=(10, 10))
         b = tf.get_variable(name='a')
     
     [out]
     ValueError: Variable s/a already exists, disallowed. Did you mean to set reuse=True in VarScope?
     

     • 可以在需要復用變量之前改變scope的reuse狀態

       with tf.variable_scope('s') as s:
           a = tf.get_variable(name='a', shape=(10, 10))
           s.reuse_variables()
           b = tf.get_variable(name='a')
       print(a == b)
       
       [out]
       True
       

     • 也可以設置tf.variable_scope的reuse參數為True來復用已經定義過的同名變量，但如果沒定義過而設置reuse=True也是會報錯的

       with tf.variable_scope('s', reuse=True):
           a = tf.get_variable(name='a')
       
       [out]
       ValueError: Variable s/a does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?
       

       with tf.variable_scope('s'):
           a = tf.get_variable(name='a', shape=(10, 10))
       with tf.variable_scope('s', reuse=True):
           b = tf.get_variable(name='a')
       print(a == b)
       
       [out]
       True
       

3. 初始化？

   a = tf.get_variable(name='a', shape=(1, 2), initializer=tf.constant_initializer([1, 2]), dtype=tf.int64)
   b = tf.get_variable(name='b', shape=(1, 2), initializer=tf.random_uniform_initializer())
   c = tf.get_variable(name='c', shape=(1, 2), initializer=tf.zeros_initializer(), dtype=tf.int64)
   d = tf.get_variable(name='d', shape=(1, 2), initializer=slim.xavier_initializer())
   

   可見，只要給定相應的initializer就可以了，但是要注意dtype的設置，只有設置tf.get_variable的dtype參數才能正確生效，設置initializer的dtype參數是無效的

slim層里面的variable

1. 注意，slim里面的variable生成機制實際上是和tf.get_variable是一樣的，所以特性也是一樣的，比如說變量域只受tf.variable_scope影響而不受tf.name_scope影響
2. 層的命名
   1. 自動命名變量域，每一個slim層都有一個scope參數，如果不設置這個參數(默認為None)，會有以下兩種情況
      • 在同一個上下問管理器中(with tf.variable_scope('s'):)定義層，slim會按生成順序自動命名變量域(本質上就是因為slim層里面利用了with tf.variable_scope(None, default_name, ...)的機制)

        x = tf.placeholder(tf.float32, shape=[None, 10])
        with tf.variable_scope('s'):
            a = slim.fully_connected(x, 10)
            b = slim.fully_connected(a, 10)
        for var in tf.trainable_variables():
            print(var.name)
        
        [out]  
        s/fully_connected/weights:0
        s/fully_connected/biases:0
        s/fully_connected_1/weights:0
        s/fully_connected_1/biases:0
        

      • 在不同的上下問管理器中定義層，但域名是一樣的，slim將報錯
        • 報錯的例子

          x = tf.placeholder(tf.float32, shape=[None, 10])
          with tf.variable_scope('s'):
              a = slim.fully_connected(x, 10)
          with tf.variable_scope('s'):
              b = slim.fully_connected(x, 10)
          for var in tf.trainable_variables():
              print(var.name)
          
          [out]                
          Variable s/fully_connected/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

        • 報錯的例子

          x = tf.placeholder(tf.float32, shape=[None, 10])
          with tf.variable_scope('s'):
              a = slim.layer_norm(x)
          with tf.variable_scope('s'):
              b = slim.layer_norm(x)
          for var in tf.trainable_variables():
              print(var.name)
          
          [out]
          ValueError: Variable s/LayerNorm/beta already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

   2. 手動命名變量域，顧名思義。需要注意以下情況
      • 在同一個域中，如果兩個層設置的scope參數是同一個名字，那么slim將報錯

        • 報錯的例子

          x = tf.placeholder(tf.float32, shape=[None, 2])
          with tf.variable_scope('s'):
              a = slim.fully_connected(x, 2, scope='a')
          with tf.variable_scope('s'): # 在這個例子中，這一行可有可無，效果相同
              b = slim.fully_connected(x, 2, scope='a')
          for var in tf.trainable_variables():
              print(var.name)
          
          [out]
          Variable s/a/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

        • 報錯的例子

          x = tf.placeholder(tf.float32, shape=[None, 10])
          with tf.variable_scope('s'):
              a = slim.layer_norm(x, scope='a')
          with tf.variable_scope('s'): # 在這個例子中，這一行可有可無，效果相同
              b = slim.layer_norm(x, scope='a')
          
          [out]
          ValueError: Variable s/a/beta already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

3. 那么最合理的變量共享方式？？？實際上是和tf.get_variable定義的變量的共享機制是一樣的，用reuse參數

   x = tf.placeholder(tf.float32, shape=[None, 2])
   with tf.variable_scope('s'):
       y = slim.fully_connected(x, 2, weights_initializer=tf.random_normal_initializer())
       a = slim.layer_norm(y)
   with tf.variable_scope('s', reuse=True):
       y = slim.fully_connected(x, 2)
       b = slim.layer_norm(y)
   for var in tf.trainable_variables():
       print(var.name)
   
   sess = tf.Session()
   sess.run(tf.global_variables_initializer())
   print(sess.run(a, feed_dict={x: [[1, 7]]}))
   print(sess.run(b, feed_dict={x: [[1, 7]]}))
   
   [out] 
   s/fully_connected/weights:0
   s/fully_connected/biases:0
   s/LayerNorm/beta:0
   s/LayerNorm/gamma:0
   [[-1.          1.00000012]]
   [[-1.          1.00000012]]