Disruptor笔记(五)-FAQ

1.  MultiThreadedClaimStrategy.java中pendingPublication的用处：

参考：http://blogs.lmax.com/

How does this apply to our concurrentsequencing problem? We could allow threads to continue to make progress whilewaiting for other threads to catch by maintaining a list of sequences that arepending publication. If a thread tries to publish a sequence that is greaterthan 1 higher than current cursor (i.e. it would need to wait for anotherthread to publish its sequence) it could place that sequence into the pendinglist and return. The thread that is currently running behind would publish itsown sequence, then check the pending list and publish those sequences beforeexiting.

 

2.  Sequencer.java中gatingSequences的作用?

主要是防止发布申请序列越界。

 /**
      * Set the sequences that will gatepublishers to prevent the buffer wrapping.
      *
      * This method must be called prior toclaiming sequences otherwise
      * a NullPointerException will be thrown.
      *
      * @param sequences to be to be gatedon.
      */
     public void setGatingSequences(final Sequence...sequences)
     {
         this.gatingSequences = sequences;
     }

 

例如：MultiThreadedClaimStrategy.java中

 private void waitForFreeSlotAt(final long sequence, final Sequence[] dependentSequences, final MutableLongminGatingSequence)
     {
         final long wrapPoint = sequence - bufferSize;//可以预先放满整个BufferSize
         if (wrapPoint > minGatingSequence.get())//说明minGatingSequence位置的event还没有被消费掉。它的值为minSequence
         {
             long minSequence;
             while (wrapPoint > (minSequence = getMinimumSequence(dependentSequences)))//说明eventProcessor的处理速度跟不上，minSequence为 eventProcessor.sequence中最小的
             {
                 LockSupport.parkNanos(1L);
             }

             minGatingSequence.set(minSequence);//重置minGatingSequence
         }
 }

 

3.  BatchEventProcessor如何更新sequence来满足eventPublisher继续发布event？

 /**
      * It is ok to have another thread rerunthis method after a halt().
      */
     @Override
     public void run()
     {
         if (!running.compareAndSet(false, true))//置运行标志为true
         {
             throw new IllegalStateException("Thread is already running");
         }
         sequenceBarrier.clearAlert();//重置alert标志

         notifyStart();//调用所有eventHandler的onStart()

         T event = null;
         long nextSequence = sequence.get() + 1L;//需要等待nextSequence对应的消息到达
         while (true)
         {
             try
             {
                 final long availableSequence = sequenceBarrier.waitFor(nextSequence);//参考下面BlockingWaitStrategy的代码说明,调用BlockingWaitStrategy.java中的waitFor
                 while (nextSequence <= availableSequence)
                 {
                     event = ringBuffer.get(nextSequence);
                     eventHandler.onEvent(event, nextSequence,nextSequence == availableSequence);
                     nextSequence++;
                 }

                 sequence.set(nextSequence - 1L);//设置gate sequence，说明已经消费到sequence的位置（该位置已经消费event）
             }
             catch (final AlertException ex)
             {
                if (!running.get())
                {
                    break;
                }
             }
             catch (final Throwable ex)
             {
                 exceptionHandler.handleEventException(ex,nextSequence, event);
                 sequence.set(nextSequence);//如果有exceptionHandler,则继续，sequence递增。
                 nextSequence++;
             }
         }

         notifyShutdown();//调用所有的eventHandler的onShutdown()

         running.set(false);//置运行标志为false
     }

 

 

BlockingWaitStrategy.java中的

 @Override
     public long waitFor(final long sequence, final Sequence cursor, final Sequence[] dependents, final SequenceBarrierbarrier)//cursor 为ringbuffer的cursor sequence，表示当前发布到的序列位置
         throws AlertException,InterruptedException
     {
         long availableSequence;
         if((availableSequence = cursor.get()) < sequence)//如何申请消费序列大于当前发布到的序列，则等待新的消息发布。
         {
             lock.lock();
             try
             {
                 ++numWaiters;
                 while ((availableSequence = cursor.get())
                 {
                     barrier.checkAlert();
                     processorNotifyCondition.await();
                 }
             }
             finally
             {
                 --numWaiters;
                 lock.unlock();
             }
         }

         if (0 != dependents.length)
         {
             while ((availableSequence = getMinimumSequence(dependents))< sequence)//如果依赖的序列还没有消费到sequence位置，则等待。此场景为多个串行的sequenceBarrier。
             {
                 barrier.checkAlert();
             }
         }

         return availableSequence;
 }

 

4.  RingBuffer(Sequencer)如果发布的event超过BufferSize会出现什么情况？

在调用next时，如果超过buffersize，

 /**
      * Claim the next event in sequence forpublishing.
      *
      * @return the claimed sequence value
      */
     public long next()
     {
         if (null == gatingSequences)
         {
             throw new NullPointerException("gatingSequences must be set before claiming sequences");
         }

         return claimStrategy.incrementAndGet(gatingSequences);
 }

 

调用MultiThreadedClaimStrategy.java的incrementAndGet

@Override

  

 "code" class="java">"code" class="java">@Override
     public long incrementAndGet(final Sequence[] dependentSequences)
     {
         final MutableLong minGatingSequence = minGatingSequenceThreadLocal.get();
         waitForCapacity(dependentSequences, minGatingSequence);

         final long nextSequence = claimSequence.incrementAndGet();
 waitForFreeSlotAt(nextSequence, dependentSequences, minGatingSequence);//这里同2 Sequencer.java中gatingSequences的作用?
 的描述了,等待消费序列增加可以继续发布。

         return nextSequence;
     }







 "left">
 "left">
 "left">
 "left">5.  Sequence的无限增长超过Long.MAX_VALUE=(2<<63)-1怎么办？
 "left">从代码看，为了简化seqence的比较(头尾的判断逻辑)，一直递增，总会超过Long.MAX_VALUE,后续会出现负数，进而出现死循环代码。可以靠定期重启系统来解决。
 "left">Martin Flower的解释：This does mean that if they process a billion transactionsper second the counter will wrap in 292 years, causing some hell to breakloose. They have decided that fixing this is not a high priority.

 我的思路：
 类似halt的实现方式()，增加booleanisReset  和checkReset()
 "left">
 "left">由RingBuffer判断需要重置时来发出指令isReset = true,加一个CountDownLatch(n) n位1（RingBuffercursor）+其他eventProcessor，用来保证所有的sequence都重置, RingBuffer在next()控制，如果isReset为true,cursor&=indexMask(bufferSize-1),eventProcessor根据自身的waitStradegy来暂停,sequence&= indexMask(bufferSize-1),这里主要是和ringbuffer中的entry保持一致 。RingBuffer
   latch.wait(),等待所有sequence和自身的cursor更新完成。
 "left">
 "left">
 "left">6.  暂停处理后是否会抛弃未处理的消息？
 "left">
 "left">
 "code" class="java">/**
      * Calls {@link com.lmax.disruptor.EventProcessor#halt()} on all of the event processors created via this disruptor.
      */
     public void halt()
     {
         for (EventProcessorInfoeventprocessorInfo : eventProcessorRepository)
         {
            eventprocessorInfo.getEventProcessor().halt();//所有event processor halt(),并且激活alert标志，让sequenceBarrier.waitFor时抛出异常，跳出循环，结束线程处理逻辑。
         }
 }


 "left">这种方式比较粗暴，对于交易型应用没有问题，因为调用端会收到SocketException.
 "left">
 "left">个人觉得这个是可以优化的shutdown策略，如果要让消息继续处理完再停止，可以在停止publish event，做
 "left">
 "code" class="java">claimSequence = ringBuffer.next();
 oldEvent = ringBuffer.get(claimSequence);
 oldEvent.copy(expectedEvent);
 ringBuffer.publish(claimSequence);
 int checkSequence  = claimSequence;//暂停发布event后

 while(true) {
        long sequence = sequenceBarrier.waitFor(claimSequence, 30, TimeUnit.SECONDS);
 if (sequence> checkSequence  ) {
     checkSequence  = sequence;
 } else {
     //说明已经没有消费了。
    halt();
 }
 }


 "left">
 "left">7.  多线程并发访问RingBuffer来获取可以用的Sequence发布event，如果避免冲突？
 "left">MultiThreadedClaimStrategy.java中
 "left">a.有针对每个线程的守护序列
 "left">
 "code" class="java">private finalThreadLocal minGatingSequenceThreadLocal = newThreadLocal()
     {
         @Override
         protected MutableLong initialValue()
         {
             return new MutableLong(Sequencer.INITIAL_CURSOR_VALUE);
         }
 };


 "left">
 "left">private final PaddedAtomicLongclaimSequence = new PaddedAtomicLong(Sequencer.INITIAL_CURSOR_VALUE);//为atomic的，它保证原子性，因为是递增的，所以不会出现ABA现象。
 "left">
 "left">
 "left">8.  那个双生产者的情况下，当1号生产者提交失败时会怎么样？2号生产者会被一直阻止完成提交吗？那样的话ring buffer会出现死锁吗？
 "left">"">(disruptor开发者)生产者应该小心应付这种情况，因为它们确实需要注意它们阻塞了其他生产者。
 如果其中一个生产者因为坏掉了而提交失败，那你的整个系统会有比死锁更大的问题。但是，如果它是因为事务失败而提交失败的话，有两点需要留意：1）重试直到提交成功（其它生产者会被阻塞直到成功）或者2）向ring buffer提交一个“死消息”以使被阻塞的生产者可以继续，并且消费者会忽略这个死消息，仍然让序列号像预期的那样递增。
 "left">
 "left">9.你们对于耗时的消费者或生产者有应对策略吗（多生产者的情况）？
 "left">Martin Fowler的文章给出了关于LMAX架构的更多内容，但是我们目前还没有打算公开全部秘密;)
 "left">http://martinfowler.com/articles/lmax.html
 "left">(disruptor开发者)先不管你的架构怎么样，如果你的消费者一直都比生产者慢，那你的系统就应该加固。解决这问题的一个方法是你可以用两个消费者来做同样的事，其中一个用偶数序列，另一个用奇数的。这样你就可以有潜力并行地处理更多东西（当然你可以扩展2个以上）。
 如果你的生产者慢，这可能是你的设计中把很多事情放在一个地方做。我们的生产者通常做简单的事情－它们从一个地方拿数据然后插入到ring buffer。如果你的生产者在做大量的工作，你可以考虑一下把那个逻辑移到一个较早的消费者中。你的生产者可以向ring buffer写入原始数据，新的消费者可以从ring buffer读取原始数据，处理它，把处理过的数据写到Entry，拥有所有依赖它的下线消费者。这也许可以给你些并行处理这个事的建议。
 "left">