集成剪枝分类算法的Bagging与Adaboost示例

Bagging (Bootstrap Aggregation)

Pruning Classification is one of the simplest classification algorithms. It works just like if-then. However, when aggregating a lot of prunnings we are able to create a powerful classifier.

The process of Bagging based on pruning is really simple but not trivial:

1. For j=1,…,b j = 1 , … , b ,
   
   1. Pick up m m samples from a sample set with n n samples {(xi,yi)}ni=1 { ( x i , y i ) } i = 1 n . Repeating is permitted. Then we get a new sample set.
   2. Train the pruning classifier ψj ψ j with the new sample set.
2. For all of the pruning classifiers {ψj}bj=1 { ψ j } j = 1 b , calculate their average and get f f :
   f(x)←1b∑j=1bψj(x) f ( x ) ← 1 b ∑ j = 1 b ψ j ( x )

n=50; x=randn(n,2); 
y=2*(x(:,1)>x(:,2))-1;
b=5000; a=50; Y=zeros(a,a);
X0=linspace(-3,3,a);
[X(:,:,1), X(:,:,2)]=meshgrid(X0);

for j=1:b
    db=ceil(2*rand); r=ceil(n*rand(n,1));
    xb=x(r,:); yb=y(r); [xs,xi]=sort(xb(:,db));
    el=cumsum(yb(xi)); eu=cumsum(yb(xi(end:-1:1)));
    e=eu(end-1:-1:1)-el(1:end-1);
    [em,ei]=max(abs(e)); c=mean(xs(ei:ei+1));
    s=sign(e(ei)); Y=Y+sign(s*(X(:,:,db)-c))/b;
end

figure(1); clf; hold on; axis([-3,3,-3,3]);
colormap([1 0.7 1; 0.7 1 1]);
contourf(X0,X0,sign(Y));
plot(x(y==1,1),x(y==1,2),'bo');

Adaboost (Adaptive Boosting) Classifier

Boosting algorithms try to aggregate a couple of poor classifiers by order to make a powerful one. They assign weights to every labeled sample. When one of the poor classifier fails to correctly classify a sample, the weight of that sample is boosted. Then it tries another poor classifier.
Let’s take Adaboost and Pruning algorithms for example:

1. For the training set {(xi,yi)}ni=1 { ( x i , y i ) } i = 1 n , initialize their weights {wi}ni=1 { w i } i = 1 n as 1/n 1 / n . And let f←0 f ← 0 .
2. For j=1,…,b j = 1 , … , b :
   
   1. Based on current sample weights {wi}ni=1 { w i } i = 1 n , pick up the classifier with the smallest weighted error rate R R :
      φj=argminφR(φ),R(φ)=∑j=1nwi2(1−φ(xi)yi) φ j = arg ⁡ min φ R ( φ ) , R ( φ ) = ∑ j = 1 n w i 2 ( 1 − φ ( x i ) y i )
   2. Calculate the weight of classifier φj φ j :
      θj=12log1−R(φj)R(φj) θ j = 1 2 log ⁡ 1 − R ( φ j ) R ( φ j )
   3. Update the aggregated classifier f f :
      f←f+θjφj f ← f + θ j φ j
   4. Update the weights of samples {wi}ni=1 { w i } i = 1 n :
      wi←exp(−f(xi)yi)∑nk=1exp(−f(xk)yk),∀i=1,2,…,n w i ← exp ⁡ ( − f ( x i ) y i ) ∑ k = 1 n exp ⁡ ( − f ( x k ) y k ) , ∀ i = 1 , 2 , … , n

n=50; x=randn(n,2); 
y=2*(x(:,1)>x(:,2))-1;
b=5000; a=50; Y=zeros(a,a);
yy=zeros(size(y)); w=ones(n,1)/n;
X0=linspace(-3,3,a);
[X(:,:,1), X(:,:,2)]=meshgrid(X0);

for j=1:b
    wy=w.*y; d=ceil(2*rand); [xs,xi]=sort(x(:,d)); 
    el=cumsum(wy(xi)); eu=cumsum(wy(xi(end:-1:1)));
    e=eu(end-1:-1:1)-el(1:end-1);
    [em,ei]=max(abs(e)); c=mean(xs(ei:ei+1));s=sign(e(ei));
    yh=sign(s*(x(:,d)-c)); R=w'*(1-yh.*y)/2;
    t=log((1-R)/R)/2; yy=yy+yh*t; w=exp(-yy.*y); w=w/sum(w);
    Y=Y+sign(s*(X(:,:,d)-c))*t;
end

figure(1); clf; hold on; axis([-3,3,-3,3]);
colormap([1 0.7 1; 0.7 1 1]);
contourf(X0,X0,sign(Y));
plot(x(y==1,1),x(y==1,2),'bo');
plot(x(y==-1,1),x(y==-1,2),'rx');