首先，要明白Cart生成算法。Cart生成算法的核心是以基尼系數(Gini Index)最小化為準則生成分類樹。理解下Gini Index，它用來衡量Pure程度，即一個節點中包含y因變量值的差異程度。Gini Index越小，說明y的值越一致，分類效果好，選擇這樣的特征作為節點，樹的效率才高。

Cart算法的基本思路（遞歸過程）：
Step 1: 選定training data，遍歷每一個特征A，對每個特征A可取的值a，根據A=a測試是否劃分為兩部分，并計算Gini Index。

Step 2: 在step 1中計算得到的Gini Index中，選擇最小的Gini Index對應的A=a作為最有特征與最優切分點，由此training data被分配到了兩個子節點中。

Step 3: 重復以上步驟，直到滿足停止條件。

R 中的rpart package能夠實現Cart 算法。

R code：

# raw data has 4521 rows and 17 columns; the last column is y
bank <- read.csv("C:/working/summer/機器學習/決策樹/bank/bank.csv",header=TRUE,sep=';')

# seprate as training set & valication set
bank_train <- bank[1:4000,]
bank_test <- bank[4001:4521,1:16]
bank_test1 <- bank[4001:4521,]

# build tree
library(rpart)
fit <- rpart(y~age+job+marital+education+default+balance+housing+loan+contact  
             +day+month+duration+campaign+pdays+previous+poutcome,method="class",  
             data=bank_train)  # method=class represent build classification tree
plot(fit, uniform = TRUE,main="Classification Tree for Bank")
text(fit,use.n = TRUE,all=TRUE)

#######################################################################################################

#use validation data to test the accuracy
result <- predict(fit, bank_test,type = "class")

#use a function to calculate accuracy rate
source("C:/working/summer/機器學習/決策樹/accurate rate.r")
count_result(result,bank_test1)

#######################################################################################################

# deal with missing value
# na.action 默認保留自變量缺失的觀測值，刪除因變量缺失的觀測值
# 但是不明白怎么保留自變量缺失的觀測值？？這樣保留了怎么建的樹？
summary(bank) #The 4th, 9th,16th column have unknown value
n <- nrow(bank)
for (i in 1:n){
  if (bank[i,4]=="unknown"){
    bank[i,4]=NA
  }
  if (bank[i,9]=="unknown"){
    bank[i,9]=NA
  }
  if (bank[i,16]=="unknown"){
    bank[i,16]=NA
  }
}

fit2 <- rpart(y~.,method = "class", data=bank_train,na.action=na.rpart)  
plot(fit,,use.n=TRUE,all=TRUE)  
text(fit,use.n = TRUE,all=TRUE)
result2 <- predict(fit2,bank_test,type="class")
count_result(result2,bank_test1)

########################################################################################################
fit3 <- rpart(y~age+job+marital+education+default+balance+housing+loan+contact+day+month+duration+campaign+
                pdays+previous+poutcome,method="class",data=bank_train,na.action=na.rpart,
              control=rpart.control(minsplit=40,cp=0.001))   # minsplit越大樹越簡單，它表示當分類小到這個值時就停止
result3 <- predict(fit3,bank_test,type="class")  
count_result(result3,bank_test1)
plot(fit3,use.n=TRUE,all=TRUE)

count_result function 用來計算分類的正確率

count_result <- function(result,data_test){
  n <- length(result)
  count_right<-0
  i <-1
  for (i in 1:n){
    if (result[i]==data_test[i,17]){
      count_right=count_right+1
    }
  }
  print(count_right/n)

剪枝：

library(rpart)
fit <- rpart(y~age+job+marital+education+default+balance+housing+loan+contact  
             +day+month+duration+campaign+pdays+previous+poutcome,method="class",  
             data=bank_train,control=rpart.control(minsplit=140,cp=0.001))  # method=class represent build classification tree
plot(fit, uniform = TRUE,main="Classification Tree for Bank")
text(fit,use.n = TRUE,all=TRUE)

# more beautiful plot
library(rpart.plot)
rpart.plot(fit, branch=1, branch.type=2, type=1, extra=102,  
           shadow.col="gray", box.col="green",  
           border.col="blue", split.col="red",  
           split.cex=1.2, main="Kyphosis決策樹");  

# prune
printcp(fit)
fit$cptable
fit2 <- prune(fit, cp= fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"]) 
rpart.plot(fit2, branch=1, branch.type=2, type=1, extra=102,  
           shadow.col="gray", box.col="green",  
           border.col="blue", split.col="red",  
           split.cex=1.2, main="Kyphosis決策樹");

剪枝前：4層

Paste_Image.png

剪枝后：3層

Paste_Image.png