肥胖风险的多类预测——CatBoost模型的89%

项目网址（具体描述）：
https://www.kaggle.com/competitions/playground-series-s4e2

（一）数据导入与预览

import time
import pandas as pd 
import numpy as np
import seaborn as sns 
import matplotlib.pylab as plt
from scipy.stats import mode
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.model_selection import StratifiedKFold
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')

train.head()

	id	Gender	Age	Height	Weight	family_history_with_overweight	FAVC	FCVC	NCP	CAEC	SMOKE	CH2O	SCC	FAF	TUE	CALC	MTRANS	NObeyesdad
0	0	Male	24.443011	1.699998	81.669950	yes	yes	2.000000	2.983297	Sometimes	no	2.763573	no	0.000000	0.976473	Sometimes	Public_Transportation	Overweight_Level_II
1	1	Female	18.000000	1.560000	57.000000	yes	yes	2.000000	3.000000	Frequently	no	2.000000	no	1.000000	1.000000	no	Automobile	Normal_Weight
2	2	Female	18.000000	1.711460	50.165754	yes	yes	1.880534	1.411685	Sometimes	no	1.910378	no	0.866045	1.673584	no	Public_Transportation	Insufficient_Weight
3	3	Female	20.952737	1.710730	131.274851	yes	yes	3.000000	3.000000	Sometimes	no	1.674061	no	1.467863	0.780199	Sometimes	Public_Transportation	Obesity_Type_III
4	4	Male	31.641081	1.914186	93.798055	yes	yes	2.679664	1.971472	Sometimes	no	1.979848	no	1.967973	0.931721	Sometimes	Public_Transportation	Overweight_Level_II

目标变量：NObeyesdad

（二）数据预处理

缺失值检验、重复值检验：

test.isnull().sum()

id                                0
Gender                            0
Age                               0
Height                            0
Weight                            0
family_history_with_overweight    0
FAVC                              0
FCVC                              0
NCP                               0
CAEC                              0
SMOKE                             0
CH2O                              0
SCC                               0
FAF                               0
TUE                               0
CALC                              0
MTRANS                            0
dtype: int64

train.duplicated().sum()

0

train.isnull().sum()

id                                0
Gender                            0
Age                               0
Height                            0
Weight                            0
family_history_with_overweight    0
FAVC                              0
FCVC                              0
NCP                               0
CAEC                              0
SMOKE                             0
CH2O                              0
SCC                               0
FAF                               0
TUE                               0
CALC                              0
MTRANS                            0
NObeyesdad                        0
dtype: int64

整体预览：

train.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20758 entries, 0 to 20757
Data columns (total 18 columns):#   Column                          Non-Null Count  Dtype  
---  ------                          --------------  -----  0   id                              20758 non-null  int64  1   Gender                          20758 non-null  object 2   Age                             20758 non-null  float643   Height                          20758 non-null  float644   Weight                          20758 non-null  float645   family_history_with_overweight  20758 non-null  object 6   FAVC                            20758 non-null  object 7   FCVC                            20758 non-null  float648   NCP                             20758 non-null  float649   CAEC                            20758 non-null  object 10  SMOKE                           20758 non-null  object 11  CH2O                            20758 non-null  float6412  SCC                             20758 non-null  object 13  FAF                             20758 non-null  float6414  TUE                             20758 non-null  float6415  CALC                            20758 non-null  object 16  MTRANS                          20758 non-null  object 17  NObeyesdad                      20758 non-null  object 
dtypes: float64(8), int64(1), object(9)
memory usage: 2.9+ MB

（三）EDA

特征分类（数值型、类别型）：

cats = train.select_dtypes('object').columns.tolist()
nums = train.select_dtypes('float').columns.tolist()

绘制分布图，可以直观发现数据的模式和异常，加深对数据的熟悉程度，也是为特征工程做准备，比如需要处理哪些特征：

plt.figure(figsize=(15, 10))
for i, col in enumerate(cats, 1): ax = plt.subplot(len(cats)//3+1, 3, i)sns.histplot(train[col], ax=ax)ax.tick_params(axis='x', rotation=30)ax.set_title(col, color='r')plt.tight_layout()
plt.show()

plt.figure(figsize=(15, 10))
for i, col in enumerate(nums, 1): ax = plt.subplot(len(nums)//3+1, 3, i)sns.histplot(train[col], ax=ax)ax.set_title(col, color='r')plt.tight_layout()
plt.show()

（四）特征工程

def feature_job(df): # 二分类特征编码for col in ['family_history_with_overweight', 'FAVC', 'SMOKE', 'SCC']: df[col] = df[col].map({'yes': 1, 'no': 0}).astype(int)df['Gender'] = df['Gender'].map({'Male': 1, 'Female': 0}).astype(int)# 有序多类别编码 for col in ['CAEC', 'CALC']:df[col] = df[col].map({'Sometimes': 1, 'Frequently': 2, 'no': 0, 'Always':3}).astype(int)# 无序多类别编码 - 独热编码df = pd.get_dummies(df, columns=['MTRANS'])# 数值型特征处理df['Age'] = df['Age'].apply(lambda x: x/20 + 1).astype(int)df['Height'] = pd.cut(df['Height'], bins=[0, 1.5, 1.6, 1.7, 1.8, 1.9, 2.5], labels=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6]).astype(float)df['Weight'] = pd.cut(df['Weight'], bins=[0, 40, 60, 80, 100, 120, 140, 160, 200], labels=[w/10 for w in range(1,9)]).astype(float)return df train = feature_job(train)
test = feature_job(test)

train.sample(3)

	id	Gender	Age	Height	Weight	family_history_with_overweight	FAVC	FCVC	NCP	CAEC	SMOKE	CH2O	SCC	FAF	TUE	CALC	NObeyesdad	MTRANS_Automobile	MTRANS_Bike	MTRANS_Motorbike	MTRANS_Public_Transportation	MTRANS_Walking
5118	5118	0	3	0.2	0.4	1	1	2.020502	1.0	1	0	1.972551	0	1.926381	0.000000	0	Obesity_Type_I	True	False	False	False	False
19376	19376	0	2	0.3	0.5	1	1	3.000000	3.0	1	0	2.094901	0	0.067329	0.599441	1	Obesity_Type_III	False	False	False	True	False
6189	6189	0	2	0.2	0.2	1	1	2.000000	1.0	1	0	2.000000	0	1.000000	1.000000	1	Normal_Weight	False	False	False	True	False

建模前的数据处理：划分数据，并对目标进行标签编码，由于准备使用逻辑回归模型和SVM模型，而模型又对特征尺度敏感，所以又对特征进行了标准化处理：

X = train.drop(columns=['NObeyesdad', 'id'])
y= train['NObeyesdad']
# 目标 - 标签编码
label_enc = LabelEncoder()
y_enc = label_enc.fit_transform(y)
# 特征 - 标准化
scaler = StandardScaler()
X_scaler = scaler.fit_transform(X)
test_scaler = scaler.transform(test.drop(columns='id'))

（五）模型构建与预测

项目要求的模型评价指标： Accuracy - 准确率

模型一：逻辑回归模型

方法：5折交叉验证 + L1正则项
参数调节：简单测试了不同的参数，发现模型效果不好，所以没有进行过多参数调节。

n_folds = 5
accu_valids = []    # 存储每个 fold 的准确率start = time.time()
folds = StratifiedKFold(n_splits=n_folds, random_state=42, shuffle=True)
for fold, (train_idx, valid_idx) in enumerate(folds.split(X_scaler, y_enc)):X_train, y_train = X_scaler[train_idx], y_enc[train_idx]X_valid, y_valid = X_scaler[valid_idx], y_enc[valid_idx]model = LogisticRegression(class_weight='balanced',penalty='l1',solver='liblinear',C=5,max_iter=5000, random_state=42)model.fit(X_train, y_train)# 验证集性能y_pred_valids = model.predict(X_valid)accu_valid = accuracy_score(y_valid, y_pred_valids)print(f'Fold {fold} accuracy: {accu_valid}')accu_valids.append(accu_valid)end = time.time()
print(f'\nTime cost: {end-start: .3f} Seconds')
print(f'Avg accurary: {np.mean(accu_valids): .4f}')

结果输出：

Fold 0 accuracy: 0.7215799614643545
Fold 1 accuracy: 0.7124277456647399
Fold 2 accuracy: 0.7297687861271677
Fold 3 accuracy: 0.7087448807516261
Fold 4 accuracy: 0.7154902433148639Time cost:  5.921 Seconds
Avg accurary:  0.7176

评价：准确率只有0.7左右，效果很差，先试试其他模型，如果也很差，就是数据预处理和特征工程出现问题。

模型二：SVM

方法：5折交叉验证
参数调节：也是简单测试几个不同的参数，不进行复杂调参，而且SVM效率很慢（对当前数据集），训练时间长，可以优先考虑其他模型
说明：选择使用svm是因为想练习一下该模型，其实可以选择自己喜欢的模型，比如树模型等。

n_folds = 5
accu_valids = []start = time.time()
folds = StratifiedKFold(n_splits=n_folds, random_state=42, shuffle=True)
for fold, (train_idx, valid_idx) in enumerate(folds.split(X_scaler, y_enc)):X_train, y_train = X_scaler[train_idx], y_enc[train_idx]X_valid, y_valid = X_scaler[valid_idx], y_enc[valid_idx]model = SVC(class_weight='balanced',kernel='rbf', max_iter=6000,random_state=42)model.fit(X_train, y_train)# 验证集性能y_pred_valids = model.predict(X_valid)accu_valid = accuracy_score(y_valid, y_pred_valids)print(f'Fold {fold} accuracy: {accu_valid}')accu_valids.append(accu_valid)end = time.time()
print(f'\nTime cost: {end-start: .3f} Seconds')
print(f'Avg accuracy: {np.mean(accu_valids):.4f}')

Fold 0 accuracy: 0.7952793834296724
Fold 1 accuracy: 0.7776974951830443
Fold 2 accuracy: 0.799373795761079
Fold 3 accuracy: 0.7949891592387377
Fold 4 accuracy: 0.7865574560346904Time cost:  30.196 Seconds
Avg accuracy: 0.7908

评价：效果提升了不少，有0.79左右，接下来我选择使用CatBoost模型（因为是我个人比较喜欢的模型，最终我也是采用了CatBoost模型）

模型三：CatBoost模型

方法：5折交叉验证 + L2正则项 + 早停 + GPU加速
参数调节：简单测试了不同的参数取值，如学习率、正则化强度

import time
import optuna
from catboost import Pool, CatBoostClassifier
import pandas as pd 
import numpy as np
import seaborn as sns 
import matplotlib.pylab as plt
from scipy.stats import mode
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import accuracy_score, confusion_matrix

cat_features = X.select_dtypes('object').columns.tolist()n_fold = 5
folds = StratifiedKFold(n_splits=n_fold, random_state=42, shuffle=True) 
accu_valids = []    # 储存每个 fold 的 AUC
y_pred = np.empty((n_fold, len(test)))    # 存储每个fold的test预测值start = time.time()
for fold, (train_index, valid_index) in enumerate(folds.split(X, y_enc)): # 训练集X_train, y_train = X.iloc[train_index], y_enc[train_index]# 测试集X_vilid, y_vilid = X.iloc[valid_index], y_enc[valid_index]# 模型训练train_pool = Pool(X_train, y_train, cat_features=cat_features)valid_pool = Pool(X_vilid, y_vilid, cat_features=cat_features)clf = CatBoostClassifier(eval_metric='Accuracy',task_type='GPU', learning_rate=0.2, l2_leaf_reg=5,iterations=1000, early_stopping_rounds=100, verbose=200)clf.fit(train_pool, eval_set=valid_pool)# 验证集测试y_pred_valid = clf.predict(X_vilid)accu_valid = accuracy_score(y_vilid, y_pred_valid)print(f'Fold {fold} accurary: {accu_valid}')accu_valids.append(accu_valid)# 整个测试集性能y_pred[fold, :] = clf.predict(test)[: , 0]print('-' * 60)end = time.time()
print(f'\nTime cost: {end-start: .3f} Seconds')
print(f'Avg accurary: {np.mean(accu_valids): .4f}')

结果输出：

0:	learn: 0.6945080	test: 0.6958092	best: 0.6958092 (0)	total: 8.9ms	remaining: 8.89s
200:	learn: 0.8775142	test: 0.8338150	best: 0.8345376 (113)	total: 1.31s	remaining: 5.23s
bestTest = 0.8345375723
bestIteration = 113
Shrink model to first 114 iterations.
Fold 0 accurary: 0.8345375722543352
------------------------------------------------------------
0:	learn: 0.6959533	test: 0.6900289	best: 0.6900289 (0)	total: 7.07ms	remaining: 7.06s
200:	learn: 0.8749849	test: 0.8400771	best: 0.8415222 (198)	total: 1.34s	remaining: 5.31s
bestTest = 0.8427263969
bestIteration = 277
Shrink model to first 278 iterations.
Fold 1 accurary: 0.8427263969171483
------------------------------------------------------------
0:	learn: 0.7006504	test: 0.6912331	best: 0.6912331 (0)	total: 9.46ms	remaining: 9.46s
200:	learn: 0.8717933	test: 0.8458574	best: 0.8458574 (200)	total: 1.33s	remaining: 5.31s
bestTest = 0.8458574181
bestIteration = 200
Shrink model to first 201 iterations.
Fold 2 accurary: 0.8458574181117534
------------------------------------------------------------
0:	learn: 0.6952490	test: 0.6928451	best: 0.6928451 (0)	total: 7.58ms	remaining: 7.57s
200:	learn: 0.8737882	test: 0.8376295	best: 0.8388340 (188)	total: 1.31s	remaining: 5.21s
bestTest = 0.8397976391
bestIteration = 235
Shrink model to first 236 iterations.
Fold 3 accurary: 0.8397976391231029
------------------------------------------------------------
0:	learn: 0.6989221	test: 0.6979041	best: 0.6979041 (0)	total: 6.9ms	remaining: 6.89s
200:	learn: 0.8748118	test: 0.8308841	best: 0.8344977 (171)	total: 1.3s	remaining: 5.19s
bestTest = 0.8344977114
bestIteration = 171
Shrink model to first 172 iterations.
Fold 4 accurary: 0.8344977113948446
------------------------------------------------------------Time cost:  11.381 Seconds
Avg accurary:  0.8395

评价：效果又提升了不少，准确率突破了0.80，但是无论我后续我怎么调节参数，准确率都在 0.8~0.85 之间，没有得到明显提升。
后续策略：不浪费太多时间在调参上（当前特征不足以拟合出优秀的模型），而是将重心转向特征工程，优化特征工程策略。

（六）优化过程

优化内容：特征工程
策略：尝试构造更多新特征

• BMI（体质指数）
• Is_Obese：依据BMI，BMI>30，则肥胖
• Family_FAVC特征交互：家族肥胖 + 高热量饮食
• Health_Score综合评分：蔬菜摄入 + 运动频率 + 饮水

def feature_job(df): # 构造新特征 df['BMI'] = df['Weight'] / (df['Height']) ** 2df['Is_Obese'] = df['BMI'].apply(lambda x: 1 if x >= 30 else 0)df['Family_FAVC'] = df['family_history_with_overweight'].astype(str) + '_' + df['FAVC'].astype(str)df['Health_Score'] = df['FCVC'] + df['FAF'] + df['CH2O'] # 二分类特征编码for col in ['family_history_with_overweight', 'FAVC', 'SMOKE', 'SCC']: df[col] = df[col].map({'yes': 1, 'no': 0}).astype(int)df['Gender'] = df['Gender'].map({'Male': 1, 'Female': 0}).astype(int)# 有序多类别编码 for col in ['CAEC', 'CALC']:df[col] = df[col].map({'Sometimes': 1, 'Frequently': 2, 'no': 0, 'Always':3}).astype(int)# 无序多类别编码 - 独热编码df = pd.get_dummies(df, columns=['MTRANS', 'Family_FAVC'] ) # 数值型特征处理df['Age'] = df['Age'].apply(lambda x: x/20 + 1).astype(int)df['Height'] = pd.cut(df['Height'], bins=[0, 1.5, 1.6, 1.7, 1.8, 1.9, 2.5], labels=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6]).astype(float)df['Weight'] = pd.cut(df['Weight'], bins=[0, 40, 60, 80, 100, 120, 140, 160, 200], labels=[w/10 for w in range(1,9)]).astype(float)return df train = feature_job(train)
test = feature_job(test)

应用到CatBoost模型，输出如下：

cat_features = X.select_dtypes('object').columns.tolist()n_fold = 5
folds = StratifiedKFold(n_splits=n_fold, random_state=42, shuffle=True) 
accu_valids = []    # 储存每个 fold 的 AUC
y_pred = np.empty((n_fold, len(test)))    # 存储每个fold的test预测值start = time.time()
for fold, (train_index, valid_index) in enumerate(folds.split(X, y_enc)): # 训练集X_train, y_train = X.iloc[train_index], y_enc[train_index]# 测试集X_vilid, y_vilid = X.iloc[valid_index], y_enc[valid_index]# 模型训练train_pool = Pool(X_train, y_train, cat_features=cat_features)valid_pool = Pool(X_vilid, y_vilid, cat_features=cat_features)clf = CatBoostClassifier(eval_metric='Accuracy',task_type='GPU', learning_rate=0.2, l2_leaf_reg=5,iterations=1000, early_stopping_rounds=100, verbose=200)clf.fit(train_pool, eval_set=valid_pool)# 验证集测试y_pred_valid = clf.predict(X_vilid)accu_valid = accuracy_score(y_vilid, y_pred_valid)print(f'Fold {fold} accurary: {accu_valid}')accu_valids.append(accu_valid)# 整个测试集性能y_pred[fold, :] = clf.predict(test)[: , 0]print('-' * 60)end = time.time()
print(f'\nTime cost: {end-start: .3f} Seconds')
print(f'Avg accurary: {np.mean(accu_valids): .4f}')

(原代码基础上，加入了测试集的folds预测，用于kaggle上提交结果)

输出结果：

0:	learn: 0.7897146	test: 0.7998555	best: 0.7998555 (0)	total: 7.28ms	remaining: 7.27s
200:	learn: 0.9151512	test: 0.8947495	best: 0.8964355 (196)	total: 1.33s	remaining: 5.28s
bestTest = 0.8973988439
bestIteration = 270
Shrink model to first 271 iterations.
Fold 0 accurary: 0.8973988439306358
------------------------------------------------------------
0:	learn: 0.7934482	test: 0.7887765	best: 0.7887765 (0)	total: 8.24ms	remaining: 8.23s
200:	learn: 0.9193063	test: 0.8872832	best: 0.8892100 (119)	total: 1.31s	remaining: 5.23s
bestTest = 0.8892100193
bestIteration = 119
Shrink model to first 120 iterations.
Fold 1 accurary: 0.8892100192678227
------------------------------------------------------------
0:	learn: 0.7923642	test: 0.7885356	best: 0.7885356 (0)	total: 6.13ms	remaining: 6.13s
200:	learn: 0.9166566	test: 0.8928227	best: 0.8935453 (192)	total: 1.33s	remaining: 5.27s
400:	learn: 0.9359870	test: 0.8964355	best: 0.8964355 (400)	total: 2.63s	remaining: 3.93s
bestTest = 0.8964354528
bestIteration = 400
Shrink model to first 401 iterations.
Fold 2 accurary: 0.8964354527938343
------------------------------------------------------------
0:	learn: 0.7939423	test: 0.7836666	best: 0.7836666 (0)	total: 6.63ms	remaining: 6.63s
200:	learn: 0.9188896	test: 0.8855697	best: 0.8867743 (169)	total: 1.3s	remaining: 5.19s
bestTest = 0.8874969887
bestIteration = 234
Shrink model to first 235 iterations.
Fold 3 accurary: 0.8874969886774271
------------------------------------------------------------
0:	learn: 0.7917143	test: 0.7964346	best: 0.7964346 (0)	total: 6.79ms	remaining: 6.79s
200:	learn: 0.9197326	test: 0.8879788	best: 0.8899060 (145)	total: 1.33s	remaining: 5.3s
bestTest = 0.8899060467
bestIteration = 145
Shrink model to first 146 iterations.
Fold 4 accurary: 0.8899060467357264
------------------------------------------------------------Time cost:  12.632 Seconds
Avg accurary:  0.8921

评价：不需要太复杂的调参，模型效果就得到了很大的提升，准确率提升到了0.892，模型效果很好了。

（七）结果保存

# 最终预测结果
y_pred_final = mode(y_pred, axis=0)[0].astype(int)
print(f'Test final prediction: {y_pred_final}')

Test final prediction: [3 5 4 ... 0 1 3]

将结果保存到csv文件, 用于kaggle上提交：

submission = pd.DataFrame({'id': test_id, 'NObeyesdad': le.inverse_transform(y_pred_final)
})
submission.to_csv('submission.csv', index=False)

kaggle上的最终评分（目前排行榜最高分，私人数据集0.91157，公共数据集0.92341）：

说明：当然后续还可以继续进行优化，使得分数去到0.9以上。

# 如果大家有更好的策略，欢迎一起探讨啊
# 文章到此就结束了，我们下期再见叭！