Kaggle playground 练习项目 New York City Taxi Trip Duration

项目 New York City Taxi Trip Duration"/>

Kaggle playground 练习项目 New York City Taxi Trip Duration

       最近接触了一些机器学习知识，想在kaggle上找入门项目做做练手。于是选择了New York City Taxi Trip Duration这个预测出租车行驶时间的练习赛。

     训练集特征包括以下部分，目的是建立模型预测出租车每次行程的行驶时间。

    id - 每次旅行的唯一标识符
    vendor_id - 指示与旅行记录关联的提供者的代码
    pickup_datetime - 仪表启用的日期和时间
    dropoff_datetime - 仪表脱离的日期和时间
    passenger_count - 车辆中的乘客数量（驾驶员输入值）
    pickup_longitude - 仪表所用的经度
    pickup_latitude - 仪表所处的纬度
    dropoff_longitude - 仪表脱离的经度
    dropoff_latitude - 仪表脱离的纬度
    store_and_fwd_flag - 该标志指示在发送给供应商之前是否将行程记录保存在车辆存储器中，因为车辆没有连接到服务器

                                       Y =存储和转发; N =不是商店和前瞻旅行
    trip_duration - 行程的持续时间，以秒为单位

    评分根据为RMSE（均方根误差）。

        以下是一个使用随机森林预测的简单初步模型。

# -*- coding: utf-8 -*-import pandas as pdtrain = pd.read_csv("train.csv", header=0)
test = pd.read_csv("test.csv", header=0)# 查看数据的字段信息, dropoff_datetime,id可以去掉
# print(train.columns)
# print(test.columns)# 查看数据是否有缺失
# print(train.info())
# print(test.info())# 选取特征值
X_train = train.drop(['dropoff_datetime', 'trip_duration', 'id'], axis=1)
y_train = train['trip_duration']
X_test = test.drop(['id'], axis=1)# print(X_train.shape)
# print(X_test.shape)
# print(y_train.head())# 特征值处理
X_train['month'] = pd.DatetimeIndex(X_train.pickup_datetime).month
X_train['day'] = pd.DatetimeIndex(X_train.pickup_datetime).dayofweek
X_train['hour'] = pd.DatetimeIndex(X_train.pickup_datetime).hour
X_train['store_and_fwd_flag'].replace('Y', 1, inplace=True)
X_train['store_and_fwd_flag'].replace('N', 0, inplace=True)
X_train = X_train.drop(['pickup_datetime'], axis=1)X_test['month'] = pd.DatetimeIndex(X_test.pickup_datetime).month
X_test['day'] = pd.DatetimeIndex(X_test.pickup_datetime).dayofweek
X_test['hour'] = pd.DatetimeIndex(X_test.pickup_datetime).hour
X_test['store_and_fwd_flag'].replace('Y', 1, inplace=True)
X_test['store_and_fwd_flag'].replace('N', 0, inplace=True)
X_test = X_test.drop(['pickup_datetime'], axis=1)
# print(X_test['store_and_fwd_flag'].value_counts())
# print(X_train.head())
# print(X_test.head())
# print(X_train.shape)
# print(X_test.shape)# 使用RandomForestRegressor进行回归预测
from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor()
rfr.fit(X_train, y_train)
rfr_y_predict = rfr.predict(X_test)# 输出结果
gbr_submission = pd.DataFrame({'id': test['id'], 'trip_duration': rfr_y_predict})
gbr_submission.to_csv('rfr_submission.csv', index=False)

最终分数为0.55480，还有很大的提升空间。

下面是一个进行了比较完善的特征工程，并且使用了模型融合的解法。

 导入各种计算包

import pandas as pd
pd.set_option('display.max_columns', None)
import numpy as np
import tensorflow as tffrom sklearn.ensemble import RandomForestRegressor as RFR
import lightgbm as lgb
from catboost import CatBoostRegressorfrom collections import namedtuple
from pandas.tseries.holiday import USFederalHolidayCalendar
from pandas.tseries.offsets import CustomBusinessDayimport time
import operator
import haversine
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from datetime import timedelta
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline  

读入数据

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

确定没有重复和错误数据

print(train.duplicated().sum())
print(train.id.duplicated().sum())
print(test.id.duplicated().sum())sum(train.dropoff_datetime < train.pickup_datetime)

数据清洗

train = train.drop('dropoff_datetime',1)
train.trip_duration.describe()# Values are in minutes
print(np.percentile(train.trip_duration, 99)/60)
print(np.percentile(train.trip_duration, 99.5)/60)
print(np.percentile(train.trip_duration, 99.6)/60)
print(np.percentile(train.trip_duration, 99.8)/60)
print(np.percentile(train.trip_duration, 99.85)/60)
print(np.percentile(train.trip_duration, 99.9)/60)
print(np.percentile(train.trip_duration, 99.99)/60)
print(np.percentile(train.trip_duration, 99.999)/60)
print(np.percentile(train.trip_duration, 99.9999)/60)
print(train.trip_duration.max() / 60)

通过上面的操作可以看见，有些旅程记录耗费时间太多，应该作为离群点删去，否则会对模型预测造成影响。

# Check how many trips remain with each limit
print(len(train[train.trip_duration <= np.percentile(train.trip_duration, 99.9)]))
print(len(train[train.trip_duration <= np.percentile(train.trip_duration, 99.99)]))
print(len(train[train.trip_duration <= np.percentile(train.trip_duration, 99.999)]))# Remove outliers
train = train[train.trip_duration <= np.percentile(train.trip_duration, 99.999)]

对训练集作图，查找离群点。

# Plot locations - look for outliers
n = 100000 # number of data points to displayf, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(10, 5))
ax1.scatter(train.pickup_longitude[:n], train.pickup_latitude[:n],alpha = 0.1)
ax1.set_title('Pickup')
ax2.scatter(train.dropoff_longitude[:n], train.dropoff_latitude[:n],alpha = 0.1)
ax2.set_title('Dropoff')

图像如下所示

从图像上看还算相对比较集中，但是我们还是要对数据进行一定的修剪，从而提高模型训练效果。

# The values are not too wild, but we'll trim them back a little to be conservative
print(train.pickup_latitude.max())
print(train.pickup_latitude.min())
print(train.pickup_longitude.max())
print(train.pickup_longitude.min())
print()
print(train.dropoff_latitude.max())
print(train.dropoff_latitude.min())
print(train.dropoff_longitude.max())
print(tra