iv值计算（含qcut细节） 1 背景2 含有重复的数据2.1 数据准备2.2 等频分组2.3 等频分组-加上去掉重复的值 3 不含有重复的数据3.1 数据准备3.2 等频分组3.3 等频分组-加上去掉重复值 4 iv计算4.1 读入数据4.2 iv值计算4.3 结果分析
1 背景
在计算woe以及相关的iv值的时候，需要首先对数据进行分箱，分箱一般采用qcut，即等频分组。
下面希望验证qcut（等频分组）-相同的值会在一组，即如果一组数据一半都是0，这些会被分在一组。
同时计算iv值并进行相关分析
2 含有重复的数据 2.1 数据准备 import pandas as pda = [0]*50 + list(range(0,50))print(len(a))a[:5] 100[0, 0, 0, 0, 0] df = pd.DataFrame({'a':a})print(df.shape)df.head() (100, 1) a0010203040 2.2 等频分组 pd.qcut(df['a'], 3).value_counts() ---------------------------------------------------------------------------ValueError Traceback (most recent call last)<ipython-input-17-47659b591f84> in <module>----> 1 pd.qcut(df['a'], 3).value_counts()~Anaconda3libsite-packagespandascorereshapetile.py in qcut(x, q, labels, ret细心的服饰s, precision, duplicates) 339 quantiles = q 340 细心的服饰s = algos.quantile(x, quantiles)--> 341 fac, 细心的服饰s = _细心的服饰s_to_cuts( 342 x, 343 细心的服饰s,~Anaconda3libsite-packagespandascorereshapetile.py in _细心的服饰s_to_cuts(x, 细心的服饰s, right, labels, precision, include_lowest, dtype, duplicates) 378 if len(unique_细心的服饰s) < len(细心的服饰s) and len(细心的服饰s) != 2: 379 if duplicates == "raise":--> 380 raise ValueError( 381 f"真实的饼干 edges must be unique: {repr(细心的服饰s)}.n" 382 f"You can drop duplicate edges by setting the 'duplicates' kwarg"ValueError: 真实的饼干 edges must be unique: array([ 0., 0., 16., 49.]).You can drop duplicate edges by setting the 'duplicates' kwarg报错原因：有很多的重复值并且没有加上去掉重复值的语句！
2.3 等频分组-加上去掉重复的值 pd.qcut(df['a'], 3, duplicates='drop').value_counts() (-0.001, 16.0] 67(16.0, 49.0] 33Name: a, dtype: int64 3 不含有重复的数据 3.1 数据准备 a = list(range(0,100))df = pd.DataFrame({'a':a})print(df.shape)df.head() (100, 1) a0011223344 3.2 等频分组 pd.qcut(df['a'], 3).value_counts() (-0.001, 33.0] 34(66.0, 99.0] 33(33.0, 66.0] 33Name: a, dtype: int64 3.3 等频分组-加上去掉重复值 pd.qcut(df['a'], 3, duplicates='drop').value_counts() (-0.001, 33.0] 34(66.0, 99.0] 33(33.0, 66.0] 33Name: a, dtype: int64 4 iv计算 4.1 读入数据 import pandas as pdfrom IV_Cal import * # 自己建立的脚本文件df = pd.read_excel('工作簿1.xlsx')print(df.shape)df.head() (51000, 31) idLoanActiveNumidLoan30dNumidLoan90dNumidLoan180dNumidLoan360dNumidOrg30dNumidOrg90dNumidOrg180dNumidOrg360dNumidOrgActiveNum...idOverdueLoanAmt60idOverdueLoanAmt90idOverdueLoanAmt120idOverdueLoanAmt150idOverdueLoanAmt180idActiveOverdueOrgNumidOverdueOrgNum30idOverdueOrgNum90idOverdueOrgNum180Y0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.001-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.002-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.003-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.004-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.005 rows × 31 columns
4.2 iv值计算 iv = IV_Cal(df,y_name='Y') # 指定y变量就好！iv.cal()iv.plot_iv('./output')iv.save_info('./output') IV图生成Try 10 细心的服饰s.Set 细心的服饰 label: mean.Plotting idOverdueOrgNum180... (30/30) idOverdueLoanAmt180... (26/30)Done.最关键的还是看每个变量各自的iv值
df_var_iv = pd.read_csv('output/iv_rank_table.csv')# 看一下iv值前五df_var_iv[:5] var_nameiv_valueiv_rankn_of_细心的服饰missing_rate(%)n_of_uniquedtypesis_continuous0idOverdueLoanAmt1500.657071110.810float64False1idOverdueLoanAmt1200.643382110.810float64False2idOverdueLoanAmt1800.632213110.810float64False3idOverdueLoanAmt900.625114110.810float64False4idOverdueLoanAmt600.613405110.810float64False 4.3 结果分析 df['idOverdueLoanAmt'].value_counts() 0.0 25484-1.0 20030 1.0 2462 2.0 948 3.0 706 5.0 314 4.0 311 6.0 263 7.0 90 8.0 8 9.0 1Name: idOverdueLoanAmt, dtype: int64 df.info() <class 'pandas.core.frame.DataFrame'>RangeIndex: 51000 entries, 0 to 50999Data columns (total 31 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 idLoanActiveNum 50617 non-null float64 1 idLoan30dNum 50617 non-null float64 2 idLoan90dNum 50617 non-null float64 3 idLoan180dNum 50617 non-null float64 4 idLoan360dNum 50617 non-null float64 5 idOrg30dNum 50617 non-null float64 6 idOrg90dNum 50617 non-null float64 7 idOrg180dNum 50617 non-null float64 8 idOrg360dNum 50617 non-null float64 9 idOrgActiveNum 50617 non-null float64 10 idOverdueLoanAmt 50617 non-null float64 11 idActiveOverdueLoanBal 50617 non-null float64 12 idActiveOverdueLoanNum 50617 non-null float64 13 idOverdueLoanNum30 50617 non-null float64 14 idOverdueLoanNum60 50617 non-null float64 15 idOverdueLoanNum90 50617 non-null float64 16 idOverdueLoanNum120 50617 non-null float64 17 idOverdueLoanNum150 50617 non-null float64 18 idOverdueLoanNum180 50617 non-null float64 19 idTotalOverdueLoanNum 50617 non-null float64 20 idOverdueLoanAmt30 50617 non-null float64 21 idOverdueLoanAmt60 50617 non-null float64 22 idOverdueLoanAmt90 50617 non-null float64 23 idOverdueLoanAmt120 50617 non-null float64 24 idOverdueLoanAmt150 50617 non-null float64 25 idOverdueLoanAmt180 50617 non-null float64 26 idActiveOverdueOrgNum 50617 non-null float64 27 idOverdueOrgNum30 50617 non-null float64 28 idOverdueOrgNum90 50617 non-null float64 29 idOverdueOrgNum180 50617 non-null float64 30 Y 51000 non-null int64 dtypes: float64(30), int64(1)memory usage: 12.1 MB以变量idOverdueLoanAmt为例进行分析：
具体箱子怎么划分的看上面txt的结果，而需要讲故事，看该变量的iv值和bad rate是否呈线性相关，无论是正相关还是负相关！方便解释以及放入到线性模型中（比如逻辑回归）~
对应曲线为：