数据清理- 重复数据判断删除1.函数============================================================df.duplicated(subset=None, keep='first') # 指定列数据重复项判断； # 返回：指定列重复行boolean Seriesdf.drop_duplicates(subset=None, keep='first', # 删除重复数据 inplace=False) # 返回：副本或替代参数：subset=None：列标签或标签序列，可选# 只考虑某些列来识别重复项;默认使用所有列keep='first'：{'first'，'last'，False}# - first：将第一次出现重复值标记为True# - last：将最后一次出现重复值标记为True# - False：将所有重复项标记为True============================================================2.实例# 实例1：重复数据判断df= pd.DataFrame({'k1': [ 's1']* 3 + ['s2']* 5,'k2' : [1, 1, 2, 3, 3, 4, 4,4]})result1=df.duplicated()result2=df.duplicated(keep='last')result3=df.duplicated(keep=False)result4=df.duplicated('k1')result5=df.duplicated(['k1','k2'])# df result1 result2 result3 result4 result5# first last False ['k1'] ['k1','k2'] k1 k2 0 s1 1 0 False 0 True 0 True 0 False 0 False1 s1 1 1 True 1 False 1 True 1 True 1 True2 s1 2 2 False 2 False 2 False 2 True 2 False3 s2 3 3 False 3 True 3 True 3 False 3 False4 s2 3 4 True 4 False 4 True 4 True 4 True5 s2 4 5 False 5 True 5 True 5 True 5 False6 s2 4 6 True 6 True 6 True 6 True 6 True7 s2 4 7 True 7 False 7 True 7 True 7 True============================================================# 实例2.1：重复数据清理- 副本df.drop_duplicates() #保留第一个值，返回副本df.drop_duplicates(keep='last') #保留最后一个值，返回副本df.drop_duplicates(keep=False) #删除所有重复值，返回副本df.drop_duplicates('k1') #删除第一列重复值，返回副本df.drop_duplicates(['k1','k2']) #删除全部列重复值，返回副本# df result1 result2 result3 result4 result5# first last False ['k1'] ['k1','k2'] k1 k2 k1 k2 k1 k2 k1 k2 k1 k2 k1 k20 s1 1 0 s1 1 1 s1 1 2 s1 2 0 s1 1 0 s1 11 s1 1 2 s1 2 2 s1 2 3 s2 3 2 s1 22 s1 2 3 s2 3 4 s2 3 3 s2 33 s2 3 5 s2 4 7 s2 4 5 s2 44 s2 3 5 s2 4 6 s2 4 7 s2 4# 实例2.2：重复数据清理- 就地修改df.drop_duplicates(inplace=True)#就地修改df k1 k20 s1 12 s1 23 s2 35 s2 4============================================================