本地处理
#!/usr/bin/python # -*- coding: utf-8 -*- # UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a in position 12的暂时解决方法——修改默认encoding import sys reload(sys) sys.setdefaultencoding('utf-8') from pyspark.sql import SparkSession from pyspark import SparkContext from pyspark.sql.functions import * # 为使用dataframe的方法 import re from pyspark.sql.types import * import datetime import pandas as pd conf = SparkConf().setAppName("miniProject").setMaster("local") sc1 = SparkContext.getOrCreate(conf) spark = SparkSession(sc1)
讯享网
1. 客户经理信息简单查询
讯享网# 客户经理数据——已经将代码的csv传到本地,读取 cm_df = pd.read_csv('file:///home/hadoop/xxx/project_0509_khjl/secret.csv', sep=',', encoding='utf-8') cm_df = cm_df.ix[:,1:-1]
cm_df.info()
讯享网<class 'pandas.core.frame.DataFrame'> RangeIndex: entries, 0 to Data columns (total 12 columns): staff_id non-null int64 staff_name non-null object user_id_card_edit non-null object id_county non-null object id_province non-null object id_city non-null object id non-null int64 role_id non-null int64 presona_id non-null int64 status non-null int64 city_number non-null int64 num non-null int64 dtypes: int64(7), object(5) memory usage: 9.6+ MB
cm_df.describe()
| staff_id | id | role_id | presona_id | status | city_number | num | |
|---|---|---|---|---|---|---|---|
| count | .000000 | .000000 | .000000 | .000000 | .000000 | .0 | .000000 |
| mean | . | . | 58. | 3135. | 1. | 0.0 | -0.000029 |
| std | 78064. | 78064. | 0. | 2029. | 0. | 0.0 | 0.031703 |
| min | 218.000000 | 218.000000 | 0.000000 | 43.000000 | 1.000000 | 0.0 | -4.000000 |
| 25% | . | . | 59.000000 | 1603.000000 | 2.000000 | 0.0 | 0.000000 |
| 50% | .000000 | .000000 | 59.000000 | 2374.000000 | 2.000000 | 0.0 | 0.000000 |
| 75% | . | . | 59.000000 | 4828.000000 | 2.000000 | 0.0 | 0.000000 |
| max | .000000 | .000000 | 155.000000 | 9002.000000 | 2.000000 | 0.0 | 4.000000 |
版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容,请联系我们,一经查实,本站将立刻删除。
如需转载请保留出处:https://51itzy.com/kjqy/112641.html