--- sidebar_position: 10 title: Doris说明 --- # Doris说明 ### DWD表开发 ```properties 1.加yml配置 2.加step sql文件 3.create_table_all.sql写好记录 decimalv3 小数精度 BIGINT 主键 INT 常规数字 datetime varchar 字符 4.定时任务加执行地址 http://10.1.67.76:19999/ 5.window执行完定时自动加入 ``` ### 网页说明 ##### 数据集 ``` 1.新增数据集: 录入已创建的dwd表 2.dwd表新增了字段: 图表来源-更多-编辑数据集, 选择列-从原数据同步列 (交换数据集是啥?) 3.增加指标: 图表来源-更多-编辑数据集, 选择指标-增加条件 4.删除亮哥的指标: 后缀加 _del ``` ##### 图表 ``` ``` ##### 看板 ``` 1.可以全局筛选, 可以局部筛选, 可以联动筛选 2.展示: 数量不要小数 金额不要小数 比率用1位小数 ``` ##### sql工具箱 ``` ``` ### code对应表 ``` city_id -> city_name 1 北京 2 杭州 5 苏州 6 太原 7 天津 8 南京 9 上海 15 成都 18 郑州 19 无锡 20 武汉 67 广州 672 广州PARK contract_state_code -> contract_state 1000100070001 已签约 1000100070003 已作废 1000100070004 已续约 1000100070005 解约中 1000100070006 已解约 1000100070008 续约中 1000100070009 签约中 1000100070010 草稿 signing_route_code -> signing_route --非太原/太原 1000100790001/1000100820001 AMS纸版签约 1000100790002/1000100820002 APP电子签约 contract_type_code -> contract_type SF 委托 CF 承租 property_check_state_code -> property_check_state 1000300270001 待审核 1000300270002 审核已通过 1000300270003 审核未通过 1000300270004 待补充 1000300270005 待客户确认 jy_type_code -> jy_type --非太原/太原 1000100800010001/1000100830010001 AMS纸质解约 1000100800010002/1000100830010001 APP电子解约 ``` ##### ODS是原材料，DWD是清洗好的零件，DWS是组装好的半成品，那么ADS就是摆在货架上成品。 ### ods ```properties 设计ods表,数据类型按全城市的最大值创建 #流程 定时去拉去AMS的数据到ods表 , 链路猜测是ogg->spark->ods 全量定时: 每天全刷 增量定时: 几分钟一刷 ``` ### dwd ```properties application.yml 里配置了dwd需要执行哪些sql dwd-micro: 业务 dws-ads-micro: 数据组用 zipper: ods备份表,比如监控到发生变化,就insert一条,时间是zip_valid_start,对应字段在项目的superset_fields_xxx文件 classpath配置里dwd的建表语句,写法像视图,11 ${group}: sfContract ${dwdTable}: dwd_fact_sf ${odsTable}: ods_wk_sf_contract ${odsTables}: 都写的这个,但是实际用的时候还是odsTable ${isFull}: 是否全量 ${lookbackSeconds}: 增量数据,前60秒的,一般不用 ${jobParam}: XXL-JOB参数 ${initSplitPredicate}: isFull=true且数据量很大时, 拆成多个小任务执行 ``` ### 自己尝试 ```yml dwd_fact_agreement: enabled: true ods-tables: ods_wiwj_agreement_info: enabled: true sql-files: - classpath:sql/dwd/sfContract/dwd_fact_sf_agreement_step1.sql ``` ```sql --委托协议测试表 DROP TABLE IF EXISTS warehouse.dwd_fact_agreement; CREATE TABLE IF NOT EXISTS warehouse.dwd_fact_agreement ( dw_cityid INT COMMENT '城市ID', agreement_id BIGINT COMMENT '协议ID', city_name VARCHAR(20) COMMENT '城市', contract_id BIGINT COMMENT '合同ID', contract_type VARCHAR(40) COMMENT '协议类型(收房出房)', confer_id BIGINT COMMENT '协议类型字典', confer_name BIGINT COMMENT '协议类型名称', status VARCHAR(100) COMMENT '协议风控审核状态', letdatechange VARCHAR(2) COMMENT '首个招租期是否变更', cycleLetDaysChange VARCHAR(2) COMMENT '托管招租期是否变更', create_time DATETIME COMMENT '创建时间', --共通 op_type VARCHAR(20) COMMENT '是否被物理删除', etl_time DATETIMEV2(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'ETL 加载时间' ) UNIQUE KEY(dw_cityid, agreement_id) COMMENT '委托协议测试表' DISTRIBUTED BY HASH(dw_cityid, agreement_id) BUCKETS 16 PROPERTIES ( "replication_num" = "3", "enable_unique_key_merge_on_write" = "true" ); ``` ##### [dwd_fact_sf_agreement_step1.sql](C:\project\doris-microbatch-executor\src\main\resources\sql\dwd\sfContract\dwd_fact_sf_agreement_step1.sql) ```sql INSERT INTO dwd_fact_agreement ( dw_cityid, agreement_id, city_name, contract_id, contract_type, confer_id, confer_name, status, letdatechange, cycleLetDaysChange, create_time, op_type, etl_time ) SELECT s.dw_cityid as dw_cityid, s.ID as agreement_id, dc.dict_name as city_name, s.CONTRACTSID as contract_id, s.CONTRACTTYPE as contract_type, s.CONFERID as confer_id, ods_conferid.DICTNAME as confer_name, s.STATUS as status, detail.letdatechange AS letdatechange, detail.cycleLetDaysChange AS cycleLetDaysChange, s.CREATETIME as create_time, s.op_type, s.etl_time FROM ${odsTable} s left join dict_config dc on dc.dict_id = s.dw_cityid and dc.dict_type = 'city' left join ods_taw_system_dicttype ods_conferid on ods_conferid.dw_cityid = s.dw_cityid and ods_conferid.dictid = s.conferid LEFT JOIN ( SELECT dw_cityid, CONFERID, MAX(CASE WHEN name = 'letDateChange' AND value = 'Y' THEN 'Y' ELSE 'N' END) AS letdatechange, MAX(CASE WHEN name = 'cycleLetDaysChange' AND value = 'Y' THEN 'Y' ELSE 'N' END) AS cycleLetDaysChange FROM ods_wiwj_agreement_detail_info WHERE name IN ('letDateChange', 'cycleLetDaysChange') GROUP BY dw_cityid, CONFERID ) detail ON detail.dw_cityid = s.dw_cityid AND detail.CONFERID = s.id INNER JOIN dwd_batch_window b ON b.dwd_group = '${group}' AND b.dwd_table = '${dwdTable}' AND b.ods_table = '${odsTable}' WHERE s.pos > b.batch_start_pos AND s.pos <= b.batch_end_pos AND s.CONTRACTTYPE = 'SF'; ```