爬虫入门

既然sakila的数据不是很好用,干脆自己爬点数据吧。

  • 实战-sakila数据库项目 介绍了sakila数据开发应用,结果发现只有文字数据,而且影片数据也不理想,所以这里需要使用爬虫从万能的互联网上抓点数据。

目标选择

  • 腾讯视频

抓取类型

  • 电影

技术选型

  • jsoup

代码实现

简单的单元测试入门。jsoup的用法可以直接查看官方文档。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
private static final String UA_PHONE = "Mozilla/5.0 (Linux; Android 4.3; Nexus 10 Build/JSS15Q) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.23 Safari/537.36";

private static final String UA_PC = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36";

private static final String QQ_PAGE_MOVIE = "http://v.qq.com/x/list/movie?itype=-1&offset=1";

@Test
public void test() throws IOException {
Document document = Jsoup.connect(QQ_PAGE_MOVIE)
.userAgent(UA_PC)
.timeout(TIME_OUT)
.ignoreContentType(true).get();
Elements elements = document.select("li.list_item a.figure");
for (Element element : elements) {
String url = element.attr("href");
String title = element.select("img").attr("alt");
String image = element.select("img").attr("r-lazyload");
System.out.println("href " + url + " title " + title + " image " + image);
}
}

输出结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

href https://v.qq.com/x/cover/5lmuabxmb5jf1fl.html title 云南虫谷 image //puui.qpic.cn/vcover_vt_pic/0/5lmuabxmb5jf1fl1550480109/220

href https://v.qq.com/x/cover/3oe086g3e8pvmm1.html title 神谕通天 image //puui.qpic.cn/vcover_vt_pic/0/3oe086g3e8pvmm11526636789/220

href https://v.qq.com/x/cover/nvy9lp6e061ecu2.html title 湮灭 image //puui.qpic.cn/vcover_vt_pic/0/nvy9lp6e061ecu21522377505/220

href https://v.qq.com/x/cover/vq6l5tg1blwcquk.html title 小悟空 image //puui.qpic.cn/vcover_vt_pic/0/vq6l5tg1blwcquk1528942630/220

href https://v.qq.com/x/cover/xuy45yvwdgajywr.html title 小偷家族 image //puui.qpic.cn/vcover_vt_pic/0/xuy45yvwdgajywr1534475242/220

href https://v.qq.com/x/cover/v097i2zl7bekihf.html title 遇见你真好 image //puui.qpic.cn/vcover_vt_pic/0/v097i2zl7bekihf1520840657/220

href https://v.qq.com/x/cover/5crb4lhjxu5irm0.html title 摘金奇缘 image //puui.qpic.cn/vcover_vt_pic/0/5crb4lhjxu5irm01540260494/220

href https://v.qq.com/x/cover/70y5f38tjcmtw2t.html title 吃货宇宙 image //puui.qpic.cn/vcover_vt_pic/0/70y5f38tjcmtw2t1530070491/220

href https://v.qq.com/x/cover/og0eputlxwet1cn.html title 我不是药神 image //puui.qpic.cn/vcover_vt_pic/0/og0eputlxwet1cn1524011116/220

href https://v.qq.com/x/cover/c32gmogdsrjwi5t.html title 求求你爱上我 image //puui.qpic.cn/vcover_vt_pic/0/c32gmogdsrjwi5t1527907276/220

href https://v.qq.com/x/cover/m871wzl4wsan735.html title 地球最后的夜晚 image //puui.qpic.cn/vcover_vt_pic/0/m871wzl4wsan7351548821409/220

href https://v.qq.com/x/cover/5i3ppwn4q8ykk2v.html title 午夜十二点 image //puui.qpic.cn/vcover_vt_pic/0/5i3ppwn4q8ykk2v1527728549/220

href https://v.qq.com/x/cover/wagzbx91asjomnu.html title 无问西东 image //puui.qpic.cn/vcover_vt_pic/0/wagzbx91asjomnu1527151117/220

href https://v.qq.com/x/cover/fhz98sq3ecwh2bm.html title 恐怖浴室 image //puui.qpic.cn/vcover_vt_pic/0/fhz98sq3ecwh2bm1531122381/220

href https://v.qq.com/x/cover/tpa0c309erq2p7t.html title 无名之辈 image //puui.qpic.cn/vcover_vt_pic/0/tpa0c309erq2p7t1545014750/220

href https://v.qq.com/x/cover/h8lp8zokyi0y960.html title 疯狂的公牛 image //puui.qpic.cn/vcover_vt_pic/0/h8lp8zokyi0y9601548833946/220

href https://v.qq.com/x/cover/7fjv56lqblm6jhd.html title 熊出没·变形记 image //puui.qpic.cn/vcover_vt_pic/0/7fjv56lqblm6jhd1518058675/220

href https://v.qq.com/x/cover/5y95zy4idzqf6hc.html title 捉妖记2 image //puui.qpic.cn/vcover_vt_pic/0/5y95zy4idzqf6hc1515143734/220

href https://v.qq.com/x/cover/xyaud3hk6mmi6ss.html title 追龙 image //puui.qpic.cn/vcover_vt_pic/0/xyaud3hk6mmi6ss1513156112/220

href https://v.qq.com/x/cover/s4fnybn8ttiqerl.html title 神奇动物:格林德沃之罪 image //puui.qpic.cn/vcover_vt_pic/0/s4fnybn8ttiqerl1541656334/220

href https://v.qq.com/x/cover/sdmcdbqecsgbqpq.html title 南极之恋 image //puui.qpic.cn/vcover_vt_pic/0/sdmcdbqecsgbqpq1514943903/220

href https://v.qq.com/x/cover/fl98wya3y3vit90.html title 闺蜜的战争 image //puui.qpic.cn/vcover_vt_pic/0/fl98wya3y3vit901548665650/220

href https://v.qq.com/x/cover/ewh18dn62hvddln.html title 昼颜 image //puui.qpic.cn/vcover_vt_pic/0/ewh18dn62hvddln1490692208/220

href https://v.qq.com/x/cover/frs5qyzh8aeo7mj.html title 来电狂响 image //puui.qpic.cn/vcover_vt_pic/0/frs5qyzh8aeo7mj1547171319/220

href https://v.qq.com/x/cover/cp8jhn5o7r5j0aa.html title 英雄本色2018 image //puui.qpic.cn/vcover_vt_pic/0/cp8jhn5o7r5j0aa1519977118/220

href https://v.qq.com/x/cover/ia95960ibzdqly9.html title 细思极恐 image //puui.qpic.cn/vcover_vt_pic/0/ia95960ibzdqly91530059937/220

href https://v.qq.com/x/cover/8taoglu73bcpz3u.html title 猫与桃花源 image //puui.qpic.cn/vcover_vt_pic/0/8taoglu73bcpz3u1524019282/220

href https://v.qq.com/x/cover/zxsr2hae16abq7v.html title 妈妈咪鸭 image //puui.qpic.cn/vcover_vt_pic/0/zxsr2hae16abq7v1515552446/220

href https://v.qq.com/x/cover/hhp6u9obgnkvpqh.html title 地下凶猛 image //puui.qpic.cn/vcover_vt_pic/0/hhp6u9obgnkvpqht1463720650.jpg/220

href https://v.qq.com/x/cover/begymcfom3fw25j.html title 谜巢 image //puui.qpic.cn/vcover_vt_pic/0/begymcfom3fw25j1521084275/220