爬虫入门

既然sakila的数据不是很好用，干脆自己爬点数据吧。

• 在 实战-sakila数据库项目 介绍了sakila数据开发应用，结果发现只有文字数据，而且影片数据也不理想，所以这里需要使用爬虫从万能的互联网上抓点数据。

目标选择

• 腾讯视频

抓取类型

• 电影

技术选型

• jsoup

代码实现

简单的单元测试入门。jsoup的用法可以直接查看官方文档。

private static final String UA_PHONE = "Mozilla/5.0 (Linux; Android 4.3; Nexus 10 Build/JSS15Q) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.23 Safari/537.36";

private static final String UA_PC = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36";

private static final String QQ_PAGE_MOVIE = "http://v.qq.com/x/list/movie?itype=-1&offset=1";
	
	@Test
    public void test() throws IOException {
	Document document = Jsoup.connect(QQ_PAGE_MOVIE)
            .userAgent(UA_PC)
            .timeout(TIME_OUT)
            .ignoreContentType(true).get();
      Elements elements = document.select("li.list_item a.figure");
        for (Element element : elements) {
            String url = element.attr("href");
            String title = element.select("img").attr("alt");
            String image = element.select("img").attr("r-lazyload");
            System.out.println("href  " + url + "   title  " + title + "  image  " + image);
        }
    }

输出结果

href  https://v.qq.com/x/cover/5lmuabxmb5jf1fl.html   title  云南虫谷  image  //puui.qpic.cn/vcover_vt_pic/0/5lmuabxmb5jf1fl1550480109/220

href  https://v.qq.com/x/cover/3oe086g3e8pvmm1.html   title  神谕通天  image  //puui.qpic.cn/vcover_vt_pic/0/3oe086g3e8pvmm11526636789/220

href  https://v.qq.com/x/cover/nvy9lp6e061ecu2.html   title  湮灭  image  //puui.qpic.cn/vcover_vt_pic/0/nvy9lp6e061ecu21522377505/220

href  https://v.qq.com/x/cover/vq6l5tg1blwcquk.html   title  小悟空  image  //puui.qpic.cn/vcover_vt_pic/0/vq6l5tg1blwcquk1528942630/220

href  https://v.qq.com/x/cover/xuy45yvwdgajywr.html   title  小偷家族  image  //puui.qpic.cn/vcover_vt_pic/0/xuy45yvwdgajywr1534475242/220

href  https://v.qq.com/x/cover/v097i2zl7bekihf.html   title  遇见你真好  image  //puui.qpic.cn/vcover_vt_pic/0/v097i2zl7bekihf1520840657/220

href  https://v.qq.com/x/cover/5crb4lhjxu5irm0.html   title  摘金奇缘  image  //puui.qpic.cn/vcover_vt_pic/0/5crb4lhjxu5irm01540260494/220

href  https://v.qq.com/x/cover/70y5f38tjcmtw2t.html   title  吃货宇宙  image  //puui.qpic.cn/vcover_vt_pic/0/70y5f38tjcmtw2t1530070491/220

href  https://v.qq.com/x/cover/og0eputlxwet1cn.html   title  我不是药神  image  //puui.qpic.cn/vcover_vt_pic/0/og0eputlxwet1cn1524011116/220

href  https://v.qq.com/x/cover/c32gmogdsrjwi5t.html   title  求求你爱上我  image  //puui.qpic.cn/vcover_vt_pic/0/c32gmogdsrjwi5t1527907276/220

href  https://v.qq.com/x/cover/m871wzl4wsan735.html   title  地球最后的夜晚  image  //puui.qpic.cn/vcover_vt_pic/0/m871wzl4wsan7351548821409/220

href  https://v.qq.com/x/cover/5i3ppwn4q8ykk2v.html   title  午夜十二点  image  //puui.qpic.cn/vcover_vt_pic/0/5i3ppwn4q8ykk2v1527728549/220

href  https://v.qq.com/x/cover/wagzbx91asjomnu.html   title  无问西东  image  //puui.qpic.cn/vcover_vt_pic/0/wagzbx91asjomnu1527151117/220

href  https://v.qq.com/x/cover/fhz98sq3ecwh2bm.html   title  恐怖浴室  image  //puui.qpic.cn/vcover_vt_pic/0/fhz98sq3ecwh2bm1531122381/220

href  https://v.qq.com/x/cover/tpa0c309erq2p7t.html   title  无名之辈  image  //puui.qpic.cn/vcover_vt_pic/0/tpa0c309erq2p7t1545014750/220

href  https://v.qq.com/x/cover/h8lp8zokyi0y960.html   title  疯狂的公牛  image  //puui.qpic.cn/vcover_vt_pic/0/h8lp8zokyi0y9601548833946/220

href  https://v.qq.com/x/cover/7fjv56lqblm6jhd.html   title  熊出没·变形记  image  //puui.qpic.cn/vcover_vt_pic/0/7fjv56lqblm6jhd1518058675/220

href  https://v.qq.com/x/cover/5y95zy4idzqf6hc.html   title  捉妖记2  image  //puui.qpic.cn/vcover_vt_pic/0/5y95zy4idzqf6hc1515143734/220

href  https://v.qq.com/x/cover/xyaud3hk6mmi6ss.html   title  追龙  image  //puui.qpic.cn/vcover_vt_pic/0/xyaud3hk6mmi6ss1513156112/220

href  https://v.qq.com/x/cover/s4fnybn8ttiqerl.html   title  神奇动物：格林德沃之罪  image  //puui.qpic.cn/vcover_vt_pic/0/s4fnybn8ttiqerl1541656334/220

href  https://v.qq.com/x/cover/sdmcdbqecsgbqpq.html   title  南极之恋  image  //puui.qpic.cn/vcover_vt_pic/0/sdmcdbqecsgbqpq1514943903/220

href  https://v.qq.com/x/cover/fl98wya3y3vit90.html   title  闺蜜的战争  image  //puui.qpic.cn/vcover_vt_pic/0/fl98wya3y3vit901548665650/220

href  https://v.qq.com/x/cover/ewh18dn62hvddln.html   title  昼颜  image  //puui.qpic.cn/vcover_vt_pic/0/ewh18dn62hvddln1490692208/220

href  https://v.qq.com/x/cover/frs5qyzh8aeo7mj.html   title  来电狂响  image  //puui.qpic.cn/vcover_vt_pic/0/frs5qyzh8aeo7mj1547171319/220

href  https://v.qq.com/x/cover/cp8jhn5o7r5j0aa.html   title  英雄本色2018  image  //puui.qpic.cn/vcover_vt_pic/0/cp8jhn5o7r5j0aa1519977118/220

href  https://v.qq.com/x/cover/ia95960ibzdqly9.html   title  细思极恐  image  //puui.qpic.cn/vcover_vt_pic/0/ia95960ibzdqly91530059937/220

href  https://v.qq.com/x/cover/8taoglu73bcpz3u.html   title  猫与桃花源  image  //puui.qpic.cn/vcover_vt_pic/0/8taoglu73bcpz3u1524019282/220

href  https://v.qq.com/x/cover/zxsr2hae16abq7v.html   title  妈妈咪鸭  image  //puui.qpic.cn/vcover_vt_pic/0/zxsr2hae16abq7v1515552446/220

href  https://v.qq.com/x/cover/hhp6u9obgnkvpqh.html   title  地下凶猛  image  //puui.qpic.cn/vcover_vt_pic/0/hhp6u9obgnkvpqht1463720650.jpg/220

href  https://v.qq.com/x/cover/begymcfom3fw25j.html   title  谜巢  image  //puui.qpic.cn/vcover_vt_pic/0/begymcfom3fw25j1521084275/220