也就是文章作者管理
客户端提供文章作者首页地址,
通过如下方式可以获取到作者信息:
- http://www.toutiao.com/c/user/6310889977/
- public void test2() {
- //写手主页
- String url = "http://www.toutiao.com/c/user/6310889977/";
- // 文章详情页的前缀(由于今日头条的文章都是在group这个目录下,所以定义了前缀,而且通过请求获取到的html页面)
- // 链接到该网站
- Connection connection = Jsoup.connect(url);
- Document content = null;
- try {
- Map<String, String> cookies = new HashMap<String, String>();
- cookies.put("UM_distinctid",
- "15d604b19ca3a2-0e6085b9900ead-1c197450-1fa400-15d604b19cb7b8");
- cookies.put("uuid", "w:a881896fbbd446bf9fc7c0c97434e78f");
- cookies.put("OUTFOX_SEARCH_USER_ID_NCOO", "1631109872.7671177");
- cookies.put("csrftoken", "59bd6e9979d9a0e447159c09c69ec182");
- cookies.put("_ga", "GA1.2.1128248025.1501125011");
- cookies.put("_gid", "GA1.2.1377010915.1501125011");
- cookies.put("__utmt", "1");
- cookies.put("__utma",
- "24953151.1128248025.1501125011.1501125474.1501125474.1");
- cookies.put("__utmb", "24953151.8.10.1501125474");
- cookies.put("__utmc", "24953151");
- cookies.put("__utmz",
- "24953151.15011254741.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)");
- cookies.put("tt_webid", "6444852342776890893");
- cookies.put("CNZZDATA1259612802",
- "478646998-1500557369-|1501121800");
- connection
- .header("User-Agent",
- "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3100.0 Safari/537.36");
- connection.cookies(cookies);
- // 获取内容
- content = connection.get();
- } catch (IOException e) {
- e.printStackTrace();
- }
- // 转换成字符串
- String htmlStr = content.html();
- // 因为今日头条的文章展示比较奇葩,都是通过js定义成变量,所以无法使用获取dom元素的方式获取值
- String jsonStr = StringUtils.substringBetween(htmlStr,
- "var userInfo = ", ";");
- System.out.println(jsonStr);
- Map parse = (Map) JSONObject.parse(jsonStr);
- Object id = parse.get("id");
- //通过用户id拼接查询该作者的文章列表url
- String articleUrl = "http://www.toutiao.com/c/user/article/?page_type=1&user_id="
- + id
- + "&max_behot_time=0&count=30&as=A18539E7A918279&cp=5979A8F29759AE1";
- System.out.println("详情url:"+articleUrl);
- }
输出如下
- {
- id: 6310889977,
- name: '爱豆表情',
- avatarUrl: 'http://p3.pstatp.com/thumb/411000fa6b181d3a25c',
- isPgc: true,
- isBanned: false,
- isOwner: false,
- mediaId: 6311316427,
- type: '1'
- }详情url: http: //www.toutiao.com/c/user/article/?page_type=1&user_id=6310889977&max_behot_time=0&count=30&as=A18539E7A918279&cp=5979A8F29759AE1
抓取选择作者的文章列表,在这里需要使用到上面拼接的 url,
这个 url 返回 json,通过我们的 json 请求方式即可。
- http://www.toutiao.com/c/user/article?page_type=1&user_id=6310889977&max_behot_time=0&count=30&as=A18539E7A918279&cp=5979A8F29759AE1
这里建议使用爬取的当前的时间戳作为
字段的参数,为什么要这样,因为如果使用默认的值,则是按照阅读数排序的,当重复爬取的时候不容易判断重复。下面的数据是通过
- max_behot_time
获取的
- http://www.toutiao.com/c/user/article/?page_type=1&user_id=6310889977&max_behot_time=1501146065&count=2&as=A18539E7A918279&cp=5979A8F29759AE1
- {
- "login_status": false,
- "has_more": true,
- "next": {
- "max_behot_time": 1501046661
- },
- "page_type": 1,
- "message": "success",
- "data": [
- {
- "image_url": "http://p1.pstatp.com/list/190x124/31c60002fe07a841314d",
- "single_mode": true,
- "abstract": "要知道90后年龄最大的都有27岁了,正是适婚的时候,不过说起结婚,礼金这个问题就避免不了,不管是朋友结婚也好同学结婚也好,这份子钱还是都得随了,而明星结婚也是如此,收礼金收到手软,当梁朝伟和刘嘉玲结婚,王菲给15万还算少的,但来看看赵本山,那才是真的豪气!",
- "middle_mode": false,
- "more_mode": true,
- "tag": "news_entertainment",
- "tag_url": "news_entertainment",
- "title": "明星结婚给多少礼金?王菲给15万算少的,赵本山才是真豪气!",
- "has_video": false,
- "chinese_tag": "娱乐",
- "source": "爱豆表情",
- "group_source": 2,
- "comments_count": 78,
- "media_url": "/m6311316427/",
- "go_detail_count": 198964,
- "image_list": [
- {
- "url": "http://p1.pstatp.com/list/31c60002fe07a841314d",
- "width": 1080,
- "url_list": [
- {
- "url": "http://p1.pstatp.com/list/31c60002fe07a841314d"
- },
- {
- "url": "http://pb3.pstatp.com/list/31c60002fe07a841314d"
- },
- {
- "url": "http://pb9.pstatp.com/list/31c60002fe07a841314d"
- }
- ],
- "uri": "list/31c60002fe07a841314d",
- "height": 799
- },
- {
- "url": "http://p1.pstatp.com/list/31b9000303c0e7e03e26",
- "width": 1080,
- "url_list": [
- {
- "url": "http://p1.pstatp.com/list/31b9000303c0e7e03e26"
- },
- {
- "url": "http://pb3.pstatp.com/list/31b9000303c0e7e03e26"
- },
- {
- "url": "http://pb9.pstatp.com/list/31b9000303c0e7e03e26"
- }
- ],
- "uri": "list/31b9000303c0e7e03e26",
- "height": 727
- },
- {
- "url": "http://p1.pstatp.com/list/2ee300032db03ae5b319",
- "width": 1080,
- "url_list": [
- {
- "url": "http://p1.pstatp.com/list/2ee300032db03ae5b319"
- },
- {
- "url": "http://pb3.pstatp.com/list/2ee300032db03ae5b319"
- },
- {
- "url": "http://pb9.pstatp.com/list/2ee300032db03ae5b319"
- }
- ],
- "uri": "list/2ee300032db03ae5b319",
- "height": 795
- }
- ],
- "gallary_image_count": 0,
- "detail_play_effective_count": 11,
- "source_url": "/item/6446946448201220622/",
- "item_id": "6446946448201220622",
- "article_genre": "article",
- "display_url": "http://toutiao.com/preview_article/?pgc_id=6446946448201220622",
- "behot_time": 1501046691,
- "has_gallery": false,
- "group_id": "6446942793298510094"
- },
- {
- "image_url": "http://p1.pstatp.com/list/190x124/31c70000e30f63f73da5",
- "single_mode": true,
- "abstract": "时下《春风十里不如你》的开播在观众心目中可是千呼万唤始出来。早在开拍之时,这部剧就引发了众多关注和观众们的无限期待。张一山和周冬雨两位年轻戏骨的组合让观众纷纷好奇他们之间会擦出什么样的火花。而在荧幕上是小清新女神的周冬雨,微博晒照立马暴露了自己的本性。",
- "middle_mode": false,
- "more_mode": true,
- "tag": "news_entertainment",
- "tag_url": "news_entertainment",
- "title": "周冬雨自拍晒豪宅,游戏房抢镜,但客厅却暴露本性!",
- "has_video": false,
- "chinese_tag": "娱乐",
- "source": "爱豆表情",
- "group_source": 2,
- "comments_count": 281,
- "media_url": "/m6311316427/",
- "go_detail_count": 427439,
- "image_list": [
- {
- "url": "http://p1.pstatp.com/list/31c70000e30f63f73da5",
- "width": 550,
- "url_list": [
- {
- "url": "http://p1.pstatp.com/list/31c70000e30f63f73da5"
- },
- {
- "url": "http://pb3.pstatp.com/list/31c70000e30f63f73da5"
- },
- {
- "url": "http://pb9.pstatp.com/list/31c70000e30f63f73da5"
- }
- ],
- "uri": "list/31c70000e30f63f73da5",
- "height": 654
- },
- {
- "url": "http://p3.pstatp.com/list/2ee300032d6e1bbe7d4e",
- "width": 600,
- "url_list": [
- {
- "url": "http://p3.pstatp.com/list/2ee300032d6e1bbe7d4e"
- },
- {
- "url": "http://pb9.pstatp.com/list/2ee300032d6e1bbe7d4e"
- },
- {
- "url": "http://pb1.pstatp.com/list/2ee300032d6e1bbe7d4e"
- }
- ],
- "uri": "list/2ee300032d6e1bbe7d4e",
- "height": 506
- },
- {
- "url": "http://p1.pstatp.com/list/31c70000e30ea71c1110",
- "width": 400,
- "url_list": [
- {
- "url": "http://p1.pstatp.com/list/31c70000e30ea71c1110"
- },
- {
- "url": "http://pb3.pstatp.com/list/31c70000e30ea71c1110"
- },
- {
- "url": "http://pb9.pstatp.com/list/31c70000e30ea71c1110"
- }
- ],
- "uri": "list/31c70000e30ea71c1110",
- "height": 448
- }
- ],
- "gallary_image_count": 0,
- "detail_play_effective_count": 5,
- "source_url": "/item/6446946316734956046/",
- "item_id": "6446946316734956046",
- "article_genre": "article",
- "display_url": "http://toutiao.com/group/6446942632664498445/",
- "behot_time": 1501046661,
- "has_gallery": false,
- "group_id": "6446942632664498445"
- }
- ],
- "is_self": false
- }
主要字段介绍:
- /**
- * 爬去文章详情信息
- */
- @Test public void test1() {
- String url2 = "http://www.toutiao.com/i6277404061113254402/";
- Connection connection = null;
- connection = Jsoup.connect(url2);
- try {
- Document document = connection.get();
- // 获取文章标题
- Elements title = document.select("[class=article-title]");
- System.out.println(title.html());
- // 获取文章来源和文章发布时间
- Elements articleInfo = document.select("[class=articleInfo]");
- Elements src = articleInfo.select("[class=src]");
- System.out.println(src.html());
- Elements time = articleInfo.select("[class=time]");
- System.out.println(time.html());
- // 获取文章内容
- Elements contentEle = document.select("[class=article-content]");
- String content = contentEle.html();
- System.out.println(content);
- //查找文章的img标签,并替换url
- Elements select = contentEle.select("img[src]");
- Iterator < Element > iterator = select.iterator();
- while (iterator.hasNext()) {
- Element next = iterator.next();
- String attr = next.attr("src");
- content = content.replace(attr, attr + "/abc");
- }
- System.out.println("\n替换url后\n");
- System.out.println(content);
- } catch(IOException e) {
- e.printStackTrace();
- }
- }
- 看到周星驰这些表情,我竟然想哭了
- 爱豆表情
- 2016-04-25 18:07
- <div>
- <p>我心中永远的艺术家,才华横溢的导演就是周星驰了;虽然最近的这个美人鱼不是我的喜欢,但是很是期待接下来更多周氏风格的作品!</p>
- <p>周星驰陪伴我们一路成长,下面这些动图,你能回忆起几张?</p>
- <p><img src="http://p3.pstatp.com/large/52d0005cda7a3dff492" img_width="400" img_height="224" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>来来来,还有一个鸡腿留给你的</p>
- <p><img src="http://p1.pstatp.com/large/52e0005d01bb43cde93" img_width="300" img_height="184" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷越吹越过瘾啊</p>
- <p><img src="http://p1.pstatp.com/large/52f0005cfc30f970ecc" img_width="400" img_height="225" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>亲个吻再拍</p>
- <p><img src="http://p1.pstatp.com/large/52c0005cddb2f3e19f2" img_width="400" img_height="225" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>隔山打牛,隔的人越多越厉害</p>
- <p><img src="http://p3.pstatp.com/large/41100099da8b6aca739" img_width="298" img_height="224" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷见到美女流口水</p>
- <p><img src="http://p3.pstatp.com/large/52f0005cfc4266c0ea4" img_width="397" img_height="214" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷要多帅就有多帅</p>
- <p><img src="http://p9.pstatp.com/large/5300005d205804a37bc" img_width="375" img_height="200" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>哇靠!吓得我辫子都直了</p>
- <p><img src="http://p1.pstatp.com/large/52f0005cfc5e4c4ac42" img_width="317" img_height="171" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>老兄,当心把肺都吐出来了</p>
- <p><img src="http://p1.pstatp.com/large/5300005d20690cb8ea2" img_width="296" img_height="168" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷教你如何优雅的比试柔道</p>
- <p><img src="http://p1.pstatp.com/large/52f0005cfc7ceeb3408" img_width="320" img_height="179" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷赌圣帅气出场</p>
- <p><img src="http://p1.pstatp.com/large/5300005d207dc534168" img_width="320" img_height="179" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>国产007帅气特工</p>
- <p><img src="http://p1.pstatp.com/large/5300005d209fd260b71" img_width="320" img_height="179" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>茶餐厅的灵魂,蛋挞王子!</p>
- <p><img src="http://p3.pstatp.com/large/52c0005cdddd9d27482" img_width="261" img_height="142" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>国产007周星驰看美女</p>
- <p><img src="http://p3.pstatp.com/large/41100099dacc7a9ab23" img_width="330" img_height="175" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>食神,我来了!</p>
- <p><img src="http://p3.pstatp.com/large/52c0005cddf788d6322" img_width="300" img_height="217" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>《师兄撞鬼》周星驰经典的的邪恶片段</p>
- <p><img src="http://p3.pstatp.com/large/41100099daedc49bd47" img_width="444" img_height="250" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>我左青龙,右边虎,老牛在腰间,龙头在心口...</p>
- <p><img src="http://p9.pstatp.com/large/52f0005cfc8e6b00351" img_width="320" img_height="180" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷,好一只大力金刚腿</p>
- <p><img src="http://p1.pstatp.com/large/41100099db024fb4fb8" img_width="320" img_height="168" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>如花茹凉</p>
- <p><img src="http://p3.pstatp.com/large/4120009a1722dade61d" img_width="280" img_height="151" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>周星驰弹小JJ</p>
- <p>一路陪伴我们的周星驰,曾教我们如何耍帅,如何撩妹,如果为自己所爱热血奋斗!</p>
- <p>每次看到这些图,不得不感叹,伴随我们的是一份多麽美好的记忆!</p>
- </div>
- 替换url后
- <div>
- <p>我心中永远的艺术家,才华横溢的导演就是周星驰了;虽然最近的这个美人鱼不是我的喜欢,但是很是期待接下来更多周氏风格的作品!</p>
- <p>周星驰陪伴我们一路成长,下面这些动图,你能回忆起几张?</p>
- <p><img src="http://p3.pstatp.com/large/52d0005cda7a3dff492/abc" img_width="400" img_height="224" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>来来来,还有一个鸡腿留给你的</p>
- <p><img src="http://p1.pstatp.com/large/52e0005d01bb43cde93/abc" img_width="300" img_height="184" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷越吹越过瘾啊</p>
- <p><img src="http://p1.pstatp.com/large/52f0005cfc30f970ecc/abc" img_width="400" img_height="225" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>亲个吻再拍</p>
- <p><img src="http://p1.pstatp.com/large/52c0005cddb2f3e19f2/abc" img_width="400" img_height="225" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>隔山打牛,隔的人越多越厉害</p>
- <p><img src="http://p3.pstatp.com/large/41100099da8b6aca739/abc" img_width="298" img_height="224" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷见到美女流口水</p>
- <p><img src="http://p3.pstatp.com/large/52f0005cfc4266c0ea4/abc" img_width="397" img_height="214" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷要多帅就有多帅</p>
- <p><img src="http://p9.pstatp.com/large/5300005d205804a37bc/abc" img_width="375" img_height="200" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>哇靠!吓得我辫子都直了</p>
- <p><img src="http://p1.pstatp.com/large/52f0005cfc5e4c4ac42/abc" img_width="317" img_height="171" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>老兄,当心把肺都吐出来了</p>
- <p><img src="http://p1.pstatp.com/large/5300005d20690cb8ea2/abc" img_width="296" img_height="168" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷教你如何优雅的比试柔道</p>
- <p><img src="http://p1.pstatp.com/large/52f0005cfc7ceeb3408/abc" img_width="320" img_height="179" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷赌圣帅气出场</p>
- <p><img src="http://p1.pstatp.com/large/5300005d207dc534168/abc" img_width="320" img_height="179" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>国产007帅气特工</p>
- <p><img src="http://p1.pstatp.com/large/5300005d209fd260b71/abc" img_width="320" img_height="179" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>茶餐厅的灵魂,蛋挞王子!</p>
- <p><img src="http://p3.pstatp.com/large/52c0005cdddd9d27482/abc" img_width="261" img_height="142" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>国产007周星驰看美女</p>
- <p><img src="http://p3.pstatp.com/large/41100099dacc7a9ab23/abc" img_width="330" img_height="175" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>食神,我来了!</p>
- <p><img src="http://p3.pstatp.com/large/52c0005cddf788d6322/abc" img_width="300" img_height="217" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>《师兄撞鬼》周星驰经典的的邪恶片段</p>
- <p><img src="http://p3.pstatp.com/large/41100099daedc49bd47/abc" img_width="444" img_height="250" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>我左青龙,右边虎,老牛在腰间,龙头在心口...</p>
- <p><img src="http://p9.pstatp.com/large/52f0005cfc8e6b00351/abc" img_width="320" img_height="180" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>星爷,好一只大力金刚腿</p>
- <p><img src="http://p1.pstatp.com/large/41100099db024fb4fb8/abc" img_width="320" img_height="168" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>如花茹凉</p>
- <p><img src="http://p3.pstatp.com/large/4120009a1722dade61d/abc" img_width="280" img_height="151" alt="看到周星驰这些表情,我竟然想哭了" inline="0"></p>
- <p>周星驰弹小JJ</p>
- <p>一路陪伴我们的周星驰,曾教我们如何耍帅,如何撩妹,如果为自己所爱热血奋斗!</p>
- <p>每次看到这些图,不得不感叹,伴随我们的是一份多麽美好的记忆!</p>
- </div>
- <dependency>
- <groupId>org.jsoup</groupId>
- <artifactId>jsoup</artifactId>
- <version>1.10.3</version>
- </dependency>
- <dependency>
- <groupId>com.alibaba</groupId>
- <artifactId>fastjson</artifactId>
- <version>1.2.27</version>
- </dependency>
来源: http://blog.csdn.net/facekbook/article/details/76566856