htmlunit和爬虫问题

最近写了个爬虫项目，暂时放在github上。https://github.com/shenbaise/goodcrawler

对于爬虫最难的问题应该是javascript和ajax的处理。现在很多网站使用大量ajax，普通爬虫无法获取js生成的

内容。

目前大体上有2中方式来解决这个问题。

一是使用htmlunit之类的所谓handless browser。htmlunit对js支持相当不错。只是适用起来不很方便。想要知道什么时候js执行完了比较困难。

官网 FQA对这个问题的说明。htmlunit

Nothing happens when using HtmlUnit with AJAX, although page works correctly in browsers. What's wrong?

The main thread using HtmlUnit may be finishing execution before allowing background threads to run. You have a couple of options:

webClient.setAjaxController(new NicelyResynchronizingAjaxController());will tell your WebClient instance to re-synchronize asynchronous XHR.

webClient.waitForBackgroundJavaScript(10000);orwebClient.waitForBackgroundJavaScriptStartingBefore(10000);just after getting the page and before manipulating it.

Explicitly wait for a condition that is expected be fulfilled when your JavaScript runs, e.g.

我的测试：

public static void testYouku() throws Exception {
		String url = "http://v.youku.com/v_show/id_XNDc2MDkzMTIw.html";
		String xurl = "http://v.youku.com/v_vpofficiallistv5/id_119023280_showid_271942_page_2?__rt=1&__ro=listitem_page2";
		// String a = "<a page=\"2\">178-101</a>";
		// String url="http://.baidu.com";
		// 模拟一个浏览器
		final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);

		LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log","org.apache.commons.logging.impl.NoOpLog");
		java.util.logging.Logger.getLogger("net.sourceforge.htmlunit").setLevel(java.util.logging.Level.OFF);
		webClient.getOptions().setThrowExceptionOnScriptError(false);
		webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
		// final WebClient webClient=new
		// WebClient(BrowserVersion.FIREFOX_10,"http://myproxyserver",8000);
		// //使用代理
		// final WebClient webClient2=new
		// WebClient(BrowserVersion.INTERNET_EXPLORER_10);
		// 设置webClient的相关参数
		webClient.getOptions().setJavaScriptEnabled(true);
		webClient.getOptions().setActiveXNative(false);
		webClient.getOptions().setCssEnabled(false);
		webClient.getOptions().setThrowExceptionOnScriptError(false);
		webClient.waitForBackgroundJavaScript(600*1000);
		webClient.setAjaxController(new NicelyResynchronizingAjaxController());
		
		webClient.getOptions().setJavaScriptEnabled(true);  
		/*
		webClient.setJavaScriptTimeout(3600*1000);  
		webClient.getOptions().setRedirectEnabled(true);  
		webClient.getOptions().setThrowExceptionOnScriptError(true);  
		webClient.getOptions().setThrowExceptionOnFailingStatusCode(true);  
		webClient.getOptions().setTimeout(3600*1000);  
		webClient.waitForBackgroundJavaScript(600*1000);  
		*/
//		webClient.waitForBackgroundJavaScript(600*1000);
		webClient.setAjaxController(new NicelyResynchronizingAjaxController()); 
		
		// 模拟浏览器打开一个目标网址
		final HtmlPage page = webClient.getPage(url);
//		该方法在getPage()方法之后调用才能生效
		webClient.waitForBackgroundJavaScript(1000*3);
		webClient.setJavaScriptTimeout(0);
//		Thread.sleep(1000 *3L);
//		String js = "javascript:checkShowFollow('271942','2');";
//		ScriptResult sr = page.executeJavaScript(js);
//		HtmlPage newPage = (HtmlPage) sr.getNewPage();
//		System.out.println("new page.asText=" + newPage.asText());
//		System.out.println("page.asText=" + page.asText());
//		System.out.println("page.getUrl=" + page.getUrl());
		List links = (List) page.getByXPath ("//*[@id=\"groups_tab\"]/div[1]/ul/li[1]/a");
		if(null!=links){
			System.out.println(links.size());
			HtmlAnchor link = (HtmlAnchor) links.get(0);
			System.out.println(link.asXml());
			HtmlPage p = link.click();
			webClient.waitForBackgroundJavaScript(1000*3L);
//			webClient.waitForBackgroundJavaScriptStartingBefore(1000L);
//			Thread.sleep(3000L);
			System.out.println(p.asText());
		}
	}

本文来自电脑杂谈，转载请注明本文网址：
https://www.pc-fly.com/a/jisuanjixue/article-38449-1.html