Skip to content

add Popular search engines' spiders user-agent list #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bpceee wants to merge 3 commits into from
Closed

add Popular search engines' spiders user-agent list #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c7b818d
add '.woff',
levionchain Jan 20, 2015
56ee7b2
add Popular search engines' spiders user-agent list
levionchain Jan 20, 2015
3139b81
ignore case for user-agent test
levionchain Jan 22, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,16 @@ If your URLs use push-state:

# Customization

### Popular search engines' spiders user-agent list

Many search engines do not follow google's _escaped_fragment_ proposal, we can use user-agent to detect these spiders.
```js
var prerender = require('prerender-node');
prerender.set('robot', 'Baiduspider|Googlebot|BingBot|Slurp!|MSNBot|YoudaoBot|JikeSpider|Sosospider|360Spider|Sogou web spider|Sogou inst spider');
app.use(prerender);

```

### Whitelist

Whitelist a single url path or multiple url paths. Compares using regex, so be specific when possible. If a whitelist is supplied, only url's containing a whitelist path will be prerendered.
Expand Down
14 changes: 12 additions & 2 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,9 @@ prerender.extensionsToIgnore = [
'.iso',
'.flv',
'.m4v',
'.torrent'
'.torrent',
'.woff',
'.ttf'
];


Expand All @@ -107,6 +109,14 @@ prerender.blacklisted = function(blacklist) {
};


prerender.isRobot = function(userAgent){
if(!this.robot) return false;

var robotReg = new RegExp(this.robot, 'i');
if (robotReg.test(userAgent)) return true;
return false;
};

prerender.shouldShowPrerenderedPage = function(req) {
var userAgent = req.headers['user-agent']
, bufferAgent = req.headers['x-bufferbot']
Expand All @@ -117,7 +127,7 @@ prerender.shouldShowPrerenderedPage = function(req) {

//if it contains _escaped_fragment_, show prerendered page
var parsedQuery = url.parse(req.url, true).query;
if(parsedQuery && parsedQuery.hasOwnProperty('_escaped_fragment_')) isRequestingPrerenderedPage = true;
if(prerender.isRobot(userAgent) || (parsedQuery && parsedQuery.hasOwnProperty('_escaped_fragment_'))) isRequestingPrerenderedPage = true;

//if it is a bot...show prerendered page
if(prerender.crawlerUserAgents.some(function(crawlerUserAgent){ return userAgent.toLowerCase().indexOf(crawlerUserAgent.toLowerCase()) !== -1;})) isRequestingPrerenderedPage = true;
Expand Down