add Popular search engines' spiders user-agent list

[bpceee]

add Popular search engines' spiders user-agent list

[gillesdemey]

Can anyone have a look at this PR?

[thoop]

Sorry for the delay!

Adding .woff and .tff is good, but your isRobot function is already handled by the prerender.crawlerUserAgents check. Is there a reason you duplicated that logic?

Also, the user agent example you added contains Googlebot, which doing a user agent check on Googlebot will get you penalized for cloaking. You should only check for user agents that do not support the escaped fragment protocol.

[bpceee]

@thoop
Sorry, You are right, the isRobot function is duplicated, I didn't find prerender.crawlerUserAgents before.
And many thanks for your suggestions.

[homeyer]

@bpceee, want to pull the .woff and .ttf changes into a separate PR, or remove the other pieces from this PR?

[bpceee]

@homeyer, better remove the other pieces from this PR, but how to do it?

[homeyer]

git reset HEAD~2 --hard to remove the last two commits, and then git push -f to your fork.

[homeyer]

Went ahead and opened a new PR (#74), I'll merge tonight after the tests pass.