使用Nginx对百度蜘蛛的301重定向跳转处理

2021-11-11主机推荐 浏览

先简单说下我碰到的情况。我一个域名的http页面被墙，国内用户可以通过https方式打开。为了seo，也遵循尽量不换网站域名的前提下，现 在我想实现的目标是对国外蜘蛛（google...）和不在墙内的用户访问http页面时301重定向跳转到https，而百度蜘蛛访问原页面时301到 一个新域名，新域名反向代理原来的网页，用户通过百度搜索点击打开的是原域名https页面。


1、修改原域名nginx配置
server {
 listen 80 default;
 server_name www.2kyb.com;
 index index.html index.htm index.php default.html default.htm default.php;
 root /home/wwwroot/2kyb;
 
 #通过$http_user_agent判断是否是百度蜘蛛
 if ($http_user_agent ~* Baiduspider){
 return 301 http://www.mke6.com$request_uri;
 } 
 #排除对一个网址的301重定向
 if ($request_uri !~ ^/.*/.*notify_url.php.*){
 return 301 https://$server_name$request_uri;
 }
location ~ .*.(php|php5)?$
 {
 try_files $uri =404;
 fastcgi_pass unix:/tmp/php-cgi.sock;
 fastcgi_index index.php;
 include fcgi.conf;
 }
 
 access_log /home/wwwlogs/www.2kyb.com.log www.2kyb.com;
}
server { 
 listen 443 ;
 ......
}
2、修改新域名nginx配置
server
 {
 listen 80;
 #listen [::]:80;
 server_name www.mke6.com;
 index index.html index.htm index.php default.html default.htm default.php;
 root /home/wwwroot/mke6.com/www;
 #如果用户不是百度蜘蛛且访问的网址不是baidu.*.html格式
 set $flag 0;
 if ($http_user_agent !~* Baiduspider){
 set $flag "${flag}1";
 }
 #用做百度站长平台的验证
 if ($request_uri !~ ^.*/baidu.*.html.*){
 set $flag "${flag}2";
 }
 if ($flag = "012") {
 return 301 https://www.2kyb.com$request_uri;
 }
 #接上不，如果是百度蜘蛛，执行proxy_pass反代。
 location ^~ / {
 proxy_redirect off;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 if ($request_uri !~ ^.*/baidu.*.html.*){
 proxy_pass https://www.2kyb.com;
 }
 }
......
access_log /home/wwwlogs/www.mke6.com.log access;
 }
3、通过curl模拟百度蜘蛛、google、普通用户验证
[stocky@s1 ~]$ curl -I -A Baiduspider www.2kyb.com
HTTP/1.1 301 Moved Permanently
Server: nginx/1.0.15
Date: Wed, 04 Mar 2015 08:59:27 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: http://www.mke6.com/
[stocky@s1 ~]$ curl -I -A Baiduspider www.mke6.com
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 04 Mar 2015 09:01:29 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Powered-By: PHP/5.2.17p1
[stocky@s1 ~]$ curl -I -A google www.2kyb.com
HTTP/1.1 301 Moved Permanently
Server: nginx/1.0.15
Date: Wed, 04 Mar 2015 09:00:08 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.2kyb.com/
[stocky@s1 ~]$ curl -I -A google www.mke6.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 04 Mar 2015 09:02:07 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://www.2kyb.com/
[stocky@s1 ~]$ curl -I -A Mozilla www.2kyb.com
HTTP/1.1 301 Moved Permanently
Server: nginx/1.0.15
Date: Wed, 04 Mar 2015 09:01:59 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.2kyb.com/
[stocky@s1 ~]$ curl -I -A Mozilla www.mke6.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 04 Mar 2015 09:04:00 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://www.2kyb.com/
测试后发现满足要求。