Freelancer任务之四squid查询用户浏览记录

这个需求也比较简单:

User Browsing Log for Open VPN server

简单说就是用户连到他的openvpn服务器,通过上面的squid代理来浏览其他网站,比较特别的是需要查看用户http和https的浏览记录。

squid做透明代理,这样就可以截取浏览记录并且提供加速了

服务器是Ubuntu,缺省安装的的squid是不支持SSL的,所以需要重新编译一个

安装依赖包:

1sudo apt-get install build-essential fakeroot devscripts gawk gcc-multilib dpatch  
2sudo apt-get build-dep squid3  
3sudo apt-get build-dep openssl  
4sudo apt-get install libssl-dev  
5sudo apt-get source squid3  

下载到squid的源代码,以及ubuntu的修改包,解压并释放:

1tar zxvf squid3_3.5.12.orig.tar.gz  
2cd squid3-3.5.12  
3tar xf ../squid3_3.5.12-1ubuntu7.5.debian.tar.xz  

修改参数增加对ssl的支持:

 1vi debian/rules  
 2Add --with-openssl --enable-ssl --enable-ssl-crtd under the DEB_CONFIGURE_EXTRA_FLAGS section.
 3
 4DEB_CONFIGURE_EXTRA_FLAGS := BUILDCXXFLAGS="$(CXXFLAGS) $(LDFLAGS)" \  
 5...
 6                --with-default-user=proxy \
 7                --with-openssl \
 8                --enable-ssl \
 9                --enable-ssl-crtd
10...

编译,会生成7个deb包

 1debuild -us -uc -b  
 2cd ..  
 3ls -1 *.deb  
 4squid3_3.5.12-1ubuntu7.5_all.deb  
 5squid_3.5.12-1ubuntu7.5_amd64.deb  
 6squid-cgi_3.5.12-1ubuntu7.5_amd64.deb  
 7squidclient_3.5.12-1ubuntu7.5_amd64.deb  
 8squid-common_3.5.12-1ubuntu7.5_all.deb  
 9squid-dbg_3.5.12-1ubuntu7.5_amd64.deb  
10squid-purge_3.5.12-1ubuntu7.5_amd64.deb  

安装,先装语言包,然后安装三个自己生成的包

1sudo apt-get install squid-langpack  
2sudo dpkg -i squid_3.5.12-1ubuntu7.5_amd64.deb squid-common_3.5.12-1ubuntu7.5_all.deb squid-dbg_3.5.12-1ubuntu7.5_amd64.deb  

检查一下新的squid是否支持ssl了

1squid -v|grep ssl  
2configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' 'BUILDCXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--with-openssl' '--enable-ssl' '--enable-ssl-crtd' '--enable-build-info=Ubuntu linux' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'
3
4cd /usr/lib/squid  
5ls ssl_crtd  

Gen出ssl的证书和密钥并拷贝到正确的位置,更新ca-certificates:

 1openssl genrsa -out squid.key 2048
 2
 3openssl req -new -key squid.key -out squid.csr  
 4You are about to be asked to enter information that will be incorporated  
 5into your certificate request.  
 6What you are about to enter is what is called a Distinguished Name or a DN.  
 7There are quite a few fields but you can leave some blank  
 8For some fields there will be a default value,  
 9If you enter '.', the field will be left blank.  
10-----
11Country Name (2 letter code) [AU]:CN  
12State or Province Name (full name) [Some-State]:Beijing  
13Locality Name (eg, city) []:Beijing  
14Organization Name (eg, company) [Internet Widgits Pty Ltd]:Rendoumi.com  
15Organizational Unit Name (eg, section) []:Rendoumi.com  
16Common Name (e.g. server FQDN or YOUR name) []:159.89.116.192  
17Email Address []:
18
19Please enter the following 'extra' attributes  
20to be sent with your certificate request  
21A challenge password []:  
22An optional company name []:
23
24
25openssl x509 -req -days 3650 -in squid.csr -signkey squid.key -out squid.crt  
26Signature ok  
27subject=/C=CN/ST=Beijing/L=Beijing/O=Rendoumi.com/OU=Rendoumi.com/CN=159.89.116.192  
28Getting Private key
29
30sudo cp squid.crt /usr/local/share/ca-certificates
31
32sudo /usr/sbin/update-ca-certificates  
33Updating certificates in /etc/ssl/certs...  
341 added, 0 removed; done.  
35Running hooks in /etc/ca-certificates/update.d...  
36done.
37
38sudo cp squid.pem /etc/squid  

修改/etc/squid.conf配置文件

 1cd /etc/squid  
 2cat squid.conf|grep -v ^# | grep -v ^$
 3
 4sudo vi /etc/squid/squid.conf  
 5----------------------------------------
 6acl SSL_ports port 443  
 7acl Safe_ports port 80          # http  
 8acl Safe_ports port 21          # ftp  
 9acl Safe_ports port 443         # https  
10acl Safe_ports port 70          # gopher  
11acl Safe_ports port 210         # wais  
12acl Safe_ports port 1025-65535  # unregistered ports  
13acl Safe_ports port 280         # http-mgmt  
14acl Safe_ports port 488         # gss-http  
15acl Safe_ports port 591         # filemaker  
16acl Safe_ports port 777         # multiling http  
17acl CONNECT method CONNECT  
18acl localnet src 10.8.0.0/16
19
20http_access deny !Safe_ports  
21http_access deny CONNECT !SSL_ports
22
23http_access allow localhost manager  
24http_access deny manager
25
26http_access allow localhost  
27http_access allow localnet  
28http_access deny all
29
30coredump_dir /var/spool/squid
31
32refresh_pattern ^ftp:           1440    20%     10080  
33refresh_pattern ^gopher:        1440    0%      1440  
34refresh_pattern -i (/cgi-bin/|\?) 0     0%      0  
35refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880  
36# example lin deb packages
37#refresh_pattern (\.deb|\.udeb)$   129600 100% 129600
38refresh_pattern .               0       20%     4320
39
40shutdown_lifetime 3
41
42http_port  3128 intercept  
43https_port 3129 intercept ssl-bump  generate-host-certificates=on version=1 options=NO_SSLv2,NO_SSLv3,SINGLE_DH_USE dynamic_cert_mem_cache_size=4MB cert=/etc/squid/squid.pem
44
45always_direct allow all  
46ssl_bump none localhost  
47ssl_bump server-first all  
48sslproxy_cert_error allow all  
49sslproxy_flags DONT_VERIFY_PEER  
50sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB  
51sslcrtd_children 8 startup=1 idle=1  
52----------------------------------------

初始化ssl_db

1sudo /usr/lib/squid/ssl_crtd -c -s /var/lib/ssl_db/  
2chown -R proxy /var/lib/ssl_db  

重启squid

1sudo systemctl restart squid.service  

特别的一点,雇主写了巨多的ufw的规则,导致IPTABLE爆满,居然无法手动清除所有的规则,这也是第一次遇到这样的,只能写脚本清除,方法如下:

 1vi cl.txt  
 2-----------------------
 3# Empty the entire filter table
 4*filter
 5:INPUT ACCEPT [0:0]
 6:FORWARD ACCEPT [0:0]
 7:OUTPUT ACCEPT [0:0]
 8COMMIT  
 9-----------------------
10
11sudo iptables-restore < cl  

最后修改IPTABLES,把80和443的请求都送到squid去

1sudo vi /etc/rc.local  
2iptables -t nat -A PREROUTING -p tcp -s 10.8.0.0/24 --dport 80 -j REDIRECT --to-ports 3128  
3iptables -t nat -A PREROUTING -p tcp -s 10.8.0.0/24 --dport 443 -j REDIRECT --to-ports 3129  
4iptables -t nat -A POSTROUTING -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to 159.89.116.192  

用代理访问,验证一下:

1sudo tail -f /var/log/squid/access.log  

这样做有个问题,就是https实际是被劫持代理了,所以客户端会弹出个是否信任证书,除非在每个客户端中预埋,这样才能解决。

That is all.


Freelancer任务之五多线路聚合vpn
Freelancer任务之三:Setup Proxy on VPS for Instagram
comments powered by Disqus