среда, 26 января 2011 г.

Squid: Оптимизируем transparent proxy на Solaris 10 - неблокирующий IO

Я уже писал о том, как построить transparent proxy на базе Squid на Solaris 10. Пришла пора заняться его оптимизацией.

Лично мне, в процессе эксплуатации, не очень понравилось время отклика сервера, достигающее 0,3 секунды.

Тщательное исследование показало несколько достаточно узких мест.

Первое - это избыточное кэширование данных файловой системой.

Так как мы используем приложение, которое само по себе является кэшем, кэш второго уровня ZFS явно избыточный. Отключим его:


root @ ktulhu / # zfs get primarycache,secondarycache,recordsize data/cache
NAME PROPERTY VALUE SOURCE
data/cache primarycache all local
data/cache secondarycache none local
data/cache recordsize 2K local

Следующий шаг - собранный на www.sunfreeware.com сквид нас не устраивает. В нем не включена опция асинхронного IO, и включены некоторые избыточные опции в случае, если включить этот самый саинхронный IO. Возьмем исходники и пересоберем его. Так как сервер прозрачный, можно с чистой совестью выкинуть поддержку SSL, а также SNMP (для безопасности) и unlinkd (это лишнее при использовании асинхронного IO).

Для ленивых команда конфигурирования (у моего сервера 4 ядра и два диска в зеркале):


./configure '--disable-unlinkd' '--enable-default-err-language=Russian-1251' --enable-err-languages='Russian-1251 English' '--enable-follow-x-forwarded-for' '--enable-ipf-transparent' '--enable-storeio=ufs,diskd' '--enable-delay-pools' '--enable-async-io=4' '--prefix=/usr/local/squid' '--enable-external-acl-helpers=ldap_group' '--enable-removal-policies=lru,heap' '--enable-large-cache-files' 'CC=gcc' 'CFLAGS=-O2 -march=i686 -L/usr/local/lib -R/usr/local/lib -L/usr/local/ssl/lib -R/usr/local/ssl/lib -L/usr/openwin/lib -R/usr/openwin/lib -I/usr/local/rrdtool-1.4.2/include -I/usr/local/BerkeleyDB.4.7/include -I/usr/local/mysql/include' 'LDFLAGS=-L/usr/local/lib -R/usr/local/lib -R/usr/lib -L/usr/lib -R/usr/openwin/lib -L/usr/openwin/lib -L/usr/local/ssl/lib -R/usr/local/ssl/lib -L/usr/X11R6/lib -R/usr/X11R6/lib -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib -L/usr/local/mysql/lib -R/usr/local/mysql/lib' 'CPPFLAGS=-I/usr/local/include -I/usr/local/ssl/include -I/usr/local/include/ncurses -I/usr/openwin/include -I/usr/local/rrdtool-1.4.2/include -I/usr/local/BerkeleyDB.4.7/include -I/usr/local/include/pcap -I/usr/local/include/freetype2'

Результат сборки:


squid @ ktulhu /usr/local/squid/bin $ squid -v
Squid Cache: Version 2.7.STABLE9
configure options: '--disable-unlinkd' '--enable-default-err-language=Russian-1251' '--enable-err-languages=Russian-1251 English' '--enable-follow-x-forwarded-for' '--enable-ipf-transparent' '--enable-storeio=ufs,diskd' '--prefix=/usr/local/squid' '--enable-external-acl-helpers=ldap_group' '--enable-removal-policies=lru,heap' '--enable-large-cache-files' '--enable-cache-digests' 'CC=gcc' 'CFLAGS=-O2 -march=i686 -L/usr/local/lib -R/usr/local/lib -L/usr/local/ssl/lib -R/usr/local/ssl/lib -L/usr/openwin/lib -R/usr/openwin/lib -I/usr/local/rrdtool-1.4.2/include -I/usr/local/BerkeleyDB.4.7/include -I/usr/local/mysql/include' 'LDFLAGS=-L/usr/local/lib -R/usr/local/lib -R/usr/lib -L/usr/lib -R/usr/openwin/lib -L/usr/openwin/lib -L/usr/local/ssl/lib -R/usr/local/ssl/lib -L/usr/X11R6/lib -R/usr/X11R6/lib -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib -L/usr/local/mysql/lib -R/usr/local/mysql/lib' 'CPPFLAGS=-I/usr/local/include -I/usr/local/ssl/include -I/usr/local/include/ncurses -I/usr/openwin/include -I/usr/local/rrdtool-1.4.2/include -I/usr/local/BerkeleyDB.4.7/include -I/usr/local/include/pcap -I/usr/local/include/freetype2'

Требуется внесение некоторых изменений в конфигурацию. Главное изменение - вместо ufs в директиве cache_dir будет стоять diskd (модуль aufs на Solaris не применяется), и для полного счастья добавим четыре файловых системы, на каждую из которых будем запускать отдельный процесс IO:

root @ ktulhu / # zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 3.01G 257G 25K /data
data/backup 2.96G 257G 2.96G /data/backup
data/cache 51.1M 64.0G 31K /data/cache
data/cache/d1 7.77M 64.0G 7.77M /data/cache/d1
data/cache/d2 8.14M 64.0G 8.14M /data/cache/d2
data/cache/d3 9.56M 64.0G 9.56M /data/cache/d3
data/cache/d4 7.82M 64.0G 7.82M /data/cache/d4
data/cache/log 17.8M 64.0G 17.8M /data/cache/log

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /usr/loal/squid/var/cache 100 16 256
cache_dir diskd /data/cache/d1 16384 16 256 Q1=72 Q2=64
cache_dir diskd /data/cache/d2 16384 16 256 Q1=72 Q2=64
cache_dir diskd /data/cache/d3 16384 16 256 Q1=72 Q2=64
cache_dir diskd /data/cache/d4 16384 16 256 Q1=72 Q2=64

На каждую отдельную директорию монтирования датасетов стартует отдельный процесс ввода-вывода diskd. Разумеется, директории кэша придется пересоздать.

И немного подправим конфигурацию самого сквида - как с целью оптимизации, так и в плане функциональности (буферизация логов, LRU-полиси, кэширование флэша, RealMedia, YouTube, refresh_patterns и немного тюнинга):

#Recommended minimum configuration:
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
#
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
#
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http

acl CONNECT method CONNECT

# Local Apache server address
acl localapache dst 192.168.1.20

# Common methods acls
acl GET method GET
acl POST method POST

# Numeric IP's acl
acl numeric_IPs url_regex ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+

# Media and video acls
acl youtube dstdomain .youtube.com
acl media rep_mime_type -i video/flv
acl mediapr urlpath_regex \.flv(\?.*)?$
acl media rep_mime_type -i ^application/x-shockwave-flash$
acl mediapr urlpath_regex \.swf(\?.*)?$

# Real audio acls
acl RealAudio_mime req_mime_type application/x-pncmd

# TAG: http_access
# Allowing or Denying access based on defined access lists
#
# Access to the HTTP port:
# http_access allow|deny [!]aclname ...
#
# NOTE on default values:
#
# If there are no "access" lines present, the default is to deny
# the request.
#
# If none of the "access" lines cause a match, the default is the
# opposite of the last line in the list. If the last line was
# deny, the default is allow. Conversely, if the last line
# is allow, the default will be deny. For these reasons, it is a
# good idea to have an "deny all" or "allow all" entry at the end
# of your access lists to avoid potential confusion.
#
#Default:
# http_access deny all
#
#Recommended minimum configuration:
#
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
# Deny access to numeric IP's
http_access deny CONNECT numeric_IPs all
# Deny requests to unknown ports
http_access deny !Safe_ports
# Deny CONNECT to other than SSL ports
http_access deny CONNECT !SSL_ports
#
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
http_access deny to_localhost
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet

# Required to work caching from local cache
http_access allow localhost

# Always direct directives
always_direct allow localapache

# Cache directives
cache allow youtube
cache allow media
cache allow mediapr
cache allow POST RealAudio_mime
# Required to work common refresh patterns
cache allow all

# Hide internal networks details outside
forwarded_for off
header_access X-Forwarded-For deny all
header_access Via deny all

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128 transparent

# Cache manager
cache_mgr yourname@yourdomain.com

# Cache manager password
cachemgr_passwd XXXXXXXXXXX all

# Cache user
cache_effective_user squid
cache_effective_group squid

# We recommend you to use at least the following line.
hierarchy_stoplist cgi-bin ?

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /usr/loal/squid/var/cache 100 16 256
cache_dir diskd /data/cache/d1 16384 16 256 Q1=72 Q2=64
cache_dir diskd /data/cache/d2 16384 16 256 Q1=72 Q2=64
cache_dir diskd /data/cache/d3 16384 16 256 Q1=72 Q2=64
cache_dir diskd /data/cache/d4 16384 16 256 Q1=72 Q2=64

# Memory parameters
cache_mem 512 Mb
memory_pools off
maximum_object_size 32 MB
maximum_object_size_in_memory 512 KB

# -------------------------------------
# Tuning parameters
# -------------------------------------
ipcache_size 8192
ipcache_low 95
ipcache_high 98

fqdncache_size 8192

buffered_logs on

memory_replacement_policy heap GDSF
#cache_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
#cache_replacement_policy heap LRU

half_closed_clients off
#snmp_port 0
icp_port 0

cache_swap_high 98
cache_swap_low 95
# -------------------------------------
# Tuning parameters
# -------------------------------------

# Apparently youtube.com use 'Range' requests
# - not seen, but presumably when a video is stopped for a long while then resumed, (or fast-forwarded).
# - convert range requests into a full-file request, so squid can cache it
# NP: BUT slows down their _first_ load time.
quick_abort_min -1 KB

# Access log
access_log /data/cache/log/access.log squid

# Cache log
cache_log /data/cache/log/cache.log

# Store log
cache_store_log none
#cache_store_log /data/cache/log/store.log

# Leave coredumps in the first cache dir
#coredump_dir /usr/local/squid/var/cache
coredump_dir /var/core

# Pid file
pid_filename /usr/local/squid/var/logs/squid.pid

# Add any of your own refresh_pattern entries above these.
# Own refresh patterns
refresh_pattern -i \.(gif|png|jp?g|ico|bmp|tiff?)$ 10080 95% 43200 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(html|htm|css|js)$ 1440 75% 40320
refresh_pattern -i \.index.(html|htm)$ 0 75% 10080
refresh_pattern -i \.(class|css|js|tif)(\?.*)?$ 1440 95% 100000080 reload-into-ims override-lastmod
refresh_pattern -i \.(jpe|jpg|jpeg|png|bmp|gif)(\?.*)?$ 0 95% 1000000080 reload-into-ims override-lastmod
refresh_pattern -i \.(asp|acgi|pl|shtml|php3|php)(\?.*)?$ 2 20% 432000 reload-into-ims override-lastmod

refresh_pattern \.bz2$ 43200 100% 43200 override-lastmod override-expire ignore-reload ignore-no-cache
refresh_pattern \.exe$ 43200 100% 43200 override-lastmod override-expire ignore-reload ignore-no-cache
refresh_pattern \.gz$ 43200 100% 43200 override-lastmod override-expire ignore-reload ignore-no-cache
refresh_pattern \.tar$ 43200 100% 43200 override-lastmod override-expire ignore-reload ignore-no-cache
refresh_pattern \.tgz$ 43200 100% 43200 override-lastmod override-expire ignore-reload ignore-no-cache
refresh_pattern \.zip$ 43200 100% 43200 override-lastmod override-expire ignore-reload ignore-no-cache

# Break some HTTP standard for flash videos. Keep them in cache even if asked not to.
refresh_pattern -i \.flv$ 10080 90% 999999 ignore-no-cache ignore-private

#refresh_pattern ^ftp: 1440 20% 10080
#refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
#refresh_pattern . 0 20% 4320
refresh_pattern . 0 80% 14400

# SquidGuard rewriter
url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf
url_rewrite_children 96
redirect_rewrites_host_header on
#redirector_bypass on

Я сознательно оставил некоторые параметры, позволяющие в случае необходимости сделать сервер непрозрачным прокси.

Собственно, это все. Перезапускаем сквид и - вуаля! - отклик 0,1 секунды максимум (разумеется, после прогрева кэша).