4.6.2.12. URL filtering in HTTP

The integrated category-based URL filtering solution maintains a permanently updated database. During the installation of the URL filtering database the administrator can choose the size of the URL filtering database. The database can be a smaller-sized, optimized database (the recommended version) for usual scenarios, which requires 1 GB storage space and 300 MB daily update traffic, or a normal database for more extensive scenarios, which requires 6 GB storage space and 2 GB daily update traffic. Note, that during the upgrade, the 6 GB storage space usage will temporarily increase to 18-20 GB.

How URL filtering works

The PNS URL filter checks the URL of the HTTP requests and compares them to a categorized database. The URLs and the domains in the database are organized into thematic categories like adult, health, business, and so on. If the requested URL is listed in the database, the categories matching the URL are assigned to the request. PNS can accept, redirect, or reject the HTTP request based on the category it belongs to.

Typically, accessing a webpage involves several HTTP requests. URL filtering is applied to every single HTTP request, meaning that different parts of the webpage can belong to different categories. PNS can remove the parts belonging to unwanted categories, and permit access only to the remaining part. For example, this can be used to prohibit access to advertisements or videos, but to permit access to other parts of the website.

Note

URL filtering is handled by the PNS Http proxy, without the need of using CF. The URL filtering capability of PNS is available only after purchasing the url-filter license option.

Updates to the URL database are automatically downloaded daily from the BalaSys website using the vavupdate utility.

Configuring URL filtering in HTTP

Prior to the update to the new URL filter, it is necessary to check whether at least 1.7 GB free space is avaialble on the partition including the /var/lib/vela/urlfilter/ folder. Following the update, the partition will utilize close to 850 MB space.

To enable url-filtering, follow the forthcoming steps:

List of URL-filtering categories

The PNS URL database contains the following thematic categories by default.

Main categorySubcategory

Adult

Abortion, Abortion Pro Choice, Abortion Pro Life, Child Inappropriate, Gambling, Gay, Lesbian, Bisexual, Lingerie, Suggestive and Pinap, Nudity, Pornography, R-Rated, Sex and Erotic, Sex Education and Pregnancy, Tobacco

Aggressive

Military, Violence, Weapons, Aggressive - Other

Arts

Fine Art, Arts - Other

Automotive

Auto Parts, Auto Repair, Buying/Selling Cars, Car Culture, Certified Pre-Owned, Convertible, Coupe, Crossover, Diesel, Electric Vehicle, Hatchback, Hybrid, Luxury, MiniVan, Motorcycles, Off-Road Vehicles, Performance Vehicles, Pickup, Road-Side Assistance, Sedan, Trucks & Accessories, Vintage Cars, Wagon, Automotive - Other

Business

Agriculture, Biotechnology, Business Software, Construction, Forestry, Government, Green Solutions & Conservation, Home & Office Furnishings, Human Resources, Manufacturing, Marketing Services, Metals, Physical Security, Productivity, Retirement Homes & Assisted Living, Shipping & Logistics,Business - Other

Careers

Career Advice, Career Planning, College, Financial Aid, Job Fairs, Job Search, Nursing, Resume Writing/Advice, Scholarships, Telecommuting, U.S. Military, Careers - Other

Criminal Activities

Child Abuse Images, Criminal Skills, Hacking, Hate Speech, Illegal Drugs, Marijuana, Piracy & Copyright Theft, School Cheating, Self Harm, Torrent Repository, Criminal Activities - Other

Dynamic

Anonymizer, Chat, Community Forums, Instant Messenger, Login Screens, Personal Pages & Blogs, Photo Sharing, Professional Networking, Redirect, Social Networking, Text Messaging & SMS, Translator, Web-based Email, Web-based Greeting Card

Education

7-12 Education, Adult Education, Art History, College Administration, College Life, Distance Learning, Educational Institutions, Educational Materials & Studies, English as a 2nd Language, Graduate School, Homeschooling, Homework/Study Tips, K-6 Educators, Language Learning, Literature & Books, Private School, Reference Materials & Maps, Special Education, Studying Business, Tutoring, Wikis, Education - Other

Entertainment

Entertainment News & Celebrity Sites, Entertainment Venues & Events, Humor, Movies, Music, Streaming & Downloadable Audio, Streaming & Downloadable Video, Television, Entertainment - Other

Family and Parenting

Adoption, Babies and Toddlers, Daycare/Pre School, Eldercare, Family Internet, Parenting - K-6 Kids, Parenting Teens, Pregnancy, Special Needs Kids, Family & Parenting - Other

Fashion

Accessories, Beauty, Body Art, Clothing, Fashion, Jewelry, Swimsuits, Fashion - Other

Finance

Accounting, Banking, Beginning Investing, Credit/Debt & Loans, Financial News, Financial Planning, Hedge Fund, Insurance, Investing, Mutual Funds, Online Financial Tools & Quotes, Options, Retirement Planning, Stocks, Tax Planning, Finance - Other

Food and Drink

American Cuisine, Barbecues & Grilling, Cajun/Creole, Chinese Cuisine, Cocktails/Beer, Coffee/Tea, Cuisine-Specific, Desserts & Baking, Dining Out, Food Allergies, French Cuisine, Health/Low fat Cooking, Italian Cuisine, Japanese Cuisine, Mexican Cuisine, Vegan, Vegetarian, Winer, Food & Drink - Other

Health

A.D.D., AIDS/HIV, Allergies, Alternative Medicine, Arthritis, Asthma, Autism/PDD, Bipolar Disorder, Brain Tumor, Cancer, Children's Health, Cholesterol, Chronic Fatigue, Chronic Pain, Cold & Flu, Cosmetic Surgery, Deafness, Dental Care, Depression, Dermatology, Diabetes, Disorders, Epilepsy, Exercise, GERD/Acid Reflux, Headaches/Migraines, Heart Disease, Herbs for Health, Holistic Healing, IBS/Crohn’s Disease, Incest/Abuse Support, Incontinence, Infertility, Men’s Health, Nutrition & Diet, Orthopedics, Panic/Anxiety, Pediatrics, Pharmaceuticals, Physical Therapy, Psychology/PsychiatrySelf-help & Addiction, Senior Health, Sexuality, Sleep Disorders, Smoking Cessation, Supplements & Compounds, Syndrome, Thyroid Disease, Weight Loss, Women’s Health, Health - Other

Hobbies and Interests

Art/Technology, Arts & Crafts, Beadwork, Birdwatching, Board Games/Puzzles, Candle & Soap Making, Card Games, Cartoons, Anime & Comic Books, Chess, Cigars, Collecting, Comic Books, Drawing/Sketching, Freelance Writing, Genealogy, Getting Published, Guitar, Home Recording,Investors & Patents, Jewelry Making, Magic & Illusion, Needlework, Painting, Photography, Radio, Roleplaying Games, Sci-Fi & Fantasy, Scrapbooking, Screenwriting, Stamps & Coins, Themes, Video & Computer Games, Woodworking, Hobbies & Interests - Other

House and Garden

Appliances, Entertaining, Environmental Safety, Gardening, Home Repair., Home Theater, Interior Decorating, Landscaping, Remodeling & Construction, Home & Garden - Other

Kids

Games, Kid's Pag, Toys, Kids - Other

Lifestyle

Dating & Relationships, Divorce Support, Ethnic Specific, Marriage, Parks, Rec Facilities & Gyms, Senior Living, Teens, Weddings, Lifestyle - Other

Malicious

Ad Fraud, Botnet, Command and Control Centers, Compromised & Links To Malware, Malware Call-Home, Malware Distribution Point, Phishing/Fraud, Spam URLs, Spyware & Questionable Software

Miscellaneous

Content Server, No Content Found, Parked & For Sale Domains, Private IP Address, Unreachable, Miscellaneous - Other

News, Portal and Search

Image Search, International News, Local News, Magazines, National News, Portal Sites, Search Engines, News, Portal & Search - Other

Online Ads

Pay-to-Surf, Online Ads - Other

Pets

Aquariums, Birds, Cats, Dogs, Large Animals, Reptiles, Veterinary Medicine, Pets - Other

Public, Government and Law

Advocacy Groups & Trade Associations, Commentary, Government Sponsored, Immigration, Legal Issues, Philanthropic Organizations, Politics, Social & Affiliation Organizations, U.S. Government Resources, Public, Government & Law - Other

Real Estate

Apartments, Architects, Buying/Selling Homes, Real Estate - Other

Religion

Alternative Religions, Atheism & Agnosticism, Buddhism, Catholicism, Christianity, Hinduism, Islam, Judaism, Latter-Day Saints, Non-traditional Religion and Occult, Pagan/Wiccan, Religion - Other

Science

Anatomy, Astrology and Horoscopes, Biology, Botany, Chemistry, Weather, Geography, Geology, Paranormal Phenomena, Physics, Space/Astronomy, Science - Other

Shopping

Auctions & Marketplaces, Catalogs, Contests & Surveys, Shopping - Online, Engines, Product Reviews & Price Comparisons, Coupons, Shopping - Other

Sports

Auto Racing, Baseball, Bicycling, Bodybuilding, Boxing, Canoeing/Kayaking, Cheerleading, Climbing, Cricket, Figure Skating, Fly Fishing, Football, Freshwater Fishing, Game & Fish, Golf, Horse Racing, Horses, Inline Skating, Martial Arts, Mountain Biking, NASCAR Racing, Olympics, Sports - Other, Paintball, Power & Motorcycles, Pro Basketball, Pro Ice Hockey, Rodeo, Rugby, Running/Jogging, Sailing, Saltwater Fishing, Scuba Diving, Skateboarding, Skiing, Snowboarding, Sport Hunting, Surfing/Bodyboarding, Swimming, Table Tennis/Ping-Pong, Tennis, Volleyball, Walking, Waterski/Wakeboard, World Soccer

Technology

3-D Graphics, Animation, Antivirus Software, C/C++, Cameras & Camcorders, Computer Certification, Computer Networking, Computer Peripherals, Computer Reviews, Databases, Desktop Publishing, Desktop Video, File Repositories, Graphics Software, Home Video/DVD, Information Security, Internet Phone & VOIP, Internet Technology, Java, Javascript, Linux, Mac OS, Technology - Other, Mac Support, Mobile Phones, MP3/MIDI, Net Conferencing, Net for Beginners, Network Security, Palmtops/PDAs, PC Support, Peer-to-Peer, Personal Storage, Portable, Entertainment, Remote Access, Shareware/Freeware, Unix, Utilities, Visual Basic, Web Clip Art, Web Design/HTML, Web Hosting, ISP & Telco, Windows, Online Information Management

Travel

Adventure Travel, Africa, Air Travel, Australia & New Zealand, Bed & Breakfast, Budget Travel, Business Travel, By US Locale, Camping, Canada, Caribbean, Cruises, Eastern Europe, Europe, Travel - Other, France, Greece, Honeymoons/Getaways, Hotels, Italy, Japan, Mexico & Central America, National Parks, Navigation, South America, Spas, Theme Parks, Traveling with Kids, United Kingdom

Table 4.12. URL filter categories

Customizing the URL database

Create blacklist and whitelist if necessary. This can be achieved with the help of free text editor, on the PNS Node → New → Text editor → URL filter black and whitelists path.

Editor for URL filter blacklist and whitelist

Figure 4.5. Editor for URL filter blacklist and whitelist

The choices for blacklist, whitelist or * options are also put into the category list:

The 'whitelist' and 'blacklist' options are at the end of the category list

Figure 4.6. The 'whitelist' and 'blacklist' options are at the end of the category list