.htaccess, robots.txt, and chmod — A Practical Server Configuration Guide

Three files sit at the intersection of web server configuration and security: .htaccess, robots.txt, and Unix file permissions. Most developers encounter all three early in their career, get them working well enough, and never revisit them. That tends to leave subtle problems — misconfigured redirects, crawlers accessing pages they should not, files with permissions that are too open.

This guide covers all three practically, with the patterns you will actually use.

.htaccess — what it does and when to use it

.htaccess is an Apache configuration file that applies to the directory it sits in and all subdirectories. It lets you configure redirects, URL rewriting, access control, caching headers, and more without touching the main server configuration.

The most important thing to know about .htaccess is that it has a performance cost. Apache reads it on every request. For high-traffic sites, moving the same configuration into the main httpd.conf or a virtual host block is faster. For shared hosting where you cannot edit the main config, .htaccess is often your only option.

Essential .htaccess patterns

Force HTTPS. This should be in every production .htaccess:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Remove www (or add it). Pick one and be consistent. Remove www:

RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ https://%1%{REQUEST_URI} [L,R=301]

Laravel / framework front-controller. Route all requests through index.php:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [L]

Block access to sensitive files. Never expose these:

<FilesMatch "^(\.env|composer\.json|composer\.lock|package\.json)$">
    Order allow,deny
    Deny from all
</FilesMatch>

Custom error pages:

ErrorDocument 404 /404.html
ErrorDocument 500 /500.html

Rather than writing these by hand, the .htaccess Generator builds the rules you need through a form — HTTPS redirect, www handling, framework routing, security headers, caching, and more. Copy the output directly into your file.

If you are running Nginx instead of Apache, the Server Config Generator covers both Nginx server blocks and Apache VirtualHost configs for static sites, PHP-FPM, reverse proxy, and Node.js.

robots.txt — controlling crawlers

robots.txt sits at the root of your domain and tells search engine crawlers which pages they should and should not index. It is a convention, not a security mechanism — any crawler can ignore it, and bad bots routinely do. Never put a page behind robots.txt alone and consider it private.

A minimal robots.txt that allows all crawlers:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block specific paths from all crawlers:

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /staging/
Allow: /

Block a specific crawler entirely (useful for aggressive scrapers):

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

Common mistakes: blocking / entirely by accident, forgetting to include the Sitemap directive, and putting pages behind Disallow that you still want indexed. The robots.txt Generator builds the file through a form and prevents these mistakes.

Unix file permissions — chmod

Unix permissions control who can read, write, and execute each file and directory. They are represented as a three-digit octal number (like 755) or in symbolic notation (like rwxr-xr-x). Each digit covers owner, group, and others respectively. Read is 4, write is 2, execute is 1. Add them together for each role.

The permissions you actually need in practice:

755  — directories (owner can write, everyone can read and traverse)
644  — regular files (owner can write, everyone can read)
600  — private files like .env (only owner can read and write)
400  — read-only files (only owner can read, nobody can write)
775  — shared directories in a group context
640  — files readable by owner and group only

For a web server running as www-data, your application files should typically be owned by your deploy user with group www-data. Files should be 644, directories 755. Storage and cache directories that the web server needs to write to should be 775 with group ownership.

A common mistake is setting everything to 777 to fix a permission error. This gives every user on the server write access to your files and is a serious security issue on shared hosting. Use the chmod Calculator to get the right number without guessing — toggle the checkboxes for owner, group, and others and see the numeric and symbolic notation update instantly.

Permission problems on Laravel and shared hosting

Laravel requires the storage/ and bootstrap/cache/ directories to be writable by the web server. The correct approach on shared hosting:

chmod -R 775 storage
chmod -R 775 bootstrap/cache
chown -R your_user:www-data storage
chown -R your_user:www-data bootstrap/cache

Never chmod 777 these directories. If your hosting forces you to, that is a sign of a poorly configured shared environment.

Tools

• .htaccess Generator — build redirect, security, and rewrite rules without memorising the syntax
• robots.txt Generator — configure crawler access with a form-based interface
• chmod Calculator — get the right permission number with a visual checkbox grid
• Server Config Generator — generate Nginx and Apache configs for any stack