Tips for debugging .htaccess rewrite rules

Question

Many posters have problems debugging their RewriteRule and RewriteCond statements within their .htaccess files. Most of these are using a shared hosting service and therefore don't have access to the root server configuration. They cannot avoid using .htaccess files for rewriting and cannot enable a RewriteLogLevel" as many respondents suggest. Also there are many .htaccess-specific pitfalls and constraints are aren't covered well. Setting up a local test LAMP stack involves too much of a learning curve for most.

So my Q here is how would we recommend that they debug their rules themselves. I provide a few suggestions below. Other suggestions would be appreciated.

1. Understand that the mod_rewrite engine cycles through .htaccess files. The engine runs this loop:

   do
     execute server and vhost rewrites (in the Apache Virtual Host Config)
     find the lowest "Per Dir" .htaccess file on the file path with rewrites enabled
     if found(.htaccess)
        execute .htaccess rewrites (in the user's directory)
   while rewrite occurred
   

   So your rules will get executed repeatedly and if you change the URI path then it may end up executing other .htaccessfiles if they exist. So make sure that you terminate this loop, if necessary by adding extra RewriteCond to stop rules firing. Also delete any lower level .htaccess rewrite rulesets unless explicitly intent to use multi-level rulesets.

2. Make sure that the syntax of each Regexp is correct by testing against a set of test patterns to make sure that is a valid syntax and does what you intend with a fully range of test URIs. See answer below for more details.

3. Build up your rules incrementally in a test directory. You can make use of the "execute the deepest .htaccess file on the path feature" to set up a separate test directory (tree) and debug rulesets here without screwing up your main rules and stopping your site working. You have to add them one at a time because this is the only way to localise failures to individual rules.

4. Use a dummy script stub to dump out server and environment variables. (See Listing 2)If your app uses, say, blog/index.php then you can copy this into test/blog/index.php and use it to test out your blog rules in the test subdirectory. You can also use environment variables to make sure that the rewrite engine in interpreting substitution strings correctly, e.g.

   RewriteRule ^(.*) - [E=TEST0:%{DOCUMENT_ROOT}/blog/html_cache/$1.html]
   

   and look for these REDIRECT_* variables in the phpinfo dump. BTW, I used this one and discovered on my site that I had to use %{ENV:DOCUMENT_ROOT_REAL} instead. In the case of redirector looping REDIRECT_REDIRECT_* variables list the previous pass. Etc..

5. Make sure that you don't get bitten by your browser caching incorrect 301 redirects. See answer below. My thanks to Ulrich Palha for this.

6. The rewrite engine seems sensitive to cascaded rules within an .htaccess context, (that is where a RewriteRule results in a substitution and this falls though to further rules), as I found bugs with internal sub-requests (1), and incorrect PATH_INFO processing which can often be prevents by use of the [NS], [L] and [PT] flags.

Any more comment or suggestions?

Listing 1 -- phpinfo

<?php phpinfo(INFO_ENVIRONMENT|INFO_VARIABLES);

Accepter

These are good... Perhaps you should move them from the question into an answer. – Doublejointed

@w00t, I've split off the regexp checker as per your suggestion because I want to refer it by link in other answers. – Accepter

You might want to add the control flow diagram from the docs to your first suggestion. IMO it's a lot easier to comprehend than any pseudocode or explanation, and this is really the blackest part of the mod-rewrite voodoo. – Pectoralis

Number 6 is a huge deal. Rewrite rules behaving differently in standard apache configuration files vs in .htaccess files must catch lots of people out. – Showbread

Something that may be worth adding to these hints: I spent some time debugging a problem with it redirecting and not rewriting. It turns out I had it rewriting to "/comment" when I wanted "/comment/". It was rewriting to "/comment" and then the server was doing a redirect to "/comment/". Obvious behaviour to those used to Apache but probably less so for noobs like me. – Banal

Where is the "listing 2" mentioned in this answer? – Hornstone

If you are new to htaccess and URL rewriting and want to learn you can follow this short tutorial helponnet.com/2021/04/15/htaccess-tutorial-for-beginers – Kufic

Answer 1

Here are a few additional tips on testing rules that may ease the debugging for users on shared hosting

1. Use a Fake-user agent

When testing a new rule, add a condition to only execute it with a fake user-agent that you will use for your requests. This way it will not affect anyone else on your site.

e.g

#protect with a fake user agent
RewriteCond %{HTTP_USER_AGENT}  ^my-fake-user-agent$
#Here is the actual rule I am testing
RewriteCond %{HTTP_HOST} !^www\.domain\.com$ [NC] 
RewriteRule ^ http://www.domain.com%{REQUEST_URI} [L,R=302] 

If you are using Firefox, you can use the User Agent Switcher to create the fake user agent string and test.

2. Do not use 301 until you are done testing

I have seen so many posts where people are still testing their rules and they are using 301's. DON'T.

If you are not using suggestion 1 on your site, not only you, but anyone visiting your site at the time will be affected by the 301.

Remember that they are permanent, and aggressively cached by your browser. Use a 302 instead till you are sure, then change it to a 301.

3. Remember that 301's are aggressively cached in your browser

If your rule does not work and it looks right to you, and you were not using suggestions 1 and 2, then re-test after clearing your browser cache or while in private browsing.

4. Use a HTTP Capture tool

Use a HTTP capture tool like Fiddler to see the actual HTTP traffic between your browser and the server.

While others might say that your site does not look right, you could instead see and report that all of the images, css and js are returning 404 errors, quickly narrowing down the problem.

While others will report that you started at URL A and ended at URL C, you will be able to see that they started at URL A, were 302 redirected to URL B and 301 redirected to URL C. Even if URL C was the ultimate goal, you will know that this is bad for SEO and needs to be fixed.

You will be able to see cache headers that were set on the server side, replay requests, modify request headers to test ....

---

Woke

Ulrich, thanks a lot for this input. You've picked up some aspects that I hadn't thought of putting in my list. On the 301 debugging issue, I use Chrome in "Private Browsing" (AKA "Porn-mode") as this dumps this state info when you close the window. I hope that you don't mind me not "accepting" this as is an important point, but not a single best answer. Thanks again. :) – Accepter

To make it clear (you do have it in your code but didn't spot it) but to make sure you are using a 302 not 301 redirect you need [L,R=302] – Disject

You don't need to explicitly specify [L, R=302] just do [L,R] the default is 302 – Strung

@goodeye, also look at "Chrome > Settings > General > Disable Cache while DevTools is open" checkbox. – Prudy

Answer 2

Online .htaccess rewrite testing

I found a htaccess tester while I was Googling for RegEx help, it saved me a lot of time from having to upload new .htaccess files every time I make a small modification.

from the site:

htaccess tester

To test your htaccess rewrite rules, simply fill in the url that you're applying the rules to, place the contents of your htaccess on the larger input area and press "Check Now" button.

htaccess.madewithlove.be

Grosbeak

Thanks for the pointer to this tool, which I found the most direct way to debug my problem. – Manufacturer

If you have ssh access to your webspace, another option is to change the .htaccess directly via editor on the server. – Placia

You will need to ignore the ssl warning b/c the sites certificate is having issues. But the site is still there. This is the best and easiest solution. It gives incredible insight into what's wrong and results in fixing the issue FAST. – Weigel

Thanks for pointing to this tool. It's helpful, Sometimes debuging the apache htaccess itself is sooo damn hard. Thanks. Big Thanks – Guib

Seems the link referenced is buggy and doesn't always give you the exact output. Please check on the actual apache to be absolutely sure. – Panelist

It looks nice, first. But it misses a lot of features. Unfortunately not a reliable tool. – Use

Very helpful, thanks so much! – Toggle

Answer 3

Don't forget that in .htaccess files it is a relative URL that is matched.

In a .htaccess file the following RewriteRule will never match:

RewriteRule ^/(.*)     /something/$s

Docent

Yes the string fed into a RewriteRule is relative and therefore stripped on any leading /, but this stripping doesn't occur for match strings assembled in RewriteCond commands. – Accepter

Answer 4

Set environment variables and use headers to receive them:

You can create new environment variables with RewriteRule lines, as mentioned by OP:

RewriteRule ^(.*) - [E=TEST0:%{DOCUMENT_ROOT}/blog/html_cache/$1.html]

But if you can't get a server-side script to work, how can you then read this environment variable? One solution is to set a header:

Header set TEST_FOOBAR "%{REDIRECT_TEST0}e"

The value accepts format specifiers, including the %{NAME}e specifier for environment variables (don't forget the lowercase e). Sometimes, you'll need to add the REDIRECT_ prefix, but I haven't worked out when the prefix gets added and when it doesn't.

Dozer

Have you gained further insight on when to use or not use the REDIRECT_ prefix? Also, I see terminology about prefixes in other (htaccess) contexts as well, but it has never been clear exactly what is meant. Does it mean that you must name your variable with the prefix, or add the prefix to your named variable, when using certain commands (but not other commands)? Your example is the first that shows both the var definition, and the var usage, so from this I'm inclined to think the latter! The docs have been little help -they assume we know too much, and give too few references/links. – Vladamir

Answer 5

Make sure that the syntax of each Regexp is correct

by testing against a set of test patterns to make sure that is a valid syntax and does what you intend with a fully range of test URIs.

See regexpCheck.php below for a simple script that you can add to a private/test directory in your site to help you do this. I've kept this brief rather than pretty. Just past this into a file regexpCheck.php in a test directory to use it on your website. This will help you build up any regexp and test it against a list of test cases as you do so. I am using the PHP PCRE engine here, but having had a look at the Apache source, this is basically identical to the one used in Apache. There are many HowTos and tutorials which provide templates and can help you build your regexp skills.

Listing 1 -- regexpCheck.php

<html><head><title>Regexp checker</title></head><body>
<?php 
    $a_pattern= isset($_POST['pattern']) ? $_POST['pattern'] : "";
    $a_ntests = isset($_POST['ntests']) ? $_POST['ntests'] : 1;
    $a_test   = isset($_POST['test']) ? $_POST['test'] : array();
    
    $res = array(); $maxM=-1; 
    foreach($a_test as $t ){
        $rtn = @preg_match('#'.$a_pattern.'#',$t,$m);
        if($rtn == 1){
            $maxM=max($maxM,count($m));
            $res[]=array_merge( array('matched'),  $m );
        } else {
            $res[]=array(($rtn === FALSE ? 'invalid' : 'non-matched'));
        }
    } 
?> <p>&nbsp; </p>
<form method="post" action="<?php echo $_SERVER['SCRIPT_NAME'];?>">
    <label for="pl">Regexp Pattern: </label>
    <input id="p" name="pattern" size="50" value="<?php echo htmlentities($a_pattern,ENT_QUOTES,"UTF-8");;?>" />
    <label for="n">&nbsp; &nbsp; Number of test vectors: </label>
    <input id="n" name="ntests"  size="3" value="<?php echo $a_ntests;?>"/>
    <input type="submit" name="go" value="OK"/><hr/><p>&nbsp;</p>
    <table><thead><tr><td><b>Test Vector</b></td><td>&nbsp; &nbsp; <b>Result</b></td>
<?php 
    for ( $i=0; $i<$maxM; $i++ ) echo "<td>&nbsp; &nbsp; <b>\$$i</b></td>";
    echo "</tr><tbody>\n";
    for( $i=0; $i<$a_ntests; $i++ ){
        echo '<tr><td>&nbsp;<input name="test[]" value="', 
            htmlentities($a_test[$i], ENT_QUOTES,"UTF-8"),'" /></td>';
        foreach ($res[$i] as $v) { echo '<td>&nbsp; &nbsp; ',htmlentities($v, ENT_QUOTES,"UTF-8"),'&nbsp; &nbsp; </td>';}
        echo "</tr>\n";
    }
?> </table></form></body></html>

Accepter

Quick Note: import_request_variables was deprecated in PHP 5.3 and removed in 5.4. extract($_GET) coupled with extract($_POST) can perform the same function, but all variables would need the prefix removed from their name. Source: php.net/manual/en/function.import-request-variables.php – Nonu

@watcher, thanks. I'd updated my local version to be 5.4 compatible a year ago, but forgot to change this posting. Now done. – Accepter

oh my, even after edit, can't get good results just by copying your code... but with regex fiddlers around, i reckon your tool is obsolete anyhow. check out these cool tools: regex101.com or refiddle.com or regexr.com – Homebrew

@hexereisoftware, this post is 3 years old, so there may be subtle issues depending on the PHP version that now use and the Apache version. However there are many variants of regexp each with subtle differences. As I said the Apache code uses a PCRE engine that is very similar to the PHP engine's. I am not sure what the diffs with the othe variants such as .Net are, so whilst your suggestion of using an online resource is a good one, I would stick with one which explicitly supports apache or PHP syntax. :-) – Accepter

Perl would ne closest, but php uses same syntax – Homebrew

Answer 6

One from a couple of hours that I wasted:

If you've applied all these tips and are only going on 500 errors because you don't have access to the server error log, maybe the problem isn't in the .htaccess but in the files it redirects to.

After I had fixed my .htaccess-problem I spent two more hours trying to fix it some more, even though I simply had forgotten about some permissions.

Attitudinarian

I use a shared access hosting webservice for my personal site, but what I've done is to set up a test VM which roughly mirrors this in terms of the PHP / Apache config, home directory, etc.. However because this VM under my admin I can enable rewrite logging to diagnose any difficult .htaccess issues. – Accepter

Answer 7

Make sure you use the percent sign in front of variables, not the dollar sign.

It's %{HTTP_HOST}, not ${HTTP_HOST}. There will be nothing in the error_log, there will be no Internal Server Errors, your regexp is still correct, the rule will just not match. This is really hideous if you work with django / genshi templates a lot and have ${} for variable substitution in muscle memory.

Coastland

Yes, the $ substitution variables relate to the last RewriteRule pattern and the % ones relate to the last RewriteCond pattern and specials such as %{env:XXX} – Accepter

Answer 8

If you're creating redirections, test with curl to avoid browser caching issues. Use -I to fetch http headers only. Use -L to follow all redirections.

Mervin

Answer 9

Regarding 4., you still need to ensure that your "dummy script stub" is actually the target URL after all the rewriting is done, or you won't see anything!

A similar/related trick (see this question) is to insert a temporary rule such as:

RewriteRule (.*) /show.php?url=$1 [END]

Where show.php is some very simple script that just displays its $_GET parameters (you can display environment variables too, if you want).

This will stop the rewriting at the point you insert it into the ruleset, rather like a breakpoint in a debugger.

If you're using Apache <2.3.9, you'll need to use [L] rather than [END], and you may then need to add:

RewriteRule ^show.php$ - [L]

At the very top of your ruleset, if the URL /show.php is itself being rewritten.

Jasen

Answer 10

Some mistakes I observed happens when writing .htaccess

Using of ^(.*)$ repetitively in multiple rules, using ^(.*)$ causes other rules to be impotent in most cases, because it matches all of the url in single hit.

So, if we are using rule for this url sapmle/url it will also consume this url sapmle/url/string.

---

[L] flag should be used to ensure our rule has done processing.

---

Should know about:

Difference in %n and $n

%n is matched during %{RewriteCond} part and $n is matches on %{RewriteRule} part.

Working of RewriteBase

The RewriteBase directive specifies the URL prefix to be used for per-directory (htaccess) RewriteRule directives that substitute a relative path.

This directive is required when you use a relative path in a substitution in per-directory (htaccess) context unless any of the following conditions are true:

The original request, and the substitution, are underneath the DocumentRoot (as opposed to reachable by other means, such as Alias). The filesystem path to the directory containing the RewriteRule, suffixed by the relative substitution is also valid as a URL path on the server (this is rare). In Apache HTTP Server 2.4.16 and later, this directive may be omitted when the request is mapped via Alias or mod_userdir.

Pareu

Answer 11

I found this question while trying to debug my mod_rewrite issues, and it definitely has some helpful advice. But in the end the most important thing is to make sure you have your regex syntax correct. Due to problems with my own RE syntax, installing the regexpCheck.php script was not a viable option.

But since Apache uses Perl-Compatible Regular Expressions (PCRE)s, any tool which helps writing PCREs should help. I've used RegexPlanet's tool with Java and Javascript REs in the past, and was happy to find that they support Perl as well.

Just type in your regular expression and one or more example URLs, and it will tell you if the regex matches (a "1" in the "~=" column) and if applicable, any matching groups (the numbers in the "split" column will correspond to the numbers Apache expects, e.g. $1, $2 etc.) for each URL. They claim PCRE support is "in beta", but it was just what I needed to solve my syntax problems.

http://www.regexplanet.com/advanced/perl/index.html

I'd have simply added a comment to an existing answer but my reputation isn't yet at that level. Hope this helps someone.

Inoculation

nice tool, but awful form... check out these cool tools: regex101.com or refiddle.com or regexr.com – Homebrew

Answer 12

In case you are not working in an standard shared hosting environment, but in one to which you have administration access (maybe your local test environment), make sure that use of .htaccess and mod_rewrite are enabled. They are disabled in a default Apache installation. And in that case, no action configured in your .htaccess file works, even if the regexes are perfectly valid.

To enable the use of .htaccess:

Find file apache2.conf, on Debian/Ubuntu this is in /etc/apache2, and within the file the section

<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

and change the line AllowOverride None to AllowOverride All.

To enable module mod_rewrite:

On Debian/Ubuntu, execute

sudo a2enmod rewrite

By the way, to disable a module, you would use a2dismode instead of a2enmode.

After you did the above configuration changes, restart Apache for them to take effect:

sudo systemctl restart apache2

Talesman

Answer 13

If you're planning on writing more than just one line of rules in .htacesss,
don't even think about trying one of those hot-fix methods to debug it.

I have wasted days setting multiple rules, without feedback from LOGs, only to finally give up.
I got Apache on my PC, copied the whole site to its HDD, and got the whole rule-set sorted out, using the logs, real fast.
Then I reviewed my old rules, which been working. I saw they were not really doing what was desired. A time bomb, given a slightly different address.

There are so many pit falls in rewrite rules, it's not a straight logic thing at all.
You can get Apache up and running in ten minutes, it's 10MB, good license, *NIX/WIN/MAC ready, even without install.
Also, check the header lines of your server and get the same version of Apache from their archive if it's old. My OP is still on 2.0; many things are not supported.

Adalie

papo, I've run dedicated servers, ISP-hosted VPS and private VMs within my development structure, but I still use a shared hosting service for my public domains and email, simply because it's more convenient and cost-effective to use a fully managed service for these. This howto is really targeted at shared service users. Configuring a private VM to mirror a shared service fully is difficult. Yes, if you can then using a test VM helps, but I still use these "tricks" from time to time on my shared service. – Accepter

I would agree with this if your A had been framed as an alternative suggestion for debugging mod_rewrite rules, but the opening "don't even think about it" is simply bad advice for basic shared service users who are struggling to understand why their htaccess files aren't working the way that they thing that they should. – Accepter

I am sorry if it sounded like your job of putting together a nice collection of advises was worthless. I didn't want that. Believe me, I was very happy to read and follow many advises this thread offered. But my rules slowly became complicated and at the end, I wasted a lot of time by just not wanting to go through the trouble of installing an Apache server and doing the debugging as it should be done. More than that, I learned nothing, as I didn't see, in logs, what was really happening. And there is a lot of stuff going on. I believe it's also valuable to share this experience. – Adalie

for the second part, there is an IF. My text never started with 'don't even think about' I see now, this wording seems a bit harsh but it's all true. Especially for those who are new to this and struggling to understand. Advises here might misguide them, like me, that all I need is a solid regex, it's not so simple, like your point 6) PATH_INFO cost me a lot of trouble and it's not a bug as you say, but a feature. If you don't want it to be re-added, use [DPI]. But only if you'll look at logs, you'll see it's being added there. That's why, more than one line, and you are better off using logs – Adalie

And for the third part, we are all experts here, in one field or another, fancy wordings will wow no-one here and it's not helping as you need nothing of that. For most cases, not even scripting and databases, as you don't need fully working site locally. Getting Apache running is real easy and because I didn't believe it was, I went the wrong way. That's also why I wanted to share this. And that's what we all want here, to share and help others with advises. I think it would be more helpful to include this option in you list instead of down-voting it. – Adalie

Sorry @papo, but my reason for the -1 vote was that I think that this "don't even think about it" is bad advice, IMO. If your point was that "above a certain complexity, then you might find it easier to install a local Apache service to debug your .htaccess files" then this is more balanced. Yes, it is reasonably easy to set up a local Apache service, but getting it to reflect a service provider's shared hosting service can be complicated, and beyond the skill level of many users who may have just used a one-click setup of Wordpress, say, and have issues with their .htaccess file. – Accepter

Answer 14

Best way to debug it!

Add LogLevel notice rewrite:trace8 to the httpd.conf of apache to log all notices of mod_rewrite. If you are at shared hosting and don't have access to httpd.conf then test it locally and upload to the live site. Once enabled this generate a very large log in very short time, it means it can't be tested on live server anyway.

Petiolate

Nice. What if I just want to see the value of an expression, outside of rewriting rules? – Durfee

Answer 15

I'll leave this here, maybe obvious detail, but got me banging my head for hours: be careful using %{REQUEST_URI} because what @Krist van Besien say in his answer is totally right, but not for the REQUEST_URI string, because the out put of this TestString starts with a /. So take care:

RewriteCond %{REQUEST_URI} ^/assets/$  
                            ^
                            | check this pesky fella right here if missing

Anticline

Answer 16

(Similar to Doin idea) To show what is being matched, I use this code

$keys = array_keys($_GET);
foreach($keys as $i=>$key){
    echo "$i => $key <br>";
}

Save it to r.php on the server root and then do some tests in .htaccess
For example, i want to match urls that do not start with a language prefix

RewriteRule ^(?!(en|de)/)(.*)$ /r.php?$1&$2 [L] #$1&$2&...
RewriteRule ^(.*)$ /r.php?nomatch [L] #report nomatch and exit

Refrigerator

just using a phpinfo() stub as I mentioned in point 4 on my O/P does basically the same thing. Look for QUERY_STRING – Accepter

Answer 17

as pointed out by @JCastell, the online tester does a good job of testing individual redirects against an .htaccess file. However, more interesting is the api exposed which can be used to batch test a list of urls using a json object. However, to make it more useful, I have written a small bash script file which makes use of curl and jq to submit a list of urls and parse the json response into a CSV formated output with the line number and rule matched in the htaccess file along with the redirected url, making it quite handy to compare a list of urls in a spreadsheet and quickly determine which rules are not working.

Risner

Answer 18

Perhaps the best way to debug rewrite rules is not to use rewrite rules at all, but to defer URL processing from the htaccess file to a PHP file (let's call it router.php). Then, you can use PHP to do any manipulating you like, with proper error detection and the usual ways to do debugging. This even runs faster, too, since you don't have to use the rewriting module.

To transfer control immediately from .htaccess to router.php for any URL that is not found in the file system, just put the following line in .htaccess:

FallbackResource router.php

Yes, it's really that easy. And yes, it really works. Give it a try.

Note: You may need an ErrorDocument directive in your .htacess file to transfer control explicitly for certain URLs to your router.php file on HTTP status 404, especially if you inherit from a parent htaccess file that handles status 404. So that would make it a total of two lines to transfer control to a router file.

Durfee

This would involve passing all URL requests through the PHP engine (or whatever scripting engine you use). Surely most PHP server-side applications use something like Apache2 or nginx + fpm-php which perhaps 70% of requests bare css, js, html, etc. This can be even more if you are using heavy client-side React (etc.) frameworks. You can certainly mitigate the need for URL rewriting, but it is hard to avoid it entirely. – Accepter

The transfer of control from the htaccess file to the PHP file only happens for files that are not found in the file system, so this is the 70% you are talking about. They skip the PHP processing, so rewrite rules are not needed at all. – Durfee

There are usecases where access files are still needed. For example, I mostly run service specific VMs or containers, but I've still got a shared service account for my personal webservice and email domain. It's $s p.a. so not worth changing, but with hosting service you need to do host->subdir rewrites even for static content. If you have embedded JS manipulating the dom then you might want to pass context via URL parameters added by rewrite. ... Don't limit your thinking to PHP-only applications. – Accepter

Answer 19

If you are working with url, You might want to check if you "Enable Mod Rewrite"

Ruvolo