Is there a truly universal wildcard in Grep? [duplicate]

B

7

6

Really basic question here. So I'm told that a dot . matches any character EXCEPT a line break. I'm looking for something that matches any character, including line breaks.

All I want to do is to capture all the text in a website page between two specific strings, stripping the header and the footer. Something like HEADER TEXT(.+)FOOTER TEXT and then extract what's in the parentheses, but I can't find a way to include all text AND line breaks between header and footer, does this make sense? Thanks in advance!

Bradbury answered 13/12, 2009 at 19:4 Comment(0)

D

7

When I need to match several characters, including line breaks, I do:

[\s\S]*?

Note I'm using a non-greedy pattern

Decalogue answered 13/12, 2009 at 19:16 Comment(1)

Thanks guys! What a friendly, useful site. I forgot to mention that I was using grep search in BBEdit, this works wonderfully. You all rock! – Bradbury 13/12, 2009 at 19:39

A

3

You could do it with Perl:

$ perl -ne 'print if /HEADER TEXT/ .. /FOOTER TEXT/' file.html

To print only the text between the delimiters, use

$ perl -000 -lne 'print $1 while /HEADER TEXT(.+?)FOOTER TEXT/sg' file.html

The /s switch makes the regular expression matcher treat the entire string as a single line, which means dot matches newlines, and /g means match as many times as possible.

The examples above assume you're cranking on HTML files on the local disk. If you need to fetch them first, use get from LWP::Simple:

$ perl -MLWP::Simple -le '$_ = get "http://stackoverflow.com";
                          print $1 while m!<head>(.+?)</head>!sg'

Please note that parsing HTML with regular expressions as above does not work in the general case! If you're working on a quick-and-dirty scanner, fine, but for an application that needs to be more robust, use a real parser.

Antitragus answered 13/12, 2009 at 19:9 Comment(0)

B

3

By definition, grep looks for lines which match; it reads a line, sees whether it matches, and prints the line.

One possible way to do what you want is with sed:

sed -n '/HEADER TEXT/,/FOOTER TEXT/p' "$@"

This prints from the first line that matches 'HEADER TEXT' to the first line that matches 'FOOTER TEXT', and then iterates; the '-n' stops the default 'print each line' operation. This won't work well if the header and footer text appear on the same line.

To do what you want, I'd probably use perl (but you could use Python if you prefer). I'd consider slurping the whole file, and then use a suitably qualified regex to find the matching portions of the file. However, the Perl one-liner given by '@gbacon' is an almost exact transliteration into Perl of the 'sed' script above and is neater than slurping.

Bename answered 13/12, 2009 at 19:12 Comment(0)

M

2

The man page of grep says:

grep, egrep, fgrep, rgrep - print lines matching a pattern

grep is not made for matching more than a single line. You should try to solve this task with perl or awk.

Melinite answered 13/12, 2009 at 19:11 Comment(0)

S

2

As this is tagged with 'bbedit' and BBedit supports Perl-Style Pattern Modifiers you can allow the dot to match linebreaks with the switch (?s)

(?s).

will match ANY character. And yes, (?s).+ will match the whole text.

Simply answered 9/8, 2011 at 12:5 Comment(0)

R

1

As pointed elsewhere, grep will work for single line stuff.

For multiple-lines (in ruby with Regexp::MULTILINE, or in python, awk, sed, whatever), "\s" should also capture line breaks, so

HEADER TEXT(.*\s*)FOOTER TEXT

might work ...

Ress answered 13/12, 2009 at 19:9 Comment(2)

You'd have to be reading the file in a mode that scans multiple lines into memory for that to work. – Bename 13/12, 2009 at 19:35

Thanks, I added how you would do that in Ruby. IIRC, that's /g in perlish, isn't it ? – Ress 13/12, 2009 at 19:54

M

0

here's one way to do it with gawk, if you have it

awk -vRS="FOOTER" '/HEADER/{gsub(/.*HEADER/,"");print}' file

Madgemadhouse answered 14/12, 2009 at 0:2 Comment(0)

Recommended topics

Hot tags