How to print third column to last column?

Question

I'm trying to remove the first two columns (of which I'm not interested in) from a DbgView log file. I can't seem to find an example that prints from column 3 onwards until the end of the line. Note that each line has variable number of columns.

Johnnyjumpup

possible duplicate of Using awk to print all columns from the nth to the last – Springing

Answer 1

...or a simpler solution: cut -f 3- INPUTFILE just add the correct delimiter (-d) and you got the same effect.

Vigilantism

Note that this only works if the delimiter is exactly the same between all columns... For example, you can't use cut with a delimiter like \d+. (That I know of.) – Basil

When question is titled awk it is inappropriate to accept answer other than awk. What if people need it for awk scripts? This answer should've just been a comment. – Merissa

@SyaZ: Normally I'd agree, but with the amount of 'gratuitous awk' going on this board, I thought it's needed to show an alternative way of doing the task. Wouldn't you be thankful if someone showed you a simpler and quicker way to do the same task? Maybe the poster thought awk is the only way to do this because of number of 'not incorrect, but certainly improvable upon' answers to other questions? – Vigilantism

That's what the comment is for. Accept the best awk answer and provide better non-awk suggestions on comments. If people start posting answers that don't exactly answer questions, it will be annoying when searching (in my case). – Merissa

Not only delimiter have to be the same between all columns, but there have to be EXACTLY ONE delimiter character between columns. So if you are dealing with programs that align their output with delimiters, it is better to use awk. – Em

why do you need a dash after 3? – Bichloride

3 by itself is print field number 3. 3- is print field 3 and all the fields after it. – Vigilantism

@ZachWily, @sknaumov, the Unix way to mitigate this \s+ delimiter issue is by piping through tr -s '[:blank:]' ' ' beforehand [source]. – Casiecasilda

Just a quick update, I ran a bunch of the awk solutions from this thread and they all ran a bit around 65-78 seconds (2.2GB file), while my simple 'cut' version ran in 6 seconds. I win ;) – Vigilantism

Reran the awk scripts using mawk instead of gawk. Runtime is 12-13secs. Cut still wins on speed. – Vigilantism

The original title asked for an answer using Awk so, for this reason, I agree that this shouldn't have been given the tick. However, I don't feel tags should limit the answers as often the asker is just guessing how to tag to get relevant people looking at their question. It's not a requirement for how you must answer the question. The question is in cases where it sets a limiation. – Seedy

This doesn't do anything on my Debian 11 – Memphis

Answer 2

awk '{for(i=3;i<=NF;++i)print $i}' 

Trace

awk '{for(i=3;i<=NF;++i)print $i}' be more compact. :) – Operative

Thanks, lh3. I was just copying and pasting for the gawk manual. :) – Trace

I think that you can do something like this awk 'NF >= 3' – Prowl

this fails with multiple lines, each column is trated as a new line when printed iwth print – Sakovich

@user2571881 that is only printing for me results that have fields equal to or larger than 3, anot just the the remainder AFTEr those 3 – Swell

Splitted output is getting displayed with newline separator. Does not work for me. – Retinol

To address the splitted output issue, I propose this solution: awk '{for(i=3;i<=NF;++i)printf $i""FS ; print ""}' (printfwill not print the newline char while print "" will add newline after the other fields has been printed) – Argot

Or: echo $(seq 1 10) | awk '{for (i=3; i<=NF; i++) printf $i FS}', which gives: 3 4 5 6 7 8 9 10. – Combinative

It adds each column on a different line. See answer: https://mcmap.net/q/12543/-how-to-print-third-column-to-last-column – Bourgeois

Answer 3

awk '{ print substr($0, index($0,$3)) }'

solution found here:
http://www.linuxquestions.org/questions/linux-newbie-8/awk-print-field-to-end-and-character-count-179078/

Haehaecceity

I'm kind of late for this, but this won't work for records in which the first or second field is equal to the third (e.g., 3 2 3 4 5) – Denationalize

printing an internal range is also possible: ``` # from $3 (included) to $6 (excluded); echo "1,2,3,4,5,6,7,8,9" | awk 'BEGIN{FS=",";OFS=","}{ print substr($0, index($0,$3), length($0)-index($0,$6)-1) }'; # gives 3,4,5``` – Harrold

Answer 4

Jonathan Feinberg's answer prints each field on a separate line. You could use printf to rebuild the record for output on the same line, but you can also just move the fields a jump to the left.

awk '{for (i=1; i<=NF-2; i++) $i = $(i+2); NF-=2; print}' logfile

Catadromous

Be aware, that this only works for Gnu awk, decrementing NF is not allowed by POSIX. – Ka

@kvantour: It works in gawk, mawk, MacOS awk (nawk?). POSIX appears to be silent on whether NF can be decremented. – Catadromous

It is one of these funny dark corners of awk. – Ka

Answer 5

awk '{$1=$2=$3=""}1' file

NB: this method will leave "blanks" in 1,2,3 fields but not a problem if you just want to look at output.

Tonguing

Trail that command with ` | sed s/^\ *// | column -t` to strip leading spaces and align the remaining columns – Possessory

What does the last 1 mean? which keyword should I search with awk? – Lorislorita

@Lorislorita see example 1 of catonmat.net/blog/awk-one-liners-explained-part-one – Ka

@Nathan you solve this issue as {$1=$2=$3="";$0=$0;$1=$1}1 – Ka

This is really useful for weeding out a few unnecessary columns in the middle (while still grabbing to the end)! Thanks @Tonguing – Aphrodisiac

Answer 6

If you want to print the columns after the 3rd for example in the same line, you can use:

awk '{for(i=3; i<=NF; ++i) printf "%s ", $i; print ""}'

For example:

Mar 09:39 20180301_123131.jpg
Mar 13:28 20180301_124304.jpg
Mar 13:35 20180301_124358.jpg
Feb 09:45 Cisco_WebEx_Add-On.dmg
Feb 12:49 Docker.dmg
Feb 09:04 Grammarly.dmg
Feb 09:20 Payslip 10459 %2828-02-2018%29.pdf

It will print:

20180301_123131.jpg
20180301_124304.jpg
20180301_124358.jpg
Cisco_WebEx_Add-On.dmg
Docker.dmg
Grammarly.dmg
Payslip 10459 %2828-02-2018%29.pdf

As we can see, the payslip even with space, shows in the correct line.

Bourgeois

This is excellent, except I have a problem with $NF being excluded. When I set the condition (<=NF) I get the last field but the first character of the first field is chopped off. Am I misunderstanding something in terms of functionality? – Sackcloth

Looks like my problem is that ^M is stuck to the end of the last column. Don't see how to remove it. – Sackcloth

Answer 7

What about following line:

awk '{$1=$2=$3=""; print}' file

Based on @ghostdog74 suggestion. Mine should behave better when you filter lines, i.e.:

awk '/^exim4-config/ {$1=""; print }' file

Altruist

Short & simple. Could also pipe & add sed 's/\s\+//g' at the end of the command to trim leading spaces – Epigenesis

Answer 8

awk -v m="\x0a" -v N="3" '{$N=m$N ;print substr($0, index($0,m)+1)}'

This chops what is before the given field nr., N, and prints all the rest of the line, including field nr.N and maintaining the original spacing (it does not reformat). It doesn't mater if the string of the field appears also somewhere else in the line, which is the problem with daisaa's answer.

Define a function:

fromField () { 
awk -v m="\x0a" -v N="$1" '{$N=m$N; print substr($0,index($0,m)+1)}'
}

And use it like this:

$ echo "  bat   bi       iru   lau bost   " | fromField 3
iru   lau bost   
$ echo "  bat   bi       iru   lau bost   " | fromField 2
bi       iru   lau bost 

Output maintains everything, including trailing spaces

Works well for files where '/n' is the record separator so you don't have that new-line char inside the lines. If you want to use it with other record separators then use:

awk -v m="\x01" -v N="3" '{$N=m$N ;print substr($0, index($0,m)+1)}'

for example. Works well with almost all files as long as they don't use hexadecimal char nr. 1 inside the lines.

Ajar

Answer 9

awk '{a=match($0, $3); print substr($0,a)}'

First you find the position of the start of the third column. With substr you will print the whole line ($0) starting at the position(in this case a) to the end of the line.

Highpowered

Answer 10

The following awk command prints the last N fields of each line and at the end of the line prints a new line character:

awk '{for( i=6; i<=NF; i++ ){printf( "%s ", $i )}; printf( "\n"); }'

Find below an example that lists the content of the /usr/bin directory and then holds the last 3 lines and then prints the last 4 columns of each line using awk:

$ ls -ltr /usr/bin/ | tail -3
-rwxr-xr-x 1 root root       14736 Jan 14  2014 bcomps
-rwxr-xr-x 1 root root       10480 Jan 14  2014 acyclic
-rwxr-xr-x 1 root root    35868448 May 22  2014 skype

$ ls -ltr /usr/bin/ | tail -3 | awk '{for( i=6; i<=NF; i++ ){printf( "%s ", $i )}; printf( "\n"); }'
Jan 14 2014 bcomps 
Jan 14 2014 acyclic 
May 22 2014 skype

Backbite

Answer 11

Perl solution:

perl -lane 'splice @F,0,2; print join " ",@F' file

These command-line options are used:

• -n loop around every line of the input file, do not automatically print every line

• -l removes newlines before processing, and adds them back in afterwards

• -a autosplit mode – split input lines into the @F array. Defaults to splitting on whitespace

• -e execute the perl code

splice @F,0,2 cleanly removes columns 0 and 1 from the @F array

join " ",@F joins the elements of the @F array, using a space in-between each element

If your input file is comma-delimited, rather than space-delimited, use -F, -lane

---

Python solution:

python -c "import sys;[sys.stdout.write(' '.join(line.split()[2:]) + '\n') for line in sys.stdin]" < file

Adkins

Answer 12

Well, you can easily accomplish the same effect using a regular expression. Assuming the separator is a space, it would look like:

awk '{ sub(/[^ ]+ +[^ ]+ +/, ""); print }'

People

I'd avoid regex. It's probably slower and easier to accidentally mess up. – Unused

It shorten it like this: awk '{ sub(/([^ ]+ +){2}/, ""); print }' which takes the pattern two times away. – Bawcock

Answer 13

awk '{print ""}{for(i=3;i<=NF;++i)printf $i" "}'

Karrikarrie

Answer 14

A bit late here, but none of the above seemed to work. Try this, using printf, inserts spaces between each. I chose to not have newline at the end.

awk '{for(i=3;i<=NF;++i) printf("%s ",  $i) }'

Hurleigh

Answer 15

awk '{for (i=4; i<=NF; i++)printf("%c", $i); printf("\n");}'

prints records starting from the 4th field to the last field in the same order they were in the original file

Siret

sorry, this was not quite correct answer. it is too specific, but I do not know how to delete it – Siret

Answer 16

In Bash you can use the following syntax with positional parameters:

while read -a cols; do echo ${cols[@]:2}; done < file.txt

Learn more: Handling positional parameters at Bash Hackers Wiki

Hypoglycemia

Answer 17

If its only about ignoring the first two fields and if you don't want a space when masking those fields (like some of the answers above do) :

awk '{gsub($1" "$2" ",""); print;}' file

Byte

Answer 18

awk '{$1=$2=""}1' FILENAME | sed 's/\s\+//g'

First two columns are cleared, sed removes leading spaces.

Farah

Answer 19

In AWK columns are called fields, hence NF is the key

all rows:

awk -F '<column separator>' '{print $(NF-2)}' <filename>

first row only:

awk -F '<column separator>' 'NR<=1{print $(NF-2)}' <filename>

Ewing