mycpen

Mycpen

记录学习历程与受益知识
github
telegram
bilibili
HomeArchives

29_Linux Basics - Shell Programming - awk - sed

#Linux#Shell#awk#sed19
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
This is a summary of the given text in "en" language: The text discusses Linux basics, shell programming, awk, and sed. It provides examples and explanations on using awk with shell variables, as well as different ways of inter-process communication. It also introduces sed as a text stream editor for modifying and replacing text. The text emphasizes the importance of performance and capacity awareness when using sed. It concludes with examples of using sed to modify files directly or through piping.

awk

Input delimiter (-F, FS), output delimiter (OFS)

Referencing shell variables in awk

1. The issue of exchanging awk and shell variables#

The issue of exchanging awk and shell variables

  1. Use the -v option

  2. Use double quotes, but the $ symbols inside awk such as $0, $1 need to be escaped with \

  3. Use single quotes to enclose the variable, then add a $ symbol in front to reference the variable's value again, which is equivalent to taking the value twice

# Example: using the -v option
[root@cPen_A ~]# sg="panjinhao"
[root@cPen_A ~]# echo $sg
panjinhao
[root@cPen_A ~]# echo|awk '{print $sg}'

[root@cPen_A ~]# echo|awk -v bsg=$sg '{print bsg}'
panjinhao
[root@cPen_A ~]# ls|awk -v bsg=$sg '{print bsg}'
panjinhao			# Note: ls outputs multiple lines
panjinhao
……
panjinhao

[root@cPen_A lianxi]# vim test2.sh 
awk -v var=$1 -F: '$1==var{print NR,$0}' /etc/passwd
[root@cPen_A lianxi]# bash test2.sh root
1 root:x:0:0:root:/root:/bin/bash
[root@cPen_A lianxi]# bash test2.sh sanchuang
21 sanchuang:x:1000:1000::/home/sanchuang:/bin/bash
# Note: The first $1 is a positional variable, the second $1 is the first field

# Example: using double quotes, but the $ symbols inside awk such as $0, $1 need to be escaped with \
[root@cPen_A lianxi]# mv="zhangjie"
[root@cPen_A lianxi]# useradd zhangjie_123
[root@cPen_A lianxi]# cat /etc/passwd|awk -F: "/^$mv/{print \$1,\$3}"
zhangjie_123 12358

# Example: using single quotes to enclose the variable, then add a $ symbol in front to reference the variable's value again, which is equivalent to taking the value twice
[root@cPen_A lianxi]# sg=3
[root@cPen_A lianxi]# awk -F: '/root/{print $1,$'$sg'}' /etc/passwd
root 0
operator 11

2. Ways for processes to communicate with each other#

  1. Shared memory

  2. Semaphore

  3. Signal

  4. Pipe

  5. Queue

  6. Socket

[root@cPen_A lianxi]# cat /etc/shadow
The second field: password field
* Provided by the operating system, no password
!! Created by the user, no password
Note: The password field is * or !! indicates no password is set; an empty password field means the password has been cleared.
# Example: users without a password
[root@cPen_A lianxi]# cat /etc/shadow|awk -F: 'BEGIN{i=0}length($2)<=2{print $1,"has no password";i++}END{print "There are a total of "i" users"}'

3. Built-in functions of awk#

Built-in functions in awk:

​ length()

​ int()

​ sqrt()

​ system()

String Functions

​ sub()

​ index()

​ length()

​ split()

Numeric Functions

yum install python3 Install python3

# Example: generate a random number between 0 and 1
[root@cPen_A lianxi]# echo |awk '{print rand()}'
0.237788
# Example: generate a random number between 0 and 100
[root@cPen_A lianxi]# echo |awk '{print rand()*100}'
23.7788
[root@cPen_A lianxi]# echo |awk '{print int(rand()*100)}'
23

4. Control flow#

Control flow: control flow

for

while

case

if

if statement
Single branch
[root@cPen_A lianxi]# awk -F: '{if($1 ~ /\<...\>/)print $0}' /etc/passwd
bin:x:1:1:bin:/bin:/sbin/nologin
[root@cPen_A lianxi]# awk -F: '$1 ~ /\<.{3}\>/{print $0}' /etc/passwd
bin:x:1:1:bin:/bin:/sbin/nologin

Double branch
[root@cPen_A lianxi]# awk -F: '{if($1 ~ /\<...\>/)print $0;else print "does not meet the requirements"}' /etc/passwd
does not meet the requirements
bin:x:1:1:bin:/bin:/sbin/nologin

image-20221011220439733

# Exercise
/etc/passwd
$3 is the uid user number
If the user number is 0  --> Administrator
1~999  --> Program user
Greater than 1000 --> Ordinary user
Finally, count how many administrators, how many program users, and how many ordinary users?
-----------------------------------------------------
[root@cPen_A ~]# awk -F: 'BEGIN{x=0;y=0;z=0}{if($3=="0") {print $1,"Administrator";x++} else if($3>=1 && $3<=999) {print $1,"Program user";y++} else  {print $1,"Ordinary user";z++}}END{print "Administrators "x,"Program users "y,"Ordinary users "z}' /etc/passwd
root Administrator
bin Program user
……
Administrators 1 Program users 25 Ordinary users 40

5. awk's for loop and arrays#

image-20221011220532360

image-20221011220538243

image-20221011220551647

image-20221011220605574

image-20221011220614623

Difficulties in awk:

  1. Regular expressions

  2. if

  3. Built-in functions

  4. Built-in variables

  5. for, array

6. sed#

What is sed? what --> text stream editor
What is it used for? where how --> Modify and replace text --> modify and substitute

sed is a non-interactive stream editor that supports regular expressions

​ The best tool for modifying or replacing text in scripts

Interactive: one question one answer. For example, typing commands in python3

image-20221011220732396

2 spaces
pattern space --> the place to process data --> after processing one line of data, the pattern space will be cleared
hold space --> a temporary place to store data

Put pattern space into hold space command: h, H

Put hold space into pattern space command: g, G

Note: Performance awareness, sed runs in memory, faster than disk speed#

Note: Capacity awareness#

# Example: -i option directly modifies the file, does not output to the screen
[root@cPen_A lianxi]# sed -i 's/xiaomi/redmi/g' test.txt 
# Example: passed through a pipe
[root@cPen_A lianxi]# cat chen |sed 's/xiaomi/redmi/g'

image-20221011220845338

[root@cPen_A lianxi]# cat -n /etc/passwd|head |tail -6
     5	lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
     6	sync:x:5:0:sync:/sbin:/bin/sync
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     8	halt:x:7:0:halt:/sbin:/sbin/halt
     9	mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    10	operator:x:11:0:operator:/root:/sbin/nologin
# Example: -n option displays the matching processed lines (otherwise it will output all)
[root@cPen_A lianxi]# cat -n /etc/passwd|sed -n '5,10p'
     5	lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
     6	sync:x:5:0:sync:/sbin:/bin/sync
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     8	halt:x:7:0:halt:/sbin:/sbin/halt
     9	mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    10	operator:x:11:0:operator:/root:/sbin/nologin

image-20221011220928234

Example of sed's p command#

= : output line number

When outputting with the p command:
Continuous lines
[root@localhost lianxi]# cat /etc/passwd|sed -n '1,5p'
Non-continuous lines
[root@localhost lianxi]# cat /etc/passwd|sed -n '10p;20p'

Example: sed's p command example
sed -n ‘line number 1,line number 2 p’ output file
[root@localhost lianxi]# cat /etc/passwd|sed -n '1,5p'			# Note: continuous use of , 
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
[root@localhost lianxi]# cat /etc/passwd|sed -n '=;1,5p'		# Note: =; output line number
1
root:x:0:0:root:/root:/bin/bash
2
bin:x:1:1:bin:/bin:/sbin/nologin
3
daemon:x:2:2:daemon:/sbin:/sbin/nologin
4
adm:x:3:4:adm:/var/adm:/sbin/nologin
5
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
6
7
……
14
[root@localhost lianxi]# cat -n /etc/passwd|sed -n '1,$p'		# Note: $p last line
[root@localhost lianxi]# cat -n /etc/passwd|sed -n '5,+5p'		# Note: from line 5 add 5 lines
     5	lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
     6	sync:x:5:0:sync:/sbin:/bin/sync
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     8	halt:x:7:0:halt:/sbin:/sbin/halt
     9	mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    10	operator:x:11:0:operator:/root:/sbin/nologin
[root@localhost lianxi]# cat -n /etc/passwd|sed -n '5,100!p'		# Note: take the inverse, take lines 1 to 4
     1	root:x:0:0:root:/root:/bin/bash
     2	bin:x:1:1:bin:/bin:/sbin/nologin
     3	daemon:x:2:2:daemon:/sbin:/sbin/nologin
     4	adm:x:3:4:adm:/var/adm:/sbin/nologin
# Example: step value
	# cat -n /etc/passwd|sed -n '1~2p'
	# cat -n /etc/passwd|sed -n '2~2p'
[root@localhost lianxi]# cat -n /etc/passwd|sed -n '1~2p'	# Note: odd lines, starting from 1 every 2 lines
     1	root:x:0:0:root:/root:/bin/bash
     3	daemon:x:2:2:daemon:/sbin:/sbin/nologin
     5	lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     9	mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    11	games:x:12:100:games:/usr/games:/sbin/nologin
    13	nobody:x:99:99:Nobody:/:/sbin/nologin
    15	dbus:x:81:81:System message bus:/:/sbin/nologin
    17	sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
    19	chrony:x:998:996::/var/lib/chrony:/sbin/nologin
    21	chenpeng:x:1001:1001::/home/chenpeng:/bin/bash
    23	nginx:x:996:994:Nginx web server:/var/lib/nginx:/sbin/nologin
[root@localhost lianxi]# cat -n /etc/passwd|sed -n '2~2p'		# Note: even lines
     2	bin:x:1:1:bin:/bin:/sbin/nologin
     4	adm:x:3:4:adm:/var/adm:/sbin/nologin
     6	sync:x:5:0:sync:/sbin:/bin/sync
     8	halt:x:7:0:halt:/sbin:/sbin/halt
    10	operator:x:11:0:operator:/root:/sbin/nologin
    12	ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
    14	systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
    16	polkitd:x:999:998:User for polkitd:/:/sbin/nologin
    18	postfix:x:89:89::/var/spool/postfix:/sbin/nologin
    20	mysql:x:1000:1000::/home/mysql:/sbin/nologin
    22	redis:x:997:995:Redis Database Server:/var/lib/redis:/sbin/nologin
# Example: passing shell variables to sed
# Double quotes recommended
[root@localhost lianxi]# num1=6
[root@localhost lianxi]# num2=10
[root@localhost lianxi]# cat -n /etc/passwd|sed -n "${num1},${num2}p"
     6	sync:x:5:0:sync:/sbin:/bin/sync
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     8	halt:x:7:0:halt:/sbin:/sbin/halt
     9	mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    10	operator:x:11:0:operator:/root:/sbin/nologin
# Single quotes
[root@localhost lianxi]# cat -n /etc/passwd|sed -n ''${num1},${num2}'p'
     6	sync:x:5:0:sync:/sbin:/bin/sync
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     8	halt:x:7:0:halt:/sbin:/sbin/halt
     9	mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    10	operator:x:11:0:operator:/root:/sbin/nologin

----------------------------------------------------------
# Example: display lines containing bash
[root@localhost lianxi]# cat /etc/passwd|sed -n '/bash/p'
root:x:0:0:root:/root:/bin/bash
chenpeng:x:1001:1001::/home/chenpeng:/bin/bash
[root@localhost lianxi]# cat /etc/passwd|egrep "bash"
root:x:0:0:root:/root:/bin/bash
chenpeng:x:1001:1001::/home/chenpeng:/bin/bash
[root@localhost lianxi]# cat /etc/passwd|awk '/bash/'
root:x:0:0:root:/root:/bin/bash
chenpeng:x:1001:1001::/home/chenpeng:/bin/bash

# Example: output lines starting with #
[root@localhost lianxi]# cat /etc/ssh/ssh_config |sed -n '/^#/p'
# Example: output lines not starting with #
[root@localhost lianxi]# cat /etc/ssh/ssh_config |sed -n '/^#/!p'
# Example: display lines not starting with # and !
[root@localhost lianxi]# cat /etc/ssh/ssh_config |sed -r -n '/^#|^$/!p'	# Note: -r supports more regex
[root@localhost lianxi]# cat /etc/ssh/ssh_config |egrep -v '^#|^$'
# Example: display lines ending with /
[root@localhost lianxi]# df -h|egrep "/$"
/dev/mapper/centos-root   17G  9.9G  7.2G   58% /
[root@localhost lianxi]# df -h|sed -n '/\/$/p'
/dev/mapper/centos-root   17G  9.9G  7.2G   58% /

# Example: lines starting with a-Z
[root@localhost lianxi]# cat /etc/passwd|sed -n '/^[a-Z]/p'
# Example: lines not starting with a-Z
[root@localhost lianxi]# cat /etc/passwd|sed -n '/^[^a-Z]/p'

Question: Extract all access logs within the time period of March 18, 15:30:00 -- 16:48:10 from nginx's access.log file

[root@cPen_A lianxi]# service firewalld stop	# Note: Immediately stop the firewalld service
[root@cPen_A lianxi]# systemctl disable firewalld	# Note: Set firewalld service to not start on boot --> it will not start when the machine is restarted next time

[root@cPen_A nginx]# tail -f access.log		# Note: Dynamically monitor the last line
192.168.0.17 - - [09/Jan/2021:16:36:03 +0800] "GET /favicon.ico HTTP/1.1" 404 3650 "http://192.168.0.118/" "Mozilla/5.0 (Linux; Android 10; PCT-AL10; HMSCore 5.1.0.300; GMSCore 20.26.14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 HuaweiBrowser/11.0.5.304 Mobile Safari/537.36" "-"
[root@cPen_A nginx]# pwd
/var/log/nginx				# Note: Path of nginx log files
[root@cPen_A nginx]# ls
access.log  access.log-20201202.gz  error.log  error.log-20201202.gz
# Note: access.log is the access log file of nginx, recording the information of visits to our website
[root@cPen_A nginx]# cat access.log | awk -F'[()]' '{print $2}'|sort|uniq -c|sort -nr
     13 Windows NT 10.0; WOW64			# Note: sort and deduplicate, then sort in descending order to display the count
      4 Linux; Android 10; PCT-AL10; HMSCore 5.1.0.300; GMSCore 20.26.14
--------------------------------------------------------------------------- 
[09/Jan/2021:16:32:46 +0800] --> 09/Jan/2021:16:32:46
[09/Jan/2021:16:33:07 +0800] --> 09/Jan/2021:16:32:46
# Example: find all access logs in this time period
[root@cPen_A nginx]# cat access.log|sed -n '/09\/Jan\/2021:16:32:46/,/09\/Jan\/2021:16:33:07/p'
# Example: regex writing
[root@cPen_A nginx]# cat access.log|sed -r -n '/09\/Jan\/2021:16:(3[789]|4[0-9]|5[01]):(0[1-9]|[1-5][0-9])/p'
# Note: Minute range 37~51, second range 01~59
# 37~51
# 37~39
# 40~49
# 50~51

# 01~59
0[1-9]|[1-5][0-9]

Example of sed's d command#

Example of sed's d command
Delete operations can be performed based on line numbers and matching patterns
# Example: delete lines containing denghui
[root@cPen_A nginx]# cat /etc/passwd|sed '/denghui/d'
# Example: -i modify the original file, delete the line containing redmi
[root@cPen_A lianxi]# sed -i '/redmi/d' test.txt 
# Example: delete by line number
[root@cPen_A lianxi]# sed -i '2d' test.txt 	# Note: delete the second line
sed search methods:
	1. By line number
	2. By pattern --> regular expression = character + special symbol
	3. By string

# Example: append huawei after line 4
[root@cPen_A lianxi]# sed -i '4a huawei' test.txt 
# Example: insert apple before line 4
[root@cPen_A lianxi]# sed -i '4i apple' test.txt 
# Example: insert OPPO before the line containing VIVO
[root@cPen_A lianxi]# sed -i '/VIVO/i OPPO' test.txt 
# Example: add APPLE after the line containing VIVO, & represents everything before
[root@cPen_A lianxi]# sed -i 's/VIVO/& APPLE/' test.txt 
# Example: replace all occurrences of huawei with sanchuang
[root@cPen_A lianxi]# sed -i 's/huawei/sanchuang/g' test.txt 
sed commands:
	p
	d
	a
	i
	s
	c
	r
	w
Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.