--- title: "AWK" description: "AWK is a text processing language for pattern scanning and data extraction. Used to process text files, extract columns, and perform calculations on text data." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/cheatsheets/awk --- # AWK ## AWK complete reference guide ### Quick start ```bash # Print entire file awk '{ print }' file.txt # Print specific column awk '{ print $1 }' file.txt # Print lines matching pattern awk '/pattern/ { print }' file.txt # Sum all numbers in first column awk '{ sum += $1 } END { print sum }' numbers.txt # Replace text pattern awk '{ gsub(/old/, "new"); print }' file.txt ``` ### AWK program structure ```bash awk '[BEGIN block] [pattern { action }] [END block]' [options] [file] ``` ### Essential built-in variables | Variable | Description | | -------- | ----------------------------- | | NR | Total record number (line) | | FNR | File record number | | NF | Number of fields | | FS | Field separator (input) | | OFS | Output field separator | | RS | Record separator | | ORS | Output record separator | | FILENAME | Current filename | | ARGC | Argument count | | ARGV | Argument array | | ENVIRON | Environment variables | | RSTART | Start of match (from match()) | | RLENGTH | Length of match | ### Field variables ```bash $0 # Entire line $1 # First field $2 # Second field $NF # Last field $(NF-1) # Second-to-last field ``` ### String functions reference ```bash length(str) # String length substr(str, s, l) # Substring (start, length) index(str, substr) # Find substring position split(str, arr, sep) # Split string into array sub(regex, repl) # Replace first match gsub(regex, repl) # Replace all matches match(str, regex) # Test match, set RSTART/RLENGTH tolower(str) # Convert to lowercase toupper(str) # Convert to uppercase sprintf(fmt, ...) # Format string ``` ### Math functions reference ```bash int(x) # Integer part sqrt(x) # Square root sin(x) # Sine (radians) cos(x) # Cosine (radians) atan2(y,x) # Arctangent exp(x) # e^x log(x) # Natural logarithm rand() # Random 0-1 srand(seed) # Seed random ``` ### Operators quick reference ```bash # Arithmetic: +, -, *, /, %, ^ # Comparison: <, <=, >, >=, ==, != # Logical: &&, ||, ! # String: ~ (match), !~ (not match) # Assignment: =, +=, -=, *=, /=, %=, ^=, ++, -- # Ternary: condition ? true_val : false_val ``` ### Command-line options ```bash -F sep # Set field separator -v var=val # Set variable -f script.awk # Read program from file -W version # Show version --posix # POSIX mode ``` ### Common one-liners ```bash # Print line numbers awk '{ print NR, $0 }' file # Print lines longer than 80 characters awk 'length($0) > 80 { print }' file # Print every nth line awk 'NR % n == 0 { print }' file # Reverse lines awk '{ arr[NR] = $0 } END { for (i=NR; i>0; i--) print arr[i] }' file # Print lines where column 2 > 100 awk '$2 > 100 { print }' file # Transpose CSV awk -F',' '{ for(i=1;i<=NF;i++) a[i,NR]=$i } END { for(i=1;i<=NF;i++) { for(j=1;j<=NR;j++) printf "%s%s", a[i,j], (j==NR?"\n":","); print "" } }' file ``` ### Tips and best practices - Use single quotes to prevent shell interpretation - Use -F flag to set different field separators - Combine awk with pipes and other tools (grep, sort, uniq) - Save complex programs in .awk files with -f flag - Use printf for formatted output - Initialize variables in BEGIN block - Test on small files before processing large datasets - Use comments (#) to document complex logic - Consider performance with large files (avoid unnecessary loops) ## Getting Started Introduction to AWK and basic syntax fundamentals ### What is AWK AWK and its use cases for text processing **Keywords:** awk, text-processing, pattern-matching, data-extraction, language #### AWK overview and capabilities ```bash # AWK is a text processing language designed for pattern scanning # and manipulation of data. Key features: # - Process text files line by line # - Extract and manipulate columns # - Perform calculations on data # - Filter records based on patterns # - Generate reports and formatted output # AWK stands for: Aho, Weinberger, Kernighan (creators) # Variants: awk (original), gawk (GNU AWK), mawk (minimal AWK) ``` _exec_ ```bash which awk && awk --version ``` _output_ ```bash /usr/bin/awk GNU Awk 5.1.0, API: 3.1 ``` AWK handles text processing, data extraction, and pattern matching across files and streams. - AWK reads input line by line automatically - Supports regular expressions for pattern matching - Built-in variables track lines, fields, and more - Can be used interactively or in shell scripts #### AWK vs other text processing tools ```bash # Compare AWK with other text processing tools: # grep: Search file patterns (matches lines) # sed: Stream editor (find and replace, transformations) # awk: Full programming language (matching + calculations) # AWK is best when you need: # - Extract or reorganize columns # - Perform calculations on text data # - Process structured text (CSV, logs, reports) # - Conditional logic and complex transformations ``` _exec_ ```bash echo -e "name,age\nJohn,30\nJane,25" | awk -F',' '{print $1 " is " $2 " years old"}' ``` _output_ ```bash name is age years old John is 30 years old Jane is 25 years old ``` AWK processes structured data and applies transformations that combine pattern matching with data manipulation. - AWK is a complete Turing-complete programming language - Handles more complex text processing than grep or sed - Easier syntax than writing shell scripts for text processing **Best practices:** - Use AWK for text processing tasks over multiple grep/sed commands - Choose AWK when you need both pattern matching and calculations - Prefer AWK over Perl for text processing simplicity **Common errors:** - **awk not found**: AWK may be installed as gawk on some systems; install with apt-get install gawk ### Basic Syntax and Structure AWK program structure and execution flow **Keywords:** syntax, structure, pattern, action, BEGIN, END #### AWK program structure ```bash # Basic AWK syntax structure: # awk 'pattern { action }' input-file # Program structure with BEGIN, pattern/action, and END: awk ' BEGIN { # Initialization, runs before processing input print "Starting processing..." } pattern { # Main processing, runs for each line # pattern can be regex, expression, or range } END { # Finalization, runs after all input processed print "Processing complete" } ' input-file ``` _exec_ ```bash echo -e "apple\nbanana\ncherry" | awk 'BEGIN { print "Fruits:" } { print "- " $0 } END { print "Done" }' ``` _output_ ```bash Fruits: - apple - banana - cherry Done ``` Shows the three parts of an AWK program. The BEGIN block runs once before input, the main block runs for each line, and the END block runs once after all input. - BEGIN and END blocks are optional - Multiple patterns can match the same line - Pattern can be omitted (all lines match) - Action can be omitted (default is print) #### Inline AWK programs and scripts ```bash # Method 1: Inline program with single quotes awk '{ print NR, $0 }' file.txt # Method 2: Program from file awk -f script.awk file.txt # Method 3: Multiple input files awk '{ print FILENAME, NR, $0 }' file1.txt file2.txt # Method 4: Program with variables awk -v var=value '{ print var, $0 }' file.txt # Method 5: Pipe input directly echo "data" | awk '{ print toupper($0) }' ``` _exec_ ```bash echo -e "line1\nline2" | awk '{ print NR, $0 }' ``` _output_ ```bash 1 line1 2 line2 ``` Different ways to invoke AWK, including inline, from file, with arguments, and from pipes. - Single quotes protect AWK syntax from shell interpretation - -f flag reads program from file - -v flag passes variables to AWK - AWK can read from multiple files sequentially **Best practices:** - Use single quotes for inline programs to avoid shell variable expansion - Store complex AWK programs in .awk files with -f flag - Use -v for passing shell variables to AWK **Common errors:** - **Unmatched { or }**: Check pattern and action braces are balanced ### Installation and Setup Installing AWK and verifying functionality **Keywords:** install, setup, gawk, mawk, variants #### Install AWK on Linux systems ```bash # Ubuntu/Debian sudo apt-get update sudo apt-get install -y gawk # CentOS/RHEL sudo yum install -y gawk # macOS brew install gawk # Verify installation awk --version ``` _exec_ ```bash gawk --version | head -1 ``` _output_ ```bash GNU Awk 5.1.0, API: 3.1 (GNU MPFR 4.1.0, GNU MP 6.2.1) ``` Installation of GNU AWK (gawk) on various Linux distributions and macOS. - Most systems have awk or gawk installed by default - gawk (GNU AWK) is most feature-complete implementation - mawk is faster but supports fewer features - Compatibility: gawk includes all mawk features #### Creating AWK script file ```bash # Create a simple AWK script file cat > process.awk << 'EOF' BEGIN { print "Processing file..." } { total += $1 count++ } END { print "Total: " total print "Average: " (count > 0 ? total / count : 0) } EOF # Run the script awk -f process.awk numbers.txt # Make script executable (requires shebang) echo '#!/usr/bin/awk -f' | cat - process.awk > /tmp/process chmod +x /tmp/process ./process numbers.txt ``` _exec_ ```bash cat process.awk ``` _output_ ```bash BEGIN { print "Processing file..." } { total += $1 count++ } END { print "Total: " total print "Average: " (count > 0 ? total / count : 0) } ``` Creating and using AWK script files for more complex programs. - Use -f flag to load program from file - Script files can be made executable with proper shebang - Shebang line: #!/usr/bin/awk -f - Scripts are useful for complex logic and reusability **Best practices:** - Use gawk for maximum compatibility and features - Store complex programs in .awk files - Use executable script files for frequently used tasks **Common errors:** - **awk: can't open program text file**: Check the file exists and the path passed to -f is correct ## Field Processing Working with fields, separators, and field manipulation ### Field Variables and NF Field access and manipulation with field variables **Keywords:** fields, $0, $1, $2, NF, field-separator, columns #### Accessing fields in AWK ```bash # Field variables reference: # $0 = entire line # $1 = first field # $2 = second field # ... $NF = last field # NF = total number of fields in the line # Examples: echo "John Doe 30 Engineer" | awk '{ print $1 }' # John echo "John Doe 30 Engineer" | awk '{ print $NF }' # Engineer echo "John Doe 30 Engineer" | awk '{ print $(NF-1) }' # 30 echo "John Doe 30 Engineer" | awk '{ print NF }' # 4 ``` _exec_ ```bash echo "apple banana cherry" | awk '{ print "Last field:", $NF; print "Field count:", NF }' ``` _output_ ```bash Last field: cherry Field count: 3 ``` Field variables allow accessing individual fields by number, with $NF accessing the last field dynamically. - Fields are separated by whitespace by default (FS) - $0 is the entire line - Use $(NF-1) for second-to-last field, $(NF-2) for third-to-last, etc. - NF changes if you modify fields #### Modifying fields ```bash # Change field values echo "apple 5 basket" | awk '{ $2 = 10; print }' # apple 10 basket # Add fields echo "apple basket" | awk '{ $(NF+1) = "10"; print }' # apple basket 10 # Reconstruct line with modified fields echo "a b c" | awk '{ $2 = "B"; print $0 }' # a B c # Remove fields (truncate) echo "a b c d e" | awk '{ NF = 3; print }' # a b c ``` _exec_ ```bash echo "test 100 active" | awk '{ $2 = 200; NF = 2; print }' ``` _output_ ```bash test 200 ``` Modifying, adding, and removing fields by direct assignment or changing NF value. - Modifying a field updates $0 automatically - Increasing NF adds empty fields - Decreasing NF removes fields from the end - Output uses OFS to join modified fields **Best practices:** - Use $NF for the last field instead of hardcoding field numbers - Use $(NF-n) for fields relative to the end - Remember that modifying fields changes $0 **Common errors:** - **Reference to undefined field**: Check field number is within NF range ### Field Separators Setting and using different field separator patterns **Keywords:** separator, FS, OFS, delimiter, csv, custom-separators #### Setting field separators ```bash # FS (Field Separator) - what separates fields in input # OFS (Output Field Separator) - what separates fields in output # Default FS is whitespace. Set custom FS with -F flag: echo "apple,banana,cherry" | awk -F',' '{ print $2 }' # banana # Or use BEGIN to set FS echo "apple:banana:cherry" | awk 'BEGIN { FS=":" } { print $2 }' # banana # Set both FS and OFS echo "apple-banana-cherry" | awk -F'-' -v OFS=',' '{ print $1, $2, $3 }' # apple,banana,cherry # Use regex as FS (multiple delimiters) echo "apple,banana;cherry" | awk -F'[,;]' '{ print $2 }' # banana ``` _exec_ ```bash echo "name:age:city" | awk -F':' '{ print "Name=" $1 ", Age=" $2 }' ``` _output_ ```bash Name=name, Age=age ``` Custom field separators allow parsing different delimited formats like CSV, TSV, and colon-separated data. - Default FS matches one or more whitespace characters - FS can be a single character or regex pattern - -F flag overrides default FS - BEGIN block can set FS for variable input #### Processing CSV and complex formats ```bash # Process CSV with quoted fields echo 'name,age,city "Smith, John",30,"New York" "Doe, Jane",25,"Los Angeles"' | awk -F',' ' NR > 1 { gsub(/"/, "", $1) # Remove quotes gsub(/ +$/, "", $1) # Remove trailing spaces print $1 " is " $2 " years old" }' # Tab-separated values echo -e "col1\tcol2\tcol3" | awk -F'\t' '{ print $2 }' # Multi-character delimiter echo "apple::banana::cherry" | awk -F'::' '{ print $1 }' ``` _exec_ ```bash echo "a:b:c" | awk -F':' -v OFS='-' '{ print $1, $2, $3 }' ``` _output_ ```bash a-b-c ``` Different separators for input parsing and output formatting with FS and OFS. - -v OFS flags sets output field separator - FS regex can handle multiple delimiter types - gsub useful for cleaning CSV quote characters - Use -F'\t' for tab-delimited files **Best practices:** - Use -F flag for quick FS changes - Set both FS and OFS for format conversion - Use regex FS for flexible delimiter matching **Common errors:** - **Field contains unexpected data**: Check FS matches your actual delimiter in the file ### Field Manipulation and Reconstruction Transform and reorganize field data **Keywords:** manipulation, reorganize, transform, extract, rearrange, concatenate #### Rearranging and extracting fields ```bash # Extract and rearrange columns echo "John Doe john@example.com 30" | awk '{ print $3, $2, $1 }' # john@example.com Doe John # Extract specific fields echo "apple 100 USD red" | awk '{ print "Item: " $1 ", Price: " $2 " " $4 }' # Item: apple, Price: 100 red # Concatenate fields echo "John Doe" | awk '{ full_name = $1 " " $2; print "Hello, " full_name }' # Hello, John Doe # Extract substring from field echo "example@domain.com" | awk -F'@' '{ print "User: " $1 }' # User: example ``` _exec_ ```bash echo "2025-02-28" | awk -F'-' '{ print "Year: " $1 ", Month: " $2 ", Day: " $3 }' ``` _output_ ```bash Year: 2025, Month: 02, Day: 28 ``` Extracting and rearranging fields to create custom output formats. - Fields can be rearranged in any order - Easy to combine fields into new strings - Using FS and field numbers simplifies parsing #### Computing values from fields ```bash # Sum and average fields echo -e "100\n200\n300" | awk '{ sum += $1 } END { print "Total: " sum ", Avg: " sum/NR }' # Total: 600, Avg: 200 # Product of fields echo "5 4 3" | awk '{ print "Product: " $1 * $2 * $3 }' # Product: 60 # Calculate percentage echo "75 100" | awk '{ print ($1 / $2 * 100) "%" }' # 75% # String operations on fields echo "javascript PYTHON bash" | awk '{ print toupper($1), tolower($2), tolower($3) }' # JAVASCRIPT python bash ``` _exec_ ```bash echo "10 20" | awk '{ print "Sum: " $1 + $2 ", Diff: " $1 - $2 ", Prod: " $1 * $2 }' ``` _output_ ```bash Sum: 30, Diff: -10, Prod: 200 ``` Performing calculations and transformations on field values. - AWK automatically converts fields to numbers when needed - Can perform arithmetic directly on fields - String functions can transform field values **Best practices:** - Extract and process fields close to their source - Use meaningful variable names for complex field operations - Validate field count before accessing specific fields **Common errors:** - **Pattern not found**: Check delimiter matches actual field separator character ## Patterns and Matching Pattern matching and regular expressions in AWK ### Pattern Types and Matching Different pattern types for matching records **Keywords:** pattern, match, regex, expression, condition, filtering #### AWK pattern types ```bash # Pattern types in AWK: # 1. Regex patterns awk '/pattern/ { action }' file # Match lines with pattern awk '!/pattern/ { action }' file # Match lines without pattern # 2. Expression patterns (boolean) awk '$1 > 100 { action }' file # Field comparison awk 'NR >= 2 && NR <= 5' file # Line number range awk '$1 == "active" { action }' file # Exact match # 3. BEGIN/END patterns awk 'BEGIN { action }' # Before processing input awk 'END { action }' # After processing input # 4. Range patterns awk '/start/,/end/ { action }' file # From start to end pattern # 5. No pattern awk '{ action }' file # All lines match ``` _exec_ ```bash echo -e "apple 50\nbanana 120\ncherry 75" | awk '$2 > 100 { print }' ``` _output_ ```bash banana 120 ``` Different pattern types allow flexible filtering and matching of records. - Patterns determine which lines trigger the action block - Multiple patterns can target different lines - Patterns are optional (all lines match by default) - Expression patterns use standard operators #### Regex patterns and matching ```bash # Regex pattern matching awk '/linux/' file # Contains "linux" awk '/^linux/' file # Starts with "linux" awk '/linux$/' file # Ends with "linux" awk '/[0-9]+/' file # Contains digits awk '!/error/' file # Does not contain "error" # Regex with field matching (~) awk '$1 ~ /john/' file # First field matches regex awk '$2 !~ /test/' file # Second field doesn't match # Case-insensitive matching awk 'tolower($0) ~ /error/' file # Case-insensitive search ``` _exec_ ```bash echo -e "error: issue\nwarning: notice\nerror: problem" | awk '/error/' ``` _output_ ```bash error: issue error: problem ``` Regex patterns filter lines based on pattern matching and operator ~. - ~ operator checks if field matches regex - !~ operator checks if field doesn't match - Use tolower() for case-insensitive matching - ^ and $ anchor patterns to start and end **Best practices:** - Use specific patterns to avoid unnecessary processing - Combine regex with field matching for precise filtering - Use expression patterns for numeric comparisons **Common errors:** - **Regex not matching**: Check escaping of special characters; use \\( instead of ( in regex ### BEGIN and END Blocks Initialization and finalization with BEGIN and END **Keywords:** BEGIN, END, initialization, setup, cleanup, summary #### BEGIN and END blocks ```bash # BEGIN block - runs once before processing input awk 'BEGIN { print "Processing started..." count = 0 } { count++ } END { print "Total lines: " count }' file # Example: Sum numbers and show statistics awk 'BEGIN { sum = 0; count = 0 } NR > 1 { sum += $1; count++ } END { avg = (count > 0) ? sum / count : 0 print "Sum: " sum "\nAverage: " avg }' data.txt ``` _exec_ ```bash echo -e "10\n20\n30" | awk 'BEGIN { print "Numbers:" } { print "- " $0 } END { print "Done" }' ``` _output_ ```bash Numbers: - 10 - 20 - 30 Done ``` BEGIN runs before processing input and END runs after, useful for setup, cleanup, and reporting. - BEGIN block runs even if no input is provided - END block runs even if no lines match patterns - Variables initialized in BEGIN persist through all input #### Header and summary generation ```bash # Generate report with header and footer awk 'BEGIN { printf "%-15s %-10s %-10s\n", "Name", "Age", "City" printf "%-15s %-10s %-10s\n", "----", "---", "----" } { printf "%-15s %-10s %-10s\n", $1, $2, $3 } END { print "" print "Total records: " NR }' people.txt # Calculate running statistics awk 'BEGIN { header = "Line\tValue\tRunSum" print header } { sum += $1 printf "%d\t%d\t%d\n", NR, $1, sum } END { print "Final sum: " sum }' numbers.txt ``` _exec_ ```bash echo -e "5\n10\n15" | awk 'BEGIN { print "Input values:" } { sum += $1; print NR, $1 } END { print "Total: " sum }' ``` _output_ ```bash Input values: 1 5 2 10 3 15 Total: 30 ``` Using BEGIN and END to create formatted reports with headers, footers, and summary statistics. - printf in BEGIN/END creates formatted output - Variables maintain state across BEGIN, main, and END - Good pattern for generating summaries and reports **Best practices:** - Use BEGIN for initialization and variable setup - Use END for final calculations and report generation - Initialize all variables in BEGIN to avoid undefined behavior **Common errors:** - **Variable undefined**: Initialize variables in BEGIN block ### Range and Compound Patterns Using range patterns and combining conditions **Keywords:** range, compound, logical, &&, condition #### Range patterns ```bash # Range patterns match from first pattern to second pattern # Syntax: /start/,/end/ { action } # Print lines between BEGIN and END markers awk '/BEGIN/,/END/' file # Process sections of a file awk '/^Chapter 1/,/^Chapter 2/' book.txt # Extract code blocks between markers awk '/```/,/```/' doc.md | head -20 # Line number ranges awk 'NR==10, NR==20 { print NR, $0 }' file ``` _exec_ ```bash echo -e "start\ndata1\ndata2\nend\nignore" | awk '/start/,/end/ { print }' ``` _output_ ```bash start data1 data2 end ``` Range patterns select a block of lines from first matching pattern to second matching pattern. - Range state toggles when start pattern matches - Range stays true until end pattern matches - Both start and end lines are included #### Compound conditions ```bash # Logical AND (&&) - both conditions must be true awk '$1 == "error" && $2 > 100 { print }' logs.txt # Logical OR (||) - either condition can be true awk '$1 == "error" || $1 == "warning" { print }' logs.txt # Negation (!) - condition must be false awk '!($1 == "info") { print }' logs.txt # Complex expressions awk '($1 ~ /^ERR/ || $1 ~ /^FAIL/) && NR > 100 { print }' logs.txt # Multiple conditions on fields awk '$2 >= 500 && $2 <= 1000 && $3 == "active"' data.txt ``` _exec_ ```bash echo -e "error 150\nerror 50\nwarning 200" | awk '$1 == "error" && $2 > 100 { print }' ``` _output_ ```bash error 150 ``` Compound conditions combine multiple patterns with logical operators for complex filtering. - && has higher precedence than || - Use parentheses for clarity in complex expressions - Patterns are evaluated left to right **Best practices:** - Use range patterns for selecting related lines - Combine && for restrictive filtering - Use || for inclusive filtering **Common errors:** - **Pattern not matching**: Check range start condition appears before end condition ## Variables and Operators Built-in variables, operators, and expressions ### Built-in Variables The built-in variables AWK maintains **Keywords:** NR, NF, FS, OFS, FILENAME, FNR, variables #### Built-in variables reference ```bash # NR - Total record number (line number across all files) awk '{ print NR, $0 }' file # FNR - File record number (line number within current file) awk '{ print FNR, FILENAME, $0 }' file1 file2 # NF - Number of fields in current record awk '{ print "Fields: " NF, "Last field: " $NF }' file # FS - Field separator (input) awk 'BEGIN { FS=":" } { print $1 }' # OFS - Output field separator awk -v OFS="-" '{ print $1, $2, $3 }' file # RS - Record separator (usually newline) awk 'BEGIN { RS=";" } { print NR, $0 }' # ORS - Output record separator awk 'BEGIN { ORS=";" } { print $0 }' # FILENAME - Current filename being processed awk '{ print FILENAME ":" NR ":" $0 }' file1 file2 # ARGC/ARGV - Command line argument count and values awk 'BEGIN { print "Args: " ARGC; for (i=0; i greater than # >= greater than or equal # == equal # != not equal # LOGICAL OPERATORS # && logical AND # || logical OR # ! logical NOT # STRING OPERATORS # (space) concatenation: "hello" " " "world" # ~ regex match: $0 ~ /pattern/ # !~ regex not match: $0 !~ /pattern/ # ASSIGNMENT OPERATORS # = assignment # += add and assign # -= subtract and assign # *= multiply and assign # /= divide and assign # %= modulo and assign # ^= exponentiate and assign # ++ increment # -- decrement ``` _exec_ ```bash echo "10 3" | awk '{ print $1 + $2, $1 - $2, $1 * $2, $1 / $2, $1 % $2, $1 ^ $2 }' ``` _output_ ```bash 13 7 30 3.33333 1 1000 ``` Complete set of AWK operators for arithmetic, comparison, logical, and string operations. - String concatenation is implicit (space between values) - Comparison returns 1 (true) or 0 (false) - Regex operators ~ and !~ test field patterns #### Operator examples ```bash # Arithmetic with increment/decrement awk 'BEGIN { x = 5; print x++, ++x, x--, --x }' # 5 7 7 5 # Compound assignment awk 'BEGIN { x = 10; x += 5; print x }' # 15 # String concatenation awk '{ print $1 " is " $2 " years old" }' # Concatenate strings # Ternary operator awk '{ print ($1 > 50) ? "large" : "small" }' # Conditional value # Precedence example awk 'BEGIN { print 2 + 3 * 4 }' # 14 (not 20) awk 'BEGIN { print (2 + 3) * 4 }' # 20 ``` _exec_ ```bash awk 'BEGIN { x = 5; y = 3; print (x > y) ? "x is greater" : "y is greater" }' ``` _output_ ```bash x is greater ``` Practical examples of operators including compound assignment, concatenation, and ternary operator. - Ternary operator syntax: condition ? true_value : false_value - Post-increment (x++) returns old value - Pre-increment (++x) returns new value **Best practices:** - Use parentheses to clarify operator precedence - Use compound assignment operators for brevity - Use ternary operator for simple conditionals **Common errors:** - **Unexpected type**: AWK automatically converts types; check for unintended string comparisons ### User-Defined Variables Creating and managing custom variables **Keywords:** variables, declare, scope, global, local, initialization #### Variable declaration and scope ```bash # Variables don't need declaration - created on use awk 'BEGIN { x = 5; y = 10; print x + y }' # 15 # Uninitialized variables are 0 or empty string awk 'BEGIN { print "x=" x, "y=" y }' # x=0 y= # Variables are global by default awk 'BEGIN { x = 5 } { print x, $0 } END { print x }' # Function parameters and local variables awk 'function add(a, b, local1, local2) { local1 = a local2 = b return local1 + local2 } BEGIN { print add(3, 4) }' # 7 # Command-line variable assignment awk -v var=value 'BEGIN { print var }' awk -v name="John" '{ print name, $0 }' ``` _exec_ ```bash awk 'BEGIN { count = 0 } { count++ } END { print "Total lines: " count }' /etc/hostname ``` _output_ ```bash Total lines: 1 ``` Variables in AWK are created automatically and are global by default. - Uninitialized variables are 0 (numeric) or "" (string) - Variables created on first use - All variables are global except function parameters - -v flag passes variables to AWK program #### Local and global variable conventions ```bash # Global variables (accessible everywhere) awk 'BEGIN { global_var = 10 } function test() { return global_var * 2 } BEGIN { print test() }' # 20 # Function parameters are local awk 'function f(x) { y = 100 # y is global (created inside function) return x + y } BEGIN { x = 1; y = 2; print f(5); print x, y }' # 105, 1 100 # Local variables (extra parameters) awk 'function f(x, y, local1, local2) { local1 = 10 local2 = 20 return x + y + local1 + local2 } BEGIN { print f(1, 2) }' # 33 # Array variables awk 'BEGIN { arr[1] = "one" arr[2] = "two" for (i in arr) print i, arr[i] }' ``` _exec_ ```bash awk 'BEGIN { total = 0; for (i = 1; i <= 3; i++) total += i; print "Sum 1-3: " total }' ``` _output_ ```bash Sum 1-3: 6 ``` Global variables persist across functions, while function parameters and extra parameters are local. - Extra function parameters after named parameters act as local variables - Global variables accessible throughout program - Local variables should be separated by extra whitespace in function signature **Best practices:** - Use meaningful variable names - Initialize variables in BEGIN block - Use local variables in functions to avoid side effects **Common errors:** - **Modifying global accidentally**: Use extra parameters for local variables in functions ## Control Flow Conditional statements, loops, and flow control ### If-Else Statements Conditional execution with if, else, and else if **Keywords:** if, else, condition, conditional, branching, ternary #### If-else statements ```bash # Basic if statement awk '{ if ($1 > 50) print "Large: " $1 }' file # if-else statement awk '{ if ($1 > 50) print "Large" else print "Small" }' file # Multiple conditions with else-if awk '{ if ($1 > 100) print "Very large" else if ($1 > 50) print "Large" else if ($1 > 10) print "Medium" else print "Small" }' file # Nested conditions awk '{ if ($1 > 50) { if ($2 == "active") print "Active and large" else print "Inactive and large" } }' file ``` _exec_ ```bash echo -e "apple 75\nbanana 30" | awk '{ if ($2 > 50) print $1 " is popular"; else print $1 " is unpopular" }' ``` _output_ ```bash apple is popular banana is unpopular ``` If-else statements allow conditional execution based on boolean expressions. - Condition must evaluate to true (non-zero) or false (zero) - Braces required for multiple statements in block - String comparisons use ==, !=, ~, !~ #### Ternary operator ```bash # Ternary operator: condition ? true_value : false_value awk '{ print ($1 > 50) ? "large" : "small" }' file # Nested ternary awk '{ status = ($1 > 100) ? "very large" : ($1 > 50) ? "large" : "small" print $1 " is " status }' file # Ternary in variable assignment awk '{ message = (NR == 1) ? "First line" : "Not first" print message }' file # Ternary with string values awk '{ result = ($1 ~ /^[0-9]+$/) ? "number" : "text" print result }' file ``` _exec_ ```bash echo -e "5\n150\n75" | awk '{ print $1 " is " (($1 > 100) ? "large" : "small") }' ``` _output_ ```bash 5 is small 150 is large 75 is small ``` Ternary operator provides compact conditional value selection. - Ternary operator: condition ? value_if_true : value_if_false - Useful for simple conditional assignments - Can be nested for multiple conditions **Best practices:** - Use braces even for single statements (clarity) - Use ternary operator for simple condition values - Use if-else for complex conditional logic **Common errors:** - **Unexpected { or }**: Check all if/else blocks have matching braces ### Loops (For, While, Do-While) Loop structures for iterating over data **Keywords:** for, while, do-while, loop, iteration, break, continue #### For loops ```bash # C-style for loop awk 'BEGIN { for (i = 1; i <= 5; i++) print i }' # For loop with array awk 'BEGIN { arr[1] = "apple" arr[2] = "banana" arr[3] = "cherry" for (i = 1; i <= 3; i++) print arr[i] }' # For-in loop (iterate array keys) awk 'BEGIN { data["name"] = "John" data["age"] = "30" data["city"] = "NYC" for (key in data) print key ": " data[key] }' # For loop with field iteration awk '{ for (i = 1; i <= NF; i++) print "Field " i ": " $i }' file ``` _exec_ ```bash awk 'BEGIN { for (i=1; i<=3; i++) for (j=1; j<=3; j++) print i, j }' ``` _output_ ```bash 1 1 1 2 1 3 2 1 2 2 2 3 3 1 3 2 3 3 ``` For loops iterate with counter (C-style) or iterate array keys (for-in). - For-in loop iterates over array keys in arbitrary order - C-style for has init, condition, and update - Empty for (;;) creates infinite loop #### While and do-while loops ```bash # While loop awk 'BEGIN { i = 1 while (i <= 5) { print i i++ } }' # While loop reading from file awk '{ i = 1 while (i <= NF) { print "Field " i ": " $i i++ } }' file # Do-while loop (always runs at least once) awk 'BEGIN { i = 1 do { print i i++ } while (i <= 3) }' # While loop with break awk 'BEGIN { i = 1 while (i <= 10) { if (i == 5) break print i i++ } }' ``` _exec_ ```bash awk 'BEGIN { i=1; while (i<=3) { print "i=" i; i++ } }' ``` _output_ ```bash i=1 i=2 i=3 ``` While loops execute while condition is true; do-while always executes at least once. - While checks condition before executing - Do-while checks condition after executing - Both support break and continue statements **Best practices:** - Use for loop for known iteration count - Use while loop for condition-based iteration - Use for-in for array iteration **Common errors:** - **Infinite loop**: Modify the loop variable inside the body so the condition eventually fails ### Flow Control (Break, Continue, Next) Control loop and program flow with break, continue, next, and exit **Keywords:** break, continue, next, nextfile, exit, return, flow-control #### Break and continue ```bash # Break - exit loop immediately awk 'BEGIN { for (i = 1; i <= 10; i++) { if (i == 5) break print i } }' # Output: 1 2 3 4 # Continue - skip to next iteration awk 'BEGIN { for (i = 1; i <= 5; i++) { if (i == 3) continue print i } }' # Output: 1 2 4 5 # Break from nested loop awk 'BEGIN { for (i = 1; i <= 3; i++) { for (j = 1; j <= 3; j++) { if (j == 2) break print i, j } } }' ``` _exec_ ```bash awk 'BEGIN { for(i=1; i<=5; i++) { if(i==3) continue; print i } }' ``` _output_ ```bash 1 2 4 5 ``` Break exits loops immediately, continue skips to next iteration. - Break only exits innermost loop - Continue skips rest of loop body and goes to next iteration #### Next, nextfile, and exit ```bash # next - skip to next input line awk ' { if ($1 == "skip") next print "Processing: " $0 }' file # nextfile - skip to next file awk ' { if ($1 == "EOF") nextfile print $0 }' file1 file2 # exit - terminate program awk ' { if (NR > 10) exit print $0 }' largefile # exit with status code awk 'BEGIN { exit 0 }' # Success awk 'BEGIN { exit 1 }' # Error # exit runs END block awk 'BEGIN { exit } END { print "Cleanup" }' ``` _exec_ ```bash echo -e "a\nb\nc\nd" | awk 'NR==2 { next } { print }' ``` _output_ ```bash a c d ``` Next skips to next line, nextfile skips remaining lines in file, exit terminates program. - Next applies to main pattern-action block - Nextfile skips to next file in ARGV - Exit runs END blocks before terminating **Best practices:** - Use next to skip unwanted lines - Use exit for error conditions or early termination - Use break to exit when condition met **Common errors:** - **break outside loop**: Break/continue only valid inside loops ## Built-in Functions String, mathematical, and array functions ### String Functions Functions for string manipulation and processing **Keywords:** length, substr, index, split, sub, gsub, match, tolower, toupper, sprintf #### String length and substring ```bash # length(str) - string length awk 'BEGIN { print length("hello") }' # 5 awk '{ print NF, length($1) }' file # Fields and field length # substr(str, start, len) - extract substring awk 'BEGIN { print substr("hello world", 7) }' # world awk 'BEGIN { print substr("hello world", 1, 5) }' # hello awk '{ print substr($0, 1, 10) }' file # First 10 chars of line # index(str, substr) - find position of substring awk 'BEGIN { print index("hello world", "world") }' # 7 awk '{ print index($0, "error") }' # Find "error" position # Practical example: extract domain from email awk -F'@' '{ domain = substr($2, 1, length($2)-4); print domain }' emails.txt ``` _exec_ ```bash awk 'BEGIN { s="hello"; print "len=" length(s), "sub=" substr(s, 2, 3), "idx=" index(s, "ll") }' ``` _output_ ```bash len=5 sub=ell idx=3 ``` Length, substr, and index extract string information and substrings. - length() returns character count - substr starts at 1 (not 0) - index() returns 1-based position or 0 if not found #### String replacement and formatting ```bash # sub(regex, repl, target) - replace first match awk 'BEGIN { s="hello hello"; sub(/hello/, "hi", s); print s }' # hi hello awk '{ sub(/old/, "new"); print }' file # Replace first occurrence # gsub(regex, repl, target) - replace all matches awk 'BEGIN { s="hello hello"; gsub(/hello/, "hi", s); print s }' # hi hi awk '{ gsub(/ +/, " "); print }' file # Normalize spaces # match(str, regex) - test if matches and set RSTART, RLENGTH awk 'BEGIN { if (match("hello world", /wor/)) print "Found at " RSTART " length " RLENGTH }' # sprintf(format, ...) - formatted string awk 'BEGIN { printf "%s=%d\n", "count", 42 }' awk 'BEGIN { s = sprintf("%.2f", 3.14159); print s }' # 3.14 # tolower() and toupper() awk 'BEGIN { print tolower("HELLO") }' # hello awk 'BEGIN { print toupper("hello") }' # HELLO ``` _exec_ ```bash awk 'BEGIN { s="hello world"; gsub(/l/, "L", s); print s }' ``` _output_ ```bash heLLo worLd ``` Sub and gsub replace patterns; match tests for matches; toupper/tolower change case. - sub replaces first occurrence, gsub replaces all - If target omitted, uses $0 - match sets RSTART and RLENGTH variables - sprintf formats strings without printing **Best practices:** - Use gsub for multiple replacements - Use match to find position before substring extraction - Use sprintf for formatted string generation **Common errors:** - **Regex not matching**: Check regex syntax and escaping ### Mathematical Functions Arithmetic and trigonometric functions **Keywords:** int, sqrt, sin, cos, atan2, exp, log, rand, srand, abs #### Basic math functions ```bash # int(x) - integer part awk 'BEGIN { print int(3.7) }' # 3 # sqrt(x) - square root awk 'BEGIN { print sqrt(16) }' # 4 # sin(x), cos(x), atan2(y,x) - trigonometric awk 'BEGIN { print sin(0), cos(0) }' # 0 1 awk 'BEGIN { pi = atan2(0, -1) print "Pi = " pi print "Sin(pi/2) = " sin(pi/2) }' # exp(x) - e^x awk 'BEGIN { print exp(1) }' # 2.71828... # log(x) - natural logarithm awk 'BEGIN { print log(2.71828) }' # 1 # Practical: Calculate percentage awk 'BEGIN { print (75/100) * 100 "%"}' # 75% ``` _exec_ ```bash awk 'BEGIN { print "sqrt(25)=" sqrt(25), "int(3.99)=" int(3.99), "exp(1)=" exp(1) }' ``` _output_ ```bash sqrt(25)=5 int(3.99)=3 exp(1)=2.71828 ``` Math functions perform calculations on numbers. - int() truncates toward zero - Trigonometric functions use radians - exp(1) is approximately e #### Random numbers ```bash # rand() - random number 0 to 1 awk 'BEGIN { for (i = 1; i <= 3; i++) print rand() }' # Random integer 1-10 awk 'BEGIN { for (i = 1; i <= 5; i++) print int(rand() * 10) + 1 }' # srand(seed) - seed random generator awk 'BEGIN { srand(123) print rand() }' # srand() with no seed uses current time awk 'BEGIN { srand() print rand() }' ``` _exec_ ```bash awk 'BEGIN { srand(42); for(i=1; i<=3; i++) print int(rand()*10) }' ``` _output_ ```bash 4 8 2 ``` Random functions generate random numbers for simulations and sampling. - rand() returns float between 0 and 1 - srand() seeds with current time if no seed given - Seeding enables reproducible random sequences **Best practices:** - Use int(rand() * N) + 1 for random integers 1 to N - Use srand() with fixed seed for testing - Use srand() with no seed for different sequences each run **Common errors:** - **NaN or inf in calculations**: Check for negative sqrt, division by zero, etc. ### Arrays and Array Functions Creating and manipulating arrays **Keywords:** arrays, elements, keys, values, delete, in, for-in, split, multi-dimensional #### Array basics ```bash # Create and access array elements awk 'BEGIN { arr[1] = "one" arr[2] = "two" arr[3] = "three" for (i = 1; i <= 3; i++) print arr[i] }' # Associative array (string keys) awk 'BEGIN { person["name"] = "John" person["age"] = "30" person["city"] = "NYC" for (key in person) print key ": " person[key] }' # Array from file awk '{ arr[NR] = $0 } END { for (i in arr) print i ": " arr[i] }' file # Check if key exists awk 'BEGIN { arr["key"] = "value" if ("key" in arr) print "Found" if ("missing" in arr) print "Not there" }' ``` _exec_ ```bash awk 'BEGIN { a[1]=10; a[2]=20; a[3]=30; for(i in a) sum+=a[i]; print "Sum=" sum }' ``` _output_ ```bash Sum=60 ``` Arrays store multiple values with numeric or string indices. - Array indices can be numbers or strings - For-in loop iterates in arbitrary order - in operator tests key existence #### Array operations ```bash # Delete array element awk 'BEGIN { a[1] = "one"; a[2] = "two" delete a[1] for (i in a) print i, a[i] }' # Output: 2 two # Delete entire array awk 'BEGIN { a[1] = 1; a[2] = 2 delete a for (i in a) print i }' # No output # Multi-dimensional arrays awk 'BEGIN { a[1,1] = "top-left" a[1,2] = "top-right" a[2,1] = "bot-left" for (key in a) print key ": " a[key] }' # split() function creates array awk 'BEGIN { count = split("a:b:c:d", arr, ":") for (i = 1; i <= count; i++) print arr[i] }' ``` _exec_ ```bash awk 'BEGIN { n=split("one,two,three", a, ","); for(i=1;i<=n;i++) print a[i] }' ``` _output_ ```bash one two three ``` Delete removes elements, split creates arrays from strings, multi-dimensional arrays possible with composite keys. - delete removes single element or entire array - split returns count of elements created - Multi-dimensional: a[i,j,k] key is i\034j\034k (SUBSEP) **Best practices:** - Use associative arrays with meaningful keys - Check key existence with in operator - Use split to parse strings into arrays **Common errors:** - **Array index out of bounds**: AWK arrays auto-expand; any index is valid ## Advanced Features User-defined functions, file handling, and advanced techniques ### User-Defined Functions Creating and using custom functions **Keywords:** function, define, parameters, return, local, scope, recursion #### Defining and calling functions ```bash # Basic function definition and example awk 'function add(a, b) { return a + b } BEGIN { print "5 + 3 = " add(5, 3) }' # Function with multiple statements awk 'function greet(name) { greeting = "Hello, " name "!" return greeting } BEGIN { print greet("Alice") }' # Function without return statement awk 'function print_info(x) { print "Value: " x } BEGIN { print_info(42) }' # Function with local variables (extra parameters) awk 'function calculate(x, y, local_result) { local_result = x * y + 10 return local_result } BEGIN { print calculate(3, 4) }' ``` _exec_ ```bash awk 'function double(x) { return x * 2 } BEGIN { print double(21) }' ``` _output_ ```bash 42 ``` Functions encapsulate logic for reuse and modularity. - Functions must be defined before use - Parameters are passed by value - Extra parameters act as local variables - Return statement optional (returns 0/"" if omitted) #### Advanced function patterns ```bash # Recursive function (factorial) awk 'function fact(n) { if (n <= 1) return 1 return n * fact(n - 1) } BEGIN { print "5! = " fact(5) }' # Function modifying array parameter (arrays passed by reference) awk 'function fill_array(arr, n, i) { for (i = 1; i <= n; i++) arr[i] = i * i } BEGIN { fill_array(data, 5) for (i in data) print i, data[i] }' # Helper functions for validation awk 'function is_number(s) { return s ~ /^[0-9]+$/ } function is_empty(s) { return length(s) == 0 } BEGIN { print "5 is_number: " is_number("5") print empty is_empty: " is_empty("") }' ``` _exec_ ```bash awk 'function max(a, b) { return (a > b) ? a : b } BEGIN { print max(15, 23) }' ``` _output_ ```bash 23 ``` Functions support recursion, array parameters, and complex logic patterns. - Recursive functions must have base case - Arrays passed by reference (modifications persist) - Scalars passed by value (modifications local) **Best practices:** - Keep functions small and focused - Document function purpose and parameters - Use meaningful recursive base cases **Common errors:** - **Call to undefined function**: Define function before first use ### Advanced Input and Output Multiple file handling, redirection, and pipes **Keywords:** files, FILENAME, FNR, getline, pipes, redirection, multiple-files, command-line #### Processing multiple files ```bash # FILENAME variable shows current file awk '{ print FILENAME ":" NR ":" $0 }' file1 file2 # FNR is line number within each file awk 'FNR == 1 { print "Starting " FILENAME } { print FILENAME ":" FNR ":" $0 }' file1 file2 # ARGIND (gawk) - current argument index awk '{ print ARGIND, FILENAME, NR, FNR }' file1 file2 # Process specific files only awk 'FILENAME == "file1" { print }' file1 file2 # Skip files based on pattern awk 'FNR == 1 && FILENAME ~ /skip/ { nextfile } { print }' file* ``` _exec_ ```bash echo "a" > /tmp/f1 && echo "b" > /tmp/f2 && awk '{ print FILENAME, FNR, $0 }' /tmp/f1 /tmp/f2 ``` _output_ ```bash /tmp/f1 1 a /tmp/f2 1 b ``` Process multiple files with automatic file tracking. - NR tracks total lines across all files - FNR resets for each file - FILENAME shows current filename #### Output redirection and pipes ```bash # Redirect output to file awk '{ print > "output.txt" }' input.txt # Overwrite awk '{ print >> "output.txt" }' input.txt # Append # Redirect specific pattern to file awk '/error/ { print > "errors.log" } !/error/ { print > "clean.log" }' logfile # Pipe output to command awk '{ print | "sort" }' unsorted.txt awk '{ print | "mail -s report user@host" }' data # Close file or pipe awk '{ print > "output.txt" close("output.txt") # Flush and close file }' input # getline - read next line awk '{ print "Current: " $0 getline next_line print "Next: " next_line }' file ``` _exec_ ```bash echo -e "hello\nworld" | awk '{ print | "sort -r" }' ``` _output_ ```bash world hello ``` Redirect output to files or pipes; getline reads additional lines. - > creates new file, >> appends to file - | pipes output to command - close() flushes file and allows reopening - getline reads next input line into a variable **Best practices:** - Use FNR for per-file line numbers - Close files to avoid resource leaks - Use pipes carefully for system performance **Common errors:** - **Too many open files**: Use close() to close files after writing ## Practical Examples Real-world use cases and data processing scenarios ### Common Text Processing Tasks Practical examples of typical AWK operations **Keywords:** examples, practical, use-cases, real-world, data-processing #### Extracting and reformatting data ```bash # Extract specific columns from CSV awk -F',' '{ print $1, $3 }' data.csv # Convert CSV to TSV (tab-separated) awk -F',' -v OFS='\t' '{ print $1, $2, $3 }' data.csv # Extract IP address from log files awk '{ print $1 }' /var/log/apache2/access.log # Count occurrences of each word awk '{ for(i=1; i<=NF; i++) count[$i]++ } END { for (w in count) print w, count[w] }' file.txt # Filter and print lines with specific pattern awk '/^ERROR|^WARN/ { print }' system.log # Sum column of numbers awk '{ sum += $1 } END { print "Total: " sum }' numbers.txt ``` _exec_ ```bash echo -e "a:1\nb:2\nc:3" | awk -F':' '{ sum+=$2 } END { print "Sum=" sum }' ``` _output_ ```bash Sum=6 ``` Common data extraction and transformation operations. - awk handles column extraction and reformatting - Easy to change field separators for format conversion #### Log processing and analysis ```bash # Count requests by hour from Apache log awk '{ time = $4 sub(/\[/, "", time) gsub(/:/, " ", time) hour = substr(time, 13, 2) count[hour]++ } END { for (h in count) print h ":00 - " count[h] " requests" }' /var/log/apache2/access.log # Extract and count HTTP status codes awk '{ print $(NF-1) }' access.log | sort | uniq -c # Find top users by request count awk '{ users[$1]++ } END { for (u in users) print u, users[u] }' access.log | sort -k2 -rn | head -10 # Calculate average response time awk '{ time = $(NF) sum += time count++ } END { print "Average: " sum / count " ms" }' response-times.log ``` _exec_ ```bash echo -e "error\nerror\nwarn\nerror" | awk '{ count[$1]++ } END { for (t in count) print t, count[t] }' ``` _output_ ```bash error 3 warn 1 ``` Processing logs to extract metrics and statistics. - AWK ideal for parsing structured log formats - Pattern matching finds relevant log lines - Aggregation generates summary statistics **Best practices:** - Use awk for quick data extraction instead of Perl - Combine awk with grep for complex filtering - Save awk program in file for reusable scripts **Common errors:** - **Unexpected output**: Check field numbers and separators match file format ### Advanced Data Processing Complex data transformations and analysis **Keywords:** advanced, complex, calculations, transformations, statistics, reports #### Data validation and cleaning ```bash # Remove duplicate lines (preserve order) awk '!seen[$0]++' file # Remove leading/trailing whitespace awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' file # Validate email format awk '/^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/ { print }' emails.txt # Check if all fields are numeric awk '{ for(i=1; i<=NF; i++) if ($i !~ /^[0-9.]+$/) next; print }' data.txt # Remove lines with specific patterns awk '!/^#/ && !/^$/ && !/^;/ { print }' config.txt # Normalize whitespace (multiple spaces to single) awk '{ gsub(/ +/, " "); $1=$1; print }' file ``` _exec_ ```bash echo -e "a\nb\na\nc\nb" | awk '!seen[$0]++' ``` _output_ ```bash a b c ``` Data cleaning operations remove duplicates, whitespace, and invalid records. - !seen[$0]++ is the standard duplicate removal pattern - gsub with start/end anchors trims whitespace - Email validation uses complex regex pattern #### Statistical calculations and reports ```bash # Calculate mean, median, and mode awk '{ sum += $1; count++ arr[count] = $1 } END { printf "Mean: %.2f\n", sum/count }' numbers.txt # Find min and max values awk 'NR==1 { min=$1; max=$1; next } { if ($1 < min) min=$1; if ($1 > max) max=$1 } END { print "Min: " min ", Max: " max }' numbers.txt # Generate formatted report awk 'BEGIN { printf "%-20s %-10s %-10s\n", "Name", "Age", "Score" printf "%-20s %-10s %-10s\n", "----", "---", "-----" } { total += $3; count++ printf "%-20s %-10d %-10d\n", $1, $2, $3 } END { printf "%-20s %-10s %-10.1f\n", "Average", "", total/count }' data.txt # Percentile calculation awk '{ arr[NR] = $1 sum += $1 } END { mean = sum / NR for (i=1; i<=NR; i++) dev += (arr[i] - mean)^2 stddev = sqrt(dev / NR) print "Mean: " mean ", StdDev: " stddev }' data.txt ``` _exec_ ```bash echo -e "5\n10\n15\n20\n25" | awk '{ sum+=$1; arr[NR]=$1 } END { print "Sum=" sum " Avg=" sum/NR " Count=" NR }' ``` _output_ ```bash Sum=75 Avg=15 Count=5 ``` Statistical operations including mean, min/max, and formatted report generation. - Array needed for min/max or median calculations - printf enables precise formatting for reports - Mathematical functions enable statistical analysis **Best practices:** - Save awk programs in files for complex operations - Test with small datasets before running on large files - Use printf for column-aligned output **Common errors:** - **Type conversion errors**: Check fields are numeric before arithmetic