A Tale of ChatGPT Failure

The hype around ChatGPT has receeded somewhat, but it's still strong. One of the things you keep hearing is that tools like ChatGTP will replace (human) software developers. In this article I will show you how asking ChatGTP to write a simple Bash shell script resulted in failure after failure. For now it would seem that we (the human software developers) are safe. No doubt AI tools like ChatGPT will improve, but my feeling increasingly is that this will only increase the value of experienced (human) developers who can review the code generated by such tools.

I used to be quite good writing shell scripts. But that time was 25 years ago and it has long since passed. I don't use shell scripting much anymore and anything non trivial I have to do, I usually do in PHP. For reasons of consistency I wanted a short shell script to do some file processing when generating SQL migration code and turned to ChatGPT for help. Here is the almost humorous exchange that followed:

Me: Write a bash shell script which, if a file contains the string "CREATE VIEW ABC" adds the line "DROP VIEW IF EXISTS ABC before this line?

ChatGTP: Certainly! Here's a bash shell script that can accomplish this task

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    sed -i "/$string/i $line" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`ABC\`"
line_to_add="DROP VIEW IF EXISTS \`ABC\`"

if file_contains_string "$file_path" "$string_to_find"; then
    add_line_before_string "$file_path" "$string_to_find" "$line_to_add"
    echo "Line added successfully."
else
    echo "String not found in the file."
fi

This was a basic skeleton and it worked, but of course this wasn't quite what I needed. So, back to ChatGPT for a refinement.

Me: *Change it so that instead of matching ABC, it will take the name between the `` quotes and use this in the "DROP VIEW IF EXISTS" statement?

ChatGPT: Certainly! Here's an updated version of the script that extracts the name between the backticks and uses it in the "DROP VIEW IF EXISTS" statement:

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    sed -i "/$string/i $line" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`([^`]+)\`"
line_template="DROP VIEW IF EXISTS \`%s\`"

if file_contains_string "$file_path" "$string_to_find"; then
    while read -r match; do
        view_name="${match#*\`}"
        view_name="${view_name%*\`}"
        line_to_add=$(printf "$line_template" "$view_name")
        add_line_before_string "$file_path" "$match" "$line_to_add"
        echo "Added line for view: $view_name"
    done < <(grep -Eo "$string_to_find" "$file_path")
else
    echo "String not found in the file."
fi

Me: I get the following error using this: "createMigrations.sh: 43: Syntax error: EOF in backquote substitution"

ChatGPT: I apologize for the error. It seems there was a mistake in the script. Let's update it to use a different approach without backquote substitution. Here's the revised version:

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    sed -i "/$string/i $line" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`([^`]+)\`"
line_template="DROP VIEW IF EXISTS `%s`"

if file_contains_string "$file_path" "$string_to_find"; then
    while IFS= read -r match; do
        view_name=$(echo "$match" | sed -E "s/$string_to_find/\1/")
        line_to_add=$(printf "$line_template" "$view_name")
        add_line_before_string "$file_path" "$match" "$line_to_add"
        echo "Added line for view: $view_name"
    done < <(grep -Eo "$string_to_find" "$file_path")
else
    echo "String not found in the file."
fi

Me: I still get the same error

ChatGPT: I apologize for the inconvenience. It seems there was still an issue with the previous script. Here's an alternative version that should work without causing the "EOF in backquote substitution" error:

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    awk -v str="$string" -v ln="$line" '{print} $0~str{print ln}' "$file" > "$file.tmp" && mv "$file.tmp" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`([^`]+)\`"
line_template="DROP VIEW IF EXISTS `%s`"

if file_contains_string "$file_path" "$string_to_find"; then
    while IFS= read -r match; do
        view_name=$(echo "$match" | sed -E "s/$string_to_find/\1/")
        line_to_add=$(printf "$line_template" "$view_name")
        add_line_before_string "$file_path" "$match" "$line_to_add"
        echo "Added line for view: $view_name"
    done < <(grep -Eo "$string_to_find" "$file_path")
else
    echo "String not found in the file."
fi

Me: I now get "redirection unexpected"

ChatGPT: Apologies for the inconvenience caused. The error you encountered is likely due to the use of the < <(...) process substitution syntax, which may not be supported in your current shell. Let's modify the script to avoid this issue by using a temporary file instead. Here's the updated version:

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    awk -v str="$string" -v ln="$line" '{print} $0~str{print ln}' "$file" > "$file.tmp" && mv "$file.tmp" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`([^`]+)\`"
line_template="DROP VIEW IF EXISTS `%s`"

if file_contains_string "$file_path" "$string_to_find"; then
    grep -Eo "$string_to_find" "$file_path" | while IFS= read -r match; do
        view_name=$(echo "$match" | sed -E "s/$string_to_find/\1/")
        line_to_add=$(printf "$line_template" "$view_name")
        add_line_before_string "$file_path" "$match" "$line_to_add"
        echo "Added line for view: $view_name"
    done
else
    echo "String not found in the file."
fi

Me: I now get "1: %s: not found"

ChatGPT: Apologies for the oversight. It appears that the error is due to the use of the printf function without proper argument substitution. Let's update the script to resolve this issue. Here's the revised version:

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    awk -v str="$string" -v ln="$line" '{print} $0~str{print ln}' "$file" > "$file.tmp" && mv "$file.tmp" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`([^`]+)\`"
line_template="DROP VIEW IF EXISTS '%s'"

if file_contains_string "$file_path" "$string_to_find"; then
    grep -Eo "$string_to_find" "$file_path" | while IFS= read -r match; do
        view_name=$(echo "$match" | sed -E "s/$string_to_find/\1/")
        line_to_add=$(printf "$line_template" "$view_name")
        add_line_before_string "$file_path" "$match" "$line_to_add"
        echo "Added line for view: $view_name"
    done
else
    echo "String not found in the file."
fi

Me: I still get the same error

ChatGPT: Apologies for the persistence of the error. It appears there was a mistake in the script. To resolve the issue, we'll modify the way the line template is passed to awk for insertion. Here's an updated version of the script:

#!/bin/bash

# Function to check if a file contains a specific string
file_contains_string() {
    if grep -q "$2" "$1"; then
        return 0  # String found
    else
        return 1  # String not found
    fi
}

# Function to add a line before a specific string in a file
add_line_before_string() {
    local file="$1"
    local string="$2"
    local line="$3"
    awk -v str="$string" -v ln="$line" '{print} $0~str{print ln}' "$file" > "$file.tmp" && mv "$file.tmp" "$file"
}

# Main script

file_path="path/to/your/file.sql"  # Replace with the actual file path
string_to_find="CREATE VIEW \`([^`]+)\`"
line_template='DROP VIEW IF EXISTS `%s`'

if file_contains_string "$file_path" "$string_to_find"; then
    grep -Eo "$string_to_find" "$file_path" | while IFS= read -r match; do
        view_name=$(echo "$match" | sed -E "s/$string_to_find/\1/")
        line_to_add=$(printf "$line_template" "$view_name")
        add_line_before_string "$file_path" "$match" "$line_to_add"
        echo "Added line for view: $view_name"
    done
else
    echo "String not found in the file."
fi

This pattern continued for a few more times and eventually I gave up and decided to code what I needed using PHP. It took me less than 5 minutes to do so, less time that I had spent trying to get ChatGPT to generate a working result.

Conclusion

I'd say for now we're safe. Even more so, the skillset of an experienced engineer just became more valuable, because they can evaluate and fix the code that such AI tools generate. A few years from now, the story might be different, but for now, AIs are a potentially useful tool, but you need to be aware of the pitfalls when using them. It's sort of like copying/pasting code from Stackoverflow, it might work, it might not work, depends on your exact requirement and how lucky you are.