# Chatgpt Generate Ruby Code to Check User Exists in Github

On saturday night, I want to work on a side project. To find a name for the project and I wanted to create an Github organization in the same name. I started trying out the name one after the other, all the names were taken and thought about writing small script in ruby. Then from nowhere I decided to let chatgpt to write the code for me.

In this blog post, I’ll share some of the code generated by chatgpt for the checking whether a profile name exists in the Github with increasing complexity.

You can signup for chatgpt from openai site.

# Case 1: Simple case

Prompt

Can you write a ruby program to check a a list of profile names exists in github?


Generated Code

require 'octokit'

ACCESS_TOKEN = "ACCESS_TOKEN"

# Replace with the list of profile names
profiles = ["profile1", "profile2", "profile3"]

client = Octokit::Client.new(access_token: ACCESS_TOKEN)

profiles.each do |profile|
user = client.user(profile)
if user
puts "#{profile} exists on GitHub"
else
puts "#{profile} does not exist on GitHub"
end
end


The code looks legit and may run, but there are two issues

• There is no instruction to install octokit.
• Not reading the secret from the environment variable is a bad sign.

Let’s enhance the instruction by explicitly stating it.

Prompt

Can you write a ruby program to check a a list of profile names exists in github?
If you use external gem, specify the installation instruction. Also read secrets from environment variable.


Generated code

require 'octokit'

# Replace with the list of profile names
profiles = ["profile1", "profile2", "profile3"]

access_token = ENV["ACCESS_TOKEN"]

client = Octokit::Client.new(access_token: access_token)

profiles.each do |profile|
user = client.user(profile)
if user
puts "#{profile} exists on GitHub"
else
puts "#{profile} does not exist on GitHub"
end
end


Along with the code, there is an installation line, To install the octokit gem, run gem install octokit in your terminal.

Output

$ruby case_1.rb To use retry middleware with Faraday v2.0+, install faraday-retry gem profile1 exists on GitHub profile2 exists on GitHub profile3 exists on GitHub  The code actually works! Now let’s slightly modify the requirements to read list of profile names from a file. # Case 2: Read list of profile names from a file, check the profiles and write to JSON file Prompt Can you write a ruby program to read a list of github profile from a text file, check each profile exists in github and write the result to a JSON file? If you use external gem, specify the installation instruction. Also read secrets from environment variable.  Generated code require 'octokit' require 'json' # Read profile names from text file profiles = File.readlines("profiles.txt").map(&:chomp) # Read access token from environment variable access_token = ENV["ACCESS_TOKEN"] client = Octokit::Client.new(access_token: access_token) # Create a new hash to store the result result = {} profiles.each do |profile| user = client.user(profile) if user result[profile] = true else result[profile] = false end end # Write the result to a JSON file File.open("result.json","w") do |f| f.write(result.to_json) end  Output $cat profiles.txt
kracekumar
zzzzzz
404
$ruby case_2.rb To use retry middleware with Faraday v2.0+, install faraday-retry gem Traceback (most recent call last): 15: from case_2.rb:15:in <main>' 14: from case_2.rb:15:in each' 13: from case_2.rb:16:in block in <main>' 12: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/octokit-6.0.1/lib/octokit/client/users.rb:34:in user' 11: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/octokit-6.0.1/lib/octokit/connection.rb:19:in get' 10: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/octokit-6.0.1/lib/octokit/connection.rb:156:in request' 9: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sawyer-0.9.2/lib/sawyer/agent.rb:99:in call' 8: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/faraday-2.7.1/lib/faraday/connection.rb:200:in get' 7: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/faraday-2.7.1/lib/faraday/connection.rb:445:in run_request' 6: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/faraday-2.7.1/lib/faraday/rack_builder.rb:153:in build_response' 5: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/octokit-6.0.1/lib/octokit/middleware/follow_redirects.rb:61:in call' 4: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/octokit-6.0.1/lib/octokit/middleware/follow_redirects.rb:73:in perform_with_redirection' 3: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/faraday-2.7.1/lib/faraday/middleware.rb:17:in call' 2: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/faraday-2.7.1/lib/faraday/response.rb:42:in on_complete' 1: from /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/faraday-2.7.1/lib/faraday/middleware.rb:18:in block in call' /Users/krace/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/octokit-6.0.1/lib/octokit/response/raise_error.rb:14:in on_complete': GET https://api.github.com/users/404: 404 - Not Found // See: https://docs.github.com/rest/reference/users#get-a-user (Octokit::NotFound)  Ah ha. the profile 404 returns 404 and the program fails, now enhance the prompt to modify it. Prompt Can you write a ruby program to read a list of github profile from a text file, check each profile exists in github and write the result to a JSON file? The code should handle exceptions. If you use external gem, specify the installation instruction. Also read secrets from environment variable.  Generated code require 'octokit' require 'json' # Read profile names from text file profiles = File.readlines("profiles.txt").map(&:chomp) # Read access token from environment variable access_token = ENV["ACCESS_TOKEN"] client = Octokit::Client.new(access_token: access_token) # Create a new hash to store the result result = {} profiles.each do |profile| begin user = client.user(profile) if user result[profile] = true else result[profile] = false end rescue Octokit::NotFound result[profile] = false end end # Write the result to a JSON file File.open("result.json","w") do |f| f.write(result.to_json) end  • Now the model generates the code with exception handling but only handles 404 without explicitly mentioning the specific exception in the prompt. Neat! Output $ruby case_2.rb
To use retry middleware with Faraday v2.0+, install faraday-retry gem
$cat result.json {"kracekumar":true,"zzzzzz":true,"404":false}  # Case 3: Read from dictionary, check the profile in github, handle rate-limit and write the results to file. Prompt Can you write a ruby program to read a list of github profile from a text file in the location /usr/share/dict/words, check each profile exists in github, check for rate limit and write the result to a JSON file? Print result of each user check in the terminal. The code should handle exceptions. If you use external gem, specify the installation instruction. Also read secrets from environment variable.  Generated Code require 'octokit' require 'json' # Read profile names from text file profiles = File.readlines("/usr/share/dict/words").map(&:chomp) # Read access token from environment variable access_token = ENV["ACCESS_TOKEN"] client = Octokit::Client.new(access_token: access_token) # Create a new hash to store the result result = {} profiles.each do |profile| # Check for rate limit rate_limit = client.rate_limit if rate_limit.remaining == 0 puts "Rate limit exceeded. Waiting for reset at #{rate_limit.reset_at}..." sleep rate_limit.resets_in end begin user = client.user(profile) if user result[profile] = true puts "#{profile} exists on GitHub" else result[profile] = false puts "#{profile} does not exist on GitHub" end rescue Octokit::NotFound result[profile] = false puts "#{profile} does not exist on GitHub" end end # Write the result to a JSON file File.open("result.json","w") do |f| f.write(result.to_json) end  Output $ruby case_3.rb
To use retry middleware with Faraday v2.0+, install faraday-retry gem
A exists on GitHub
a exists on GitHub
aa exists on GitHub
aal exists on GitHub
...
abarthrosis does not exist on GitHub
Traceback (most recent call last):
2: from case_3.rb:15:in <main>'
1: from case_3.rb:15:in each'
case_3.rb:19:in block in <main>': undefined method reset_at' for #<Octokit::RateLimit:0x0000000107ba8760> (NoMethodError)
Did you mean?  resets_at
resets_at=


Oops, the code breaks at line number 19 - rate_limit.reset_at.

Let’s give feedback to the chatgpt, saying the code fails at line 19.

Prompt

the above code failed with the following error, undefined method reset_at' for #<Octokit::RateLimit:0x000000010498f210> (NoMethodError)
Did you mean?  resets_at
resets_at=

Can you fix it?


Generated code

require 'octokit'
require 'json'

# Read profile names from text file

access_token = ENV["ACCESS_TOKEN"]

client = Octokit::Client.new(access_token: access_token)

# Create a new hash to store the result
result = {}

profiles.each do |profile|
# Check for rate limit
rate_limit = client.rate_limit
if rate_limit.remaining == 0
puts "Rate limit exceeded. Waiting for reset at #{rate_limit.resets_at}..."
sleep rate_limit.resets_in
end

begin
user = client.user(profile)
if user
result[profile] = true
puts "#{profile} exists on GitHub"
else
result[profile] = false
puts "#{profile} does not exist on GitHub"
end
rescue Octokit::NotFound
result[profile] = false
puts "#{profile} does not exist on GitHub"
end
end

# Write the result to a JSON file
File.open("result.json","w") do |f|
f.write(result.to_json)
end


The model generated above code with a message, The error occurred because the Octokit::RateLimit object does not have a method named reset_at. The correct method is resets_at. Great!

Output

$ruby case_3.rb To use retry middleware with Faraday v2.0+, install faraday-retry gem Rate limit exceeded. Waiting for reset at 2022-12-04 19:46:29 +0000...  Now the bug is fixed and worked as expected! I don’t say issues. OF course! there can be all sort of exceptions that need to be handled - socket timeout, slow response time, 500s etc… Saving the results to a file after checking all profiles is problematic. # Case 4: Let’s tell the model to refactor the code into functions. Prompt Thanks, that fixed exception. Can you refactor the code into functions?  Generated Code require 'octokit' require 'json' # Read profile names from text file def read_profiles File.readlines("/usr/share/dict/words").map(&:chomp) end # Check if user exists on GitHub def check_user(user) begin client.user(user) true rescue Octokit::NotFound false end end # Check for rate limit and wait if necessary def check_rate_limit rate_limit = client.rate_limit if rate_limit.remaining == 0 puts "Rate limit exceeded. Waiting for reset at #{rate_limit.resets_at}..." sleep rate_limit.resets_in end end # Read access token from environment variable access_token = ENV["ACCESS_TOKEN"] client = Octokit::Client.new(access_token: access_token) # Create a new hash to store the result result = {} profiles = read_profiles profiles.each do |profile| check_rate_limit result[profile] = check_user(profile) puts "#{profile} exists on GitHub" if result[profile] puts "#{profile} does not exist on GitHub" unless result[profile] end # Write the result to a JSON file File.open("result.json","w") do |f| f.write(result.to_json) end  The generated code is synatically invalid, since client variable is undefined in 21 Prompt I get the error when I run the above code, Traceback (most recent call last): 3: from case_3.rb:38:in <main>' 2: from case_3.rb:38:in each' 1: from case_3.rb:39:in block in <main>' case_3.rb:21:in check_rate_limit': undefined local variable or method client' for main:Object (NameError)  Generated Code require 'octokit' require 'json' # Read profile names from text file def read_profiles File.readlines("/usr/share/dict/words").map(&:chomp) end # Check if user exists on GitHub def check_user(client, user) begin client.user(user) true rescue Octokit::NotFound false end end # Check for rate limit and wait if necessary def check_rate_limit(client) rate_limit = client.rate_limit if rate_limit.remaining == 0 puts "Rate limit exceeded. Waiting for reset at #{rate_limit.resets_at}..." sleep rate_limit.resets_in end end # Read access token from environment variable access_token = ENV["ACCESS_TOKEN"] client = Octokit::Client.new(access_token: access_token) # Create a new hash to store the result result = {} profiles = read_profiles profiles.each do |profile| check_rate_limit(client) result[profile] = check_user(client, profile) puts "#{profile} exists on GitHub" if result[profile] puts "#{profile} does not exist on GitHub" unless result[profile] end # Write the result to a JSON file File.open("result.json","w") do |f| f.write(result.to_json) end  Fix message is better: The error occurred because the check_rate_limit and check_user functions did not have access to the client object. To fix the error, the client object must be passed as an argument to these functions. Output $ruby case_3.rb
To use retry middleware with Faraday v2.0+, install faraday-retry gem
Rate limit exceeded. Waiting for reset at 2022-12-04 19:46:29 +0000...


# Case 5: Write the test cases

Prompt

Can you write the test cases for the above code?


Generated Code

require 'minitest/autorun'
require 'octokit'
require 'json'

class TestGitHubProfileChecker < Minitest::Test
def setup
@access_token = ENV["ACCESS_TOKEN"]
@client = Octokit::Client.new(access_token: @access_token)
end

expected_profiles = ["test1", "test2"]
end
end

def test_check_user_exists
user = "octocat"
assert check_user(@client, user)
end

def test_check_user_does_not_exist
user = "not_a_real_user"
refute check_user(@client, user)
end

def test_check_rate_limit
rate_limit = @client.rate_limit
rate_limit.stub :remaining, 0 do
rate_limit.stub :resets_at, Time.now do
rate_limit.stub :resets_in, 0 do
assert_output "Rate limit exceeded. Waiting for reset at #{rate_limit.resets_at}...\n" do
check_rate_limit(@client)
end
end
end
end
end
end


After adding the previously generated code, a test case failed.

ruby test_case.rb
To use retry middleware with Faraday v2.0+, install faraday-retry gem
Run options: --seed 34383

# Running:

..F.

Finished in 0.782971s, 5.1087 runs/s, 5.1087 assertions/s.

1) Failure:
TestGitHubProfileChecker#test_check_rate_limit [test_case.rb:83]:
In stdout.
--- expected
+++ actual
@@ -1,2 +1 @@
-"Rate limit exceeded. Waiting for reset at 2022-12-04 20:24:05 +0000...
-"
+""

4 runs, 4 assertions, 1 failures, 0 errors, 0 skips


Looking at the code, rate limit code was stubbed but doesn’t behave as expected. Again updating the prompt, a couple of times, the model produces same test code and it is unable to fix it. After multiple prompts, the model stop generating complete code.

Final generated code

require 'minitest/autorun'
require 'octokit'
require 'json'

class TestGitHubProfileChecker < Minitest::Test
def setup
@access_token = ENV["ACCESS_TOKEN"]
@client = Octokit::Client.new(access_token: @access_token)
end

expected_profiles = ["test1", "test2"]
end
end

def test_check_user_exists
user = "octocat"
assert check_user(@client, user)
end

def test_check_user_does_not_exist
user = "not_a_real_user"
refute check_user(@client, user)
end

def test_check_rate_limit
now = Time.now
rate_limit = Octokit::RateLimit.new(remaining: 0, resets_at: now.to_i, resets_in: 0)
@client.stub :rate_limit, rate_limit do
assert_output "Rate limit exceeded. Waiting for reset at #{now}...\n" do
check_rate_limit(@client)


Final conversation

Me: How many lines of code are you allowed to generate in a given time?
Model: There is no specific limit on the number of lines of code that can be generated in a given time.
The quality and accuracy of the code is more important than the quantity of code generated.
Me: But your previous code generation was incomplete. can you tell me what happened?
Model: I apologize if my previous code generation was incomplete. There can be a number of reasons why this may have happened.
For example, the prompt may not have provided enough information to accurately generate the code, or there may have been a technical issue with the code generation system. In any case, I will do my best to provide complete and accurate code in response to future prompts.


# Takeway

• Given enough context and iteration the model is able to correct the simple errors.
• It take many attempts to get something working, but it’s far away from production quality code(could be pure model safe-guard).
• It’s able to remember the previous context and improvise based on feedback.
• Once things get slightly complicated the model fails to generate syntactically correct code.

It was definitely fun to play around with the model but making the model to produce fool proof code seems very far away.