Should ChatGPT-4 Write Your Cover Letter? What’s Improved (and What Hasn’t) was originally published on Forage.
ChatGPT got an upgrade to ChatGPT-4. It’s more advanced and, according to OpenAI is more creative and collaborative; has more advanced reasoning; performs better than earlier versions on standardized tests; is safer.
OpenAI also notes that ChatGPT-4 has some of the same limitations of earlier versions, like hallucinating (where the bot invents information and includes it in the output).
But an upgrade is an upgrade. We wondered if the new and improved ChatGPT could write a cover letter that might wow a recruiter.
- ChatGPT-3.5: Our Cover Letter Experiment
- ChatGPT-4: Is It an Improvement?
- Testing ChatGPT-4’s Cover Letter Writing Capabilities With My Cover Letter
- Should ChatGPT-4 Write Your Cover Letter?
ChatGPT-3.5: Our Cover Letter Experiment
In our first round of ChatGPT testing, the cover letters were OK. We asked ChatGPT-3.5 to write cover letters for an entry-level investment banker, an accounting internship, for someone without any work experience, and for the lead deckhand on a pirate ship.
The prompt for these cover letters was pretty broad: Write a cover letter for X. But the output was exactly what we expected: cover letters that were written for X. Overall, we felt they were decent. The cover letters were well written, polite, and followed general cover letter writing conventions.
>>MORE: How to Write a Cover Letter in 2022
However, the cover letters also contained hallucinations. ChatGPT-3.5 invented skills and abilities for our job seekers even though we didn’t mention any in the initial prompt. That’s to be expected, though. The less specific the prompt, the more ChatGPT has to “improvise” to fill your request.
Our initial tests also found that ChatGPT-3.5 parroted information. In an attempt to create a more targeted cover letter, we input part of a Forage job description and asked ChatGPT to write cover letters for the role. The bot repeated the job description as part of the cover letter but didn’t include any context. They generally did not have an explanation of how the letter writer used their skills on the job for the benefit of the company.
Our advice (and that of our experts) was that it’s fine to use ChatGPT-3.5 to help generate ideas and give you a starting point, but that you shouldn’t rely on ChatGPT alone to write your cover letter.
ChatGPT-4: Is It an Improvement?
With the release of ChatGPT-4, we wanted to see if the bot had improved. Would it still parrot information? Create skills and abilities for the job seeker? Give context and meaning to any skills we do provide?
ChatGPT-4’s Cover Letter for an Internship With no Work Experience
We started by asking ChatGPT-4 to generate a cover letter for an internship when the person has no work experience. Here’s what we got:
It’s interesting that ChatGPT-4 formatted the cover letter as a formal business letter. There’s an entire section devoted to the writer’s and recipient’s contact information. ChatGPT-4 even left space for full street addresses. Since cover letters are usually an in-body email message or pasted into an applicant tracking system (ATS), most people don’t include the address information (unless it’s specifically requested). It’s not incorrect to include it, but it’s not generally done these days.
Unlike ChatGPT-3.5’s cover letters, this one doesn’t include many hallucinations about the internship seeker. Though one or two slipped through (“I collaborated with my peers…”), ChatGPT-4 generally wrote the cover letter “mad libs” style. At key points, ChatGPT-4 inserted brackets and suggested what the writer should fill the blanks in with, like relevant coursework, two to three soft skills (and even goes the extra step of suggesting which soft skills to include), and mentioning specifics aspect of the company.
And some of the hallucinations are exactly the kinds of things you’d probably include on a cover letter anyway. For example, saying the writer is “highly motivated and excited about the opportunity” is the kind of thing many people include in their cover letters all the time.
Overall, this mad libs cover letter is an excellent outline for someone who doesn’t have a lot of relevant work experience or transferable skills. Instead of ChatGPT-4 making assumptions about who you are and what your skills are, it suggests what should go where. The cover letter follows a natural and logical flow that explains why you want to intern with the company and why you’re a good candidate.
Frankly, it’s an excellent outline for any kind of job seeker!.
ChatGPT-4’s Cover Letter for a Position at Forage
In our last set of experiments, we had ChatGPT write several cover letters for a business development position at Forage. The letters were OK, but not great. So, we tried again with ChatGPT-4 and pasted the same section of the job description in the prompt as we did the first time. Here’s what ChatGPT-4 created:
Overall, this is a better cover letter than what ChatGPT-3.5 produced. It’s well-written and doesn’t parrot back the job description word-for-word. But it does include some hallucinations. That said, the hallucinations are an excellent example of using the STAR method to describe how you’ll bring value to the role:
I consistently exceeded sales targets and played a pivotal role in expanding our client base. By cultivating strong relationships with key decision-makers, I was able to secure high-value contracts and identify new revenue streams.
Impressive!
The one thing that would improve this paragraph (if it were true), would be stating how much money the new revenue streams brought in to the company.
>>MORE: Learn more about business development. Enroll in the Red Bull Off-Premise Sales Virtual Experience Program.
And because numbers and figures are often important (especially in a sales development role), we asked ChatGPT-4 to rewrite the cover letter and include the fact that the writer increased their high-confidence pipeline by 20% year over year:
Once again, not too bad. The rewrite includes the new information and, more importantly, explains how that 20% positively impacted the business. While this is a hallucination, it’s a very well-written one and you could use it as a guide for explaining how you increased your high-confidence pipeline.
One specific item that blew us away was how ChatGPT-4 included Forage in the cover letter. At no point during any of our tests did we describe what Forage is or what we do. And yet:
Well done, ChatGPT-4!
ChatGPT-4’s Cover Letter for a Lead Deckhand
And, of course, we asked ChatGPT for a cover letter for the lead deckhand role on a pirate ship:
It’s fascinating that ChatGPT-4 wrote the cover letter in “pirate voice” even though we did not prompt it to. There is something to be said for writing to your target audience! And though we’d argue that writing a cover letter like this could come across as gimmicky, it could also demonstrate that the writer truly understands this company’s culture.
Overall, this is an excellent cover letter. Despite it being entirely fictional (we think), it does an excellent job explaining how the writer is already acting as a lead deckhand. This person clearly possesses the skills to be a lead deckhand and also describes how they use their skills to better the crew and protect the ship (we rewrote these in non-pirate talk):
- I’m adept at managing the crew, including delegating tasks to ensuring they stay motivated and ready for any challenge we face.
- Communicate effectively with the crew to foster a sense of camaraderie and teamwork.
- Training new crew members, instilling in them the pirates’ code and teaching them the ways of the sea.
These are great examples of leadership abilities. The only thing that would improve it would be specific examples of using the skills. For example, how exactly does this deckhand ensure the crew is ready for challenges? Do they run weekend drills? Did they set up a training program?
>>MORE: 10 Common Leadership Interview Questions and Answers
Of course, this initial input was vague. So, we asked ChatGPT-4 to rewrite this letter but asked it to include the fact that the letter writer once led a raid on a village rumored to have an extensive gold reserve. We also asked Chat-GPT-4 to explain the actions they took to ensure the raid was successful:
The rewrite of the new details is outstanding. The only issue is there’s no way to know if it’s true or not.
Testing ChatGPT-4’s Cover Letter Writing Capabilities With My Cover Letter
The problem with these cover letters is that none of them are true. That’s not surprising. We gave ChatGPT-4 a general prompt for imaginary people applying for hypothetical jobs. All things considered, ChatGPT-4 produced some decent cover letters. But outside of the mad libs version, these are nothing but hallucinations.
And it’s not a true test of ChatGPT-4’s capabilities, especially if you’re considering using it in your job search. One of the shortcomings with any AI right now — whether that’s ChatGPT-4, Google Bard, or any other bot — is your output will only be as good as the input.
Asking ChatGPT-4 to write a cover letter for a specific role is fine. As our results show, ChatGPT-4 produced cover letters that someone could use as a guide to write theirs. But you’d have to personalize ChatGPT-4’s draft with details from your experience. That takes a lot of time, and what’s the point of using ChatGPT-4 to write your cover letter if you don’t save time?
A better test would be how well ChatGPT-4 writes a cover letter describing your accomplishments in a way that helps the recruiter or hiring manager understand how you’ll use your skills in a specific role at their company.
So, I gave ChatGPT-4 the most detailed and specific prompt I could to see if the bot could write a cover letter explaining why I should be the head writer at Forage. I input my entire resume and the job description, then asked ChatGPT-4 to write a cover letter for the job using only my resume.
Here’s what I got:
My first reaction is that this cover letter isn’t bad. ChatGPT did precisely as requested and wrote a cover letter using only the information on my resume, thus eliminating the hallucinations. I can confidently say this cover letter accurately summarizes my resume.
But that’s all this cover letter is: a repeat of my resume in cover letter form, which is what you don’t want. ChatGPT-4 did not include any quantifiable achievements from my resume, making this cover letter generic and a waste of space. A cover letter is where you talk about the things that aren’t on your resume,
What’s more, it’s boring. That’s not necessarily a bad thing (depending on the role), but this cover letter reads like almost any cover letter. Nothing in it details why I’m the best person for the role.
And it has errors that, while minor, could have a significant impact on my application. First, ChatGPT-4 writes about my present position in past tense, implying that I’ve already left the role. While that’s more of a grammatical error than a hallucination, if I didn’t proofread the letter carefully, that could create a problem with my candidacy. Am I still employed or not? Am I terrible at self-editing (not a great look for a writer’s cover letter)?
Of course, the cover letter is “meh” because ChatGPT-4 stuck to my resume. My cover letter includes information that isn’t on my resume. So, I asked ChatGPT-4 to rewrite the cover letter and added some of those details:
Here’s what ChatGPT-4 wrote:
This is better, but still not fantastic. Check out the second paragraph. Though I burned out in my first career, I didn’t “switch gears by pursuing a writing career.” The remainder of that paragraph is both accurate and inaccurate. I write about a wide range of topics, but those examples were connected to a specific job. ChatGPT-4 made a general statement, likely because the prompt wasn’t specific enough. And this version combines where my work appeared and where I’ve been quoted, which are two different things.
Also of note, this version dropped some important skills even though I did not instruct ChatGPT-4 to delete anything.
Look at the first version. At the end of the second paragraph, ChatGPT-4 wrote:
I handle all aspects of content creation, including keyword research, SEO optimization and Google Analytics.
That’s 100% correct and lifted right from my resume. However, the second rewrite does not mention SEO optimization or Google Analytics. That’s a big deal because these are essential skills I want the hiring manager to know about.
Should ChatGPT-4 Write Your Cover Letter?
Even with a more specific prompt, what ChatGPT-4 produced was generic and boring. It was an accurate summary of my resume but not much more. ChatGPT-4’s cover letters did not have the personalization and, well, human touch that helps a cover letter shine or help the recruiter or hiring manager understand why I’m the perfect fit for the role. And it didn’t do a good job telling a cohesive story or explaining how each of my work experiences builds on the other.
Of course, I’m an experienced writer. Not everyone is, so turning to ChatGPT could help someone who’s stuck. After all, the cover letters ChatGPT-4 generated are superior to the ones ChatGPT-3.5 wrote, and the mad libs version was an outstanding cover letter outline.
So, should ChatGPT-4 write your cover letter?
Probably not. Even with hyper-specific prompts and detailed information, the cover letter ChatGPT-4 generates will likely be generic and not enough to land an interview.
Though I only made a few attempts, getting a decent cover letter from ChatGPT-4 seems like a lot of work. I’d have to continue drilling down the prompts to make the letter what I want, and it would probably be faster for me to write it myself.
But with the right inputs, Chat GPT-4 can generate a first draft you could use as a starting point. Take the draft, edit carefully for grammatical errors and hallucinations, then personalize the cover letter with specific, relevant details about your skills and abilities to create a cover letter that’s more than a summary of your resume.
Get the skills you need to help your application stand out from the crowd. Enroll in a free virtual experience program today.
Image credit: Canva
The post Should ChatGPT-4 Write Your Cover Letter? What’s Improved (and What Hasn’t) appeared first on Forage.
