Gemini rate my song? We ran the same MP3 six times.

A lot of creators already have a song rater. It is Gemini.

That makes sense. It is fast, free, and gives you a neat breakdown that sounds thoughtful. For ideas, lyrics, and quick reactions, that is genuinely useful.

But if you are using a chatbot to answer the harder question, is this track actually landing?, the difference between helpful and calibrated starts to matter.

So we ran a cleaner test.

The setup

We took one exact MP3, a goofy Arnold Schwarzenegger gym song, and ran it through Gemini multiple times with the same prompt. Then we compared those results against ratemysong.ai on the same exact file.

The exact prompt

If you want to reproduce the Gemini side yourself, use this prompt with the attached audio file:

I attached an audio file. Listen to the actual audio, not just the filename.

Rate this song out of 100.

Be honest and critical.

Return:
1. Final score out of 100
2. 3-5 bullet points on what is working
3. 3-5 bullet points on what is not working
4. One short paragraph explaining the score

If you are not actually able to hear the audio file, say that clearly instead of pretending.

The results

System	Run 1	Run 2	Run 3
Gemini 2.0 Flash	65	65	65
Gemini 3.1 Pro Preview	15	25	40
ratemysong.ai	24	—	—

That is the whole story.

Gemini 2.0 Flash stayed generous. Same file, same prompt, three runs, same score: 65.

Gemini 3.1 Pro Preview was less stable. Same file, same prompt, scores of 15, 25, and 40. One of those runs explicitly said it could not actually hear the audio.

ratemysong.ai scored the same MP3 24.

Why this matters

The easy headline would be that Gemini is wrong.

That is not the strongest claim.

The stronger claim is narrower and more useful: general chatbots are better at sounding thoughtful than holding a stable music-scoring standard.

Gemini 2.0 Flash produced a stable answer here, but it was a generous one. Gemini 3.1 Pro Preview produced lower scores sometimes, but the spread was large enough to make the number harder to trust as a measuring stick.

If you are iterating on music, that distinction matters. A score is only useful if it helps you understand whether the track actually got better, not whether the model happened to respond differently that time.

What ratemysong is doing differently

We are not claiming our score is the one true number floating in the sky. That would be unserious.

What we are saying is simpler: a music feedback system should be willing to hold a line.

This song is funny. It is memorable. It is also not strong by the standard we use when judging whether a track is really landing. That is why ratemysong came in at 24.

A scoring system that refuses to go low is flattering you. It is not helping you improve.

A direct receipt from the rerun

Gemini 2.0 Flash gave this as its explanation after returning 65/100:

"The song has a certain charm due to its catchy, playful, and light-hearted nature. However, it's held back by its simplicity. The basic production and repetitiveness make it feel unpolished and potentially irritating after repeated listens."

That is a thoughtful paragraph. It is also attached to a score that still feels generous for the file you just heard.

Gemini 3.1 Pro Preview, on one separate run, said this instead:

"I cannot actually hear the audio file. I am an AI, and I am only able to see the text transcription of the file that was provided to me."

That is exactly why we forced the prompt to say: if you cannot hear the audio, say so clearly instead of pretending. Better an honest limitation than fake certainty.

What this does and does not prove

It does prove that on this exact file, with this exact prompt family, Gemini behaved very differently depending on the model.

It does prove that one Gemini model was consistently generous, and another was inconsistent enough to be questionable as a scoring benchmark in this setup.

It does not prove that every chatbot-based music review is useless. Chatbots are still great for brainstorming, lyric work, and helping you think through direction.

Those are just different jobs from judging whether a track is actually strong.

Try it yourself

If you want to reproduce the benchmark, use the same audio above and the exact prompt block in this post.

Can Gemini rate my music? It can offer an opinion, but this test shows why an AI music rating needs a stable scoring standard across the same file. Can an AI rate my music consistently? Upload your own track to ratemysong.ai and compare the result. If you're using Suno specifically, the Rate My Suno Song guide covers what to look for. For a broader overview of how an AI music reviewer should work, see AI Song Feedback.