My company blocked access to all AI chatbots through our firewall - except Copilot. We're encouraged to use Copilot to solve our coding and engineering problems.
A typical use case:
Me: I'm writing a powershell script to solve such-and-so need in our on-prem Azure Devops Server instance. What is the right API endpoint to get this information?
Co: Sure, just use {nonexistent powershell function} to go to your.base.url/_apis/non-existent-endpoint
Me: Wait, {nonexistent powershell function}? That's not documented anywhere, and powershell crashes when it hits that command.
Co: Oopsies! I meant {other nonexistent function}
Me: That didn't work either. I googled it and found {correct function}
Co: Goodness gracious me, that's right.
Me: I tried that endpoint and nonexistent-endpoint isn't real
Co: Gosh, I don't know what happened! I do apologize. Try {other nonexistent-endpoint} instead
Yesterday was classic Copilot
Me: I'm running this pipeline in our on-prem Visual Studio Server instance (no matter how many times I say that, as often as not it gives me answers that only apply to the cloud Services). There are two pipeline agents installed on server {blurp}. When the pipeline uses agent A, it works correctly. But when it uses agent B, it throws this error saying it can't find the right version of Visual Studio. As I mentioned, they are both on the same server. They also both use the same service account, so they both have the same environment settings, the same versions of VStudio installed, identical agent versions, identical .NET assemblies, etc. If I open the agent configuration screen in the Azure agent pool setup screen, both of them have identical capabilities. How is it that agent A runs it correctly but B fails?
Co: Compare the versions of Visual Studio and make sure it is correct for both of them.
Me: Are you unable to read? I just told you, they are both on the same machine with the same VS available.
Co: Righto, try looking at the agent's capabilities and make sure they are the same.
Me: What part of "capabilities are identical" do you not understand?
Co: Look in the agents' config folder.
Me: In C:\agents\A and C:\agents\B, there is no folder called config
Co: My mistake, I meant to look at config.json.
Me: There is no file called config.json.
Co: Compare the versions of Visual Studio and make sure it is correct for both of them.
I lose more time trying to get a straight answer out of Copilot than if I had just googled instead. And considering my low opinion of the quality of google search results, that's a pretty low bar.
So, my fellow coders, beware: don't fall into the trap of letting Copilot do your thinking for you. That way lies madness (and really crappy unusable code)
Postscript for those wondering how I resolved the problem: I had to uninstall the "bad" agent and reinstall it to get the pipeline working correctly.
Post-postscript: Yes, the above dialogue is paraphrased. Copilot does apologise profusely every time I call out its mistakes, but it doesn't use so much colloquialism. However, my angry sarcastic responses are verbatim.