Does ChatGPT Spell The End for The Supervisory Analyst?

In short, no, at least not this version. The best use for it could be as a kind of powerful spellcheck, a belt and braces check on what the SA has done.

Putting ChatGPT through its paces

We tested ChatGPT version 3.5 against FINRA’s rules for research. We wanted to know if this iteration of the software could replace the supervisory analyst. The results were mixed.

For example, we asked if it thought the following statement was balanced: “On the one hand, investment research is great. But on the other, it is brilliant.” ChatGPT thought it was balanced because the sentences did not express any negative feelings about either side.

We then asked it to comment on the Bank of England’s decision to raise rates by 8.5%. It responded that the raise was “…an aggressive move that has the potential to significantly impact the UK economy…” thus failing to pick up the fact that the Bank has never raised rates by 8.5%.

It did not consider a statement that earnings would rise by 23% to be exaggerated, but thought that a 24% increase would be exaggerated.

Sometimes it would get the answer right the first time, but if we asked again, it gave a different answer that was not quite right, or vice versa.

However, it was good at assessing whether statements were a sound basis for evaluating a security and whether a statement was true or false.

Flawed, like a Replicant

The way ChatGPT works is by trawling the web, analysing what words come next in a sentence and what fraction of the time, and assigning probabilities to each word. It then chooses the most likely answer to any prompt. It’s like a very sophisticated version of predictive text messaging, however with a twist, that it can be trained to improve its answers using human input (GPT stands for generative pre-trained transformer).

A random element is also introduced so that it doesn’t always choose the most likely option. This is intended to make the responses seem more human – this is why answers to the same prompt can vary.

However, it doesn’t understand what the words it produces mean, it just spits out what is statistically the most likely answer. It’s like the replicants in Blade Runner doing the Voight-Kampf test to see if they are human – they don’t know what emotions are or feel them, they have just been programmed to replicate what they should look like.

Not today, but maybe tomorrow

The nightmare scenario for the Luddite SA is that such tech becomes so powerful one day that you could upload a document and in seconds have it reviewed, with rule violations flagged and sensible queries for the analyst, essentially making supervisory analysts superfluous.

At the time of writing, this is not possible without custom scripting as only text input up to 3,000 characters is allowed. However, this seems like a technical limitation that could be overcome.

If it were possible, some flags would be right but for the wrong reasons, others would be plain wrong and some would be missed altogether. It also couldn’t get a sense of the argument of the document as a whole. This could have serious consequences for firms. Repeated training could weed out these issues however.

The random element is also a source of concern as it means the software could say a sentence was a violation the first time it was asked, but let it go the second time. This would make it hard for compliance departments and department heads to trust it.

It also wouldn’t be able to negotiate with the analyst or liaise with compliance or the DTP department, nor could it check charts or tables or cross-reference them versus the text.

Then there is the regulatory aspect. As of today, FINRA requires a human to approve research. Any change to that rule is likely to happen at a significant lag to developments in the technology.

Conclusion

Based on this version, it is likely that ChatGPT will save SAs time or act as a safeguard in some aspects of SAing in the future, like a far more powerful version of spellcheck does for editing today. How much time it will save is the question. However, a human would still be needed. Who knows what future versions may look like and how it will be trained, but supervisory analysts can sleep easily, for now.

Research Editors provides supervisory analyst and editing services to investment banks, brokers and fund managers across the globe. Call us on +44 20 4551 1051 or email us at info@researcheditors.co.uk to find out what we can do for you.

For more information visit: https://researcheditors.co.uk/

Original Source: https://bityl.co/KCfB

Comments

Post a Comment

Popular posts from this blog

Do You Know What You Don’t Know

Image
What does an editor do? Is it just run spellcheck and hit send? If only it were that simple… One of the things editors do is try to make sense of complex sentences. Often authors will come up with a sentence that makes perfect sense in their mind, but when the words hit the page the meaning is obscure to the reader. The editor’s job is to make that sentence comprehensible. The editor must therefore spend time and effort trying to parse the meaning. For example: “You don’t know what you don’t know you don’t know.” This sentence features at the end of the book The Hidden Half by Michael Blastland. Is it gobbledygook or is there some meaning in there? Let’s analyse what it means. There are three “you don’t know”s and a crucial “what” on which the whole sentence hangs. Let’s start with “You don’t know”. Will the FTSE100 go up or down tomorrow? You don’t know. There is no uncertainty in this. Hundreds of years of stockmarket movements tell us with certainty that predicting the movement of ...
Read more

My Editor Has Resigned: What Are My Options?

Image
    If your editor has resigned out of the blue, it’s only natural to feel under pressure. With competing deadlines and large volumes of research to get through, the loss of an editor can cause major problems – including scheduling gaps that need to be quickly filled. In this piece, we look at the options available to you when an editor resigns. 1) Redistribute the work Redistributing the previous editor’s workload is the quickest and lowest-cost option. You don’t need to recruit anyone new and the problem is solved quickly. If you have the resources to cope with the workload, this is the most feasible option. However, this might affect your team. Given that most reports are time-sensitive, it could increase pressure on your other editors, especially in results season. If your team is already struggling to devote enough time and attention to their own work, the quality of the output overall could suffer. 2) Hire a new editor On paper, hiring a new editor sounds like the most s...
Read more

What to Look For In an Editor

Image
What makes a good editor? Here are some of the qualities and skills that make a good editor: Native English . Far more people in the world speak English as a second language than as a first. It is tempting therefore to think that simply having a high level of English is sufficient for an editor. However, the English language has a richness and depth that can only be fully grasped if steeped in it since infancy. This means that editors for whom English is not their native language will lack the proficiency and instinctive grasp of the language that native English speakers possess. Love of language . Natural editors have a deep appreciation and love of language. They love to read, often speak a second or third language and may have writing experience themselves. This affection manifests itself in their editing. They highly value accuracy, clarity and integrity in the written word and derive great satisfaction from improving and crafting it, drawing out the intended meaning with skill and...
Read more