Category: Big Question

Talking about ChatGPT

There has been a lot of discussion in the FMS TEL office about ChatGPT over the past couple of weeks. Michael and Eleanor have spent some time putting this post together to help our readers to learn a bit more about ChatGPT, and to catch up on the discourse around it in HE in general.

We used ChatGPT to generate an about page for the blog last week, which you may have seen. It provides a good example of an introduction for a website. It feels somewhat generic, it has gotten a few things wrong, and expresses itself a little strangely in places. At the same time, it’s a decent starting point to work from.

What is ChatGPT?

ChatGPT is one of many advanced Artificial Intelligence models which has been trained to understand inputs (e.g. questions) in natural language (hence ‘chat’) and answer them. Interacting with ChatGPT feels similar to interacting with a chatbot or virtual assistant on the surface, but the technologies are completely different.

How does Chat GPT work?

The data that ChatGPT draws from is an offline dataset which was created in 2021. The exact content of the dataset is not clear, however ChatGPT is able to formulate responses to a massive range of topics, so it is safe to assume the dataset is enormous, and most likely taken from public internet sites (i.e. not password-protected sites). Part of the training process involved feeding computer code into the model, which helped it learn logic and syntax – something that is present in natural languages in the form of grammar. Feedback was also given to the model to help it improve – this is called ‘supervised learning’.

Q: How were you trained? A: I was trained by OpenAI using a deep learning technique called Transformer-based neural network architecture on a massive amount of text data sourced from the internet. The training process involved inputting large amounts of text data into the model and updating the model's parameters to minimize the difference between the model's predicted output and the actual text in the data. This process was repeated multiple times until the model learned to generate text that is similar to the input data. The end result is a model that can generate human-like text based on the input it receives.
Some of the language is somewhat technical, but the summary is quite clear!

While ChatGPT can produce extended responses drawing on its huge dataset, it doesn’t understand what it is producing – similar to a parrot repeating after you. It can mimic the sounds but has no true understanding of the meaning.

What are people saying about ChatGPT and University Education?

Assessment Security

With its ability to generate text that looks like a human wrote it, it is natural to be worried that students may use ChatGPT for assessed work. Many automated text editors and translators are already in this market, though they work in a different way. Tools like Word’s spellchecker and Grammarly can both be employed to boost writing skills – though these do not generate text. ChatGPT is different in this respect, and it is free, quick, and easy for anyone to use.

“…The big change is that this technology is wrapped up in a very nice interface where you can interact with it, almost like speaking to another human. So it makes it available to a lot of people.”

Dr Thomas Lancaster, Imperial College London in The Guardian

Assessment security has always been a concern, and as with any new technology, there will be ways we can adapt to its introduction. Some people are already writing apps to detect AI text, and OpenAI themselves are exploring watermarking their AI’s creations. Google Translate has long been a challenge for language teachers with its ability to generate translations, but a practiced eye can spot the deviations from a student’s usual style, or expected skill level.

Within HE, clear principles are already in place around plagiarism, essay mills and other academic misconduct, and institutions are updating their policies all the time. One area in which ChatGPT does not perform well is in the inclusion of references and citations – a cornerstone of academic integrity for many years.

Authentic assessment may be another element of the solution in some disciplines, and many institutions have been doing work in this area for some time, our own included. On the other hand for some disciplines, the ability to write structured text is a key learning outcome and is inherently an authentic way to assess.

Consider ChatGPT’s ability to summarise and change the style of the language it uses.

  • Could ChatGPT be used to generate lay summaries of research for participants in clinical trials?
  • Would it do as good a job as a clinician?
  • How much time could be saved by generating these automatically and then simply tweaking the text to comply with good practice?

The good practice would still need to be taught and assessed, but perhaps this is a process that will become standard practice in the workplace.

Digital skills, critical thinking and accessibility

Prompting AI is certainly a skill in itself – just as knowing what to ask Google to get your required answer. ChatGPT reacts well to certain prompts such as ‘let’s go step by step’ which tells it you’d like a list of steps. You can use that prompt to get a broad view of the task ahead. A clear example of this would be to get a structure for a business plan, or outline what to learn in a given subject. As a tool, ChatGPT can be helpful in collating information and combining it into an easily readable format. This means that time spent searching can instead be spent reading. At the same time, it is important to be conscious of the fact that ChatGPT does not understand what it is writing and does not check its sources for bias, or factual correctness.

ChatGPT can offer help to students with disabilities, or neurodivergent students who may find traditional learning settings more challenging. It can also parse spelling errors and non-standard English to produce its response, and tailor its response to a reading level if prompted correctly.

Conclusions

ChatGPT in its current free-to-use form prompts us to change how we think about many elements of HE. While naturally it creates concerns around assessment security, we have always been able to meet these challenges in the past by applying technical solutions, monitoring grades, and teaching academic integrity. Discussions are already ongoing in every institution on how to continue this work, with authentic assessment coming to the fore as a way of breaking our heavy reliance on the traditional essay.

As a source of student assistance, ChatGPT offers a wealth of tools to help students gather information and access it more easily. It is also a challenge for students’ critical thinking skills, just like the advent the internet or Wikipedia. It is well worth taking the time to familiarise oneself with the technology, and to explore how it may be applied in education, and in students’ future workplaces.

Resources

  • Try ChatGPT – you will need to make an account with OpenAI and possibly wait quite a while as the service is very busy.
  • Try DALL-E – this AI generates images based on your inputs.

Sources and Further Reading

Captioning and Transcribing – What Standard Should I Aim for?

When captioning and transcribing, what is meant by ‘accuracy’? When are captions good enough?

In FMS TEL and LTDS many team members regularly work with captioning videos, in particular for our own instructional videos or webinars. Recently a few of us have been talking about how we caption videos and how we decide what to correct. After discovering we all had differences of opinion about what to keep and what to edit, it seemed like a good idea to think through the issues.

This webinar from the University of Kent features Nigel Megitt from the BBC talking about priorities when captioning and audio describing TV programme. It includes research on how people with different levels of hearing feel about captions.

Note: These discussions refer to materials created for staff training and other internal uses. For student materials, please see the university policies on captioning materials for students and the captioning disclaimer to help with your decision-making.

Different Types of Captioning and Transcription

Commercial captioning companies offer a range of levels of detail. We do not outsource these tasks, but the predefined service levels can help clarify what decisions are made when captioning. Is verbatim captioning better than a lightly edited video? An accurate set of captions or transcript should include hesitations and false starts, but a more readable one might remove these for fast comprehensibility and more closely resemble the script of a speech.

Key Considerations

  • Destination – who is the audience? What do they need?
  • Speaker(s) – how can they be best represented? How do they feel about you editing their speech for clarity (e.g. removing filler words) vs correcting captions to verbatim?
  • Timescale – how fast do you need to turn this around? Longer videos and heavier editing takes longer.
  • Longevity – will this resource be around for a long time and reach a wider audience? If so it may merit extra polish.

Once you have broadly decided on the above, you can deal with the nitty-gritty of deciding what to fix, edit or remove. Deciding on your approach to these common issues means you won’t have to make a decision each time you find an error in your transcript. If working with a few other colleagues on a larger project you might want to agree with each other what standard you are aiming for to create uniformity.

Editing Decisions

The ASR occasionally misunderstands speech and adds incorrect captions that may be distracting, embarrassing or inappropriate, for example adding swearing or discriminatory language that the speaker has not in fact used. Checking the captions for these is a great start, and is likely to be appreciated by all speakers!

We don’t usually speak in the same way we write. Normal speech is full of little quirks that don’t appear in text. Some of these include…

  • False starts (If we take… no actually let’s start with… yes, OK, if we take question 4 next…)
  • Hesitations (um….ah…)
  • Filler Words (you know, like, so…)
  • Repeated words (You can do this by… by reading the text)

Other Considerations for Captioning

Remember that captions will be read on screen at the pace of the video. This means that anything that you can do to increase readability may be useful for the viewer. This includes simple things like…

  • Fixing initialisms and acronyms (PGR not p g r, SAgE not sage)
  • Fixing web and email addresses (abc1@ncl.ac.uk, not A B C One At Newcastle Dot A See Dot UK)
  • Adding quotation marks around quotes.

You may also consider…

  • Presenting numbers using figures rather than words (99% not ninety-nine percent)
  • Removing awkward breaks (When Panopto separates a final word from its sentence.)
  • Fixing inaccurate punctuation like full stops in the wrong places, or commas and apostrophes (this is quite time consuming).

Considerations for Transcription

As well as the editing and tidying jobs above, before beginning to work with your file, consider whether or not the timing points are going to be important, and how you are going to denote different speakers, or break up the text. For example, for an interview you may need to denote various speakers very clearly. By contrast, for a training webinar, even if there are two presenters it might not be crucial to distinguish them. Instead it might be better to add headings for each slide so that the two resources can be used side by side.

Once you have decided on what to edit and what to ignore, your process will move along much faster as you won’t need to decide on the fly.

Keep an eye on the blog over the next few weeks for tips on how to quickly manage and edit your caption and transcription files.

Ethical Framework for Digital Teaching and Learning

In December 2020 I had the opportunity to attend the Association for Learning Technologies’ Winter Conference. One of the presentations at the conference really struck a chord with me and I would like to share a synopsis of what was discussed.

Presenters Sharon Flynn, Natalie Lafferty, John Traxler, Bella Abrams, and Lyshi Rodrigo sat on a panel discussing an Ethical Framework for learning technology. They discussed what they perceived as the biggest issues around ethical teaching and learning digitally.

One of the primary concerns driving the development of an Ethical Framework is the inevitable power relationship learning technologies create between teachers and their students. For example, how can monitoring work in the right way, where it is not there as a policing tool, but rather as a tool for aiding engagement and learning. One of the panellists suggested a simplified form of terms and conditions could go a long way to pacifying student concerns over any form of monitoring.

There are inherent principles about trust and reliability in the digital world. This is evident in many sectors but likely not more than in the surveillance culture of the digital world. We have, therefore, the responsibility to help protect students, and colleagues, as we become more aware of ethical challenges in the digital world.

Another concern relates to fair access. What ethical role does the institution have in ensuring all students have access to the digital tools, such as laptops and broadband internet? What is considered adequate and equitable? How logistically can this be accomplished? And, this is not simply a problem for students. Some teachers will also experience digital tools poverty. This would also include training for students and teachers in the systems, programs, and tools they would be expected to use. (Something that Newcastle University is working hard to ensure exists to support students and teachers in the unique set of circumstances following of from Covid-19.)

Another question brought up was what constitutes harm? This question would be at the heart of an Ethical Framework. How do we as institutions identify harm caused by digital teaching and learning and mitigate it? For example, how does proctoring and the use of e-resources impact students. What about productivity measures? These could potentially be arbitrary and misrepresent what really matters. Some people think these are easy solutions for the current challenges, but they invite the need for an Ethical Framework.

The implications of GDPR and its potential successor also impact the need for an Ethical Framework. Professional bodies are not necessarily thinking of the problems related to approaches like proctoring. So, any Ethical Framework must be rooted in context of principles and be ever aware of the needs and where importance lies withing various other cultures.

This all leads to the need to develop an Ethical Framework for teaching and learning digitally. The panellist suggested that we start from a position of respect and use our values to build an Ethical Framework including student voice.

This summary of the impetus and content of what may be needed in an Ethical Framework for teaching and learning online is certainly worth considering as we enter into the new normal that will likely contain more online teaching than we had pre-Covid. I would be interested to hear (reply below) what you think about what the ALT panellists had to say and what your views on such an Ethical Framework should and could be.