Ta prenosni računalnik je zasnovan za uporabo vnaprej pripravljenega modela transformatorjev in njegovo natančno nastavitev pri nalogi klasifikacije. Poudarek te vadnice bo na sami kodi in na tem, kako jo prilagoditi svojim potrebam.
Ta prenosni računalnik uporablja Avtorazredi iz transformator by Objemni obraz funkcionalnost. Ta funkcionalnost lahko ugane konfiguracijo modela, tokenizer in arhitekturo samo s podajanjem imena modela. To omogoča ponovno uporabo kode na velikem številu modelov transformatorjev!
Zagotovil sem dovolj navodil in komentarjev, da jih lahko spremljam skupaj z minimalnim znanjem kodiranja v Pythonu.
Ker uporabljam PyTorch za natančno nastavitev naših modelov transformatorjev, je vsako znanje o PyTorchu zelo koristno. Vedeti nekaj o transformatorji pomaga tudi knjižnica.
Kako uporabljati ta zvezek?
Ta zvezek sem izdelal z mislijo na možnost ponovne uporabe. Način nalaganja nabora podatkov v PyTorch Podatkovni niz razred je precej standarden in ga je mogoče zlahka ponovno uporabiti za kateri koli drug niz podatkov.
Edine spremembe, potrebne za uporabo lastnega nabora podatkov, bodo branje v naboru podatkov znotraj MovieReviewsDataset razred, ki uporablja PyTorch Podatkovni niz. DataLoader bo vrnil slovar serija vhodni format, tako da ga je mogoče vnesti neposredno v model z uporabo izjave: outputs = model(**batch). Dokler ta izjava velja, bo ostala koda delovala!
Kateri modeli transformatorjev delujejo s tem prenosnikom?
Obstajajo redki primeri, ko uporabljam drugačen model kot Bert, ko se ukvarjam s klasifikacijo iz besedilnih podatkov. Ko je treba zagnati drugačno arhitekturo modela transformatorja, kateri bi deloval s to kodo?
Ker je ime zvezkov finetune_transformatorji mora delovati z več kot eno vrsto transformatorjev.
Preletel sem ta zvezek vsi vnaprej pripravljeni modeli najdeno na Hugging Face Transformer. Tako boste vnaprej vedeli, ali model, ki ga nameravate uporabiti, deluje s to kodo brez kakršnih koli sprememb.
Seznam vnaprej pripravljenih modelov transformatorjev, ki delujejo s tem prenosnikom, je na voljo tukaj. Obstajajo 73 modelov, ki so delovali? in 33 modelov, ki niso delovali? s tem zvezkom.
Podatkovni niz
Ta zvezek bo pokrival natančno nastavitev transformatorjev za nalogo binarne klasifikacije. Uporabil bom kritike znanih filmov pozitivno — negativno Velik nabor podatkov za pregled filma.
Opis na spletnem mestu Stanford:
To je nabor podatkov za binarno klasifikacijo razpoloženja, ki vsebuje bistveno več podatkov kot prejšnji nabori primerjalnih podatkov. Ponujamo nabor 25,000 zelo polarnih filmskih ocen za usposabljanje in 25,000 za testiranje. Obstajajo tudi dodatni neoznačeni podatki za uporabo. Na voljo sta neobdelano besedilo in že obdelana oblika zapisa besed. Za več podrobnosti si oglejte datoteko README v izdaji.
Zakaj ta niz podatkov? Menim, da je nabor podatkov za klasifikacijo enostaven za razumevanje in uporabo. Mislim, da je s podatki o razpoloženju vedno zabavno delati.
Kodiranje
Zdaj pa naredimo nekaj kodiranja! Pregledali bomo vsako kodirno celico v zvezku in opisali, kaj počne, kakšna je koda in kdaj je relevantna – pokažite rezultat.
To obliko sem naredil tako, da vam je enostavno slediti, če se odločite zagnati vsako kodno celico v svojem prenosnem računalniku python.
Ko se učim iz vadnice, vedno poskušam ponoviti rezultate. Verjamem, da je enostavno slediti, če imate kodo poleg razlag.
Moji Prenosi
Prenos Velik nabor podatkov za pregled filma in ga lokalno razpakirajte.
# Download the dataset.
!wget -q -nc http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
# Unzip the dataset.
!tar -zxf /content/aclImdb_v1.tar.gz
Namesti
transformatorji knjižnico je treba namestiti, če želite uporabiti vso čudovito kodo Hugging Face. Za pridobitev najnovejše različice jo bom namestil neposredno iz GitHuba.
ml_stvari knjižnica, ki se uporablja za različne naloge, povezane s strojnim učenjem. To knjižnico sem ustvaril, da zmanjšam količino kode, ki jo moram napisati za vsak projekt strojnega učenja. Poskusi!
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done |████████████████████████████████| 2.9MB 6.7MB/s |████████████████████████████████| 890kB 48.9MB/s |████████████████████████████████| 1.1MB 49.0MB/s Building wheel for transformers (PEP 517) ... done
Building wheel for sacremoses (setup.py) ... done |████████████████████████████████| 71kB 5.2MB/s Building wheel for ml-things (setup.py) ... done
Building wheel for ftfy (setup.py) ... done
Uvoz
Uvozite vse potrebne knjižnice za ta zvezek.
Navedite parametre, uporabljene za ta zvezek:
set_seed(123) – Zaradi ponovljivosti je vedno dobro nastaviti fiksno seme.
epochs – Število vadbenih obdobij (avtorji priporočajo med 2 in 4).
batch_size – Število paketov – odvisno od največje dolžine zaporedja in pomnilnika GPE. Za dolžino zaporedja 512 serija 10 OBIČAJNO deluje brez težav s spominom cuda. Za majhno dolžino zaporedja lahko poskusite serijo 32 ali več.
max_length – Obložite ali skrajšajte besedilna zaporedja na določeno dolžino. Nastavil ga bom na 60 žetonov, da pospešim trening.
device – Poiščite GPU za uporabo. bom uporabil cpu privzeto, če št gpu najdemo.
model_name_or_path – Ime modela transformatorjev – bo uporabil že vnaprej usposobljen model. Pot modela transformatorja – naloži vaš model z lokalnega diska. Vedno rad začnem z bert-base-cased: 12-slojni, 768-skriti, 12-glav, 109M parametrov. Usposobljen za angleško besedilo s malimi in malimi črkami.
labels_ids – Slovar oznak in njihovih id – to bo uporabljeno za pretvorbo oznak nizov v številke.
n_labels – Koliko oznak uporabljamo v tem naboru podatkov. To se uporablja za določitev velikosti glave klasifikacije.
import io
import os
import torch
from tqdm.notebook import tqdm
from torch.utils.data import Dataset, DataLoader
from ml_things import plot_dict, plot_confusion_matrix, fix_text
from sklearn.metrics import classification_report, accuracy_score
from transformers import (AutoConfig, AutoModelForSequenceClassification, AutoTokenizer, AdamW, get_linear_schedule_with_warmup, set_seed, ) # Set seed for reproducibility,
set_seed(123) # Number of training epochs (authors recommend between 2 and 4)
epochs = 4 # Number of batches - depending on the max sequence length and GPU memory.
# For 512 sequence length batch of 10 works without cuda memory issues.
# For small sequence length can try batch of 32 or higher.
batches = 32 # Pad or truncate text sequences to a specific length
# if `None` it will use maximum sequence of word piece tokens allowed by model.
max_length = 60 # Look for gpu to use. Will use `cpu` by default if no gpu found.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # Name of transformers model - will use already pretrained model.
# Path of transformer model - will load your own model from local disk.
model_name_or_path = 'bert-base-cased' # Dicitonary of labels and their id - this will be used to convert.
# String labels to number ids.
labels_ids = {'neg': 0, 'pos': 1} # How many labels are we using in training.
# This is used to decide size of classification head.
n_labels = len(labels_ids)
Pomožne funkcije
Vse razrede in funkcije, ki jih bom uporabljal v tem zvezku, želim ohraniti v tem razdelku, da pomagam ohranjati čist videz zvezka:
MovieReviewsDataset
Če ste prej delali s PyTorch, je to precej standardno. Ta razred potrebujemo za branje v našem naboru podatkov, ga razčlenjujemo, uporabljamo tokenizer, ki pretvori besedilo v številke in ga spravimo v lepo obliko, da se vnese v model.
Srečno za uporabo, Objemni obraz pomislil na vse in naredil tokenizer opravlja vse težke naloge (razdeli besedilo na žetone, oblazini, obrezuje, kodira besedilo v številke) in je zelo enostaven za uporabo!
V tem razredu moram samo prebrati vsebino vsake datoteke, use popravi_besedilo za odpravo morebitnih težav z Unicode in spremljanje pozitivnih in negativnih občutkov.
Vsa besedila in oznake bom dodal na sezname, ki jih bom pozneje posredoval tokenizerju in ID-jem oznak, da bom vse pretvoril v številke.
Obstajajo trije glavni deli tega PyTorcha Podatkovni niz razred:
v() kjer beremo v naboru podatkov ter pretvorimo besedilo in oznake v številke.
__len__() kjer moramo vrniti število primerov, ki smo jih prebrali. To se uporablja pri klicu len(MovieReviewsDataset()) .
__getitem__() kot vhod vedno vzame int valuet hat, ki predstavlja, kateri primer iz naših primerov vrne iz našega nabora podatkov. Če je podana vrednost 3, bomo vrnili primer iz našega nabora podatkov na položaj 3. Vrniti mora objekt s formatom, ki ga je mogoče vnesti v naš model. Na srečo naš tokenizer to naredi namesto nas in vrne slovar spremenljivk, pripravljenih za vnos v model na ta način:model(**inputs).
class MovieReviewsDataset(Dataset): r"""PyTorch Dataset class for loading data. This is where the data parsing happens and where the text gets encoded using loaded tokenizer. This class is built with reusability in mind: it can be used as is as long as the `dataloader` outputs a batch in dictionary format that can be passed straight into the model - `model(**batch)`. Arguments: path (:obj:`str`): Path to the data partition. use_tokenizer (:obj:`transformers.tokenization_?`): Transformer type tokenizer used to process raw text into numbers. labels_ids (:obj:`dict`): Dictionary to encode any labels names into numbers. Keys map to labels names and Values map to number associated to those labels. max_sequence_len (:obj:`int`, `optional`) Value to indicate the maximum desired sequence to truncate or pad text sequences. If no value is passed it will used maximum sequence size supported by the tokenizer and model. """ def __init__(self, path, use_tokenizer, labels_ids, max_sequence_len=None): # Check if path exists. if not os.path.isdir(path): # Raise error if path is invalid. raise ValueError('Invalid `path` variable! Needs to be a directory') # Check max sequence length. max_sequence_len = use_tokenizer.max_len if max_sequence_len is None else max_sequence_len texts = [] labels = [] print('Reading partitions...') # Since the labels are defined by folders with data we loop # through each label. for label, label_id, in tqdm(labels_ids.items()): sentiment_path = os.path.join(path, label) # Get all files from path. files_names = os.listdir(sentiment_path)#[:10] # Sample for debugging. print('Reading %s files...' % label) # Go through each file and read its content. for file_name in tqdm(files_names): file_path = os.path.join(sentiment_path, file_name) # Read content. content = io.open(file_path, mode='r', encoding='utf-8').read() # Fix any unicode issues. content = fix_text(content) # Save content. texts.append(content) # Save encode labels. labels.append(label_id) # Number of exmaples. self.n_examples = len(labels) # Use tokenizer on texts. This can take a while. print('Using tokenizer on all texts. This can take a while...') self.inputs = use_tokenizer(texts, add_special_tokens=True, truncation=True, padding=True, return_tensors='pt', max_length=max_sequence_len) # Get maximum sequence length. self.sequence_len = self.inputs['input_ids'].shape[-1] print('Texts padded or truncated to %d length!' % self.sequence_len) # Add labels. self.inputs.update({'labels':torch.tensor(labels)}) print('Finished!n') return def __len__(self): r"""When used `len` return the number of examples. """ return self.n_examples def __getitem__(self, item): r"""Given an index return an example from the position. Arguments: item (:obj:`int`): Index position to pick an example to return. Returns: :obj:`Dict[str, object]`: Dictionary of inputs that feed into the model. It holddes the statement `model(**Returned Dictionary)`. """ return {key: self.inputs[key][item] for key in self.inputs.keys()}
Ustvaril sem to funkcijo za izvedbo popolnega prehoda skozi DataLoader predmet ( DataLoader predmet je ustvarjen iz našega Podatkovni niz vnesite predmet z uporabo MovieReviewsDataset razred). To je v bistvu ena epoha skozi celoten nabor podatkov.
O nalagalnik podatkov je ustvarjen iz PyTorcha DataLoader ki vzame predmet, ustvarjen iz MovieReviewsDataset razreda in vsak primer razdeli v skupine. Na ta način lahko našemu modelu dodamo pakete podatkov!
O optimizator_ in razporejevalnik_ so v PyTorchu zelo pogosti. Od njih se zahteva posodobitev parametrov našega modela in posodobitev naše stopnje učenja med usposabljanjem. Obstaja veliko več kot to, vendar se ne bom spuščal v podrobnosti. To je pravzaprav lahko ogromna zajčja luknja, saj se za temi funkcijami zgodi VELIKO, kar nam ni treba skrbeti. Hvala PyTorch!
V procesu spremljamo dejanske oznake in predvidene oznake skupaj z izgubo.
def train(dataloader, optimizer_, scheduler_, device_): r""" Train pytorch model on a single pass through the data loader. It will use the global variable `model` which is the transformer model loaded on `_device` that we want to train on. This function is built with reusability in mind: it can be used as is as long as the `dataloader` outputs a batch in dictionary format that can be passed straight into the model - `model(**batch)`. Arguments: dataloader (:obj:`torch.utils.data.dataloader.DataLoader`): Parsed data into batches of tensors. optimizer_ (:obj:`transformers.optimization.AdamW`): Optimizer used for training. scheduler_ (:obj:`torch.optim.lr_scheduler.LambdaLR`): PyTorch scheduler. device_ (:obj:`torch.device`): Device used to load tensors before feeding to model. Returns: :obj:`List[List[int], List[int], float]`: List of [True Labels, Predicted Labels, Train Average Loss]. """ # Use global variable for model. global model # Tracking variables. predictions_labels = [] true_labels = [] # Total loss for this epoch. total_loss = 0 # Put the model into training mode. model.train() # For each batch of training data... for batch in tqdm(dataloader, total=len(dataloader)): # Add original labels - use later for evaluation. true_labels += batch['labels'].numpy().flatten().tolist() # move batch to device batch = {k:v.type(torch.long).to(device_) for k,v in batch.items()} # Always clear any previously calculated gradients before performing a # backward pass. model.zero_grad() # Perform a forward pass (evaluate the model on this training batch). # This will return the loss (rather than the model output) because we # have provided the `labels`. # The documentation for this a bert model function is here: # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification outputs = model(**batch) # The call to `model` always returns a tuple, so we need to pull the # loss value out of the tuple along with the logits. We will use logits # later to calculate training accuracy. loss, logits = outputs[:2] # Accumulate the training loss over all of the batches so that we can # calculate the average loss at the end. `loss` is a Tensor containing a # single value; the `.item()` function just returns the Python value # from the tensor. total_loss += loss.item() # Perform a backward pass to calculate the gradients. loss.backward() # Clip the norm of the gradients to 1.0. # This is to help prevent the "exploding gradients" problem. torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # Update parameters and take a step using the computed gradient. # The optimizer dictates the "update rule"--how the parameters are # modified based on their gradients, the learning rate, etc. optimizer.step() # Update the learning rate. scheduler.step() # Move logits and labels to CPU logits = logits.detach().cpu().numpy() # Convert these logits to list of predicted labels values. predictions_labels += logits.argmax(axis=-1).flatten().tolist() # Calculate the average loss over the training data. avg_epoch_loss = total_loss / len(dataloader) # Return all true labels and prediction for future evaluations. return true_labels, predictions_labels, avg_epoch_loss
validacija (nalagalnik podatkov, naprava_)
To funkcijo sem implementiral na zelo podoben način kot vlak vendar brez posodobitve parametrov, prehoda nazaj in spodobnega dela gradienta. Ni nam treba opraviti vseh teh ZELO računsko intenzivnih nalog, ker nas zanimajo le napovedi našega modela.
Jaz raba DataLoader na podoben način kot pri usposabljanju za pridobivanje serij za dovajanje našega modela.
Pri tem spremljam dejanske oznake in predvidene oznake skupaj z izgubo.
def validation(dataloader, device_): r"""Validation function to evaluate model performance on a separate set of data. This function will return the true and predicted labels so we can use later to evaluate the model's performance. This function is built with reusability in mind: it can be used as is as long as the `dataloader` outputs a batch in dictionary format that can be passed straight into the model - `model(**batch)`. Arguments: dataloader (:obj:`torch.utils.data.dataloader.DataLoader`): Parsed data into batches of tensors. device_ (:obj:`torch.device`): Device used to load tensors before feeding to model. Returns: :obj:`List[List[int], List[int], float]`: List of [True Labels, Predicted Labels, Train Average Loss] """ # Use global variable for model. global model # Tracking variables predictions_labels = [] true_labels = [] #total loss for this epoch. total_loss = 0 # Put the model in evaluation mode--the dropout layers behave differently # during evaluation. model.eval() # Evaluate data for one epoch for batch in tqdm(dataloader, total=len(dataloader)): # add original labels true_labels += batch['labels'].numpy().flatten().tolist() # move batch to device batch = {k:v.type(torch.long).to(device_) for k,v in batch.items()} # Telling the model not to compute or store gradients, saving memory and # speeding up validation with torch.no_grad(): # Forward pass, calculate logit predictions. # This will return the logits rather than the loss because we have # not provided labels. # token_type_ids is the same as the "segment ids", which # differentiates sentence 1 and 2 in 2-sentence tasks. # The documentation for this `model` function is here: # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification outputs = model(**batch) # The call to `model` always returns a tuple, so we need to pull the # loss value out of the tuple along with the logits. We will use logits # later to to calculate training accuracy. loss, logits = outputs[:2] # Move logits and labels to CPU logits = logits.detach().cpu().numpy() # Accumulate the training loss over all of the batches so that we can # calculate the average loss at the end. `loss` is a Tensor containing a # single value; the `.item()` function just returns the Python value # from the tensor. total_loss += loss.item() # get predicitons to list predict_content = logits.argmax(axis=-1).flatten().tolist() # update list predictions_labels += predict_content # Calculate the average loss over the training data. avg_epoch_loss = total_loss / len(dataloader) # Return all true labels and prediciton for future evaluations. return true_labels, predictions_labels, avg_epoch_loss
Naloži model in tokenizator
Nalaganje treh bistvenih delov vnaprej pripravljenih transformatorjev: konfiguracija, tokenizer in Model. Prav tako moram naložiti model v napravo, ki jo nameravam uporabiti (GPE / CPE).
Ker uporabljam AutoClass funkcionalnost od Objemni obraz Skrbeti moram le za ime modela kot vnos, ostalo pa ureja knjižnica transformatorjev.
# Get model configuration.
print('Loading configuraiton...')
model_config = AutoConfig.from_pretrained(pretrained_model_name_or_path=model_name_or_path, num_labels=n_labels) # Get model's tokenizer.
print('Loading tokenizer...')
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model_name_or_path) # Get the actual model.
print('Loading model...')
model = AutoModelForSequenceClassification.from_pretrained(pretrained_model_name_or_path=model_name_or_path, config=model_config) # Load model to defined device.
model.to(device)
print('Model loaded to `%s`'%device)
Loading configuraiton...
Loading tokenizer...
Loading model...
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Model loaded to `cuda`
Dataset in DataLoader
Tukaj ustvarim PyTorch Podatkovni niz in DataLoader objekte, ki bodo uporabljeni za dovajanje podatkov v naš model.
Tukaj uporabljam MovieReviewsDataset razred in ustvarite spremenljivke nabora podatkov. Ker so podatki razdeljeni za vlak in test, bom ustvaril PyTorch Podatkovni niz in PyTorch DataLoader predmet za vlak in test. SAMO zaradi enostavnosti bom uporabil test kot potrditev. V praksi NIKOLI NE UPORABLJAJTE TESTNIH PODATKOV ZA VALIDACIJO!
print('Dealing with Train...')
# Create pytorch dataset.
train_dataset = MovieReviewsDataset(path='/content/aclImdb/train', use_tokenizer=tokenizer, labels_ids=labels_ids, max_sequence_len=max_length)
print('Created `train_dataset` with %d examples!'%len(train_dataset)) # Move pytorch dataset into dataloader.
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
print('Created `train_dataloader` with %d batches!'%len(train_dataloader)) print() print('Dealing with ...')
# Create pytorch dataset.
valid_dataset = MovieReviewsDataset(path='/content/aclImdb/test', use_tokenizer=tokenizer, labels_ids=labels_ids, max_sequence_len=max_length)
print('Created `valid_dataset` with %d examples!'%len(valid_dataset)) # Move pytorch dataset into dataloader.
valid_dataloader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)
print('Created `eval_dataloader` with %d batches!'%len(valid_dataloader))
Dealing with Train...
Reading partitions...
100%|████████████████████████████████|2/2 [00:34<00:00, 17.28s/it]
Reading neg files...
100%|████████████████████████████████|12500/12500 [00:34<00:00, 362.01it/s] Reading pos files...
100%|████████████████████████████████|12500/12500 [00:23<00:00, 534.34it/s] Using tokenizer on all texts. This can take a while...
Texts padded or truncated to 40 length!
Finished! Created `train_dataset` with 25000 examples!
Created `train_dataloader` with 25000 batches! Dealing with ...
Reading partitions...
100%|████████████████████████████████|2/2 [01:28<00:00, 44.13s/it]
Reading neg files...
100%|████████████████████████████████|12500/12500 [01:28<00:00, 141.71it/s] Reading pos files...
100%|████████████████████████████████|12500/12500 [01:17<00:00, 161.60it/s] Using tokenizer on all texts. This can take a while...
Texts padded or truncated to 40 length!
Finished! Created `valid_dataset` with 25000 examples!
Created `eval_dataloader` with 25000 batches!
Vlak
Ustvarim optimizator in razporejevalnik, ki ju bo PyTorch uporabljal pri usposabljanju.
Pobrskam po številu definiranih epoh in pokličem vlak in potrjevanje funkcije.
Po vsaki epohi bom izpisal podobne podatke kot v Kerasu: train_loss: — val_loss: — train_acc: — valid_acc.
Po usposabljanju narišem vlak ter krivulje izgube in natančnosti validacije, da preverim, kako je potekalo usposabljanje.
# Note: AdamW is a class from the huggingface library (as opposed to pytorch) # I believe the 'W' stands for 'Weight Decay fix"
optimizer = AdamW(model.parameters(), lr = 2e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5 eps = 1e-8 # args.adam_epsilon - default is 1e-8. ) # Total number of training steps is number of batches * number of epochs.
# `train_dataloader` contains batched data so `len(train_dataloader)` gives # us the number of batches.
total_steps = len(train_dataloader) * epochs # Create the learning rate scheduler.
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps = 0, # Default value in run_glue.py num_training_steps = total_steps) # Store the average loss after each epoch so we can plot them.
all_loss = {'train_loss':[], 'val_loss':[]}
all_acc = {'train_acc':[], 'val_acc':[]} # Loop through each epoch.
print('Epoch')
for epoch in tqdm(range(epochs)): print() print('Training on batches...') # Perform one full pass over the training set. train_labels, train_predict, train_loss = train(train_dataloader, optimizer, scheduler, device) train_acc = accuracy_score(train_labels, train_predict) # Get prediction form model on validation data. print('Validation on batches...') valid_labels, valid_predict, val_loss = validation(valid_dataloader, device) val_acc = accuracy_score(valid_labels, valid_predict) # Print loss and accuracy values to see how training evolves. print(" train_loss: %.5f - val_loss: %.5f - train_acc: %.5f - valid_acc: %.5f"%(train_loss, val_loss, train_acc, val_acc)) print() # Store the loss value for plotting the learning curve. all_loss['train_loss'].append(train_loss) all_loss['val_loss'].append(val_loss) all_acc['train_acc'].append(train_acc) all_acc['val_acc'].append(val_acc) # Plot loss curves.
plot_dict(all_loss, use_xlabel='Epochs', use_ylabel='Value', use_linestyles=['-', '--']) # Plot accuracy curves.
plot_dict(all_acc, use_xlabel='Epochs', use_ylabel='Value', use_linestyles=['-', '--'])
Epoch 100%|████████████████████████████████|4/4[13:49<00:00, 207.37s/it] Training on batches... 100%|████████████████████████████████|782/782[02:40<00:00,4.86it/s] Validation on batches... 100%|████████████████████████████████|782/782[00:46<00:00,16.80it/s] train_loss: 0.44816 - val_loss: 0.38655 - train_acc: 0.78372 - valid_acc: 0.81892 Training on batches... 100%|████████████████████████████████|782/782[02:40<00:00,4.86it/s] Validation on batches... 100%|████████████████████████████████|782/782 [02:13<00:00,5.88it/s] train_loss: 0.29504 - val_loss: 0.43493 - train_acc: 0.87352 -valid_acc: 0.82360 Training on batches... 100%|████████████████████████████████|782/782[02:40<00:00, 4.87it/s] Validation on batches... 100%|████████████████████████████████|782/782[01:43<00:00,7.58it/s] train_loss: 0.16901 - val_loss: 0.48433 - train_acc: 0.93544 -valid_acc: 0.82624 Training on batches... 100%|████████████████████████████████|782/782[02:40<00:00, 4.87it/s] Validation on batches... 100%|████████████████████████████████|782/782[00:46<00:00,16.79it/s]
train_loss: 0.09816 - val_loss: 0.73001 - train_acc: 0.96936 - valid_acc: 0.82144
Videti je, da je malo več kot eno obdobje dovolj usposabljanja za ta model in nabor podatkov.
oceniti
Ko se ukvarjate s klasifikacijo, je koristno pogledati natančnost,odpoklic in rezultat f1. Še ena dobra stvar, ki jo je treba upoštevati pri ocenjevanju modela, je matrika zmede.
# Get prediction form model on validation data. This is where you should use
# your test data.
true_labels, predictions_labels, avg_epoch_loss = validation(valid_dataloader, device) # Create the evaluation report.
evaluation_report = classification_report(true_labels, predictions_labels, labels=list(labels_ids.values()), target_names=list(labels_ids.keys()))
# Show the evaluation report.
print(evaluation_report) # Plot confusion matrix.
plot_confusion_matrix(y_true=true_labels, y_pred=predictions_labels, classes=list(labels_ids.keys()), normalize=True, magnify=3, );
