📰 Building a news classifier with weak supervision¶
In this tutorial, we will build a news classifier using rules and weak supervision:
📰 For this example, we use the AG News dataset but you can follow this process to programmatically label any dataset.
🤿 The train split without labels is used to build a training set with rules, Rubrix and Snorkel’s Label model.
🔧 The test set is used for evaluating our weak labels, label model and downstream news classifier.
🤯 We achieve 0.82 macro avg. f1-score without using a single example from the original dataset and using a pretty lightweight model (scikit-learn’s
MultinomialNB
).
The following diagram shows the overall process for using Weak supervision with Rubrix:
Introduction¶
Weak supervision is a branch of machine learning where noisy, limited, or imprecise sources are used to provide supervision signal for labeling large amounts of training data in a supervised learning setting. This approach alleviates the burden of obtaining hand-labeled data sets, which can be costly or impractical. Instead, inexpensive weak labels are employed with the understanding that they are imperfect, but can nonetheless be used to create a strong predictive model. [Wikipedia]
For a broader introduction to weak supervision, as well as further references, we recommend the excellent overview by Alex Ratner et al..
This tutorial aims to be a practical introduction to weak supervision and will walk you through its entire process. First we will generate weak labels with Rubrix, combine these labels with Snorkel, and finally train a classifier with Scikit Learn.
Setup¶
Rubrix, is a free and open-source tool to explore, annotate, and monitor data for NLP projects.
If you are new to Rubrix, check out the ⭐ Github repository.
If you have not installed and launched Rubrix yet, check the Setup and Installation guide.
For this tutorial we also need some third party libraries that can be installed via pip:
[ ]:
%pip install snorkel datasets sklearn -qqq
Note
If you want to skip the first three sections of this tutorial, and only prepare the training set and train a downstream model, you can load the records directly from the Hugging Face Hub:
import rubrix as rb
from datasets import load_dataset
# this replaces the `records = label_model.predict()` line of section 4
records = rb.read_datasets(
load_dataset("rubrix/news", split="train"),
task="TextClassification",
)
1. Load test and unlabelled datasets into Rubrix¶
First, let’s download the ag_news
data set and have a quick look at it.
[ ]:
from datasets import load_dataset
# load our data
dataset = load_dataset("ag_news")
# get the index to label mapping
labels = dataset["test"].features["label"].names
[10]:
import pandas as pd
# quick look at our data
with pd.option_context('display.max_colwidth', None):
display(dataset["test"].to_pandas().head())
text | label | |
---|---|---|
0 | Fears for T N pension after talks Unions representing workers at Turner Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul. | 2 |
1 | The Race is On: Second Private Team Sets Launch Date for Human Spaceflight (SPACE.com) SPACE.com - TORONTO, Canada -- A second\team of rocketeers competing for the #36;10 million Ansari X Prize, a contest for\privately funded suborbital space flight, has officially announced the first\launch date for its manned rocket. | 3 |
2 | Ky. Company Wins Grant to Study Peptides (AP) AP - A company founded by a chemistry researcher at the University of Louisville won a grant to develop a method of producing better peptides, which are short chains of amino acids, the building blocks of proteins. | 3 |
3 | Prediction Unit Helps Forecast Wildfires (AP) AP - It's barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar. | 3 |
4 | Calif. Aims to Limit Farm-Related Smog (AP) AP - Southern California's smog-fighting agency went after emissions of the bovine variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure. | 3 |
Now we will log the test split of our data set to Rubrix, which we will be using for testing our label and downstream models.
[ ]:
import rubrix as rb
# build our test records
records = [
rb.TextClassificationRecord(
text=record["text"],
metadata={"split": "test"},
annotation=labels[record["label"]]
)
for record in dataset["test"]
]
# log the records to Rubrix
rb.log(records, name="news")
In a second step we log the train split without labels. Remember, our goal is to programmatically build a training set using rules and weak supervision.
[ ]:
# build our training records without labels
records = [
rb.TextClassificationRecord(
text=record["text"],
metadata={"split": "unlabelled"},
)
for record in dataset["train"]
]
# log the records to Rubrix
rb.log(records, name="news")
The result of the above is the following dataset in Rubrix, with 127,600 records (120,000 unlabelled and 7,600 for testing).
You can use the web app to find good rules for programmatic labeling!
2. Interactive weak labeling: Finding and defining rules¶
After logging the dataset, you can find and save rules directly with the UI. Then, you can read the rules with Python to train a label or downstream model, as we’ll see in the next step.
3. Denoise weak labels with Snorkel’s Label Model¶
The goal at this step is to denoise the weak labels we’ve just created using rules. There are several approaches to this problem using different statistical methods.
In this tutorial, we’re going to use Snorkel but you can actually use any other Label model or weak supervision method, such as FlyingSquid for example (see the Weak supervision guide for more details). For convenience, Rubrix defines a simple wrapper over Snorkel’s Label Model so it’s easier to use with Rubrix weak labels and datasets
Let’s first read the rules defined in our dataset and create our weak labels:
[ ]:
from rubrix.labeling.text_classification import WeakLabels
weak_labels = WeakLabels(dataset="news")
[22]:
weak_labels.summary()
[22]:
label | coverage | annotated_coverage | overlaps | conflicts | correct | incorrect | precision | |
---|---|---|---|---|---|---|---|---|
money | {Business} | 0.008276 | 0.008816 | 0.002437 | 0.001936 | 30 | 37 | 0.447761 |
financ* | {Business} | 0.019655 | 0.017763 | 0.005893 | 0.005188 | 80 | 55 | 0.592593 |
dollar* | {Business} | 0.016591 | 0.016316 | 0.003542 | 0.002908 | 87 | 37 | 0.701613 |
war | {World} | 0.011779 | 0.013289 | 0.003213 | 0.001348 | 75 | 26 | 0.742574 |
gov* | {World} | 0.045078 | 0.045263 | 0.010878 | 0.006270 | 170 | 174 | 0.494186 |
minister* | {World} | 0.030031 | 0.028289 | 0.007531 | 0.002821 | 193 | 22 | 0.897674 |
conflict | {World} | 0.003041 | 0.002895 | 0.001003 | 0.000102 | 18 | 4 | 0.818182 |
footbal* | {Sports} | 0.013166 | 0.015000 | 0.004945 | 0.000439 | 107 | 7 | 0.938596 |
sport* | {Sports} | 0.021191 | 0.021316 | 0.007045 | 0.001223 | 139 | 23 | 0.858025 |
game | {Sports} | 0.038879 | 0.037763 | 0.014083 | 0.002375 | 216 | 71 | 0.752613 |
play* | {Sports} | 0.052453 | 0.050000 | 0.016889 | 0.005063 | 268 | 112 | 0.705263 |
sci* | {Sci/Tech} | 0.016552 | 0.018421 | 0.002735 | 0.001309 | 114 | 26 | 0.814286 |
techno* | {Sci/Tech} | 0.027218 | 0.028289 | 0.008433 | 0.003174 | 155 | 60 | 0.720930 |
computer* | {Sci/Tech} | 0.027320 | 0.028026 | 0.011058 | 0.004459 | 159 | 54 | 0.746479 |
software | {Sci/Tech} | 0.030243 | 0.029605 | 0.009655 | 0.003346 | 184 | 41 | 0.817778 |
web | {Sci/Tech} | 0.015376 | 0.013289 | 0.004067 | 0.001607 | 76 | 25 | 0.752475 |
total | {World, Sports, Business, Sci/Tech} | 0.317022 | 0.311447 | 0.053582 | 0.019561 | 2071 | 774 | 0.727944 |
[ ]:
from rubrix.labeling.text_classification import Snorkel
# create the label model
label_model = Snorkel(weak_labels)
# fit the model
label_model.fit()
[28]:
print(label_model.score(output_str=True))
precision recall f1-score support
Sports 0.79 0.96 0.87 632
Sci/Tech 0.77 0.77 0.77 773
World 0.70 0.80 0.74 509
Business 0.65 0.36 0.46 453
accuracy 0.75 2367
macro avg 0.73 0.72 0.71 2367
weighted avg 0.74 0.75 0.73 2367
4. Prepare our training set¶
Now, we already have a “denoised” training set, which we can prepare for training a downstream model. The label model predict returns TextClassificationRecord
objects with the predictions
from the label model.
We can either refine and review these records using the Rubrix web app, use them as is, or filter them by score, for example.
In this case, we assume the predictions are precise enough and use them without any revision. Our training set has ~38,000 records, which corresponds to all records where the label model has not abstained.
[ ]:
import pandas as pd
# get records with the predictions from the label model
records = label_model.predict()
# you can replace this line with
# records = rb.read_datasets(
# load_dataset("rubrix/news", split="train"),
# task="TextClassification",
# )
# we could also use the `weak_labels.label2int` dict
label2int = {'Sports': 0, 'Sci/Tech': 1, 'World': 2, 'Business': 3}
# extract training data
X_train = [rec.text for rec in records]
y_train = [label2int[rec.prediction[0][0]] for rec in records]
[16]:
# quick look at our training data with the weak labels from our label model
with pd.option_context('display.max_colwidth', None):
display(pd.DataFrame({"text": X_train, "label": y_train}))
text | label | |
---|---|---|
0 | FA Cup: Third round draw - joy for Yeading and Exeter The great big fat bullies from across the playground enter the FA Cup arena at the third round stage, for which the draw was made yesterday (December 5). | 0 |
1 | Rats May Help Unravel Human Drug Addiction Mysteries By LAURAN NEERGAARD WASHINGTON (AP) -- Rats can become drug addicts. That's important to know, scientists say, and has taken a long time to prove... | 1 |
2 | Palmer Passes Test Bengals quarterback Carson Palmer enjoyed his breakthrough game at the expense of the Super Bowl champion Patriots, racking up 179 yards on 12-of-19 passing in a 31-3 triumph on Saturday night. | 0 |
3 | Compromises urged amid deadlock in Darfur talks ABUJA, Nigeria -- Peace talks on Sudan's violence-torn Darfur region are deadlocked, a mediator said yesterday, as the chief of the African Union appealed to the Sudanese government and rebels to compromise. | 2 |
4 | CAPELLO FED UP WITH FEIGNING Juventus coach Fabio Capello has ordered his players not to kick the ball out of play when an opponent falls to the ground apparently hurt because he believes some players fake injury to stop the match. | 0 |
... | ... | ... |
38080 | Apple ships Mac OS X update The 26MB upgrade to version 10.3.6 is available now via the OS #39; Software Update control panel and from Apple #39;s support web site. | 1 |
38081 | Bob Evans, mainframe pioneer, dies at 77 Bob Evans, an IBM computer scientist who helped to develop the modern mainframe computer, died Thursday. He was 77. Evans died of heart failure at his at his home in the San Francisco suburb of Hillsborough, his son Evan told the Associated Press. | 1 |
38082 | For Brazil's Economy, the Doctor Is In Antonio Palocci, a doctor from Brazil's farm belt, has found himself presiding as the country's finance minister during the most robust economic expansion in a decade. | 2 |
38083 | UbiSoft Get Ready With The Next Rainbox Six Installment Ubisoft today announced its plans to launch the next installment in the Tom Clancys Rainbow Six franchise in Spring 2005. The next Rainbow Six game follows Team Rainbow, the worlds most | 0 |
38084 | PM #39;s visit to focus on reconstruction of Kashmir New Delhi: The two-day visit of Prime Minister Manmohan Singh to Jammu and Kashmir, starting on Wednesday will focus more on reconstruction and development of the state, Parliamentary Affairs Minister Ghulam Nabi Azad has said. | 2 |
38085 rows × 2 columns
5. Train a downstream model with scikit-learn¶
Now, let’s train our final model using scikit-learn
:
[ ]:
from sklearn.feature_extraction.text import TfidfTransformer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
# define our final classifier
classifier = Pipeline([
('vect', CountVectorizer()),
('clf', MultinomialNB())
])
# fit the classifier
classifier.fit(
X=X_train,
y=y_train,
)
To test our trained model, we use the records with validated annotations, that is the original ag_news test set.
[ ]:
# retrieve records with annotations
test_ds = weak_labels.records(has_annotation=True)
# you can replace this line with
# test_ds = rb.read_datasets(
# load_dataset("rubrix/news_test", split="train"),
# task="TextClassification",
# )
# extract text and labels
X_test = [rec.text for rec in test_ds]
y_test = [label2int[rec.annotation] for rec in test_ds]
[77]:
# compute the test accuracy
accuracy = classifier.score(
X=X_test,
y=y_test,
)
print(f"Test accuracy: {accuracy}")
Test accuracy: 0.8182894736842106
Not too bad! 🥳
We have achieved around 0.82 accuracy without even using a single example from the original ag_news
train set and with a small set of 16 rules. Also, we’ve improved over the 0.75 accuracy of our Label Model.
Finally, let’s take a look at more detailed metrics:
[76]:
from sklearn import metrics
# get predictions for the test set
predicted = classifier.predict(X_test)
print(metrics.classification_report(y_test, predicted, target_names=label2int.keys()))
precision recall f1-score support
Sports 0.86 0.98 0.91 1900
Sci/Tech 0.76 0.84 0.80 1900
World 0.80 0.89 0.84 1900
Business 0.88 0.57 0.69 1900
accuracy 0.82 7600
macro avg 0.82 0.82 0.81 7600
weighted avg 0.82 0.82 0.81 7600
At this point, we could go back to the UI to define more rules for those labels with less performance. Looking at the above table, we might want to add some more rules for increasing the recall of the Business
label.
Summary¶
In this tutorial, we saw how you can leverage weak supervision to quickly build up a large training data set, and use it for the training of a first lightweight model.
Rubrix is a very handy tool to start the weak supervision process by making it easy to find a good set of starting rules, and to reiterate on them dynamically. Since Rubrix also provides built-in support for the most common label models, you can get from rules to weak labels in a few straight forward steps. For more suggestions on how to leverage weak labels, you can checkout our weak supervision guide where we describe an interesting approach to jointly train the label and a transformers downstream model.
Next steps¶
If you are interested in the topic of weak supervision check our weak supervision guide.
⭐ Rubrix Github repo to stay updated.¶
📚 Rubrix documentation for more guides and tutorials.¶
Appendix I: Create rules and weak labels from Python¶
For some use cases, you might want to use Python for defining labeling rules and generating weak labels. Rubrix provides you with the ability to define and test rules and labeling functions directly using Python. This might be useful for combining it with rules defined in the UI, and for leveraging structured resources such as lexicons and gazeteers which are easier to use directly a programmatic environment.
In this section, we define the rules we’ve defined in the UI, this time directly using Python:
[ ]:
from rubrix.labeling.text_classification import Rule
# define queries and patterns for each category (using ES DSL)
queries = [
(["money", "financ*", "dollar*"], "Business"),
(["war", "gov*", "minister*", "conflict"], "World"),
(["footbal*", "sport*", "game", "play*"], "Sports"),
(["sci*", "techno*", "computer*", "software", "web"], "Sci/Tech")
]
# define rules
rules = [
Rule(query=term, label=label)
for terms,label in queries
for term in terms
]
[ ]:
from rubrix.labeling.text_classification import WeakLabels
# generate the weak labels
weak_labels = WeakLabels(
rules=rules,
dataset="news"
)
On our machine it took around 24 seconds to apply the rules and to generate weak labels for the 127,600 examples.
Typically, you want to iterate on the rules and check their statistics. For this, you can use weak_labels.summary
method:
[78]:
weak_labels.summary()
[78]:
label | coverage | annotated_coverage | overlaps | conflicts | correct | incorrect | precision | |
---|---|---|---|---|---|---|---|---|
money | {Business} | 0.008276 | 0.008816 | 0.002437 | 0.001936 | 30 | 37 | 0.447761 |
financ* | {Business} | 0.019655 | 0.017763 | 0.005893 | 0.005188 | 80 | 55 | 0.592593 |
dollar* | {Business} | 0.016591 | 0.016316 | 0.003542 | 0.002908 | 87 | 37 | 0.701613 |
war | {World} | 0.011779 | 0.013289 | 0.003213 | 0.001348 | 75 | 26 | 0.742574 |
gov* | {World} | 0.045078 | 0.045263 | 0.010878 | 0.006270 | 170 | 174 | 0.494186 |
minister* | {World} | 0.030031 | 0.028289 | 0.007531 | 0.002821 | 193 | 22 | 0.897674 |
conflict | {World} | 0.003041 | 0.002895 | 0.001003 | 0.000102 | 18 | 4 | 0.818182 |
footbal* | {Sports} | 0.013166 | 0.015000 | 0.004945 | 0.000439 | 107 | 7 | 0.938596 |
sport* | {Sports} | 0.021191 | 0.021316 | 0.007045 | 0.001223 | 139 | 23 | 0.858025 |
game | {Sports} | 0.038879 | 0.037763 | 0.014083 | 0.002375 | 216 | 71 | 0.752613 |
play* | {Sports} | 0.052453 | 0.050000 | 0.016889 | 0.005063 | 268 | 112 | 0.705263 |
sci* | {Sci/Tech} | 0.016552 | 0.018421 | 0.002735 | 0.001309 | 114 | 26 | 0.814286 |
techno* | {Sci/Tech} | 0.027218 | 0.028289 | 0.008433 | 0.003174 | 155 | 60 | 0.720930 |
computer* | {Sci/Tech} | 0.027320 | 0.028026 | 0.011058 | 0.004459 | 159 | 54 | 0.746479 |
software | {Sci/Tech} | 0.030243 | 0.029605 | 0.009655 | 0.003346 | 184 | 41 | 0.817778 |
web | {Sci/Tech} | 0.015376 | 0.013289 | 0.004067 | 0.001607 | 76 | 25 | 0.752475 |
total | {World, Sports, Business, Sci/Tech} | 0.317022 | 0.311447 | 0.053582 | 0.019561 | 2071 | 774 | 0.727944 |
From the above, we see that our rules cover around 30% of the original training set with an average precision of 0.73. Our hope is that the label and downstream models will improve both the recall and the precision of the final classifier.
Appendix II: Log datasets to the Hugging Face Hub¶
Here we will show you how we pushed our Rubrix datasets (records) to the Hugging Face Hub. In this way you can effectively version any of your Rubrix datasets.
[ ]:
train_rb = rb.DatasetForTextClassification(label_model.predict())
train_rb.to_datasets().push_to_hub("rubrix/news")
[ ]:
test_rb = rb.load("news", query="status:Validated", as_pandas=False)
test_rb.to_datasets().push_to_hub("rubrix/news_test")