Author: Paul Duan
Skip the run test because the ROME version has to be updated to make it work in the exported repository. TODO: Update ROME and remove the skiptest flag.
Problem statement: This notebook is an exploration of how we could normalize the genderization of (French) job titles in ROME (Répertoire Opérationnel des Métiers et des Emplois), which is published by Pôle Emploi. These are often inconsistently specified, which leads to confusion and poorer user experience. In addition it makes the job title longer than it should be, which makes them harder to identify at a glance.
Input: ROME job titles, with hardcoded genderization in the plaintext job titles.
Desired output: A more structured format where jobs have both a masculine and a feminine version (which may be the same if the job is not genderized).
Example problem and output: An English-language example of the problem would be that a job such as "Senior Fireman" might sometimes be genderized as "Senior Fireman / Senior Firewoman" and sometimes as "Senior Fire(wo)man". We want to turn any variant of these possible genderizations into a normalized pair of two fields: ("Senior Fireman", "Senior Firewoman"). If a job title is not gendered, for example "Artist", we want to simply return ("Artist", "Artist").
Additional outputs (TODO): We could also re-define a normalized way of returning a genderized string when the user's gender is unknown (like the current ROME job titles, but enforcing a consistent notation for how genderization is handled). This, along with the rest, could be given back to Pôle Emploi so we can help them push the improvements upstream to the official ROME.
In [1]:
from itertools import chain
import pandas as pd
import re
from bob_emploi.data_analysis.lib import cleaned_data
In [2]:
jobs = cleaned_data.rome_jobs('../../../data')
In [3]:
jobs
Out[3]:
From a cursory look at the data we can see that there are at least two ways to genderize job titles:
When the adjective itself needs to be in concordance with the gender:
Remarks:
How many genderized job names are there?
As a quick way to estimate this number we'll just count the number of names containing slashes or brackets:
In [4]:
is_genderized = jobs['name'].apply(lambda x: '(' in x or '/' in x)
print('Number of genderized names :', sum(is_genderized))
print('Out of total names :', len(jobs), 'i.e.', sum(is_genderized)/len(jobs), '%')
This is the more complex case. Here let's extract brackets and the preceding word (non-greedily, to account for cases where there are multiple brackets) but do not match when the character before the bracket is non-alphabetical. This is because we want to make sure the bracket is a postfix directly appended to a noun without space, since brackets can only be used by themselves.
As such, a possible regex expression that would capture both a genderized word and its postfix is:
r"(\S+?)\((\S+?)\)"
To only capture the bracket, a regex would be:
r"(?<=\S)\(([\S]+?)\)"
With the positive lookbehind ensuring that the preceding character is not a space.
How many types of postfixes are there?
Because rules for properly handling postfixes are complicated (and postfixes for according jobs and adjectives are few), I believe it's better to simply exhaustively list them then hardcode their associated substitution rules. Let's list them:
(side note: as a bonus this also verifies that our regex is good and has no false positives)
In [5]:
postfix_rule = re.compile(r"(?<=\S)\(([\S]+?)\)")
has_bracket = jobs['name'].apply(lambda x: '(' in x)
postfixes = jobs[has_bracket].name.apply(
lambda x: re.findall(postfix_rule, x))
postfixes_types = set(chain.from_iterable(postfixes))
print(postfixes_types)
Looks like our regex is pretty good!
Both nouns and qualifiers are subject to genderization. Here are some examples of how these postfixes are meant to transform the preceding word:
Only the 's' postfix looks out of place, as it doesn't seem to be a gender postfix (it's a plural postfix). Let's see if this is a big deal:
In [6]:
jobs[jobs.name.apply(lambda x: '(s)' in x)]
Out[6]:
Looks like there is only one case, and the '(s)' plural postfix is frankly unnecessary (here the singular form already implies you can supervise many sites). So instead of making the regex needlessly complex to account for this, we'll just skip this particular case in the substitution code.
Caveat: Is the mapping of postfixes to word endings bijective? Looking at the list of postfixes, it appears that we both have "se" and "euse" as possible postfixes. This is problematic, because it means the mapping is possibly inconsistent:
In [7]:
print(jobs[jobs.name.apply(lambda x: '(euse)' in x)][:3])
print(jobs[jobs.name.apply(lambda x: '(se)' in x)][:3])
It appears that masculine words like "Accastilleur" and "Contrôleur", which both end in -eur, sometimes use the "(se)" and sometimes the "(euse)" postfix!
We also see that conversely, (euse) is both used for words ending in -er (like "Manager") and -eur ("Contrôleur")! Which means the postfixes are not normalized and we unfortunately have a many-to-many mapping :(
This case is easier: we'll just split the sentence according to slashes. The left side of the slash is always the masculine case, and the right side the feminine case.
Caveat: We need to make sure we handle qualifiers properly, because sometimes they are repeated on both sides of the slash and sometimes not, in which case they are meant to be distributive. For example, in "Responsable éditorial / éditoriale web", the word "Responsable" on the left side is meant to be repeated on the right side, and conversely the word "web" on the right side must be repeated on the left side. Sometimes this is not the case (all qualifiers are already repeated on both sides), sometimes one side only features distributive qualifiers, sometimes both (as in the example in the previous sentence).
One way of handling this is to first insert any words at the beginning of the left side that is not present in the ride side in front of the string in the right side.
Then once this is done we can consider that if the left side contains n words, then these also represent the first n words on the right side, with all extra words on the ride side needing to be appended to the left side as well.
In [8]:
POSTFIX_MAP = {
'e': [''], # empty string if the postfix can be appended
'ère': ['er'],
'se': ['r'],
'sse': [''],
'euse': ['eur', 'er'],
've': ['f'],
're': ['r'],
'rice': ['eur'],
'ne': [''],
'trice': ['teur'],
'le': [''],
'ive': ['if'],
'ière': ['ier'],
}
def check_mapping_specification(postfix_map):
"""Check whether the mapping of postfix to word endings
correctly returns a list of possible word endings to substitute
that goes from more specified (longer) to more general."""
for postfix, postfix_map in postfix_map.items():
if postfix_map != sorted(postfix_map, key=len, reverse=True):
return False
return True
def substitute_postfix(word, postfix, postfix_map):
"""Perform the correct postfix substitution from a
masculine word to a feminine word, taking into account the
fact that some postfix are meant to be appended to
the base string while others are meant to be substituted.
Both nouns and adjectives may be genderized.
Examples:
- Abbateur(se) -> Abbateur, Abbateuse
- Accompagnateur(trice) -> Accompagnateur, Accompagnatrice
- social(e) -> social, sociale
- Technicien(ne) -> Technicien, Technicienne
- administratif(ive) -> administratif, administrative
etc. The integral set of postfixes in the ROME dataset is:
{'e', 'ère', 'se', 'sse', 'euse', 've', 're',
'rice', 'ne', 'trice', 'le', 's', 'ive', 'ière'}
(The 's' postfix is an exception, as it relates to
the plural form rather than gender.)
Since there are only a limited number of postfixes, we can
exhaustively list them and the word ending they are meant to
substitute, which is safer in case new unaccounted ones pop up.
Note that the same postfix can be specified multiple ways
for multiple types of word endings.
For example, "Manager(euse)" is feminized as "Manageuse", but
both "Masseur(euse)" and "Masseur(se)" both translate to "Masseuse".
Therefore the substitution dictionary has the form:
{postfix: [possible_word_ending1, possible_word_ending2, ...]}
Here we perform the first substitution that matches the actual
word ending. This means the list of possible word endings must
therefore be ordered by decreasing length, so more specific cases
are always checked before more general ones.
"""
if not len(word):
raise ValueError("word is empty")
if postfix in POSTFIX_MAP:
known_endings = POSTFIX_MAP[postfix]
for known_ending in known_endings:
if word.endswith(known_ending):
root = word[:len(word) - len(known_ending)]
return root + postfix
error_string = "{0}: unmapped word ending for postfix '{1}'"
raise ValueError(error_string.format(word, postfix))
else:
raise ValueError("unknown postfix:" + postfix)
def extract_bracket_notation(raw_job_name, postfix_map):
"""Extract the genderized strings from a job title
with the bracket notation, by going through the string
and replacing brackets.
Return None if the item doesn't appear to be in
bracket notation."""
# nb: we only want brackets directly following a character
bracket_regex = re.compile(r"(\S+?)\((\S+?)\)")
matches = re.findall(bracket_regex, raw_job_name)
if not matches:
return None
# masculine name is just the string without the bracket content;
# to get feminine names we also substitute the relevant words.
# we'll perform these deletions/substitutions iteratively
masculine_name = feminine_name = raw_job_name
for word, postfix in matches:
if postfix != 's': # if not the plural postfix edge case
masculine_name = masculine_name.replace("({})".format(postfix), '')
feminine_name = feminine_name.replace("({})".format(postfix), '')
new_word = substitute_postfix(word, postfix, postfix_map)
feminine_name = feminine_name.replace(word, new_word)
return masculine_name, feminine_name
In [9]:
def extract_slash_notation(raw_job_name):
"""Extract the genderized strings from a job title
with the slash notation. Simply split the raw job
name according to the slashes, then deal with the
qualifiers distribution by appending all additional
words from the feminine (right) string to the masculine one.
In addition, any leading words on the left side must be
appended to the right side as well. One subtlety is that
detecting when the leading words end (or if they exist) is
complicated by the fact that the first common word may be
gendered, so we need to fuzzy match. For our purposes simply
comparing the first few characters should be enough.
Return None if the item doesn't appear to be in
slash notation."""
chars_to_compare = 3
substrings = raw_job_name.split(' / ')
if len(substrings) != 2:
return None
masculine_name, feminine_name = substrings
feminine_words = feminine_name.split(' ')
masculine_words = masculine_name.split(' ')
n_masculine_words = len(masculine_words)
# insert until a word that looks like right side's first
# word is encountered
to_insert = []
for word in masculine_words:
if word[:chars_to_compare] != feminine_words[0][:chars_to_compare]:
to_insert.append(word)
else:
break
feminine_words = to_insert + feminine_words
feminine_name = ' '.join(feminine_words)
# append extra right-side words to the the left side
for i, word in enumerate(feminine_words):
if i > n_masculine_words - 1:
masculine_words.append(word)
masculine_name = ' '.join(masculine_words)
return masculine_name, feminine_name
In [10]:
def genderize(df, postfix_map):
"""Take a dataframe of the same form as the
one returned by cleaned_data.rome_jobs and genderize it,
adding a masculine_name and a feminine_name field to it.
By default, masculine_name = feminine_name = raw_job_name,
then we overwrite the value when either the slash notation
or bracket notation rule is successful."""
masculine_name = df['name'].copy()
feminine_name = df['name'].copy()
bracket_output = df.name.apply(extract_bracket_notation,
postfix_map=postfix_map)
slash_output = df.name.apply(extract_slash_notation)
is_bracket = bracket_output.notnull()
is_slash = slash_output.notnull()
masculine_name[is_bracket] = bracket_output[is_bracket].apply(
lambda x: x[0])
feminine_name[is_bracket] = bracket_output[is_bracket].apply(
lambda x: x[1])
masculine_name[is_slash] = slash_output[is_slash].apply(
lambda x: x[0])
feminine_name[is_slash] = slash_output[is_slash].apply(
lambda x: x[1])
df['masculine_name'] = masculine_name
df['feminine_name'] = feminine_name
return df
Now all that's left is to run it:
In [11]:
postfix_map = POSTFIX_MAP
assert check_mapping_specification(postfix_map), "ill-specified postfix map"
genderize(jobs, postfix_map)
Out[11]:
Voilà !