Tengo un dataframe que contiene noticias para cada dia y intento analizar la intensidad del sentimiento por la dia, es por decir si el sentimiento general del día a partir de la noticia es positivo, negativo o neutral. Aqui esta la dataframe df_news
:
Date name
0 2017-10-20 Gucci debuts art installation at its Ginza sto...
1 2018-08-01 Gucci Joins Paris Fashion Week for Its Spring ...
2 2018-04-20 Gucci launches its new creative hub Gucci ArtL...
3 2017-10-20 Gucci to launch homeware line Gucci Decor - CP...
4 2017-12-07 GUCCI opens new store at Miami Design District...
5 2018-01-12 Gucci opens Gucci Garden in Florence - LUXUO
6 2018-02-26 GUCCI's wild experiment with the Fall Winter 2...
7 2018-08-09 Gucci Revamped London Flagship Store | The Imp...
8 2018-08-01 Alessandro Michele Announces new Gucci Home co...
9 2017-10-20 Before He Picks Up the CFDA’s International Aw...
Intenté obtener la intensidad del sentimiento con el siguiente código que utiliza SentimentIntensityAnalyzer
de nltk.sentiment.vader
:
from nltk.sentiment.vader import SentimentIntensityAnalyzer
import unicodedata
sid = SentimentIntensityAnalyzer()
for date, row in df_news.T.iteritems():
try:
sentence = unicodedata.normalize('NFKD', df_news.loc[date, 'name']).encode('ascii','ignore')
#print((sentence))
ss = sid.polarity_scores(str(sentence))
df_news.set_value(date, 'compound', ss['compound'])
df_news.set_value(date, 'neg', ss['neg'])
df_news.set_value(date, 'neu', ss['neu'])
df_news.set_value(date, 'pos', ss['pos'])
except TypeError:
print(df_news.loc[date, 'name'])
print(date)
Sin embargo obtengo una TypeError par ciertas fechas. Gracias al try catch
no lo tiene en cuenta y me dibuja la siguiente tabla :
name compound neg neu pos
Date
2017-10-20 Gucci debuts art installation at its Ginza sto...
2018-08-01 Gucci Joins Paris Fashion Week for Its Spring ...
2018-04-20 Gucci launches its new creative hub Gucci ArtL... 0.4404 0 0.756 0.244
2017-10-20 Gucci to launch homeware line Gucci Decor - CP...
2017-12-07 GUCCI opens new store at Miami Design District... 0 0 1 0
2018-01-12 Gucci opens Gucci Garden in Florence - LUXUO 0 0 1 0
2018-02-26 GUCCI's wild experiment with the Fall Winter 2... 0 0 1 0
2018-08-09 Gucci Revamped London Flagship Store | The Imp... 0.3182 0 0.602 0.398
2018-08-01 Alessandro Michele Announces new Gucci Home co...
2017-10-20 Before He Picks Up the CFDA’s International Aw...
Pero cuando elimino try catch para entender por qué falló, recibo el siguiente error :
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-26-2e9dbfc62bce> in <module>
4 for date, row in df_news.T.iteritems():
5 # try:
----> 6 sentence = unicodedata.normalize('NFKD', df_news.loc[date, 'name']).encode('ascii','ignore')
7 #print((sentence))
8 ss = sid.polarity_scores(str(sentence))
TypeError: normalize() argument 2 must be str, not Series
Entonces pensé que el problema fue con las líneas que no fueron string pero por ejemplo con la primera
>>>type(df_news['name'][0])
str
Para obtener las noticias
doc_data = {
"size": 10,
"query": {
"bool": {
"must" : [
{"term":{"text":"gucci"}}
]
}
}
}
docs = create_doc("https://elastic:[email protected]/luxurynsight_v2/news/_search",doc_data)
information_df = pd.DataFrame.from_dict(docs.json()["hits"]["hits"])
# Reading the JSON file
df_news = pd.read_json('data.json')
# Converting the element wise _source feature datatype to dictionary
df_news._source = df_news._source.apply(lambda x: dict(x))
# Creating name column
df_news['name'] = df_news._source.apply(lambda x: x['name'])
# Creating createdAt column
df_news['createdAt'] = df_news._source.apply(lambda x: x['createdAt'])
df_news['createdAt'] = pd.to_datetime(df_news['createdAt'], unit='ms')
df_news['createdAt'] = pd.DatetimeIndex(df_news.createdAt).normalize()
#df_news.createdAt.dt.normalize()
df_news['Date'] = df_news['createdAt']
df_news = df_news[['name','Date']]
df_news = df_news.set_index('Date')
information_df._source = information_df.apply(lambda x: dict(x))
df_news.reset_index()
Y le da :
Date name
0 2017-10-20 Gucci debuts art installation at its Ginza sto...
1 2018-08-01 Gucci Joins Paris Fashion Week for Its Spring ...
2 2018-04-20 Gucci launches its new creative hub Gucci ArtL...
3 2017-10-20 Gucci to launch homeware line Gucci Decor - CP...
4 2017-12-07 GUCCI opens new store at Miami Design District...
5 2018-01-12 Gucci opens Gucci Garden in Florence - LUXUO
6 2018-02-26 GUCCI's wild experiment with the Fall Winter 2...
7 2018-08-09 Gucci Revamped London Flagship Store | The Imp...
8 2018-08-01 Alessandro Michele Announces new Gucci Home co...
9 2017-10-20 Before He Picks Up the CFDA’s International Aw...