Categories
By Aman Kharwal

Twitter Relationships Analysis with Python without API

Twitter Relationships analysis without Twitter API

In this tutorial I will explain how to scrape twitter data with Python and some relationships analysis based on followings and mentioning among a group of Twitter users.

Getting access to the official Twitter API is very challenging. Therefore, I searched for another solution and found out twint. This is a python library that enables you to scrap twitter data without API access.

Let’s start by importing the libraries :

import twint
import pandas as pd
from collections import Counter

Now we need to start with creating a user list that consists of twitter accounts. Our analysis will include the relationships of these users.

I have chosen some popular Twitter users whom everyone knows to make our analysis more interesting.

users = [
    'shakira',
    'KimKardashian',
    'rihanna',
    'jtimberlake',
    'KingJames',
    'neymarjr',
]

Following Relationships Analysis

Let’s start with relationship analysis, for this purpose we need to create a function that sends a request to twint library with a username. This function will return a list of users whom our input user follows.

def get_followings(username):

    c = twint.Config()
    c.Username = username
    c.Pandas = True

    twint.run.Following(c)
    list_of_followings = twint.storage.panda.Follow_df

    return list_of_followings['following'][username]

Now I will create a new function, to get different following lists for every person in our users list and store the results to a dictionary, and a list.

followings = {}
following_list = []
for person in users:
    print('#####\nStarting: ' + person + '\n#####')
    try:
        followings[person] = get_followings(person)
        following_list = following_list + followings[person]
    except KeyError:
        print('IndexError')
#Output
daaniloluiz
SigaRodriguinho
andremarinho_
kahverhnjak
camiiisvelasco
renatinhag_
AleeCordeiro
vjassis
dentinho
Nicole_rt
JotaC92
natigc
dani_souza_
patpilar
jugracie
jubabyyy
Yanioficiall
GilCebola
JosyQueirozz
Adalberto78
larydeoliveira
GeorgiaaTavares
danielaxavierr
RomarioOnze
Dbolina
pinhapresidente
lizibenites_
dodopixote
10Ronaldinho
sheronmenezzes
fun140
SabrinaSato
carlinhoshumor
TrioTOPPF
Babicrirossi
JujuSalimeni
LucianoHuck
LucianaGimenez
denilsonshow
BrunoAgnello
Rafaella_
nataliaabarros
karooolmachado
CRITICAL:root:twint.feed:Follow:IndexError
CRITICAL:root:twint.feed:Follow:IndexError
maikonleite17
alelabate
CarolCastroReal
tatidecarvalho_
joannadeassis
guustavo92
nanabitencourt
lauraribeiro30
thiaguinhocomth
periclesfaria
KAKA
Betao03

The for loop above creates two variables, as sometimes Twitter does not respond to our request and in this case, we get an Index Error. For such cases, I have added an exception to the code to skip these errors.

Who are mostly followed by our Users?

After getting all the following lists, we can simply calculate the most common values in the following_list variable to get the most popular accounts among our users.

To get the most followed 10 accounts, we will use Counter function from collections library.

Counter(following_list).most_common(10)
#Output
[('rihanna', 4),
 ('BarackObama', 3),
 ('Nike', 3),
 ('NBA', 3),
 ('kobebryant', 3),
 ('FCBarcelona_es', 2),
 ('FCBarcelona', 2),
 ('RecordingAcad', 2),
 ('ivanrakitic', 2),
 ('AppStore', 2)]

The result of this function is shown above. Rihanna seems to be followed by all others and in our user group, she is definitely the most popular one.

Following Relations among Users

What if we want to see who is following whom in our user group? To investigate it, I wrote a for loop that checks if anyone in the users is in the following list of another person.

As a result, it creates a dictionary of lists showing the following statuses represented by Trues and Falses.

follow_relations ={}
for following_user in followings.keys():
    follow_relation_list = []
    for followed_user in followings.keys():
        if followed_user in followings[following_user]:
            follow_relation_list.append(True)
        else:
            follow_relation_list.append(False)
    follow_relations[following_user] = follow_relation_list

In the code below, the result dictionary is transformed into a pandas dataframe for a more user-friendly visualization.

following_df = pd.DataFrame.from_dict(follow_relations, 
                                      orient='index', columns=followings.keys())
following_df

Mention Counts Analysis

Mention counts are another strong relationship indicator between Twitter users.

The function below (get_mention_count) is written for this purpose and it returns the mention counts between two users in one direction.

def get_mention_count(user, mention_word):

    c = twint.Config()
    c.Username = user
    c.Search = '@' + mention_word
    c.Store_object = True

    twint.run.Search(c)
    tweets = twint.output.tweets_list
    mention_count = len(tweets)
    tweets.clear()

    return mention_count

In the analysis, we will use two nested for loops to retrieve mention counts of every user to all others in our group.

As a result, we will get mention_relationships dictionary.

mention_relations = {}
for mentioning_user in users:
    print('#####\nStarting: ' + mentioning_user + '\n#####')
    mention_count_list = []
    for mentioned_user in users:
        mention_count = get_mention_count(mentioning_user, mentioned_user)
        mention_count_list.append(mention_count)
    mention_relations[mentioning_user] = mention_count_list
#Output
459042920428232704 2014-04-23 18:55:35 UTC <neymarjr> Bagunçando tudo ... kkkkkk RT @NeymarandLuan @neymarjr como o Davi ta
459042283854508033 2014-04-23 18:53:03 UTC <neymarjr> Já viu o video de hj ? Esse é especial...  http://youtu.be/COLSHA8EsaY  RT @inloveneymarjr @neymarjr quero 3 vídeos todas quarta, to mandando u.u
457567662768656385 2014-04-19 17:13:26 UTC <neymarjr> Welcome to the club !! RT @kobebryant Special #nikefc delivery. Guess this means I'm in the club too @neymarjr!  pic.twitter.com/zTR4DeoDmp
457552453886803968 2014-04-19 16:13:00 UTC <neymarjr> Thank you !! RT @Bless_You_MnT @neymarjr 😜😜✌️✌️✌️  https://twitter.com/Bless_You_MnT/status/457552206753853440/photo/1 pic.twitter.com/5Xfqs1XyLn 
457551488530001920 2014-04-19 16:09:10 UTC <neymarjr> Melhorando... ;-) RT @endlessneymar @neymarjr como você está?
453615610388553728 2014-04-08 19:29:23 UTC <neymarjr> ;-) RT @LojaNeymar Sucesso amanhã, @neymarjr !!
453614108978724864 2014-04-08 19:23:25 UTC <neymarjr> Às 11h do Brasil !! RT @njrendless @neymarjr é amanha que saiu episódio novo no "dias em Santos" no seu canal né?!
451371686169165824 2014-04-02 14:52:50 UTC <neymarjr> Yes !! Here ==> #DiasEmSantos  https://www.youtube.com/watch?v=uYSQlEEHdNE … RT @FutiaJR Is it your film on Youtube ? @neymarjr and is it subtitled in english ?
451369874242404352 2014-04-02 14:45:38 UTC <neymarjr> O próximo filme ? Toda quarta, 11h !! Se inscreve no  http://www.youtube.com/neymarjr  !! RT @poxajuni0r @neymarjr quando sai o próximo ??
451369177140695040 2014-04-02 14:42:52 UTC <neymarjr> Here ==> #DiasEmSantos  https://www.youtube.com/watch?v=uYSQlEEHdNE … #CanalNJr RT @SarahTounsia06 @neymarjr Can't wait for your new videos on youtube :DDD ❤
451368218851291136 2014-04-02 14:39:04 UTC <neymarjr> Para de tuitar e presta atenção na aula !! RT @NossaJoiaRara_ @neymarjr só vou ver depois que chegar da escola ok ? não fique bravo...
448547786024583168 2014-03-25 19:51:40 UTC <neymarjr> Tchau pra todo mundo !! Abss !! RT @UmaFaDoNeymar tchau Neymar  @neymarjr
448545613106647040 2014-03-25 19:43:02 UTC <neymarjr> Bonsoir !! RT @SarahTounsia06 @neymarjr BONJOUR ❤️🇫🇷 #NeymarzetesFrances
448544740192321539 2014-03-25 19:39:34 UTC <neymarjr> Recebi sim galera !! Vlw !! RT @LojaNeymar Hey @neymarjr !! Você recebeu os presentes que mandamos pra você ?? #LojaOficialNJr
448543578873012224 2014-03-25 19:34:57 UTC <neymarjr> Desiste não... ;-) RT @PartOfMeNJR vou nem falar com você, sei que não vai me responder zzz @neymarjr
448542486898946048 2014-03-25 19:30:37 UTC <neymarjr> Oi, tudo bem ? RT @Ney_Divo @neymarjr oi ney
445677303025389568 2014-03-17 21:45:23 UTC <neymarjr> Já... Fui !! RT @inloveneymarjr @neymarjr vai sair já ?
445676417398104064 2014-03-17 21:41:52 UTC <neymarjr> Hello to England !! \o/ RT @Amelia_Murphyy @neymarjr can the fans from England get a hello?!
445676157762289664 2014-03-17 21:40:50 UTC <neymarjr> O dia que o Davi nasceu !! RT @gl0wnjr @neymarjr uma data marcante na sua vida ?
445675167306756096 2014-03-17 21:36:54 UTC <neymarjr> Alegria e ousadia... RT @tudopeloneymar @neymarjr uma frase que você gosta bastante?
445674589386190849 2014-03-17 21:34:37 UTC <neymarjr> Amém !! RT @warriorneymar @neymarjr so Deus abençoando p vc me ver entre 10 milhões agora ne?!
445674241120546816 2014-03-17 21:33:13 UTC <neymarjr> Vlw galera !! Abs !! RT @LojaNeymar Parabéns !! RT @neymarjr Somos 10M ! We are 10M ! Som 10M ! Vlw ! Tks ! ¡Gracias! Gràcies! \o/
444224644435169280 2014-03-13 21:33:03 UTC <neymarjr> Tarde, né ? Vc tem razão...Vou dormir então ... Fui !! RT @LetciaNogueir11 22:30 e você acordado mocinho ? sem sono é? @neymarjr
444219694611173376 2014-03-13 21:13:23 UTC <neymarjr> Wow !! Canada !! Hello to everybody !! RT @NovaJr11 @neymarjr just say hi to your fans in Canada 😊 EU te amo 😘😘❤️
444219232755408896 2014-03-13 21:11:32 UTC <neymarjr> Tá chegando, né ?? ... RT @blessedlovenjr @neymarjr ansioso para copa meu amor?
444218364370911232 2014-03-13 21:08:05 UTC <neymarjr> Hola @Barbaritapsb !! ... @neymarjr te amo tanto el día que me respondas ese día sera el mas feliz♥ de mi vida te amoooo
441677187306237952 2014-03-06 20:50:22 UTC <neymarjr> Hi and goodbye !! Fui ... !! RT @rayhanahh_ I'm waiting for you to say hi to me @neymarjr
441676480083673088 2014-03-06 20:47:33 UTC <neymarjr> Kkkkkkkkkk RT @njrdolepolepo @neymarjr  boatos q se voce me responder eu saio correndo pela casa toda igual uma foca com asma -q
441672520056713216 2014-03-06 20:31:49 UTC <neymarjr> Do Davi... RT @gottabeneymar @neymarjr Ei!!! Do que você mais sente saudade do Brasil?
438026374692233216 2014-02-24 19:03:20 UTC <neymarjr> Hello everybody !! RT @rhystreadgold @neymarjr hello from the UK Neymar!!
438025655465562113 2014-02-24 19:00:29 UTC <neymarjr> Então vê de novo ... mas presta atenção !! Kkkk RT @juniormypride chego uma mention aqui e eu jurando que fosse o @neymarjr #fail
438024519836790784 2014-02-24 18:55:58 UTC <neymarjr> Muito !! ... é uma cidade maravilhosa !! RT @laaaauraxx @neymarjr vc esta gostando de estar em Barcelona?
438023480186249216 2014-02-24 18:51:50 UTC <neymarjr> Mês que vem eu mato a saudade... RT @brilhaneymar @neymarjr saudades de usar a amarelinha?
438023047287934976 2014-02-24 18:50:07 UTC <neymarjr> ;-) RT @Lauritaaa7 @neymarjr ney eres un crack estas dando rt a la gente me arias molt feliiç si me rt crack
438021979158085632 2014-02-24 18:45:52 UTC <neymarjr> Tô aqui ... tô lendo tudo !! ;-) RT @Forever_ney @neymarjr fica com a gente aki
431162806009090048 2014-02-05 20:29:58 UTC <neymarjr> Sería perfecto !! RT @_FabiolaMessi Es una lastima que @neymarjr no pueda jugar el día de su cumpleaños.
431123782762528768 2014-02-05 17:54:54 UTC <neymarjr> Thank you !!!! RT @mpriya01 And of course Feliz Cumpleaños to my fav player @neymarjr! X
431059373943980032 2014-02-05 13:38:58 UTC <neymarjr> #TamoJuntoSempre !! RT @likenjr @neymarjr suas neymarzetes te amam muito #Ha22AnosNasciaUmAnjoChamadoNeymarJr
431038588315508736 2014-02-05 12:16:22 UTC <neymarjr> Obrigado !! RT @yume84lb @neymarjr Parabéns Neymar🎉👏 Todo de bom pra você! Lindo❤ Deus abençoe!  https://twitter.com/yume84lb/status/431037866420862979/photo/1 pic.twitter.com/Pr1AUYTkbg 
421733648615079936 2014-01-10 20:01:51 UTC <neymarjr> Gracias RT @emiliamessi @neymarjr mucha suerte en el partido de mañana :)
421732099511832576 2014-01-10 19:55:42 UTC <neymarjr> Ainda não .. tô aqui lendo ... RT @NJRanjosemasas @neymarjr já saiu?
421729146960883713 2014-01-10 19:43:58 UTC <neymarjr> Hahaha ... !! RT @tudoporti_njr Uma mention comum, é uma coisa... Uma mention do Neymar, é outra coisa. @neymarjr
421728709335580672 2014-01-10 19:42:14 UTC <neymarjr> Tá contando aí também ?? ;-) RT @emersonftnjr @neymarjr  26 dias para 2.2 hahaha
419201311721406464 2014-01-03 20:19:15 UTC <neymarjr> Un altre per a tu!! RT @marialmasoliver @neymarjr un petó per aquesta noia que t'estima i t'admira!!!
419200327582502912 2014-01-03 20:15:21 UTC <neymarjr> É NOSSO!! RT @meutesouronjr @neymarjr esse ano é SEU!!
406890281389531137 2013-11-30 20:59:37 UTC <neymarjr> #OuseSerBrasileiro !! RT @nikefutebol A verdadeira essência do futebol moleque, na visão do @neymarjr.  https://www.youtube.com/watch?v=mX3adn8LBfA …
405801871916683264 2013-11-27 20:54:40 UTC <neymarjr> Responder e ir embora !! Fui !! RT @SempreNey_92 @neymarjr, Feito que as 18h54 você vai me responder ?
405800285182111744 2013-11-27 20:48:22 UTC <neymarjr> Boatos falsos .. não vou responder .. RT @MylifeNJR Boatos qe vc vai me responder hj, confirma??? @neymarjr
405798529068986368 2013-11-27 20:41:23 UTC <neymarjr> É nada... kkkkk RT @MylifeNjunior_ Ohhhhhhhhh neymar e feio ler e n responder @neymarjr
401435020814196736 2013-11-15 19:42:21 UTC <neymarjr> Bjo !! Tchau !!!! RT @prideneymar @neymarjr manda um beijo pra suas fãs ?
399210766190260224 2013-11-09 16:23:58 UTC <neymarjr> Nós nascemos no Brasil !! \o/ RT @ForeverMyNJR Eu não tenho culpa se eu nasci no Brasil, então pode me responder @neymarjr
399210145127071744 2013-11-09 16:21:30 UTC <neymarjr> Neymarzetes from Indonesia !!!!!! :O !!!!!! Hi !!!!!!!! RT @mithasmashblast @neymarjr say hi Neymarzetes from indonesia!! :D
399208615481184256 2013-11-09 16:15:25 UTC <neymarjr> I'll be in Canada soon !! ;-) @SuSuu24 @neymarjr say hi Canada please!!
396667349354684416 2013-11-02 15:57:20 UTC <neymarjr> Muita !! RT @meutesouronjr @neymarjr saudades do Brasil?
396667030088454144 2013-11-02 15:56:04 UTC <neymarjr> Brigado João !! RT @JoaoNorberta @neymarjr Vibrei com seu primeiro gol no barcelona Parabens!
396666276736929792 2013-11-02 15:53:04 UTC <neymarjr> It's perfect !! Kkkkk RT @SarahTounsia06 @neymarjr Como vai vocé? :) (don't know if it's right 0.o ..)
395632460052332544 2013-10-30 19:25:03 UTC <neymarjr> Oi !! RT @MylifeNjunior_ @neymarjr sds da epoca que tu dava oi

Finally, we convert this dictionary to a pandas dataframe and it becomes an understandable and more easily interpretable table.

mention_df = pd.DataFrame.from_dict(mention_relations, orient='index', columns=mention_relations.keys())
mention_df

And we see the output of the mention counts table above. Again, rows are showing the mentioning users and columns are indicating mentioned ones.

If we ignore these values, we see that Lebron James is mentioned by everyone in the group and Rihanna looks like mentioned by everyone except Neymar.

On the other side, no one in the group has ever mentioned Neymar in their tweets.

Another interesting inference might be that Shakira mentioned Rihanna 52 times in her tweets however, Rihanna mentioned her only 7 times.

I hope you find it helpful.

Follow us on Instagram for all your Queries

3 replies on “Twitter Relationships Analysis with Python without API”

Leave a Reply